P73Deep LearningAdvancedDiagnose14 min
Diagnose a Diverging Neural Network Training Run
Prioritize debugging from an exploding-loss trace.
#diagnose#deep learning#optimization
Scenario
Loss is 1.2, 1.0, 4.8, 91, NaN; gradient norms are 3, 8, 70, 900; one input is measured in millions and learning rate is 0.1.
Problem statement
Diagnose likely causes.
Your Task
- Read evidence.
- Prioritize fixes.
- Reject more epochs.
Related Concepts
Learning rateGradients
Ready to count this problem?
Mark it complete when you have reasoned through the solution in your own words.