Exception: Current loss scale already at minimum - cannot decrease scale anymore

[Z-eloto]

Thank you for sharing your codes.
When running gpt2/kd/kd_medium.sh on 2*3090, the program encountered this error. What should I do, such as adjusting the learning rate?

[shiboyu1999]

You can use fp32 to train the model or decrease the batch size to 1.

[Z-eloto]

You can use fp32 to train the model or decrease the batch size to 1.

Thanks. I will try it. :)

[t1101675]

You can also try using bfloat16 by replacing ds_config_zero1_fp16.json in this line with ds_config_zero1_bf16.json.

[Z-eloto]

You can also try using bfloat16 by replacing ds_config_zero1_fp16.json in this line with ds_config_zero1_bf16.json.

OK, I have already resolved this problem. Thanks :)
But I also want to know if this modification will affect the experimental results?

[t1101675]

This will not affect the results much. In fact, bf16 will be more stable in training than fp16 and will not suffer from the "Current loss scale already at minimum" problem.

[Z-eloto]

I see. Many thanks!

This will not affect the results much. In fact, bf16 will be more stable in training than fp16 and will not suffer from the "Current loss scale already at minimum" problem.