Use default value of initial_scale_power if FP16 scaling params not provided

[ShukantPal]

The dynamic_loss_scale_args is None if some scaling param is not specified in the config: https://github.com/microsoft/DeepSpeed/blob/9d2660d2a3fac767972f01ac96858b2605ffc0e4/deepspeed/runtime/config.py#L215

In that case, it seems like DeepSpeed is using 2^32 as the initial_scale instead of the 2^16 as specified in the docs here: https://www.deepspeed.ai/docs/config-json/#fp16-training-options

[loadams]

@ShukantPal - I know this PR is old, but I see the following error now on this PR:

            raise TimeoutError
        if self._success:
            return self._value
        else:
>           raise self._value
E           Exception: Current loss scale already at minimum - cannot decrease scale anymore. Exiting run.

[ShukantPal]

Hi @loadams, I no longer have the bandwidth to support this PR (switched jobs :)). Feel free to close if this change is no longer applicable.

[loadams]

Hi @loadams, I no longer have the bandwidth to support this PR (switched jobs :)). Feel free to close if this change is no longer applicable.

No problem @ShukantPal - thanks for the update. I'll close this PR and open an issue to track the bug and make the needed fixes.

[tjruwase]

@loadams, I will tackle this in #6976

[loadams]

@loadams, I will tackle this in #6976

Thanks, you beat me to it!