Skip to content

feat: Add AdamW bias correction parameter.#400

Merged
davidkoski merged 1 commit into
ml-explore:mainfrom
mattgallagher:feat/adamw-bias-correction
May 7, 2026
Merged

feat: Add AdamW bias correction parameter.#400
davidkoski merged 1 commit into
ml-explore:mainfrom
mattgallagher:feat/adamw-bias-correction

Conversation

@mattgallagher

Copy link
Copy Markdown
Contributor

Proposed changes

This PR adds a biasCorrection parameter to the Swift Adam and AdamW optimizers, matching the Python API and bias-corrected update rule used in MLX Python.

The main motivation is consistency with the Python implementation and easier comparison against PyTorch and other common optimizer implementations. I don't know if this falls within the goals of the mlx-swift project but I definitely encountered a situation where this was preventing close comparison with an MPSGraph implementation which was itself written to behave like PyTorch.

Internally, this requires tracking the optimizer step so the bias-correction factors can be applied:

  • 1 - beta1^t
  • 1 - beta2^t

To support that, this change replaces TupleState with an AdamState for Adam/AdamW, so the first moment, second moment, and step can be stored together.

This PR also adds tests for the bias-corrected Adam and AdamW paths and updates the relevant optimizer documentation.

Checklist

Put an x in the boxes that apply.

  • I have read the CONTRIBUTING document
  • I have run pre-commit run --all-files to format my code / installed pre-commit prior to committing changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the necessary documentation (if needed)

@davidkoski davidkoski left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks!

@davidkoski davidkoski merged commit bd196a9 into ml-explore:main May 7, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants