Skip to content

feat(LoRA): support AI Toolkit LoRA for FLUX #8071

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

feat(LoRA): support AI Toolkit LoRA for FLUX #8071

wants to merge 7 commits into from

Conversation

keturn
Copy link
Contributor

@keturn keturn commented May 30, 2025 •
edited
Loading

Summary

https://github.com/ostris/ai-toolkit is "an all in one training suite for diffusion models."

and I guess LoRA saved by it don't look exactly like the ones we support already.

Related Issues / Discussions

QA Instructions

Merge Plan

N/A

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

@github-actions github-actions bot added python PRs that change python files backend PRs that change backend files labels May 30, 2025
@keturn
Copy link
Contributor Author

keturn commented May 30, 2025 •
edited
Loading

Current state: LoRA install but don't run.

@github-actions github-actions bot added the python-tests PRs that change python tests label May 31, 2025
@keturn keturn force-pushed the feat/aitoolkit-lora branch from 43e4b2e to 0ca7a05 Compare June 1, 2025 00:06
@keturn keturn marked this pull request as ready for review June 1, 2025 00:09
keturn
keturn commented Jun 1, 2025
View reviewed changes
invokeai/backend/patches/lora_conversions/flux_aitoolkit_lora_conversion_utils.py
software = json.loads(metadata.get("software", "{}"))
except json.JSONDecodeError:
return False
return software.get("name") == "ai-toolkit"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this might not be what we need, as I have another LoRA here that reports the same metadata 'software': '{"name": "ai-toolkit", "repo": "https://github.com/ostris/ai-toolkit", "version": "0.1.0"}', but it uses transformer.single_transformer_blocks keys instead of diffusion_model.single_blocks keys.

fix: move AI Toolkit to the bottom of the detection list
b0c3520 
to avoid disrupting already-working LoRA
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lstein lstein Awaiting requested review from lstein lstein is a code owner

@blessedcoolant blessedcoolant Awaiting requested review from blessedcoolant blessedcoolant is a code owner

@hipsterusername hipsterusername Awaiting requested review from hipsterusername hipsterusername is a code owner

@jazzhaiku jazzhaiku Awaiting requested review from jazzhaiku jazzhaiku is a code owner

@psychedelicious psychedelicious Awaiting requested review from psychedelicious psychedelicious is a code owner

@maryhipp maryhipp Awaiting requested review from maryhipp maryhipp is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
backend PRs that change backend files python PRs that change python files python-tests PRs that change python tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@keturn