Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add playground experiments #2

Draft
wants to merge 29 commits into
base: main
Choose a base branch
from
Draft

Add playground experiments #2

wants to merge 29 commits into from

Conversation

rasca
Copy link
Member

@rasca rasca commented Nov 7, 2024

Add experiments while at the same time refactoring and improving the stack.

This is the first step onto adding experiments to Flou. From our first users, the feedback we gathered was mostly about adding a description to every Playground run. This prompted us to create the first iteration of Experiments: Playground Experiments where you get to interact with an LTM and iterate over it describing your journey in an Experiment.

The other type of experiment will be datasets trials, which we'll leave for the near feature.

This goes as follows:

  • Create Experiments& Trial Entities
    • Automatically assign a name "Entity Experiment N34". Create an index for every experiment and trial
    • Create Playground Trial, choose a better name
      • Creates a new LTM
      • Allow to set trial name
  • Add experiment detail
  • Add related experiment in playground
  • Replace +LTM button with +Experiment
    • Allow a commentary about what happened or what you are going to create
    • Show code diff with the last trial (when we have code as data)
  • Show rollbacks in Snapshots Table
    • And the amount of rollbacks from a particular snapshot (and derivates)
    • Show a tree of snapshots & rollbacks, but it's seen from the other side, from the snapshot, not that it roll backs, but that is has N branches from there.
    • Put the last snapshot on top
    • (also show when code changes, what changed)
  • Differentiate "Trial by playground" and "Trial by dataset"
  • You have inputs and outputs, hypothesis and results / commentaries
    • Each evaluator in a dataset creates an output
    • Add input / outputs to api + studio
      • One or several "notes"
      • You can add a field at any time, it creates it blank in the others (or it doesn't even show them)
      • You can delete fields at anytime
  • Experiment analysis
    • You can filter by trial output/input and create custom views and charts
    • You can execute "evaluators" not just in trials but in experiments as well
      • They create outputs

Refactors in this PR

  • Share session between pytest & fastapi
  • Add superforms to studio
  • Improve route handling in studio
  • Refactor fastapi into modules each with models, schemas, routes
  • Rename executor to engine
  • Unify rollback and replay

rasca added 8 commits November 4, 2024 14:46
- Extracted `database.models` and `api.router` to `engine` and `ltm`.
- Add experiments migrations
This is a big commit. I've been working vertically on adding
Experiments. It's a Work In Progress but the base is now working.

- refactor the api schemas for base classes
- add experiment & trials routes (not all of them tested yet)
- add studio experiment list and creation
- add superforms to studio for form handling
This change allows us to modify and query the db in a test that also
executes routes that modify the db.
@rasca rasca force-pushed the feature/experiments branch from b7770a3 to abdf1a0 Compare November 7, 2024 17:24
- Add trials_count
- Use formatDate
- When adding a new experiment go to the playground
@rasca rasca changed the title Add playground experiments Add playground experiments Nov 11, 2024
@rasca rasca force-pushed the feature/experiments branch from 488be84 to 07fcb1b Compare November 14, 2024 13:44
- Compose automatically selects the correct image versions that
  correspond to the installed package
- You can release to DockeHub or PyPI
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant