Skip to content

Add ProcessGraphStrategy for task-based automation #944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

Add ProcessGraphStrategy for task-based automation #944

wants to merge 13 commits into from

Conversation

abrichr
Copy link
Member

@abrichr abrichr commented Mar 15, 2025 •
edited
Loading

Summary

  • Add new ProcessGraphStrategy that uses OmniParser and Gemini 2.0 to build and maintain a process graph
  • Strategy takes natural language task descriptions instead of recording IDs
  • Implements graph construction, validation, and runtime adaptation
  • Uses Pydantic models for structured data and json_repair for robust parsing

Test plan

  • Install json-repair (pip install json-repair)
  • Set up OmniParser provider (instructions in deploy/ directory)
  • Run a replay with process graph strategy: 
    python -m openadapt.replay ProcessGraphStrategy --task_description "Log into the application and search for products"
  • Check graph construction and adaptation with various UI states

🤖 [partially] Generated with Claude Code

abrichr and others added 13 commits February 18, 2025 15:43
@abrichr
add working omniparser deploy.py, Dockerfile, pyproject.toml, README.…
22ebc39 
…md, .env.example, .dockerignore
@abrichr
add Deploy.stop
3e5251a
@abrichr
add client.py
96b0f5a
@abrichr
ruff
959c78b
@abrichr
download OCR files during build; CLEANUP_ON_FAILURE = False
6290d17
@abrichr
ruff
944549f
@abrichr
lint
b6ec99d
@abrichr
ruff
bfc49d2
@abrichr
remove unused function
bf9d548
@abrichr
replace argparse with fire
f52a33e
@abrichr
config.CONTAINER_NAME
9a49bc2
@abrichr
lint
fac368b
@abrichr
Add ProcessGraphStrategy for task-based automation
27d4d1d 
This new strategy:
1. Uses OmniParser for parsing visual state and Gemini 2.0 for evaluation
2. Takes natural language task descriptions instead of recording IDs
3. Processes coalesced actions from events.py
4. Builds and maintains a process graph where states are nodes and actions are edges

The graph is constructed before replay based on recording + task description,
and is updated during replay based on observed states. Uses Pydantic for
structured data handling and robust JSON parsing.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@abrichr