Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement MongoDB transactions for API endpoint (TBD) #884

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from
Draft

Implement MongoDB transactions for API endpoint (TBD) #884

wants to merge 7 commits into from

Conversation

eecavanna
Copy link
Collaborator

@eecavanna eecavanna commented Jan 28, 2025 •
edited
Loading

⚠️ Some of the statements in this summary are still "aspirational" (i.e. they are not true yet): On this branch, I added some foundational code for using MongoDB transactions, including tests demonstrating their use. I also updated an API endpoint so that it performs its database operations within a MongoDB transaction. Finally, I updated the configuration of the MongoDB containers in the dev/test environments to run in replica set mode, which is a requirement for using MongoDB transactions.

Details

...

Related issue(s)

...

Related subsystem(s)

  • Runtime API (except the Minter)
  • Minter
  • Dagster
  • Project documentation (in the docs directory)
  • Translators (metadata ingest pipelines)
  • MongoDB migrations
  • Other

Testing

  • I tested these changes (explain below)
  • I did not test these changes

I tested these changes by...

Documentation

  • I have not checked for relevant documentation yet (e.g. in the docs directory)
  • I have updated all relevant documentation so it will remain accurate
  • Other (explain below)

Maintainability

  • Every Python function I defined includes a docstring (test functions are exempt from this)
  • Every Python function parameter I introduced includes a type hint (e.g. study_id: str)
  • All "to do" or "fix me" Python comments I added begin with either # TODO or # FIXME
  • I used black to format all the Python files I created/modified
  • The PR title is in the imperative mood (e.g. "Do X") and not the declarative mood (e.g. "Does X" or "Did X")

@eecavanna eecavanna linked an issue Jan 28, 2025 that may be closed by this pull request
Implement MongoDB transaction for API endpoint #877
Open
@eecavanna eecavanna self-assigned this Jan 28, 2025
@eecavanna
Copy link
Collaborator Author

eecavanna commented Jan 29, 2025 •
edited
Loading

The test Docker Compose stack is not starting up successfully on the GitHub Actions runner, on this branch. The log of the failing job ends with:

docker compose --file docker-compose.test.yml run test
 Container dagster-postgresql  Running
 Container fastapi  Running
 Container dagster-daemon  Running
 Container dagster-dagit  Running
 Container mongo  Starting
 Container mongo  Started
wait-for-it.sh: waiting 300 seconds for fastapi:8000
wait-for-it.sh: timeout occurred after waiting 300 seconds for fastapi:8000
wait-for-it.sh: strict mode, refusing to execute subprocess
make: *** [Makefile:55: test-run] Error 124

I take that to mean the FastAPI application did not start up successfully (as something that was trying to access it failed to do so within 5 minutes).

For reference, here's a screenshot showing the test stack working in my local dev/test environment (on the left side of the screenshot):

image

@eecavanna
Copy link
Collaborator Author

eecavanna commented Feb 5, 2025 •
edited
Loading

In this most recent GHA failure, the newly-introduced mongo-init "init container" is not "completing successfully."

 [...]
 Container mongo-init  Started
 Container mongo-init  Waiting
 Container mongo-init  service "mongo-init" didn't complete successfully: exit 1
service "mongo-init" didn't complete successfully: exit 1
make: *** [Makefile:42: up-test] Error 1
Error: Process completed with exit code 2.

I'll look into this on Wednesday.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@eecavanna eecavanna

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Implement MongoDB transaction for API endpoint
1 participant
@eecavanna