Skip to content

Push v2 integration #262

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 44 commits into
base: master
Choose a base branch
from
Open

Push v2 integration #262

wants to merge 44 commits into from
+1,294 −268

Conversation

@MarcelGeo
Copy link
Contributor

@MarcelGeo MarcelGeo commented Aug 15, 2025 •
edited
Loading

Resolved #256

Added ➕

  • integration to v2 push chunks mechanism - spliting some factory methods for UploadJob and methods in UploadQueueItem to v2 versions (we can discuss that, if it's ok)
  • ignore 409 error in push check_only to prevent stopping upload
  • added retry mechanism for chunks - 2 attempts after 5 seconds
  • added retry mechanism for sync command - compatible with v1 and v2push version using checks for status codes, server codes and response texts. This retry is doing n attempts every 5 seconds. Retry is doing if
    • 409 AntherUploadRunning, ProjectVersionExists
    • 429 - reate limit
    • for v1 if 400 and specific text in detail
  • introducing uploaded chunks cache which is used by v2 endpoints and cleaned by successful update
  • introducing LocalChanges class to handle dictionaries with changes (which nobody knows what's there) - it's used just in push methods to finalize push (time for improvement)
  • introducing server_features which will handle in future all feature flags from server
  • getting project info directly from v2 POST versions

Updated tests:

  • added remove only files test
  • added test for low level sync methods
  • added test for retry mechanism

❗

  • not sure what to do with progress bar in sync command, as it's not possible to update maximum size (length) of that on the fly.
  • some tests will fail, becasue we are there using v2 api which is not in app.dev yet - chunks cache test is skipped

Chunks cache

Logging following to logs

2025-08-13 14:09:10,282 will upload 1 items with total size 92025-08-13 14:09:10,282 Starting upload chunks for project 459a8014-9c05-439d-9b6b-2687cfdfb4fa2025-08-13 14:09:10,283 Uploading /home/marcelkocisek/Documents/mm-workspaces/localhost/hejj7/hej3.txt part=02025-08-13 14:09:10,283 Chunk 76ade537-12f2-4e0e-b25b-c22929f10c5a already uploaded, skipping upload

Retry of sync_project

2025-08-15 13:36:32,558 Attempting sync process due to conflicts between server and local directory or another user is syncing.

Workflow
image

@MarcelGeo
Retry sync job using
42c34a2 
Introduce mapping based on chunk_id (generated by client) and server_chunk_id generated by server.
varmar05
varmar05 requested changes Aug 26, 2025
View reviewed changes
@tomasMizera tomasMizera mentioned this pull request Aug 27, 2025
Closed
tomasMizera
tomasMizera reviewed Sep 3, 2025
View reviewed changes
varmar05
varmar05 reviewed Sep 4, 2025
View reviewed changes
@MarcelGeo
Update test calling count measurement
29292da 
- comment @varmar05
@MarcelGeo
add test also for v1
c6dafca 
- +v1 is raising error just from push start (push_project_async), because there are not chunks caches
@MarcelGeo
Drop versions and permissions check
e54bde8
@MarcelGeo
get rid of project info from editor
57e854b
@MarcelGeo
black
747ea46 
prevent conflcted copy fix
@MarcelGeo MarcelGeo mentioned this pull request Sep 9, 2025
Open
@MarcelGeo
black
50b6224
@MarcelGeo
Find just one file over limit in transaction
0fc2a9e
@MarcelGeo
black swan
607e148
@MarcelGeo
Close connection to make runnable test in new python and skip test fo…
77b82e9 
…r chunks cache on v1 supported server (#276)
@MarcelGeo
Merge remote-tracking branch 'origin/push-v2-integration' into change…
85387a5 
…s_limits
@MarcelGeo
Move validation to post_init
3041e75 
- change version of python in tests
@MarcelGeo
get rid of glob.glob
350aedc
@MarcelGeo
address comments without progress bar @wonder-sk
e36ae8d
@MarcelGeo
First version for sync progress bar
0119b7d
@MarcelGeo
Merge remote-tracking branch 'origin/push-v2-integration' into change…
cf232a6 
…s_limits
@MarcelGeo
fix typo and some docstrings for localProjectChanges
7dbed14
@MarcelGeo
Merge remote-tracking branch 'origin/push-v2-integration' into change…
92d5340 
…s_limits
@MarcelGeo
move file size to consts
0b585ee
@MarcelGeo
create generator yielding function for sync project
2e38b84
@MarcelGeo
Introduce generator for getting progress
3d00631 
- final commit probably
@MarcelGeo
Merge remote-tracking branch 'origin/push-v2-integration' into change…
4fc877c 
…s_limits
@MarcelGeo
Update docstring for sync method
7440f3f
@MarcelGeo
Merge pull request #274 from MerginMaps/changes_limits
cac700e 
Upload changes limits
varmar05
varmar05 approved these changes Sep 25, 2025
View reviewed changes
Copy link
Contributor

@varmar05 varmar05 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks very good 🚀

@MarcelGeo MarcelGeo mentioned this pull request Sep 25, 2025
Open
@MarcelGeo MarcelGeo mentioned this pull request Oct 30, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@varmar05 varmar05 varmar05 approved these changes

@wonder-sk wonder-sk Awaiting requested review from wonder-sk

@harminius harminius Awaiting requested review from harminius

@tomasMizera tomasMizera Awaiting requested review from tomasMizera

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Integration to v2 push endpoints

4 participants

@MarcelGeo @tomasMizera @varmar05