Skip to content

cli - use library.storage #243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 1 commit into
base: devel
Choose a base branch
from
Draft

cli - use library.storage #243

wants to merge 1 commit into from
+90 −153

Conversation

@himdel
Copy link
Contributor

@himdel himdel commented Oct 16, 2025 •
edited
Loading

Issue: AAP-54345 / AAP-57129

Implements metrics_utility.library.storage - StorageS3, StorageDirectory, StorageCRC, StorageCRCMutual

storage = StorageS3(
    bucket=os.getenv('METRICS_UTILITY_BUCKET_NAME'),
    endpoint=os.getenv('METRICS_UTILITY_BUCKET_ENDPOINT'),
    region=os.getenv('METRICS_UTILITY_BUCKET_REGION'),
    access_key=os.getenv('METRICS_UTILITY_BUCKET_ACCESS_KEY'),
    secret_key=os.getenv('METRICS_UTILITY_BUCKET_SECRET_KEY'),
)

storage.put('my-remote-filename', filename='/tmp/file_to_upload')

with storage.get('my-remote-filename') as filename:
    open(filename)...
storage = StorageDirectory(base_path=os.getenv('METRICS_UTILITY_SHIP_PATH'))

storage.put('my-relative-filename', filename='/tmp/file_to_upload')

with storage.get('my-relative-filename') as filename:
    open(filename)...
StorageCRC(TODO)
StorageMutual(TODO)

And makes extractors, report savers and packagers use it.

wip - add paths test for storagedirectory,
clean up in Extractor/ReportSaver/Packager
move crc logic from 2f15564

FIXME: rebase after #246

@himdel himdel force-pushed the storage branch 2 times, most recently from ae2a55e to 4ec054d Compare October 28, 2025 02:32
@himdel himdel mentioned this pull request Oct 28, 2025
Merged
@himdel himdel changed the title library Storage - Directory, S3 cli - use library.storage Oct 28, 2025
@himdel
CLI - use library.storage
f6457a9 
Remove S3Handler, replace with StorageS3

upload_file caught errors, returned None on success, False on fail
download_file caught errors, returned True on success, False on fail

storage get/put doesn't catch, and doesn't return anything

but,

users of `upload_file`:
* `PackageS3` - doesn't catch, ignores return
  * gather will now fail if it fails to save the result
* `ReportSaverS3` - catches right outside, ignores return

users of `download_file`:
* `ExtractorS3` - catches right outside, ignores return
  * we no longer try to process_tarballs on download failure

update ExtractorS3, PackageS3, ReportSaverS3 to use StorageS3 instead of S3Handler

ExtractorS3 - try -> get -> tempdir -> unpack

Extractor - align ExtractorDirectory, ExtractorS3, both using storage

directory uses ship_path as the base path for storage, so we use data/Y/m/d/ as the path prefix, relative to base path
s3 uses ship_path as part of the path prefix, eg ship_path/data/Y/m/d/

Issue: AAP-54345
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@himdel