cli - use library.collectors #260
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dest was never passed to collector gather, so always None in gather_initialize, which called a _init_tmp_dir method use init_tmp_dir from the library instead (function, same code), and use the result to set gather_dir & tmp_dir (these then end up as output_dir= for fnc_collect)
CsvFileSplitter still supports it, but it wasn't inheriting the value and hardcoding 200M anyway
most slicers ignore last_gather, no need for it to be required param
…_provider_params for config, remove collection_type, unused
removing tests when a thing was moved to library and is tested there fixing slicer and collector calls to match fixing import locations
…lling_provider_params because billing_provider_params is now passed directly to the collector when it runs
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue: AAP-54476 / AAP-57129
This is the second part of #248: this makes sure we use the collectors moved to the library from the cli.
It also adds
until_slicing- which will use the last date ofdaily_slicing, unlikelimit_slicingwhich uses today - and uses it for execution environments.Also remove dead full_sync logic,
stop passing unused max_gather_size,
use init_tmp_dir helper from the library,
and adjusts test imports.