Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* add flake8 config to pyproject.toml, add flake8-docstrings plugin to repo, add .pylintrc config from Google * add flake8 config, remove flake9 and use normal flake8 with own .flake8 config file * first functions for downloading cube files * update * no verification * pylint fixes * add simple test cases for _handle_status_code * update line length from 80 to 120, change indentation from 2 to 4 * activate more rules * fix pylint errors * fix pylint errors * fix missing dependencies * fix mypy errors * remove unncessary code * fix pylint issues * fix pylint config issues * add missing dependencies to .toml * fix import error * fix jsondecode error * add demo notebook for tablefile endpoint * fix return type issue * adjust error message for http_helper * fix column header issues * remove unncessary cut footer function * integrate PR feedback * update unit tests * refactor http_helper module * set area to optional parameter * fix typo * debug failing unit tests * debug failing unit tests * add .env * restore config_loader.py * integrate feedback from PR * integrate PR feedback * remove dotenv, use configparser instead (standard library) implement a user config using configparser module upload created doc as artifact update module docstring * merge tablefile and cubefile PR and use latest config module * [#43] Cache downloaded data; Implement feature with decorator; added tests * fix pylint errors; refactor code so that common code for downloading data is now in data.py module and table.py and cube.py only hold specific parsing logic * remove helper methods from get_data, move logic to the individual moduls * added clean_cache function function that removes all files from cache directory or only one, specified by name. Likely to be moved to cache.py, once this is on dev * Added ToDos, Unified query param comparison * Updated destatis status check #45 Incorporated further response codes and response type checks. Output for Code 0 tbd * Merged updates from personal branch * Added http 5xx error check, added destatis status tests * Updated urls, unified Destatis type-style * Added generic response creation for generic http_helper test * Updated type hints, made contents rely on functionality from http_helpers * Merged changes from draft branch, including pylint fixes, Exception and Error clarification. Also moved clean_cache function. Some ToDos still left, #45. * fixed linting issues, including inconsistent returns. * hotfix of overlooked error #45. * narrowed the catched exception * Updated clean_cache, updated http_helper * Fix lint issue with assert * mypy fixes for ci/cd pipeline #45 * Updated clean_cache function, added a first version of a test * Fixed linting issues, #45 * updated with remarks from pull request #45, #46 * updated logging due to linting errors W1203 * Improved test coverage, update temporary test directory for user names with spaces and non-latin characters, added mocked API call, added mocked create_settings call, moved test_http_helper, added TODOs for the future, #45 * Added mock as dependency (as otherwise CI/CD understandably fails), #45 * Also update project.lock, forgot it before, #45 * Fixed missing config dict error (in remote CI/CD) by mocking it, moved load_config into function, #45 * Fixed missing config dict error (in remote CI/CD) by mocking it, moved load_config into function, #45 * Fixed safety issues (mako and dparse) by updating poetry.lock, #45 * Fixed linting issue: timeout for requests.get added, #45 * Feat/update cube format (#67) * update cube module to handle cubes with more than one value variable in DQI. Add correct types to value columns so they are no longer of type object. * introduce Table und Cube class for coherent and simplified user interface; both classes download data with get_data() and provide access to data via .data and access to metadata via .metadata. Cube has .cube in addition for special cube metadata * reimplement cache for download data from endpoint * use compression for caching data * new tests for cube module * add tests for cache decorator that is called with functions that don't use the data endpoint (expect no caching) * find endpoint (#66) * simple example of Statistic class * first version of parsing cube; added pandas and notebook to dependencies * adding find and hierarchy * adding find and hierarchy * adding find and hierarchy * adding find and hierarchy * created results class * created results class * created results class * updated Results class with get_metadata for statistics and variables * updated string to f-strings according to pylint * solved pylint issues * solved pylint issues * solved pylint issues * several improvements based on feedback for PR #66 * several improvements based on feedback for PR #66 * several improvements based on feedback for PR #66 * modified docstring * modified docstring Co-authored-by: Michael Aydinbas <[email protected]> * Feat/refactor cache (#68) * refactor cache; no longer a decorator function but part of the new `load_data` function. This function checks if there is a hit in cache and loads the data from there, or load the data per request from Destatis. The new cache function stores data under `<name>/<endpoint>/<method>/hash(<request-params>)/<date-today>.zip` so different params will be cached differently `load_data` has a new parameter `as_json`, which defaults to `False`, so that the response is returned as raw text. However, setting `as_json=True` will return the response as a parsed JSON dict. * Feature/#48 hello world and profile endpoint (#69) * added clean_cache function function that removes all files from cache directory or only one, specified by name. Likely to be moved to cache.py, once this is on dev * Added ToDos, Unified query param comparison * Updated destatis status check #45 Incorporated further response codes and response type checks. Output for Code 0 tbd * Merged updates from personal branch * Added http 5xx error check, added destatis status tests * Updated urls, unified Destatis type-style * Added generic response creation for generic http_helper test * Updated type hints, made contents rely on functionality from http_helpers * merged current dev into feature branch. * Merged changes from draft branch, including pylint fixes, Exception and Error clarification. Also moved clean_cache function. Some ToDos still left, #45. * fixed linting issues, including inconsistent returns. * hotfix of overlooked error #45. * narrowed the catched exception * Updated clean_cache, updated http_helper * Fix lint issue with assert * mypy fixes for ci/cd pipeline #45 * Updated clean_cache function, added a first version of a test * Fixed linting issues, #45 * updated with remarks from pull request #45, #46 * updated logging due to linting errors W1203 * Added most of the methods from helloworld and profile endpoint, without tests so far, #48 * Modified pre-commit hook (set black profile for isort) to avoid isort and black conflicting changes * Added most of the methods from helloworld and profile endpoint - this time for real, without tests so far, #48 * Merged current dev into feature branch, #48 * Updated line-length argument for isort, added tests for helloworld and profile, #48 * merged dev again and fixed merge conflicts for automatic merge, #48 * Implemented removeresult + test, #48. * updated profile password test, #48, still isort issues seem ominous. * fixed isort in run tests to same line-length as used in pre-commit-config to avoid reoccuring conflicts in profile, #48, see also #60. * Fix W0707 raise ... from, #48. * Fix mypy type-hinting error, #48. * Minor requested updates, #48. * fixed mypy type, #48. * isort toml configuration commit (check line-length specification at only one point) * fixed line-length -> line_length * Added notebooks for helloworld and profile endpoint functions, updated get_data_from_endpoint to load_data as requested, #48. * fixed profile function text outputs, fixed tests, #48. * Added casting for mypy/ ci/cd, #48 * Feat/improve cache (#71) * refactor cache to work properly for jobs: no longer cache endpoint and method but only name as identifier, so a cache hit happens when the unique name is found in the cache data, irrelevant of the endpoint or method, thus working for data/resultfile and data/tablefile * normalize an object name so we can strip off the job-id * fix cicd * fix cicd * fix isort settings * fix mypy issues * add comments * refactor all tests to use python-mock (#72) * refactor all tests to use python-mock * update mocks * Feature/#40 automatic job query (#70) * added clean_cache function function that removes all files from cache directory or only one, specified by name. Likely to be moved to cache.py, once this is on dev * Added ToDos, Unified query param comparison * Updated destatis status check #45 Incorporated further response codes and response type checks. Output for Code 0 tbd * Merged updates from personal branch * Added http 5xx error check, added destatis status tests * Updated urls, unified Destatis type-style * Added generic response creation for generic http_helper test * Updated type hints, made contents rely on functionality from http_helpers * merged current dev into feature branch. * Merged changes from draft branch, including pylint fixes, Exception and Error clarification. Also moved clean_cache function. Some ToDos still left, #45. * fixed linting issues, including inconsistent returns. * hotfix of overlooked error #45. * narrowed the catched exception * Added notes w\ Felix regarding automatic job support, #40. * Notebook for interaction & workflow with jobs * integrating jobs into http_helper.py * Simplified if selections, added TODO comments, added catches for fail cases, #40 * splitting up jobs function and writing first tests for user input * work in progress * changes in jobs params for faster tests * Changes in job due to http_helper #43 * refactor: Updated & added documentation, identified Windows input issue, #40. * Commit before merge of dev * commit before PR * last working commit, #40. * seem to have fixed issue with destatis response, however, currently am timed out and can not finally check, #40. * Fixed new implementation and updated jobs notebook. Code needs clean-up & tests need to be updated (for input), #40. * changes with respect to first PR * Removed user input, always starting job, test update, currently on timeout * pylint changes * commit before merge * Code quality http_helper.py and tests * pylint uand black changes * removing _generic_status_dict() * Changes due to bandit * Changes due to mypy * refactor load_data to handle jobs * fix tests * Removed one old comment, #40 Co-authored-by: MarcoHuebner <[email protected]> Co-authored-by: Michael Aydinbas <[email protected]> * update dependencies * rename package to pystatis * import all main functions into pystatis main namespace * fix failing tests * add authors and maintainers * fix pyproject.toml * Feat/56 57 final readme (#73) * Updating README; add sections about Installation and first setup * updated readme with main features, how to use and how to contribute. * fix tests * added new functions cube, table Co-authored-by: dpleus <[email protected]> Co-authored-by: frederik9 <[email protected]> Co-authored-by: codehering <[email protected]> Co-authored-by: MarcoHuebner <[email protected]> Co-authored-by: Daniel Pleus <[email protected]> Co-authored-by: Felix Schmitz <[email protected]> Co-authored-by: dpleus <[email protected]>
- Loading branch information
7 people
authored
Oct 12, 2022
1 parent
80a5a8a
commit 6242bdf