Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
1eaea3d
flake8 and validate path
fcollman Feb 22, 2019
034cb74
flake8 warning cleanup
fcollman Apr 23, 2018
4d9d68a
bump to 2.0
fcollman Apr 23, 2018
afa3e30
removing deprecated
fcollman Apr 23, 2018
5818c54
flake8 improvements
fcollman Apr 26, 2018
a970d48
flake8 changes
fcollman Apr 27, 2018
0d35f2d
removing more deprecated items
fcollman Apr 28, 2018
63ed42e
adding sources
fcollman Apr 28, 2018
00051f0
fixed test so that input_json is valid json
fcollman Dec 15, 2017
8f06171
merging changes with sink options
fcollman Jan 31, 2018
56223cd
removed unused exception
fcollman Dec 15, 2017
c8a641a
added pyyaml as requirement
fcollman Dec 15, 2017
a81bcc7
removing unused __init__
fcollman Dec 15, 2017
88afa43
modified code to raise exception when the more than one configuration…
fcollman Dec 15, 2017
72579ac
added exception to facilitate checking for extra configurations
fcollman Dec 15, 2017
da3cbc4
added more testing for yaml output source
fcollman Dec 15, 2017
4dbd25c
pep8
fcollman Dec 16, 2017
e66b325
fixed import error
fcollman Dec 16, 2017
a0a0048
added tests for multiple input configurations
fcollman Dec 16, 2017
a83a4cc
made it error if more than one output sink configured
fcollman Jan 2, 2018
3e63c22
fixed bug in output_sink method
fcollman Jan 2, 2018
348812e
added 2 output config failure test
fcollman Jan 2, 2018
744d16f
renaming default_configurable_sinks,sources
fcollman Jan 2, 2018
154e5ef
doc typo
fcollman Jan 2, 2018
8dca4a3
doc changes
fcollman Jan 2, 2018
84827b1
merging changes
fcollman Jan 31, 2018
619415e
merging changes
fcollman Jan 31, 2018
6c133c4
fix yaml parser
fcollman Jan 2, 2018
d520847
removing pika source
fcollman Jan 2, 2018
748233b
made ConfigSource default __init__ and added auto imports to sources …
fcollman Jan 3, 2018
9cb0580
added a test for a novel UrlSource
fcollman Jan 3, 2018
1389e9c
added requests as a test requirement
fcollman Jan 3, 2018
01e460c
added a novel test for a new source UrlSource to demonstrate plugin f…
fcollman Jan 3, 2018
5b91bbe
fixed doc include
fcollman Jan 3, 2018
286b07f
fixed doc links
fcollman Jan 3, 2018
8af291d
change tests to use normal ArgSchemaParser for urlsource
fcollman Jan 3, 2018
5bf9958
doc fix
fcollman Jan 3, 2018
f9c1ebb
doc fix
fcollman Jan 3, 2018
8a9f64d
added protocol to url_source_demo
fcollman Jan 3, 2018
55536d7
removed None's from argparse when configuring sources via smart_merge
fcollman Jan 3, 2018
08aefd3
utilized marshmallow validation in Configurable source init
fcollman Jan 3, 2018
2908dca
added optional parameter to url_source test
fcollman Jan 3, 2018
7159dcb
pep8
fcollman Jan 3, 2018
a956f79
documentation update
fcollman Jan 3, 2018
5658c09
doc changes
fcollman Jan 31, 2018
5072bc7
pep8
fcollman Jan 3, 2018
76572ab
changed to use urllib to construct url
fcollman Jan 3, 2018
0e13b30
removed FileSource pattern
fcollman Jan 3, 2018
290d0e8
moved get_input to source module
fcollman Jan 3, 2018
327c447
fixed python2 import
fcollman Jan 3, 2018
9ac9866
added docstrings
fcollman Jan 3, 2018
1085936
added doc strings to methods that need to be implemented
fcollman Jan 3, 2018
cf0821d
typo
fcollman Jan 3, 2018
877b0bd
documenation fixes
fcollman Jan 3, 2018
f488fa1
doc fixes
fcollman Jan 3, 2018
04fc787
doc update
fcollman Jan 15, 2018
4f59db4
wired up options for json sink
fcollman Jan 31, 2018
16e7c4b
fixed docstring
fcollman Jan 31, 2018
93a191f
wip
fcollman Jan 31, 2018
864d6f6
alternative way of handling indent options
fcollman Jan 31, 2018
b18545b
style changes
fcollman Apr 28, 2018
4a418de
flake8
fcollman Apr 28, 2018
f4da011
added marshmallow 3 compatability
fcollman Feb 22, 2019
ffaaae6
bump marshmallow req
fcollman Feb 22, 2019
ea82c89
remove OptionList
fcollman Feb 22, 2019
e129152
fixing test
fcollman Feb 22, 2019
5774579
making alpha version
fcollman Feb 22, 2019
4de84bf
flake8
fcollman Feb 22, 2019
58eef15
flake8
fcollman Feb 22, 2019
1297faa
flake8
fcollman Feb 22, 2019
9963abc
flake8
fcollman Feb 22, 2019
537b03d
python3 inpsect warning fix
fcollman Feb 22, 2019
2054abc
removing unused recursive checking
fcollman Feb 22, 2019
a5764a0
bumping doc reqs
fcollman Feb 22, 2019
054aea1
doc req fix
fcollman Feb 22, 2019
5053f69
merging master
fcollman Feb 5, 2020
16232cb
runtime input source selection
NileGraddis Feb 21, 2020
ca0e76e
tests for specific sources and sinks
NileGraddis Feb 22, 2020
1d414f3
docstrings for config source; some renaming
NileGraddis Feb 25, 2020
152212b
avoid security vulnerability in yaml sour e
NileGraddis Feb 25, 2020
ffdffc3
add example of multisource argschema parsing
NileGraddis Feb 25, 2020
9dafe5f
add missing docstring types; raise if url_source response is bad
NileGraddis Mar 5, 2020
d58f2ef
update url_source test class to handle raise_for_status
NileGraddis Mar 5, 2020
d520533
Merge pull request #101 from NileGraddis/input_source_tweaks
fcollman Apr 13, 2020
69d5460
Merge branch 'dev' into input_source_2.0.1
fcollman Apr 13, 2020
b02edc4
initial readme change
Apr 13, 2020
bbfb96c
edit readme
Apr 13, 2020
113b6f6
doc updates for ConfigurableSource/Sink
tmchartrand Oct 6, 2020
14f0c05
Merge pull request #106 from tmchartrand/input_source_3.0a
fcollman Jun 24, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,6 @@ jobs:
sed '/pywin32/d' test_requirements.txt > test_requirements_nowin.txt
pip install -r test_requirements_nowin.txt
pip install -e .

- save_cache:
paths:
- ./venv
Expand Down
29 changes: 21 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,6 @@ OR pass a json_dictionary directly into the module with the parameters defined

AND/OR pass parameters via the command line, in a way that will override the input_json or the json_dictionary given.

## Upgrading to version 2.0
The major change in argschema 2.0 is becoming
compatible with marshmallow 3, which changes
many of the ways your schemas and schema modifications work. Some noteable differences are that schemas are strict now by default, so tossing keys in your outputs or inputs that were ignored and stripped before now throw errors unless
def
Please read this document for more guidance
https://marshmallow.readthedocs.io/en/stable/upgrading.html

## Level of Support
We are planning on occasional updating this tool with no fixed schedule. Community involvement is encouraged through both issues and pull requests. Please make pull requests against the dev branch, as we will test changes there before merging into master.
Expand Down Expand Up @@ -75,7 +68,7 @@ You start building some code in an ipython notebook to play around with a new id

It's a mess, and you know you should migrate your code over to a module that you can call from other programs or notebooks. You start collecting your input variables to the top of the notebook and make yourself a wrapper function that you can call. However, now your mistake in filename typing is a disaster because the file doesn't exist, and your code doesn't check for the existence of the file until quite late. You start implementing some input validation checks to avoid this problem.

Now you start wanting to integrate this code with other things, including elements that aren't in python. You decide that you need to have a command line module that executes the code, because then you can use other tools to stitch together your processing, like maybe some shell scripts or docker run commands. You implement an argparse set of inputs and default values that make your python program a self-contained program, with some help documentation. Along the way, you have to refactor the parsed argparse variables into your function and strip out your old hacky validation code to avoid maintaining two versions of validation in the future.
Now you start wanting to integrate this code with other things, including elements that aren't in python. You decide that you need to have a command line module that executes the code, because then you can use other tools to stitch together your processing, like maybe some shell scripts or docker run commands. You implement an argparse set of inputs and default values that make your python program a self-contained program, with some helpful documentation. Along the way, you have to refactor the parsed argparse variables into your function and strip out your old hacky validation code to avoid maintaining two versions of validation in the future.

This module starts becoming useful enough that you want to integrate it into more complex modules. You end up copying and pasting various argparse lines over to other modules, and then 5 other modules. Later you decide to change your original module a little bit, and you have a nightmare of code replace to fix up the other modules to mirror this phenomenon.. you kick yourself for not having thought this through more clearly.

Expand All @@ -85,5 +78,25 @@ If you had only designed things from the beginning to allow for each of these us

This is what argschema is designed to do.


## Upgrading to version 3.0
The major change in argschema 3.0 is introducing a more generalized interface for reading and writing dictionaries, referred to as ConfigurableSource and ConfigurableSink. One can define customized classes that read dictionaries from any source you can code, such as making a database call, reading from a web service, reading a yaml file, etc. Argschema isn't just for json anymore. Similarly you can now dynamically tell your ArgSchemaParser to write output to a ConfigurableSink, which might write to a database, a webservice, or a messaging service. This enables those integrating modules into larger workflow management solutions more flexibility in wiring up your python modules to those systems.

It also removes features that were marked previously as deprecated.

Notably parsing List arguments with --listarg a b c, which instead should be called as --listarg a,b,c. In other words cli_as_single_argument = False is no longer an option.

It also removes the old names JsonModule, ModuleParameters, which are now ArgSchemaParser and ArgSchema respectively.

The field OptionList has been removed. The same functionality can be accomplished with the keyword, validate=mm.validate.OneOf([a,b,c...]) in the field definition.

## Upgrading to version 2.0
The major change in argschema 2.0 is becoming
compatible with marshmallow 3, which changes
many of the ways your schemas and schema modifications work. Some noteable differences are that schemas are strict now by default, so tossing keys in your outputs or inputs that were ignored and stripped before now throw errors.

Please read this document for more guidance
https://marshmallow.readthedocs.io/en/stable/upgrading.html

Copyright 2017 Allen Institute

7 changes: 3 additions & 4 deletions argschema/__init__.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
'''argschema: flexible definition, validation and setting of parameters'''
from .fields import InputFile, InputDir, OutputFile, OptionList # noQA:F401
from .schemas import ArgSchema # noQA:F401
from .argschema_parser import ArgSchemaParser # noQA:F401
from .deprecated import JsonModule, ModuleParameters # noQA:F401
from argschema.fields import InputFile, InputDir, OutputFile # noQA:F401
from argschema.schemas import ArgSchema # noQA:F401
from argschema.argschema_parser import ArgSchemaParser # noQA:F401

__version__ = "2.0.1"

Expand Down
Loading