Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
eb07891
flake8 and validate path
fcollman Feb 22, 2019
afefa32
flake8 warning cleanup
fcollman Apr 23, 2018
6bdb0ac
removing deprecated
fcollman Apr 23, 2018
2b3414b
flake8 changes
fcollman Apr 27, 2018
b6ddb98
removing more deprecated items
fcollman Apr 28, 2018
73fdff8
adding sources
fcollman Apr 28, 2018
0733114
fixed test so that input_json is valid json
fcollman Dec 15, 2017
5795d44
merging changes with sink options
fcollman Jan 31, 2018
e81d997
removed unused exception
fcollman Dec 15, 2017
309212e
removing unused __init__
fcollman Dec 15, 2017
7a5909d
modified code to raise exception when the more than one configuration…
fcollman Dec 15, 2017
cab10e3
added exception to facilitate checking for extra configurations
fcollman Dec 15, 2017
42084bf
added more testing for yaml output source
fcollman Dec 15, 2017
426ccd8
pep8
fcollman Dec 16, 2017
7289b81
fixed import error
fcollman Dec 16, 2017
e0c2d22
added tests for multiple input configurations
fcollman Dec 16, 2017
46038b9
made it error if more than one output sink configured
fcollman Jan 2, 2018
a7993be
fixed bug in output_sink method
fcollman Jan 2, 2018
cbb18dc
added 2 output config failure test
fcollman Jan 2, 2018
03be72d
renaming default_configurable_sinks,sources
fcollman Jan 2, 2018
d23616a
doc typo
fcollman Jan 2, 2018
435a94a
doc changes
fcollman Jan 2, 2018
619b71a
merging changes
fcollman Jan 31, 2018
8b4662c
merging changes
fcollman Jan 31, 2018
594c80e
fix yaml parser
fcollman Jan 2, 2018
11d6d3f
removing pika source
fcollman Jan 2, 2018
6e0a996
made ConfigSource default __init__ and added auto imports to sources …
fcollman Jan 3, 2018
dc18772
added a test for a novel UrlSource
fcollman Jan 3, 2018
669cffa
added a novel test for a new source UrlSource to demonstrate plugin f…
fcollman Jan 3, 2018
7db7a87
fixed doc include
fcollman Jan 3, 2018
9af2fe1
fixed doc links
fcollman Jan 3, 2018
bc323d0
change tests to use normal ArgSchemaParser for urlsource
fcollman Jan 3, 2018
b967ada
doc fix
fcollman Jan 3, 2018
6e3defc
doc fix
fcollman Jan 3, 2018
2a885bc
added protocol to url_source_demo
fcollman Jan 3, 2018
fd3584f
removed None's from argparse when configuring sources via smart_merge
fcollman Jan 3, 2018
7bdd4d2
utilized marshmallow validation in Configurable source init
fcollman Jan 3, 2018
0c5065b
added optional parameter to url_source test
fcollman Jan 3, 2018
1bb697e
pep8
fcollman Jan 3, 2018
f85c219
documentation update
fcollman Jan 3, 2018
504abb9
doc changes
fcollman Jan 31, 2018
c344079
pep8
fcollman Jan 3, 2018
6c48ef1
changed to use urllib to construct url
fcollman Jan 3, 2018
552c797
removed FileSource pattern
fcollman Jan 3, 2018
e5a2c29
moved get_input to source module
fcollman Jan 3, 2018
aa684e7
fixed python2 import
fcollman Jan 3, 2018
58fa353
added docstrings
fcollman Jan 3, 2018
ef31148
added doc strings to methods that need to be implemented
fcollman Jan 3, 2018
b31faea
typo
fcollman Jan 3, 2018
f0a9116
documenation fixes
fcollman Jan 3, 2018
2bb6f8e
doc fixes
fcollman Jan 3, 2018
5a43bb5
doc update
fcollman Jan 15, 2018
88c0c81
wired up options for json sink
fcollman Jan 31, 2018
67ff09a
fixed docstring
fcollman Jan 31, 2018
024c5f0
wip
fcollman Jan 31, 2018
7f895af
alternative way of handling indent options
fcollman Jan 31, 2018
4fadd6d
style changes
fcollman Apr 28, 2018
7e0540f
flake8
fcollman Apr 28, 2018
15e4cac
added marshmallow 3 compatability
fcollman Feb 22, 2019
0b82e2b
remove OptionList
fcollman Feb 22, 2019
f807072
fixing test
fcollman Feb 22, 2019
7c8f869
flake8
fcollman Feb 22, 2019
a24d0e9
flake8
fcollman Feb 22, 2019
cecaf49
flake8
fcollman Feb 22, 2019
7a90fcf
flake8
fcollman Feb 22, 2019
1bf09a7
python3 inpsect warning fix
fcollman Feb 22, 2019
3646ca9
removing unused recursive checking
fcollman Feb 22, 2019
6471b0e
bumping doc reqs
fcollman Feb 22, 2019
7b8ea3f
merging master
fcollman Feb 5, 2020
39f3553
runtime input source selection
NileGraddis Feb 21, 2020
2223222
tests for specific sources and sinks
NileGraddis Feb 22, 2020
fc8f64f
docstrings for config source; some renaming
NileGraddis Feb 25, 2020
6ad1f5d
avoid security vulnerability in yaml sour e
NileGraddis Feb 25, 2020
96b28e1
add example of multisource argschema parsing
NileGraddis Feb 25, 2020
ec50464
add missing docstring types; raise if url_source response is bad
NileGraddis Mar 5, 2020
23e9164
update url_source test class to handle raise_for_status
NileGraddis Mar 5, 2020
e777fc7
initial readme change
Apr 13, 2020
7cd3f38
edit readme
Apr 13, 2020
1db1619
making compatible with masrhmallow 3.6.1
fcollman Jul 1, 2020
9c39478
fixes test to work with configurable sink
djkapner Mar 25, 2021
7c36355
comments out troublesome deprecated list test
djkapner Mar 25, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 21 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,6 @@ OR pass a json_dictionary directly into the module with the parameters defined

AND/OR pass parameters via the command line, in a way that will override the input_json or the json_dictionary given.

## Upgrading to version 2.0
The major change in argschema 2.0 is becoming
compatible with marshmallow 3, which changes
many of the ways your schemas and schema modifications work. Some noteable differences are that schemas are strict now by default, so tossing keys in your outputs or inputs that were ignored and stripped before now throw errors unless
def
Please read this document for more guidance
https://marshmallow.readthedocs.io/en/stable/upgrading.html

## Level of Support
We are planning on occasional updating this tool with no fixed schedule. Community involvement is encouraged through both issues and pull requests. Please make pull requests against the dev branch, as we will test changes there before merging into master.
Expand Down Expand Up @@ -75,7 +68,7 @@ You start building some code in an ipython notebook to play around with a new id

It's a mess, and you know you should migrate your code over to a module that you can call from other programs or notebooks. You start collecting your input variables to the top of the notebook and make yourself a wrapper function that you can call. However, now your mistake in filename typing is a disaster because the file doesn't exist, and your code doesn't check for the existence of the file until quite late. You start implementing some input validation checks to avoid this problem.

Now you start wanting to integrate this code with other things, including elements that aren't in python. You decide that you need to have a command line module that executes the code, because then you can use other tools to stitch together your processing, like maybe some shell scripts or docker run commands. You implement an argparse set of inputs and default values that make your python program a self-contained program, with some help documentation. Along the way, you have to refactor the parsed argparse variables into your function and strip out your old hacky validation code to avoid maintaining two versions of validation in the future.
Now you start wanting to integrate this code with other things, including elements that aren't in python. You decide that you need to have a command line module that executes the code, because then you can use other tools to stitch together your processing, like maybe some shell scripts or docker run commands. You implement an argparse set of inputs and default values that make your python program a self-contained program, with some helpful documentation. Along the way, you have to refactor the parsed argparse variables into your function and strip out your old hacky validation code to avoid maintaining two versions of validation in the future.

This module starts becoming useful enough that you want to integrate it into more complex modules. You end up copying and pasting various argparse lines over to other modules, and then 5 other modules. Later you decide to change your original module a little bit, and you have a nightmare of code replace to fix up the other modules to mirror this phenomenon.. you kick yourself for not having thought this through more clearly.

Expand All @@ -85,5 +78,25 @@ If you had only designed things from the beginning to allow for each of these us

This is what argschema is designed to do.


## Upgrading to version 3.0
The major change in argschema 3.0 is introducing a more generalized interface for reading and writing dictionaries, referred to as ArgSource and ArgSink. One can define customized classes that read dictionaries from any source you can code, such as making a database call, reading from a web service, reading a yaml file, etc. Argschema isn't just for json anymore. Similarly you can now dynamically tell your ArgSchemaParser to write output to an Argsink, which might write to a database, a webservice, or a messaging service. This enables those integrating modules into larger workflow management solutions more flexibility in wiring up your python modules to those systems.

It also removes features that were marked previously as deprecated.

Notably parsing List arguments with --listarg a b c, which instead should be called as --listarg a,b,c. In other words cli_as_single_argument = False is no longer an option.

It also removes the old names JsonModule, ModuleParameters, which are now ArgSchemaParser and ArgSchema respectively.

The field OptionList has been removed. The same functionality can be accomplished with the keyword, validate=mm.validate.OneOf([a,b,c...]) in the field definition.

## Upgrading to version 2.0
The major change in argschema 2.0 is becoming
compatible with marshmallow 3, which changes
many of the ways your schemas and schema modifications work. Some noteable differences are that schemas are strict now by default, so tossing keys in your outputs or inputs that were ignored and stripped before now throw errors.

Please read this document for more guidance
https://marshmallow.readthedocs.io/en/stable/upgrading.html

Copyright 2017 Allen Institute

7 changes: 3 additions & 4 deletions argschema/__init__.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
'''argschema: flexible definition, validation and setting of parameters'''
from .fields import InputFile, InputDir, OutputFile, OptionList # noQA:F401
from .schemas import ArgSchema # noQA:F401
from .argschema_parser import ArgSchemaParser # noQA:F401
from .deprecated import JsonModule, ModuleParameters # noQA:F401
from argschema.fields import InputFile, InputDir, OutputFile # noQA:F401
from argschema.schemas import ArgSchema # noQA:F401
from argschema.argschema_parser import ArgSchemaParser # noQA:F401

__version__ = "3.0.1"

Expand Down
Loading