Documentation improvements

docs/.vuepress/_reference/autogen.py

@@ -1,4 +1,5 @@
 import ast
+import click
 import glob
 import importlib
 import inspect
@@ -7,6 +8,9 @@
 import types
 
 
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(os.path.dirname(__file__))))
+
+
 def get_file_list():
     path = os.path.dirname(__file__)
     return glob.glob(os.path.join(path, "_*.md"))
@@ -170,5 +174,23 @@ def generate(filepath=None):
         f.write(parse(filepath))
 
 
+def generate_cli():
+    from machinable.console.cli import cli
+
+    def get_help(cmd, parent=None):
+        ctx = click.core.Context(cli, info_name=cmd.name, parent=parent)
+        result = cmd.get_help(ctx) + "\n\n"
+        commands = getattr(cmd, "commands", {})
+        for sub_command in commands.values():
+            result += get_help(sub_command, ctx) + "\n\n"
+        return result
+
+    output = "\n\n## --help\n\nUse the `--help` option to inspect options\n\n```\n"
+    output += get_help(cli) + "\n```\n"
+    with open(os.path.join(ROOT, "reference", "cli.md"), "a",) as f:
+        f.write(output)
+
+
 if __name__ == "__main__":
     generate()
+    generate_cli()

docs/.vuepress/config.js

@@ -98,15 +98,22 @@ function getReferenceSidebar () {
       collapsable: false,
       sidebarDepth: 2,
       children: [
-        'component', 'execution', 'experiment', 'storage', 'engine', 'indexes'
+        'machinable_yaml',
+        'component',
+        'execution',
+        'experiment',
+        'storage',
+        'engine',
+        'indexes'
       ]
     },
     {
-      title: 'CLI',
+      title: 'System',
       collapsable: false,
       sidebarDepth: 2,
       children: [
-        'cli'
+        'cli',
+        'settings'
       ]
     }
   ]

docs/guide/execution.md

@@ -28,14 +28,17 @@ execute(
 )
 ```
 
-For even finer grained control, you can instantiate the [Execution](../reference/execution.md#execution) directly using the same arguments. In fact, `execute` is an alias for `Execution().summary().submit()`.
+For even finer grained control, you can instantiate the [Execution](../reference/execution.md#execution) object directly using the same arguments. Notably, `execute()` is an alias for `Execution().summary().submit()`.
 
 For every execution, machinable will generate a unique 6-digit experiment ID (e.g. `OY1p1o`) that will be printed at the beginning of the execution output. The ID encodes the global random seed and is used as a relative directory to write any data generated by the experiment.
 
+::: tip
+You can specify a system-wide default for `storage`, `engine` and `index`. [Learn more](../reference/settings.md)
+:::
+
 ### Experiment
 
-The experiment is the only required argument of the execution and specifies what components are ought to be executed. In the simplest case, it can be the name of a single component that will be executed using its default configuration. We will discuss the experiment specification in detail in the [following section](./experiments.md).
-The experiment is the only required argument of the execution and specifies what components are ought to be executed. In the simplest case, it can be the name of a single component that will be executed using its default configuration. We will discuss the experiment specification in detail in the [following section](./experiments.md).
+The experiment is the only required argument of the execution and specifies what components are executed. In the simplest case, it can be the name of a single component that will be executed using its default configuration. We will discuss the experiment specification in detail in the [following section](./experiments.md).
 
 ### Storage
 
@@ -75,3 +78,63 @@ If you need more control over randomness and how packages are being seeded, you
 ## Code backups
 
 machinable automatically backs up the code base at execution time in a zip file that can be used to reproduce the results. Note that the project directory needs to be under Git-version control to determine which files are included and ignored during the backup (``.gitignore`` file).
+
+## Import arguments and CLI
+
+It is often helpful to move frequently used execution arguments into modules, for example:
+```python
+# ./experiments/baseline.py
+from machinable import Experiment
+baseline_experiment = Experiment().component('example')
+```
+
+```python
+# ./engines/remote.py
+from machinable.engine import Remote
+my_remote_execution_engine = Remote(host="ssh://remote", directory="~/project")
+```
+
+```python
+# ./main.py
+from machinable import execute
+from experiments.baseline import baseline_experiment
+from engines.remote import my_remote_execution_engine
+execute(baseline_experiment, engine=my_remote_execution_engine)
+```
+You can simplify such imports by passing the module path prefixed with `@/` as an execution argument, for instance:
+```python
+from machinable import execute
+execute("@/experiments/baseline", engine="@/engines/remote")
+```
+Note that you do not need to specify the actual variable name (e.g. `baseline_experiment`) since machinable will search the module for instances automatically. 
+```python
+# ./experiments/baseline.py
+from machinable import Experiment
+# no need to assign the experiment to a variable
+Experiment().component('example')
+```
+If the module contains more than one instance, only the last one will be returned. 
+
+As a further simplification, using a simple `@` will instruct machinable to search in the following default modules.
+
+| Argument   | Module      |
+|------------|-------------|
+| experiment | experiments |
+| storage    | storages    |
+| engine     | engines     |
+| index      | indexes     |
+
+```python
+from machinable import execute
+# @baseline -> @/experiments/baseline
+# @remote -> @/engines/remote
+execute("@baseline", engine="@remote")
+```
+
+The `@`-notation is particularly useful when used in combination with the command line interface, as it allows you to specify complex arguments in a concise way. 
+
+```bash
+$ machinable execute @baseline --engine @remote
+```
+
+With the basic execution concepts out of the way, the following sections will focus on the fundamental `experiment` and `engine` arguments in more detail.

docs/guide/imports.md

@@ -62,10 +62,10 @@ Importing provides you with another level of modularity and makes it easy to mai
 
 ### Global dependencies
 
-Rather than re-downloading dependencies manually for every project, you can overwrite how dependency should be resolved in the `~/machinable/settings.yaml`:
+Rather than re-downloading dependencies manually for every project, you can overwrite how dependency should be resolved in the [`~/machinable/settings.yaml`](../reference/settings.md):
 ```yaml
 imports:
   imported_project: /opt/imported_project
 ```
 
-machinable will automatically symlink `/opt/imported_project` etc. into `vendor/` whenever a project declares the dependency (e.g. `imported_project`).
+machinable will automatically symlink `/opt/imported_project` etc. into `vendor/` whenever a project declares the dependency (e.g. `imported_project`).

docs/reference/.gitignore

@@ -1,2 +1,4 @@
 *
-!.gitignore
+!.gitignore
+!machinable_yaml.md
+!settings.md

docs/reference/machinable_yaml.md

@@ -0,0 +1,107 @@
+# machinable.yaml
+
+machinable's configuration file `machinable.yaml` must be placed at root of the project directory.
+
+    my-machinable-project
+    ├── ...
+    └── machinable.yaml
+
+The following describes an example structure
+
+- Lines starting with `#` describe the keys
+- `#!` describe the structure
+- types are denoted in brackets `[]`
+
+```yaml
+# Name of the project
+#! name: [String, must form valid Python module descriptor]
+name: example_project
+# Imports
+#! +: [List]
++:
+  #! [ String, valid Python variable name ]: [String, git/symlink/path]
+  - fooba_project:
+# List of all registered mixins and their configuration
+#! mixins: [List]
+mixins:
+  #! [String, Python import path]=[String, alias name, valid Python variable]: [Dict]
+  - mixins.trait=trait:
+      key:
+        data: True
+  #! [String, Python import path]^[String, inherit from name or alias]=[alias]:[Dict]
+  - extended^mixins.trait:
+      key:
+        extension: data
+  #! +.[ String, import project ].[ String, mixin name ]=[alias]: [Dict]
+  - +.fooba.test=test:
+      override: True
+  # Inherit from import
+  #! [ String, Import path ]^+.[ String, import project]: [Dict]
+  - inherit^+.fooba.experiments.start:
+      extended: true
+# List of all registered components and their configuration
+#! components: [List]
+components:
+  #! Same structure as Mixin list (see above)
+  - example_component:
+      alpha: False
+      beta: 
+      # Version that overrides existing keys
+      # ~[String]: [Dict]
+      ~one:
+        alpha: 1
+      ~two:
+        alpha: 2
+      ~three:
+        alpha: 3
+        ~nested:
+          alpha: 4
+          beta: nested
+          ~nestednested:
+            beta: overwritte
+      ~nested:
+        alpha: 5
+  - component_with_mixins:
+      # List of mixins. Order determines priority of mixin inheritance
+      #! _mixins_: [List]
+      _mixins_:
+        !# [String, mixin name or alias]
+        - trait
+        - test
+  - configmethods:
+      # Configuration methods that can be used as value anywhere
+      #! [String]: [config method]
+      method: hello()
+      argmethod: arghello('world')
+      nested:
+        method: hello()
+      keyarg: kwarghello(works=True)
+  - flat_notation:
+      # 'flattened' notation of nested structures
+      flat.can.be.useful: False
+      inherited.flat: value
+      ~flat_version:
+        flat:
+          nested: False
+        flat.version: 2
+  - referencing:
+      target: 'come find me'
+      # Global referencing in the entire file
+      #! $.[path.to.value]
+      globally: $.random_key
+      # Local referencing in current component
+      #! $self.[path.to.value]
+      locally: $self.target
+# Component groups
+#! components:[String, Python module name]: [List]
+components:example_module:
+  # -> example_module.example_component
+  - example_component:
+# Any other keys in the global scope are allowed
+random_key: 5e-6
+# Embedding of external files
+#! $/[filepath.yaml|.json]
+outsourcing: $/outsourced.yaml
+# Environment variables using the default bash syntax
+${envvarkey:-default}: $envvarvalue
+```

docs/reference/settings.md

@@ -0,0 +1,14 @@
+# Settings
+
+You can set system-specific settings by creating a `~/.machinable/settings.yaml` to override some or all of the default settings given below.
+
+```yaml
+cache:
+  imports: False
+imports: {}
+tmp_directory: userdata://machinable:machinable/tmp
+default_storage:
+   url: mem://
+default_engine:
+default_index:
+```