From be905d801d83490d9f674f534607b7e95be5434c Mon Sep 17 00:00:00 2001 From: frthjf Date: Fri, 7 Jun 2024 19:50:54 +0000 Subject: [PATCH] =?UTF-8?q?Deploying=20to=20gh-pages=20from=20@=20machinab?= =?UTF-8?q?le-org/machinable@eafc91c3bbd732cf8587d94af81bd05d8f5369a5=20?= =?UTF-8?q?=F0=9F=9A=80?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- 404.html | 2 +- about/approach.html | 2 +- ...les_aimstack-storage_index.md.e95360b0.js} | 2 +- ...imstack-storage_index.md.e95360b0.lean.js} | 0 ...mples_globus-storage_index.md.620b6419.js} | 2 +- ..._globus-storage_index.md.620b6419.lean.js} | 0 ...amples_mpi-execution_index.md.5075bb2c.js} | 2 +- ...s_mpi-execution_index.md.5075bb2c.lean.js} | 0 ...es_require-execution_index.md.9f45fc51.js} | 2 +- ...quire-execution_index.md.9f45fc51.lean.js} | 0 ...ples_slurm-execution_index.md.292b100f.js} | 24 +++++++++------- ...slurm-execution_index.md.292b100f.lean.js} | 0 ...9d5c.js => guide_component.md.8f272292.js} | 2 +- ...js => guide_component.md.8f272292.lean.js} | 0 ...88fc.js => guide_interface.md.72ed9d18.js} | 6 ++-- ...js => guide_interface.md.72ed9d18.lean.js} | 2 +- ...6.js => guide_introduction.md.c4095de6.js} | 4 +-- ...=> guide_introduction.md.c4095de6.lean.js} | 0 examples/aimstack-storage/index.html | 6 ++-- examples/dependent-schedules/index.html | 2 +- examples/globus-storage/index.html | 6 ++-- examples/index.html | 2 +- examples/mpi-execution/index.html | 6 ++-- examples/require-execution/index.html | 6 ++-- examples/slurm-execution/index.html | 28 +++++++++++-------- guide/component.html | 6 ++-- guide/element.html | 2 +- guide/installation.html | 2 +- guide/interface.html | 6 ++-- guide/introduction.html | 8 +++--- hashmap.json | 2 +- index.html | 2 +- reference/index.html | 2 +- 33 files changed, 72 insertions(+), 64 deletions(-) rename assets/{examples_aimstack-storage_index.md.880b0b35.js => examples_aimstack-storage_index.md.e95360b0.js} (99%) rename assets/{examples_aimstack-storage_index.md.880b0b35.lean.js => examples_aimstack-storage_index.md.e95360b0.lean.js} (100%) rename assets/{examples_globus-storage_index.md.68197128.js => examples_globus-storage_index.md.620b6419.js} (99%) rename assets/{examples_globus-storage_index.md.68197128.lean.js => examples_globus-storage_index.md.620b6419.lean.js} (100%) rename assets/{examples_mpi-execution_index.md.e2d88fe1.js => examples_mpi-execution_index.md.5075bb2c.js} (99%) rename assets/{examples_mpi-execution_index.md.e2d88fe1.lean.js => examples_mpi-execution_index.md.5075bb2c.lean.js} (100%) rename assets/{examples_require-execution_index.md.36bcc19a.js => examples_require-execution_index.md.9f45fc51.js} (98%) rename assets/{examples_require-execution_index.md.36bcc19a.lean.js => examples_require-execution_index.md.9f45fc51.lean.js} (100%) rename assets/{examples_slurm-execution_index.md.b1b1e5c2.js => examples_slurm-execution_index.md.292b100f.js} (97%) rename assets/{examples_slurm-execution_index.md.b1b1e5c2.lean.js => examples_slurm-execution_index.md.292b100f.lean.js} (100%) rename assets/{guide_component.md.771e9d5c.js => guide_component.md.8f272292.js} (99%) rename assets/{guide_component.md.771e9d5c.lean.js => guide_component.md.8f272292.lean.js} (100%) rename assets/{guide_interface.md.0cb088fc.js => guide_interface.md.72ed9d18.js} (98%) rename assets/{guide_interface.md.0cb088fc.lean.js => guide_interface.md.72ed9d18.lean.js} (92%) rename assets/{guide_introduction.md.cf46eed6.js => guide_introduction.md.c4095de6.js} (97%) rename assets/{guide_introduction.md.cf46eed6.lean.js => guide_introduction.md.c4095de6.lean.js} (100%) diff --git a/404.html b/404.html index 64640ef4..fabeaa2e 100644 --- a/404.html +++ b/404.html @@ -21,7 +21,7 @@
machinable

Appearance

404

PAGE NOT FOUND

But if you don't change your direction, and if you keep looking, you may end up where you are heading.
Take me home

MIT Licensed

Copyright © 2021-present

- diff --git a/about/approach.html b/about/approach.html index 0c3ad5c6..1ee25133 100644 --- a/about/approach.html +++ b/about/approach.html @@ -25,7 +25,7 @@
machinable

Appearance

On this page

About machinable's approach ​

Optional reading

This background discusses the big-picture approach. For a hands-on tutorial, refer to the guide.

Coming soon

This section is currently under construction

MIT Licensed

Copyright © 2021-present

- diff --git a/assets/examples_aimstack-storage_index.md.880b0b35.js b/assets/examples_aimstack-storage_index.md.e95360b0.js similarity index 99% rename from assets/examples_aimstack-storage_index.md.880b0b35.js rename to assets/examples_aimstack-storage_index.md.e95360b0.js index bea71c43..daf7b67f 100644 --- a/assets/examples_aimstack-storage_index.md.880b0b35.js +++ b/assets/examples_aimstack-storage_index.md.e95360b0.js @@ -2,7 +2,7 @@ import{_ as s,o as a,c as n,O as l}from"./chunks/framework.62020867.js";const C= get("aimstack", {"repo": "./path/to/aim-repo"}).__enter__() -# your code

Source ​

py
from typing import List, Optional
+# your code

Source ​

py
from typing import List, Optional
 
 import os
 
diff --git a/assets/examples_aimstack-storage_index.md.880b0b35.lean.js b/assets/examples_aimstack-storage_index.md.e95360b0.lean.js
similarity index 100%
rename from assets/examples_aimstack-storage_index.md.880b0b35.lean.js
rename to assets/examples_aimstack-storage_index.md.e95360b0.lean.js
diff --git a/assets/examples_globus-storage_index.md.68197128.js b/assets/examples_globus-storage_index.md.620b6419.js
similarity index 99%
rename from assets/examples_globus-storage_index.md.68197128.js
rename to assets/examples_globus-storage_index.md.620b6419.js
index 97c423c1..771dd36a 100644
--- a/assets/examples_globus-storage_index.md.68197128.js
+++ b/assets/examples_globus-storage_index.md.620b6419.js
@@ -13,7 +13,7 @@ import{_ as s,o as n,c as a,O as l}from"./chunks/framework.62020867.js";const i=
 print(matches)
 
 # download from globus
-storage.download(matches[0].uuid)

Source ​

py
from typing import TYPE_CHECKING, List
+storage.download(matches[0].uuid)

Source ​

py
from typing import TYPE_CHECKING, List
 
 import os
 import time
diff --git a/assets/examples_globus-storage_index.md.68197128.lean.js b/assets/examples_globus-storage_index.md.620b6419.lean.js
similarity index 100%
rename from assets/examples_globus-storage_index.md.68197128.lean.js
rename to assets/examples_globus-storage_index.md.620b6419.lean.js
diff --git a/assets/examples_mpi-execution_index.md.e2d88fe1.js b/assets/examples_mpi-execution_index.md.5075bb2c.js
similarity index 99%
rename from assets/examples_mpi-execution_index.md.e2d88fe1.js
rename to assets/examples_mpi-execution_index.md.5075bb2c.js
index 4658efb5..4c95a632 100644
--- a/assets/examples_mpi-execution_index.md.e2d88fe1.js
+++ b/assets/examples_mpi-execution_index.md.5075bb2c.js
@@ -1,7 +1,7 @@
 import{_ as s,o as n,c as a,O as l}from"./chunks/framework.62020867.js";const i=JSON.parse('{"title":"MPI execution","description":"","frontmatter":{},"headers":[],"relativePath":"examples/mpi-execution/index.md","filePath":"examples/mpi-execution/index.md"}'),p={name:"examples/mpi-execution/index.md"},o=l(`
MPI execution ​
Integration to launch MPI jobs.
Usage example ​
python
from machinable import get
 
 with get("mpi", {"ranks": 8}):
-    ... # your MPI ready component
Source ​
py
from typing import Literal, Optional, Union
+    ... # your MPI ready component
Source ​
py
from typing import Literal, Optional, Union
 
 import sys
 
diff --git a/assets/examples_mpi-execution_index.md.e2d88fe1.lean.js b/assets/examples_mpi-execution_index.md.5075bb2c.lean.js
similarity index 100%
rename from assets/examples_mpi-execution_index.md.e2d88fe1.lean.js
rename to assets/examples_mpi-execution_index.md.5075bb2c.lean.js
diff --git a/assets/examples_require-execution_index.md.36bcc19a.js b/assets/examples_require-execution_index.md.9f45fc51.js
similarity index 98%
rename from assets/examples_require-execution_index.md.36bcc19a.js
rename to assets/examples_require-execution_index.md.9f45fc51.js
index 2ff7ef4e..4076e430 100644
--- a/assets/examples_require-execution_index.md.36bcc19a.js
+++ b/assets/examples_require-execution_index.md.9f45fc51.js
@@ -1,7 +1,7 @@
 import{_ as s,o as a,c as n,O as l}from"./chunks/framework.62020867.js";const C=JSON.parse('{"title":"Require execution","description":"","frontmatter":{},"headers":[],"relativePath":"examples/require-execution/index.md","filePath":"examples/require-execution/index.md"}'),p={name:"examples/require-execution/index.md"},o=l(`
Require execution ​
A way to assert that components have been cached.
Usage example ​
python
from machinable import get
 
 with get("require"):
-    ... # components to check
Source ​
py
from machinable import Execution
+    ... # components to check
Source ​
py
from machinable import Execution
 
 
 class Require(Execution):
diff --git a/assets/examples_require-execution_index.md.36bcc19a.lean.js b/assets/examples_require-execution_index.md.9f45fc51.lean.js
similarity index 100%
rename from assets/examples_require-execution_index.md.36bcc19a.lean.js
rename to assets/examples_require-execution_index.md.9f45fc51.lean.js
diff --git a/assets/examples_slurm-execution_index.md.b1b1e5c2.js b/assets/examples_slurm-execution_index.md.292b100f.js
similarity index 97%
rename from assets/examples_slurm-execution_index.md.b1b1e5c2.js
rename to assets/examples_slurm-execution_index.md.292b100f.js
index ffd44a78..34b497b3 100644
--- a/assets/examples_slurm-execution_index.md.b1b1e5c2.js
+++ b/assets/examples_slurm-execution_index.md.292b100f.js
@@ -1,15 +1,17 @@
 import{_ as s,o as n,c as a,O as l}from"./chunks/framework.62020867.js";const i=JSON.parse('{"title":"Slurm execution","description":"","frontmatter":{},"headers":[],"relativePath":"examples/slurm-execution/index.md","filePath":"examples/slurm-execution/index.md"}'),o={name:"examples/slurm-execution/index.md"},p=l(`
Slurm execution ​
Integration to submit to the Slurm scheduler.
Usage example ​
python
from machinable import get
 
 with get("slurm", {"ranks": 8, 'preamble': 'mpirun'}):
-    ... # your component
Source ​
py
from typing import Literal, Optional, Union
+    ... # your component
Source ​
py
from typing import Literal, Optional, Union
 
 import os
 import subprocess
 import sys
 import time
 
+import arrow
 import yaml
 from machinable import Execution, Project
+from machinable.config import to_dict
 from machinable.errors import ExecutionFailed
 from machinable.utils import chmodx, run_and_stream
 from pydantic import BaseModel, ConfigDict
@@ -104,7 +106,7 @@ import{_ as s,o as n,c as a,O as l}from"./chunks/framework.62020867.js";const i=
         dry: bool = False
 
     def on_before_dispatch(self):
-        if self.config.confirm:
+        if self.config.confirm and not self.config.dry:
             return confirm(self)
 
     def on_compute_default_resources(self, executable):
@@ -170,8 +172,6 @@ import{_ as s,o as n,c as a,O as l}from"./chunks/framework.62020867.js";const i=
             if ranks is not None:
                 if mpi_args:
                     mpi_args = mpi_args.replace("{ranks}", str(ranks))
-                else:
-                    mpi_args = "-n " + str(executable.config.get("ranks"))
             python = self.config.python or sys.executable
 
             # usage dependencies
@@ -211,12 +211,6 @@ import{_ as s,o as n,c as a,O as l}from"./chunks/framework.62020867.js";const i=
             if self.config.preamble:
                 script += self.config.preamble
 
-            # add debug information
-            script += "\\n"
-            script += f"# {executable.module} <{executable.id}>\\n"
-            script += f"# {executable.local_directory()}>\\n"
-            script += "\\n"
-
             if mpi:
                 if mpi[-1] != " ":
                     mpi += " "
@@ -232,6 +226,16 @@ import{_ as s,o as n,c as a,O as l}from"./chunks/framework.62020867.js";const i=
             print(f"Submitting job {executable} with resources: ")
             print(yaml.dump(resources))
 
+            # add debug information
+            script += "\\n\\n"
+            script += f"# generated at: {arrow.now()}\\n"
+            script += f"# {executable.module} <{executable.id}>\\n"
+            script += f"# {executable.local_directory()}\\n\\n"
+            script += "# " + yaml.dump(executable.version()).replace(
+                "\\n", "\\n# "
+            )
+            script += "\\n"
+
             # submit to slurm
             script_file = chmodx(
                 self.save_file([executable.id, "slurm.sh"], script)
diff --git a/assets/examples_slurm-execution_index.md.b1b1e5c2.lean.js b/assets/examples_slurm-execution_index.md.292b100f.lean.js
similarity index 100%
rename from assets/examples_slurm-execution_index.md.b1b1e5c2.lean.js
rename to assets/examples_slurm-execution_index.md.292b100f.lean.js
diff --git a/assets/guide_component.md.771e9d5c.js b/assets/guide_component.md.8f272292.js
similarity index 99%
rename from assets/guide_component.md.771e9d5c.js
rename to assets/guide_component.md.8f272292.js
index 9e37020b..396a4329 100644
--- a/assets/guide_component.md.771e9d5c.js
+++ b/assets/guide_component.md.8f272292.js
@@ -17,7 +17,7 @@ import{_ as p,D as t,o as c,c as r,z as n,a as s,G as o,B as l,O as e}from"./chu
 >>> mnist.launch()
 Downloading 'mnist' ...
If the execution is successful, the component is marked as finished.
python
>>> mnist.execution.is_finished()
 True
By design, component instances can only be executed once. They are automatically assigned a timestamp, random seed, as well as a nickname for easy identification.
python
>>> mnist.seed
-1632827863
Invocations of launch() after successful execution, do not trigger another execution since the component is marked as cached. On the other hand, if the execution failed, calling launch() will resume the execution with the same configuration.
Implementing custom execution ​
Components can be executed in different ways. You may, for example, like to run components using multiprocessing or execute in a cloud environment. However, instead of adding the execution logic directly to your component code, machinable makes it easy to separate concerns. You can encapsulate the execution implementation in its own execution class that can then be used to execute the component.
`,8),C=e(`
python
from multiprocessing import Pool
+1632827863
Invocations of launch() after successful execution, do not trigger another execution since the component is marked as cached. On the other hand, if the execution failed, calling launch() will resume the execution with the same configuration.
Implementing custom execution ​
Components can be executed in different ways. You may, for example, like to run components using multiprocessing or execute in a cloud environment. However, instead of adding the execution logic directly to your component code, machinable makes it easy to separate concerns. You can encapsulate the execution implementation in its own execution class that can then be used to execute the component.
`,8),C=e(`
python
from multiprocessing import Pool
 
 from machinable import Execution
 
diff --git a/assets/guide_component.md.771e9d5c.lean.js b/assets/guide_component.md.8f272292.lean.js
similarity index 100%
rename from assets/guide_component.md.771e9d5c.lean.js
rename to assets/guide_component.md.8f272292.lean.js
diff --git a/assets/guide_interface.md.0cb088fc.js b/assets/guide_interface.md.72ed9d18.js
similarity index 98%
rename from assets/guide_interface.md.0cb088fc.js
rename to assets/guide_interface.md.72ed9d18.js
index 7a05058b..b1510015 100644
--- a/assets/guide_interface.md.0cb088fc.js
+++ b/assets/guide_interface.md.72ed9d18.js
@@ -1,4 +1,4 @@
-import{_ as t,D as p,o as c,c as r,z as a,a as s,G as l,B as o,O as e}from"./chunks/framework.62020867.js";const I=JSON.parse('{"title":"Interface","description":"","frontmatter":{},"headers":[],"relativePath":"guide/interface.md","filePath":"guide/interface.md"}'),i={name:"guide/interface.md"},y=a("h1",{id:"interface",tabindex:"-1"},[s("Interface "),a("a",{class:"header-anchor",href:"#interface","aria-label":'Permalink to "Interface"'},"​")],-1),D=a("p",null,[a("a",{href:"./element.html"},"Elements"),s(" by themselves are limited in that they are effectively stateless. You can construct and use them but any computed result or additional information will not be persisted.")],-1),F=e(`
python
from machinable import Interface
+import{_ as t,D as p,o as c,c as r,z as a,a as s,G as l,B as o,O as e}from"./chunks/framework.62020867.js";const k=JSON.parse('{"title":"Interface","description":"","frontmatter":{},"headers":[],"relativePath":"guide/interface.md","filePath":"guide/interface.md"}'),i={name:"guide/interface.md"},y=a("h1",{id:"interface",tabindex:"-1"},[s("Interface "),a("a",{class:"header-anchor",href:"#interface","aria-label":'Permalink to "Interface"'},"​")],-1),D=a("p",null,[a("a",{href:"./element.html"},"Elements"),s(" by themselves are limited in that they are effectively stateless. You can construct and use them but any computed result or additional information will not be persisted.")],-1),F=e(`
python
from machinable import Interface
 
 class MnistData(Interface):
   """A dataset of handwritten characters"""
@@ -21,7 +21,7 @@ import{_ as t,D as p,o as c,c as r,z as a,a as s,G as l,B as o,O as e}from"./chu
 ├─ estimate_gravity.py            # contains a data analysis component
 ├─ evolution/                   
 |  └─ simulate_offspring.py       # contains a evolutionary simulation
-└─ main.py                        # main script to execute
The benefit of this requirement is that you can refer to the classes via their module import path. For example, using this module convention, you can simplify the instantiation of classes that are located in different modules:
python
from machinable import get
+└─ main.py                        # main script to execute
The benefit of this requirement is that you can refer to the classes via their module import path. For example, using this module convention, you can simplify the instantiation of classes that are located in different modules:
python
from machinable import get
 
 from estimate_gravity import EstimateGravity
 from evolution.simulate_offspring import SimulateOffspring
@@ -40,4 +40,4 @@ import{_ as t,D as p,o as c,c as r,z as a,a as s,G as l,B as o,O as e}from"./chu
 mnist = get("mnist_data")
 with open(mnist.local_directory("download_script.sh"), "w") as f:
     f.write(...)
-    os.chmod(f.name, 0o755)
Overall, interfaces make it easy to associate data with code as instantiation, storage and retrieval are managed automatically behind the scenes.
`,2);function _(b,v,E,T,q,w){const n=p("Pydoc");return c(),r("div",null,[y,D,a("p",null,[s("To enable storage and retrival we can use an "),l(n,null,{default:o(()=>[s("machinable.Interface")]),_:1}),s(" class.")]),F,a("p",null,[s("Specifically, to instantiate an interface (e.g. "),A,s(") we can leverage the "),l(n,null,{default:o(()=>[s("machinable.get")]),_:1}),s(" function, which takes a class as the first argument and optional constructor arguments.")]),d,a("p",null,[s("What is happening here is that "),l(n,{caption:"get()"},{default:o(()=>[s("machinable.get")]),_:1}),s(" automatically searches the storage for an interface of type "),C,s(" with a "),u,s(" of "),h,s(". If such an instance has not been committed yet (like when initially running the code), a new instance with this configuration will be returned. But if such an instance has previously been committed, it will simply be reloaded.")]),m,a("p",null,[s("While machinable automatically commits crucial information about the interface, you can use "),l(n,null,{default:o(()=>[s("machinable.Interface.save_file")]),_:1}),s(" and "),l(n,null,{default:o(()=>[s("machinable.Interface.load_file")]),_:1}),s(" to easily store and retrieve additional custom data in different file formats:")]),f,a("p",null,[s("This may be useful to save and restore some custom state of the interface. Furthermore, you are free to implement your own methods to persist data by writing and reading from the interface's "),l(n,{caption:"local_directory()"},{default:o(()=>[s("machinable.Interface.local_directory")]),_:1}),s(":")]),g])}const S=t(i,[["render",_]]);export{I as __pageData,S as default};
+    os.chmod(f.name, 0o755)
Overall, interfaces make it easy to associate data with code as instantiation, storage and retrieval are managed automatically behind the scenes.
`,2);function _(b,v,E,q,T,w){const n=p("Pydoc");return c(),r("div",null,[y,D,a("p",null,[s("To enable storage and retrival we can use an "),l(n,null,{default:o(()=>[s("machinable.Interface")]),_:1}),s(" class.")]),F,a("p",null,[s("Specifically, to instantiate an interface (e.g. "),A,s(") we can leverage the "),l(n,null,{default:o(()=>[s("machinable.get")]),_:1}),s(" function, which takes a class as the first argument and optional constructor arguments.")]),d,a("p",null,[s("What is happening here is that "),l(n,{caption:"get()"},{default:o(()=>[s("machinable.get")]),_:1}),s(" automatically searches the storage for an interface of type "),C,s(" with a "),u,s(" of "),h,s(". If such an instance has not been committed yet (like when initially running the code), a new instance with this configuration will be returned. But if such an instance has previously been committed, it will simply be reloaded.")]),m,a("p",null,[s("While machinable automatically commits crucial information about the interface, you can use "),l(n,null,{default:o(()=>[s("machinable.Interface.save_file")]),_:1}),s(" and "),l(n,null,{default:o(()=>[s("machinable.Interface.load_file")]),_:1}),s(" to easily store and retrieve additional custom data in different file formats:")]),f,a("p",null,[s("This may be useful to save and restore some custom state of the interface. Furthermore, you are free to implement your own methods to persist data by writing and reading from the interface's "),l(n,{caption:"local_directory()"},{default:o(()=>[s("machinable.Interface.local_directory")]),_:1}),s(":")]),g])}const S=t(i,[["render",_]]);export{k as __pageData,S as default};
diff --git a/assets/guide_interface.md.0cb088fc.lean.js b/assets/guide_interface.md.72ed9d18.lean.js
similarity index 92%
rename from assets/guide_interface.md.0cb088fc.lean.js
rename to assets/guide_interface.md.72ed9d18.lean.js
index 9699c30e..48e28fb1 100644
--- a/assets/guide_interface.md.0cb088fc.lean.js
+++ b/assets/guide_interface.md.72ed9d18.lean.js
@@ -1 +1 @@
-import{_ as t,D as p,o as c,c as r,z as a,a as s,G as l,B as o,O as e}from"./chunks/framework.62020867.js";const I=JSON.parse('{"title":"Interface","description":"","frontmatter":{},"headers":[],"relativePath":"guide/interface.md","filePath":"guide/interface.md"}'),i={name:"guide/interface.md"},y=a("h1",{id:"interface",tabindex:"-1"},[s("Interface "),a("a",{class:"header-anchor",href:"#interface","aria-label":'Permalink to "Interface"'},"​")],-1),D=a("p",null,[a("a",{href:"./element.html"},"Elements"),s(" by themselves are limited in that they are effectively stateless. You can construct and use them but any computed result or additional information will not be persisted.")],-1),F=e("",6),A=a("code",null,"MnistData()",-1),d=e("",3),C=a("code",null,"MnistData",-1),u=a("code",null,"batch_size",-1),h=a("code",null,"8",-1),m=e("",7),f=e("",1),g=e("",2);function _(b,v,E,T,q,w){const n=p("Pydoc");return c(),r("div",null,[y,D,a("p",null,[s("To enable storage and retrival we can use an "),l(n,null,{default:o(()=>[s("machinable.Interface")]),_:1}),s(" class.")]),F,a("p",null,[s("Specifically, to instantiate an interface (e.g. "),A,s(") we can leverage the "),l(n,null,{default:o(()=>[s("machinable.get")]),_:1}),s(" function, which takes a class as the first argument and optional constructor arguments.")]),d,a("p",null,[s("What is happening here is that "),l(n,{caption:"get()"},{default:o(()=>[s("machinable.get")]),_:1}),s(" automatically searches the storage for an interface of type "),C,s(" with a "),u,s(" of "),h,s(". If such an instance has not been committed yet (like when initially running the code), a new instance with this configuration will be returned. But if such an instance has previously been committed, it will simply be reloaded.")]),m,a("p",null,[s("While machinable automatically commits crucial information about the interface, you can use "),l(n,null,{default:o(()=>[s("machinable.Interface.save_file")]),_:1}),s(" and "),l(n,null,{default:o(()=>[s("machinable.Interface.load_file")]),_:1}),s(" to easily store and retrieve additional custom data in different file formats:")]),f,a("p",null,[s("This may be useful to save and restore some custom state of the interface. Furthermore, you are free to implement your own methods to persist data by writing and reading from the interface's "),l(n,{caption:"local_directory()"},{default:o(()=>[s("machinable.Interface.local_directory")]),_:1}),s(":")]),g])}const S=t(i,[["render",_]]);export{I as __pageData,S as default};
+import{_ as t,D as p,o as c,c as r,z as a,a as s,G as l,B as o,O as e}from"./chunks/framework.62020867.js";const k=JSON.parse('{"title":"Interface","description":"","frontmatter":{},"headers":[],"relativePath":"guide/interface.md","filePath":"guide/interface.md"}'),i={name:"guide/interface.md"},y=a("h1",{id:"interface",tabindex:"-1"},[s("Interface "),a("a",{class:"header-anchor",href:"#interface","aria-label":'Permalink to "Interface"'},"​")],-1),D=a("p",null,[a("a",{href:"./element.html"},"Elements"),s(" by themselves are limited in that they are effectively stateless. You can construct and use them but any computed result or additional information will not be persisted.")],-1),F=e("",6),A=a("code",null,"MnistData()",-1),d=e("",3),C=a("code",null,"MnistData",-1),u=a("code",null,"batch_size",-1),h=a("code",null,"8",-1),m=e("",7),f=e("",1),g=e("",2);function _(b,v,E,q,T,w){const n=p("Pydoc");return c(),r("div",null,[y,D,a("p",null,[s("To enable storage and retrival we can use an "),l(n,null,{default:o(()=>[s("machinable.Interface")]),_:1}),s(" class.")]),F,a("p",null,[s("Specifically, to instantiate an interface (e.g. "),A,s(") we can leverage the "),l(n,null,{default:o(()=>[s("machinable.get")]),_:1}),s(" function, which takes a class as the first argument and optional constructor arguments.")]),d,a("p",null,[s("What is happening here is that "),l(n,{caption:"get()"},{default:o(()=>[s("machinable.get")]),_:1}),s(" automatically searches the storage for an interface of type "),C,s(" with a "),u,s(" of "),h,s(". If such an instance has not been committed yet (like when initially running the code), a new instance with this configuration will be returned. But if such an instance has previously been committed, it will simply be reloaded.")]),m,a("p",null,[s("While machinable automatically commits crucial information about the interface, you can use "),l(n,null,{default:o(()=>[s("machinable.Interface.save_file")]),_:1}),s(" and "),l(n,null,{default:o(()=>[s("machinable.Interface.load_file")]),_:1}),s(" to easily store and retrieve additional custom data in different file formats:")]),f,a("p",null,[s("This may be useful to save and restore some custom state of the interface. Furthermore, you are free to implement your own methods to persist data by writing and reading from the interface's "),l(n,{caption:"local_directory()"},{default:o(()=>[s("machinable.Interface.local_directory")]),_:1}),s(":")]),g])}const S=t(i,[["render",_]]);export{k as __pageData,S as default};
diff --git a/assets/guide_introduction.md.cf46eed6.js b/assets/guide_introduction.md.c4095de6.js
similarity index 97%
rename from assets/guide_introduction.md.cf46eed6.js
rename to assets/guide_introduction.md.c4095de6.js
index b529d471..a335ce5d 100644
--- a/assets/guide_introduction.md.cf46eed6.js
+++ b/assets/guide_introduction.md.c4095de6.js
@@ -1,4 +1,4 @@
-import{_ as s,o as a,c as n,O as l}from"./chunks/framework.62020867.js";const C=JSON.parse('{"title":"Introduction","description":"","frontmatter":{},"headers":[],"relativePath":"guide/introduction.md","filePath":"guide/introduction.md"}'),o={name:"guide/introduction.md"},p=l(`
Introduction ​
What is machinable? ​
machinable is a Python API for research code. It provides an object-oriented skeleton that helps you develop and experiment in a unified interface while handling tedious housekeeping behind the scenes.
The key idea is to unify the running of code and the retrieval of produced results in one abstraction. A detailed discussion of this approach can be found in the about section, but for now, here is a minimal example that illustrates the idea.
  1. Write some code
python
from random import random
+import{_ as s,o as a,c as n,O as l}from"./chunks/framework.62020867.js";const C=JSON.parse('{"title":"Introduction","description":"","frontmatter":{},"headers":[],"relativePath":"guide/introduction.md","filePath":"guide/introduction.md"}'),o={name:"guide/introduction.md"},p=l(`
Introduction ​
What is machinable? ​
machinable is a Python API for research code. It provides an object-oriented skeleton that helps you develop and experiment in a unified interface while handling tedious housekeeping behind the scenes.
The key idea is to unify the running of code and the retrieval of produced results in one abstraction. A detailed discussion of this approach can be found in the about section, but for now, here is a minimal example that illustrates the idea.
  1. Write some code
python
from random import random
 
 from pydantic import BaseModel, Field
 
@@ -26,7 +26,7 @@ import{_ as s,o as a,c as n,O as l}from"./chunks/framework.62020867.js";const C=
             print(
                 f"After {self.config.samples} samples, "
                 f"PI is approximately {self.load_file('result.json')['pi']}."
-            )
  1. Run and inspect it using a unified abstraction
python
from machinable import get
+            )
  1. Run and inspect it using a unified abstraction
python
from machinable import get
 
 # Imports component in \`montecarlo.py\` with samples=150;
 # if an component with this configuration exists, it
diff --git a/assets/guide_introduction.md.cf46eed6.lean.js b/assets/guide_introduction.md.c4095de6.lean.js
similarity index 100%
rename from assets/guide_introduction.md.cf46eed6.lean.js
rename to assets/guide_introduction.md.c4095de6.lean.js
diff --git a/examples/aimstack-storage/index.html b/examples/aimstack-storage/index.html
index dd0ca663..b751a2f2 100644
--- a/examples/aimstack-storage/index.html
+++ b/examples/aimstack-storage/index.html
@@ -11,7 +11,7 @@
   
   
   
-  
+  
   
   
   
@@ -28,7 +28,7 @@
 
 get("aimstack", {"repo": "./path/to/aim-repo"}).__enter__()
 
-# your code
Source ​
py
from typing import List, Optional
+# your code
Source ​
py
from typing import List, Optional
 
 import os
 
@@ -84,7 +84,7 @@
 
         for k, v in interface.__model__.model_dump().items():
             run[k] = v
MIT Licensed
Copyright © 2021-present

-    
     
   
diff --git a/examples/dependent-schedules/index.html b/examples/dependent-schedules/index.html
index 929317ab..1e0b8e1b 100644
--- a/examples/dependent-schedules/index.html
+++ b/examples/dependent-schedules/index.html
@@ -25,7 +25,7 @@
   
   
     
machinable
Appearance
On this page
MIT Licensed
Copyright © 2021-present

-    
     
   
diff --git a/examples/globus-storage/index.html b/examples/globus-storage/index.html
index 7630a07e..fd4202e0 100644
--- a/examples/globus-storage/index.html
+++ b/examples/globus-storage/index.html
@@ -11,7 +11,7 @@
   
   
   
-  
+  
   
   
   
@@ -39,7 +39,7 @@
 print(matches)
 
 # download from globus
-storage.download(matches[0].uuid)
Source ​
py
from typing import TYPE_CHECKING, List
+storage.download(matches[0].uuid)
Source ​
py
from typing import TYPE_CHECKING, List
 
 import os
 import time
@@ -265,7 +265,7 @@
                 found.append(item["name"])
 
         return found
MIT Licensed
Copyright © 2021-present

-    
     
   
diff --git a/examples/index.html b/examples/index.html
index 7070dea0..af3cd931 100644
--- a/examples/index.html
+++ b/examples/index.html
@@ -25,7 +25,7 @@
   
   
     
machinable
Appearance
On this page
How-to examples ​
This section presents code examples that demonstrate real-world usage and my be a good starting point when implementing your own projects.
Please select an example from the menu.
MIT Licensed
Copyright © 2021-present

-    
     
   
diff --git a/examples/mpi-execution/index.html b/examples/mpi-execution/index.html
index 5910c5b4..2295e109 100644
--- a/examples/mpi-execution/index.html
+++ b/examples/mpi-execution/index.html
@@ -11,7 +11,7 @@
   
   
   
-  
+  
   
   
   
@@ -27,7 +27,7 @@
     
machinable
Appearance
On this page
MPI execution ​
Integration to launch MPI jobs.
Usage example ​
python
from machinable import get
 
 with get("mpi", {"ranks": 8}):
-    ... # your MPI ready component
Source ​
py
from typing import Literal, Optional, Union
+    ... # your MPI ready component
Source ​
py
from typing import Literal, Optional, Union
 
 import sys
 
@@ -151,7 +151,7 @@
                     raise KeyboardInterrupt(
                         "Interrupting `" + " ".join(cmd) + "`"
                     ) from _ex
MIT Licensed
Copyright © 2021-present

-    
     
   
diff --git a/examples/require-execution/index.html b/examples/require-execution/index.html
index 8fb7c78e..f5a6bf8f 100644
--- a/examples/require-execution/index.html
+++ b/examples/require-execution/index.html
@@ -11,7 +11,7 @@
   
   
   
-  
+  
   
   
   
@@ -27,7 +27,7 @@
     
machinable
Appearance
On this page
Require execution ​
A way to assert that components have been cached.
Usage example ​
python
from machinable import get
 
 with get("require"):
-    ... # components to check
Source ​
py
from machinable import Execution
+    ... # components to check
Source ​
py
from machinable import Execution
 
 
 class Require(Execution):
@@ -62,7 +62,7 @@
                 )
             )
         )
MIT Licensed
Copyright © 2021-present

-    
     
   
diff --git a/examples/slurm-execution/index.html b/examples/slurm-execution/index.html
index e8cfb754..b05f1252 100644
--- a/examples/slurm-execution/index.html
+++ b/examples/slurm-execution/index.html
@@ -11,7 +11,7 @@
   
   
   
-  
+  
   
   
   
@@ -27,15 +27,17 @@
     
machinable
Appearance
On this page
Slurm execution ​
Integration to submit to the Slurm scheduler.
Usage example ​
python
from machinable import get
 
 with get("slurm", {"ranks": 8, 'preamble': 'mpirun'}):
-    ... # your component
Source ​
py
from typing import Literal, Optional, Union
+    ... # your component
Source ​
py
from typing import Literal, Optional, Union
 
 import os
 import subprocess
 import sys
 import time
 
+import arrow
 import yaml
 from machinable import Execution, Project
+from machinable.config import to_dict
 from machinable.errors import ExecutionFailed
 from machinable.utils import chmodx, run_and_stream
 from pydantic import BaseModel, ConfigDict
@@ -130,7 +132,7 @@
         dry: bool = False
 
     def on_before_dispatch(self):
-        if self.config.confirm:
+        if self.config.confirm and not self.config.dry:
             return confirm(self)
 
     def on_compute_default_resources(self, executable):
@@ -196,8 +198,6 @@
             if ranks is not None:
                 if mpi_args:
                     mpi_args = mpi_args.replace("{ranks}", str(ranks))
-                else:
-                    mpi_args = "-n " + str(executable.config.get("ranks"))
             python = self.config.python or sys.executable
 
             # usage dependencies
@@ -237,12 +237,6 @@
             if self.config.preamble:
                 script += self.config.preamble
 
-            # add debug information
-            script += "\n"
-            script += f"# {executable.module} <{executable.id}>\n"
-            script += f"# {executable.local_directory()}>\n"
-            script += "\n"
-
             if mpi:
                 if mpi[-1] != " ":
                     mpi += " "
@@ -258,6 +252,16 @@
             print(f"Submitting job {executable} with resources: ")
             print(yaml.dump(resources))
 
+            # add debug information
+            script += "\n\n"
+            script += f"# generated at: {arrow.now()}\n"
+            script += f"# {executable.module} <{executable.id}>\n"
+            script += f"# {executable.local_directory()}\n\n"
+            script += "# " + yaml.dump(executable.version()).replace(
+                "\n", "\n# "
+            )
+            script += "\n"
+
             # submit to slurm
             script_file = chmodx(
                 self.save_file([executable.id, "slurm.sh"], script)
@@ -385,7 +389,7 @@
                 canonicalized[prefix + "--" + k] = str(v)
 
         return canonicalized
MIT Licensed
Copyright © 2021-present

-    
     
   
diff --git a/guide/component.html b/guide/component.html
index 27bee3f5..1ce71ac1 100644
--- a/guide/component.html
+++ b/guide/component.html
@@ -11,7 +11,7 @@
   
   
   
-  
+  
   
   
   
@@ -43,7 +43,7 @@
 >>> mnist.launch()
 Downloading 'mnist' ...

If the execution is successful, the component is marked as finished.

python
>>> mnist.execution.is_finished()
 True

By design, component instances can only be executed once. They are automatically assigned a timestamp, random seed, as well as a nickname for easy identification.

python
>>> mnist.seed
-1632827863

Invocations of launch() after successful execution, do not trigger another execution since the component is marked as cached. On the other hand, if the execution failed, calling launch() will resume the execution with the same configuration.

Implementing custom execution ​

Components can be executed in different ways. You may, for example, like to run components using multiprocessing or execute in a cloud environment. However, instead of adding the execution logic directly to your component code, machinable makes it easy to separate concerns. You can encapsulate the execution implementation in its own execution class that can then be used to execute the component.

To implement an execution, implement an interface that inherits from the Execution base class, for example:

python
from multiprocessing import Pool
+1632827863

Invocations of launch() after successful execution, do not trigger another execution since the component is marked as cached. On the other hand, if the execution failed, calling launch() will resume the execution with the same configuration.

Implementing custom execution ​

Components can be executed in different ways. You may, for example, like to run components using multiprocessing or execute in a cloud environment. However, instead of adding the execution logic directly to your component code, machinable makes it easy to separate concerns. You can encapsulate the execution implementation in its own execution class that can then be used to execute the component.

To implement an execution, implement an interface that inherits from the Execution base class, for example:

python
from multiprocessing import Pool
 
 from machinable import Execution
 
@@ -66,7 +66,7 @@
 multiprocess = get("multiprocess", {'processes': 2})

Then, to use it, we can wrap the launch in the execution context:

python
with multiprocessing:
     mnist.launch()

Check out the execution examples that include generally useful implementations you may like to use in your projects.

Using the CLI ​

Components can be launched directly from the command-line. The CLI works out of the box and closely mirrors the Python interface. To run a component, type its module name and method name, optionally followed by the configuration options, for example:

bash
machinable mnist_data batch_size=4 --launch

To use multiprocessing, you may type:

bash
machinable mnist_data batch_size=4 \
   multiprocess processes=4 --launch

MIT Licensed

Copyright © 2021-present

- diff --git a/guide/element.html b/guide/element.html index 0ad042d1..85915b10 100644 --- a/guide/element.html +++ b/guide/element.html @@ -78,7 +78,7 @@ {'batch_size': 8, 'name': 'mnist'} >>> mnist.config._update_ {'batch_size': 1}

Elements support many more advanced configuration features such as typing, validation, parameter documentation, computed values, etc., which will be covered in later sections of the Guide. For now, to summarize, elements are classes with default configurations that may be modified with a list of configuration updates.

- diff --git a/guide/installation.html b/guide/installation.html index a1b51a31..43d8621f 100644 --- a/guide/installation.html +++ b/guide/installation.html @@ -25,7 +25,7 @@
machinable

Appearance

On this page

Installation ​

machinable is available via pip. Install the current release

bash
$ pip install machinable

INFO

machinable currently supports Python 3.8 and higher

Note that machinable requires the sqlite json1 extension, otherwise, you will likely see the error message: sqlite3.OperationalError: no such function: json_extract. In this case, an easy way to obtain a suitable sqlite version is to install the pysqlite package:

bash
$ pip install pysqlite3-binary

MIT Licensed

Copyright © 2021-present

- diff --git a/guide/interface.html b/guide/interface.html index e17f9972..e3723bbe 100644 --- a/guide/interface.html +++ b/guide/interface.html @@ -11,7 +11,7 @@ - + @@ -47,7 +47,7 @@ ├─ estimate_gravity.py # contains a data analysis component ├─ evolution/ | └─ simulate_offspring.py # contains a evolutionary simulation -└─ main.py # main script to execute

The benefit of this requirement is that you can refer to the classes via their module import path. For example, using this module convention, you can simplify the instantiation of classes that are located in different modules:

python
from machinable import get
+└─ main.py                        # main script to execute

The benefit of this requirement is that you can refer to the classes via their module import path. For example, using this module convention, you can simplify the instantiation of classes that are located in different modules:

python
from machinable import get
 
 from estimate_gravity import EstimateGravity
 from evolution.simulate_offspring import SimulateOffspring
@@ -67,7 +67,7 @@
 with open(mnist.local_directory("download_script.sh"), "w") as f:
     f.write(...)
     os.chmod(f.name, 0o755)

Overall, interfaces make it easy to associate data with code as instantiation, storage and retrieval are managed automatically behind the scenes.

- diff --git a/guide/introduction.html b/guide/introduction.html index 8f462d57..27c21b9f 100644 --- a/guide/introduction.html +++ b/guide/introduction.html @@ -11,7 +11,7 @@ - + @@ -24,7 +24,7 @@ -
machinable

Appearance

On this page

Introduction ​

What is machinable? ​

machinable is a Python API for research code. It provides an object-oriented skeleton that helps you develop and experiment in a unified interface while handling tedious housekeeping behind the scenes.

The key idea is to unify the running of code and the retrieval of produced results in one abstraction. A detailed discussion of this approach can be found in the about section, but for now, here is a minimal example that illustrates the idea.

  1. Write some code
python
from random import random
+    
machinable
Appearance
On this page
Introduction ​
What is machinable? ​
machinable is a Python API for research code. It provides an object-oriented skeleton that helps you develop and experiment in a unified interface while handling tedious housekeeping behind the scenes.
The key idea is to unify the running of code and the retrieval of produced results in one abstraction. A detailed discussion of this approach can be found in the about section, but for now, here is a minimal example that illustrates the idea.
  1. Write some code
python
from random import random
 
 from pydantic import BaseModel, Field
 
@@ -52,7 +52,7 @@
             print(
                 f"After {self.config.samples} samples, "
                 f"PI is approximately {self.load_file('result.json')['pi']}."
-            )
  1. Run and inspect it using a unified abstraction
python
from machinable import get
+            )
  1. Run and inspect it using a unified abstraction
python
from machinable import get
 
 # Imports component in `montecarlo.py` with samples=150;
 # if an component with this configuration exists, it
@@ -76,7 +76,7 @@
 >>> experiment.local_directory()
 './storage/24aee0fd05024400b116593d1436e9f5'
bash
$ machinable montecarlo samples=150 --launch --summary
 > After 150 samples, PI is approximately 3.1466666666666665.
The above example demonstrates the two core principles of machinable code:
  • Enforced modularity The Monte Carlo algorithm is encapsulated in its own module that can be instantiated with different configuration settings.
  • Unified representation Running code is handled through the same interface that is used to retrieve produced results; multiple invocations simply reload and display the results without re-running the experiment.
You may already have questions - don't worry. We will cover the details in the rest of the documentation. For now, please read along so you can have a high-level understanding of what machinable offers.
What it is not ​
Research is extremely diverse so machinable primarily aims to be an API-spec that leaves concrete feature implementation to the user. Check out the examples to learn what this looks like in practice.
Where to go from here ​
⚙️   Installation
We recommend installing machinable to try things out while following along.
🧑‍🎓   Continue with the Guide
Designed to learn concepts hands-on. Starts with the bare minimum of concepts necessary to start using machinable. Along the way, it will provide pointers to sections that discuss concepts in more detail or cover more advanced functionality.
➡️   Check out the How-to guides
Explore real-world examples that demonstrate advanced concepts
📖   Consult the Reference
Describes available APIs in full detail.
MIT Licensed
Copyright © 2021-present

-    
     
   
diff --git a/hashmap.json b/hashmap.json
index f67bdf34..7f5d2ee3 100644
--- a/hashmap.json
+++ b/hashmap.json
@@ -1 +1 @@
-{"guide_component.md":"771e9d5c","guide_installation.md":"59aa7ee4","reference_index.md":"a073225b","guide_introduction.md":"cf46eed6","guide_interface.md":"0cb088fc","guide_element.md":"b35f8b9f","examples_index.md":"6a333571","index.md":"ec5fddb7","about_approach.md":"40736853","examples_aimstack-storage_index.md":"880b0b35","examples_mpi-execution_index.md":"e2d88fe1","examples_globus-storage_index.md":"68197128","examples_dependent-schedules_index.md":"150d073c","examples_require-execution_index.md":"36bcc19a","examples_slurm-execution_index.md":"b1b1e5c2"}
+{"about_approach.md":"40736853","examples_index.md":"6a333571","guide_component.md":"8f272292","examples_dependent-schedules_index.md":"150d073c","index.md":"ec5fddb7","examples_require-execution_index.md":"9f45fc51","guide_installation.md":"59aa7ee4","examples_mpi-execution_index.md":"5075bb2c","reference_index.md":"a073225b","examples_globus-storage_index.md":"620b6419","guide_introduction.md":"c4095de6","examples_aimstack-storage_index.md":"e95360b0","examples_slurm-execution_index.md":"292b100f","guide_element.md":"b35f8b9f","guide_interface.md":"72ed9d18"}
diff --git a/index.html b/index.html
index df610f89..3595e326 100644
--- a/index.html
+++ b/index.html
@@ -25,7 +25,7 @@
   
   
     
machinable
Appearance
machinable
research code
A modular system to manage research code so you can move quickly while enabling reuse and collaboration.
Get Started
View on GitHub
machinable-logo
🛠️
Unified representation
Run code and inspect results using the same abstraction. Check out the example below ⏬
⚡️
Designed for rapid iteration
Spend more time experimenting while relying on machinable to keep things organized.
💡
Hackable and interactive
Tweak, extend, override while leveraging first-class support for Jupyter as well as the CLI.




   💻 
Some research code
Running code ...
python regression.py --rate=0.1 --logs=1 --name=run-01
... and loading the corresponding results ...
python plot_regression_result.py --component=run-01
... are distinct and often redundant.
This means you have to manually keep track by remembering what the component with rate=0.1 was called.

machinable research code
Running code ...
machinable regression rate=0.1 logs_=True --launch
... and loading the corresponding results ...
machinable regression rate=0.1 logs_=True --launch --plot
... are distinct but use the same abstraction.
This means no need to worry about names as machinable automatically keeps track if you ran rate=0.1 before.

➡️ Learn more about machinable's approach



logo
MIT Licensed
Copyright © 2021-present

-    
     
   
diff --git a/reference/index.html b/reference/index.html
index 0ab93c53..555519ea 100644
--- a/reference/index.html
+++ b/reference/index.html
@@ -25,7 +25,7 @@
   
   
     
machinable
Appearance
On this page
Reference documentation ​
This page contains detailed API reference documentation. It is intended to be an in-depth resource for understanding the implementation details of machinable's interfaces. You may prefer reviewing the more explanatory guide before consulting this reference.
API ​
MIT Licensed
Copyright © 2021-present

-