Skip to content

Commit

Permalink
0.9.6 (#167)
Browse files Browse the repository at this point in the history 
* 🔧 Fix workdir in log

* 🔖 0.9.5

* 🎨 Let plugins change and create workdir

* 🔧 Change the default outdir suffix from _results to -output

* 📖 Update README file and add new plugins

* 🔖 0.9.6
  • Loading branch information
@pwwang
pwwang authored Apr 18, 2023
1 parent cfcf161 commit 25c3075
Show file tree
Hide file tree
Showing 6 changed files with 38 additions and 29 deletions.
22 changes: 12 additions & 10 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -66,11 +66,11 @@ if __name__ == "__main__":
[09/13/21 04:23:37] I main
[09/13/21 04:23:37] I main version: 0.9.0
[09/13/21 04:23:37] I main
[09/13/21 04:23:37] I main ╭═════════════════════════════ MYPIPELIN ═══════════════════════════════╮
[09/13/21 04:23:37] I main ╭═════════════════════════════ MYPIPELINE ══════════════════════════════╮
[09/13/21 04:23:37] I main ║ # procs = 2 ║
[09/13/21 04:23:37] I main ║ plugins = ['main', 'verbose-0.0.1'] ║
[09/13/21 04:23:37] I main ║ profile = default ║
[09/13/21 04:23:37] I main ║ outdir = mypipeline_results ║
[09/13/21 04:23:37] I main ║ outdir = MyPipeline-output ║
[09/13/21 04:23:37] I main ║ cache = True ║
[09/13/21 04:23:37] I main ║ dirsig = 1 ║
[09/13/21 04:23:37] I main ║ error_strategy = ignore ║
Expand All @@ -91,34 +91,34 @@ if __name__ == "__main__":
[09/13/21 04:23:37] I main ╭───────────────────────────────── P1 ──────────────────────────────────╮
[09/13/21 04:23:37] I main │ Sort input file │
[09/13/21 04:23:37] I main ╰───────────────────────────────────────────────────────────────────────╯
[09/13/21 04:23:37] I main P1: Workdir: '.pipen/mypipeline/p1'
[09/13/21 04:23:37] I main P1: Workdir: '.pipen/MyPipeline/p1'
[09/13/21 04:23:37] I main P1: <<< [START]
[09/13/21 04:23:37] I main P1: >>> ['P2']
[09/13/21 04:23:37] I verbose P1: size: 1
[09/13/21 04:23:37] I verbose P1: [0/0] in.infile: /tmp/data.txt
[09/13/21 04:23:37] I verbose P1: [0/0] out.outfile:
/home/pwwang/github/pipen/.pipen/mypipeline/p1/0/output/intermediate.txt
/home/pwwang/github/pipen/.pipen/MyPipeline/p1/0/output/intermediate.txt
[09/13/21 04:23:38] I verbose P1: Time elapsed: 00:00:01.039s
[09/13/21 04:23:38] I main
[09/13/21 04:23:38] I main ╭═════════════════════════════════ P2 ══════════════════════════════════╮
[09/13/21 04:23:38] I main ║ Paste line number ║
[09/13/21 04:23:38] I main ╰═══════════════════════════════════════════════════════════════════════╯
[09/13/21 04:23:38] I main P2: Workdir: '.pipen/mypipeline/p2'
[09/13/21 04:23:38] I main P2: Workdir: '.pipen/MyPipeline/p2'
[09/13/21 04:23:38] I main P2: <<< ['P1']
[09/13/21 04:23:38] I main P2: >>> [END]
[09/13/21 04:23:38] I verbose P2: size: 1
[09/13/21 04:23:38] I verbose P2: [0/0] in.infile:
/home/pwwang/github/pipen/.pipen/mypipeline/p1/0/output/intermediate.txt
/home/pwwang/github/pipen/.pipen/MyPipeline/p1/0/output/intermediate.txt
[09/13/21 04:23:38] I verbose P2: [0/0] out.outfile:
/home/pwwang/github/pipen/mypipeline_results/P2/result.txt
/home/pwwang/github/pipen/MyPipeline-output/P2/result.txt
[09/13/21 04:23:40] I verbose P2: Time elapsed: 00:00:02.074s
[09/13/21 04:23:40] I main
PIPEN-0: 100%|████████████████████████████████████████| 2/2 [00:04<00:00, 0.56 procs/s]
```

```shell
> cat ./mypipeline_results/P2/result.txt
> cat ./MyPipeline-output/P2/result.txt
1 1
2 2
3 3
Expand All @@ -141,9 +141,10 @@ Plugins make `pipen` even better.
- [`pipen-annotate`][26]: Use docstring to annotate pipen processes
- [`pipen-args`][19]: Command line argument parser for pipen
- [`pipen-dry`][20]: Dry runner for pipen pipelines
- [`pipen-log2file`][28]: Save running logs to file for pipen
- [`pipen-board`][27]: Visualize configuration and running of pipen pipelines on the web
- [`pipen-cli-init`][21]: A pipen CLI plugin to create a pipen project (pipeline)
- [`pipen-cli-run`][22]: A pipen cli plugin to run a process or a pipeline
- [`pipen-cli-config`][27]: UI wizard to generate configuration for pipen pipelines
- [`pipen-cli-require`][24]: A pipen cli plugin check the requirements of a pipeline


Expand Down Expand Up @@ -173,4 +174,5 @@ Plugins make `pipen` even better.
[24]: https://github.com/pwwang/pipen-cli-require
[25]: https://github.com/pwwang/pipen-lock
[26]: https://github.com/pwwang/pipen-annotate
[27]: https://github.com/pwwang/pipen-cli-config
[27]: https://github.com/pwwang/pipen-board
[28]: https://github.com/pwwang/pipen-log2file
6 changes: 6 additions & 0 deletions docs/CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# Change Log

## 0.9.6

- 🎨 Let plugins change and create workdir
- 🔧 Change the default outdir suffix from `_results` to `-output`
- 📖 Update README file and add new plugins

## 0.9.5

- 🔧 Fix workdir in log
Expand Down
30 changes: 15 additions & 15 deletions docs/examples.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ When run the script the second time, you may see from the logs that jobs are cac
[09/13/21 06:10:03] I main ║ # procs = 1 ║
[09/13/21 06:10:03] I main ║ plugins = ['main', 'verbose-0.0.1'] ║
[09/13/21 06:10:03] I main ║ profile = default ║
[09/13/21 06:10:03] I main ║ outdir = pipen-0_results ║
[09/13/21 06:10:03] I main ║ outdir = Pipen-output ║
[09/13/21 06:10:03] I main ║ cache = True ║
[09/13/21 06:10:03] I main ║ dirsig = 1 ║
[09/13/21 06:10:03] I main ║ error_strategy = ignore ║
Expand All @@ -50,7 +50,7 @@ When run the script the second time, you may see from the logs that jobs are cac
[09/13/21 06:10:03] I main AProcess: >>> [END]
[09/13/21 06:10:03] I verbose AProcess: size: 1
[09/13/21 06:10:03] I verbose AProcess: [0/0] in.infile: /tmp/pipen_example_caching.txt
[09/13/21 06:10:03] I verbose AProcess: [0/0] out.outfile: /home/pwwang/github/pipen/pipen-0_results/AProcess/pipen_example_caching.txt
[09/13/21 06:10:03] I verbose AProcess: [0/0] out.outfile: /home/pwwang/github/pipen/Pipen-output/AProcess/pipen_example_caching.txt
[09/13/21 06:10:03] I main AProcess: Cached jobs: 0
[09/13/21 06:10:03] I verbose AProcess: Time elapsed: 00:00:00.040s
[09/13/21 06:10:03] I main
Expand All @@ -75,7 +75,7 @@ To "de-cache" the jobs:
[09/13/21 06:11:55] I main ║ # procs = 1 ║
[09/13/21 06:11:55] I main ║ plugins = ['main', 'verbose-0.0.1'] ║
[09/13/21 06:11:55] I main ║ profile = default ║
[09/13/21 06:11:55] I main ║ outdir = pipen-0_results ║
[09/13/21 06:11:55] I main ║ outdir = Pipen-output ║
[09/13/21 06:11:55] I main ║ cache = 0 ║
[09/13/21 06:11:55] I main ║ dirsig = 1 ║
[09/13/21 06:11:55] I main ║ error_strategy = ignore ║
Expand Down Expand Up @@ -103,7 +103,7 @@ To "de-cache" the jobs:
[09/13/21 06:11:55] D main AProcess: [0/0] Not cached (proc.cache is False)
[09/13/21 06:11:55] D main AProcess: [0/0] Clearing previous output files.
[09/13/21 06:11:55] I verbose AProcess: [0/0] in.infile: /tmp/pipen_example_caching.txt
[09/13/21 06:11:55] I verbose AProcess: [0/0] out.outfile: /home/pwwang/github/pipen/pipen-0_results/AProcess/pipen_example_caching.txt
[09/13/21 06:11:55] I verbose AProcess: [0/0] out.outfile: /home/pwwang/github/pipen/Pipen-output/AProcess/pipen_example_caching.txt
[09/13/21 06:11:56] I verbose AProcess: Time elapsed: 00:00:01.060s
[09/13/21 06:11:56] I main
```
Expand All @@ -124,7 +124,7 @@ To "de-cache" the jobs:
[09/13/21 06:13:12] I main ║ # procs = 2 ║
[09/13/21 06:13:12] I main ║ plugins = ['main', 'verbose-0.0.1'] ║
[09/13/21 06:13:12] I main ║ profile = default ║
[09/13/21 06:13:12] I main ║ outdir = pipen-0_results ║
[09/13/21 06:13:12] I main ║ outdir = Pipen-output ║
[09/13/21 06:13:12] I main ║ cache = True ║
[09/13/21 06:13:12] I main ║ dirsig = 1 ║
[09/13/21 06:13:12] I main ║ error_strategy = ignore ║
Expand Down Expand Up @@ -162,13 +162,13 @@ To "de-cache" the jobs:
[09/13/21 06:13:15] I verbose P2: size: 10
[09/13/21 06:13:15] I verbose P2: [0/9] in.infile: /home/pwwang/github/pipen/.pipen/pipen-0/p1/0/output/intermediate.txt
[09/13/21 06:13:15] I verbose P2: [0/9] in.nlines: 2
[09/13/21 06:13:15] I verbose P2: [0/9] out.outfile: /home/pwwang/github/pipen/pipen-0_results/P2/0/result.txt
[09/13/21 06:13:15] I verbose P2: [0/9] out.outfile: /home/pwwang/github/pipen/Pipen-output/P2/0/result.txt
[09/13/21 06:13:17] I verbose P2: Time elapsed: 00:00:02.192s
[09/13/21 06:13:17] I main
```

```shell
❯ cat /home/pwwang/github/pipen/pipen-0_results/P2/0/result.txt
❯ cat /home/pwwang/github/pipen/Pipen-output/P2/0/result.txt
1 0_0
2 0_1
```
Expand All @@ -189,7 +189,7 @@ To "de-cache" the jobs:
[09/13/21 06:14:57] I main ║ # procs = 1 ║
[09/13/21 06:14:57] I main ║ plugins = ['main', 'verbose-0.0.1'] ║
[09/13/21 06:14:57] I main ║ profile = default ║
[09/13/21 06:14:57] I main ║ outdir = pipen-0_results ║
[09/13/21 06:14:57] I main ║ outdir = Pipen-output ║
[09/13/21 06:14:57] I main ║ cache = True ║
[09/13/21 06:14:57] I main ║ dirsig = 1 ║
[09/13/21 06:14:57] I main ║ error_strategy = ignore ║
Expand All @@ -216,7 +216,7 @@ To "de-cache" the jobs:
[09/13/21 06:14:57] I verbose MakoProcess: size : 1
[09/13/21 06:14:57] I verbose MakoProcess: template: mako
[09/13/21 06:14:57] I verbose MakoProcess: [0/0] in.a: 1
[09/13/21 06:14:57] I verbose MakoProcess: [0/0] out.outfile: /home/pwwang/github/pipen/pipen-0_results/MakoProcess/1.txt
[09/13/21 06:14:57] I verbose MakoProcess: [0/0] out.outfile: /home/pwwang/github/pipen/Pipen-output/MakoProcess/1.txt
[09/13/21 06:14:58] I verbose MakoProcess: Time elapsed: 00:00:01.019s
[09/13/21 06:14:58] I main
```
Expand All @@ -237,7 +237,7 @@ To "de-cache" the jobs:
[09/13/21 06:16:09] I main ║ # procs = 1 ║
[09/13/21 06:16:09] I main ║ plugins = ['main', 'verbose-0.0.1'] ║
[09/13/21 06:16:09] I main ║ profile = default ║
[09/13/21 06:16:09] I main ║ outdir = pipen-0_results ║
[09/13/21 06:16:09] I main ║ outdir = Pipen-output ║
[09/13/21 06:16:09] I main ║ cache = True ║
[09/13/21 06:16:09] I main ║ dirsig = 1 ║
[09/13/21 06:16:09] I main ║ error_strategy = ignore ║
Expand Down Expand Up @@ -265,7 +265,7 @@ To "de-cache" the jobs:
[09/13/21 06:16:10] I verbose MultiJobProc: cache: False
[09/13/21 06:16:10] I verbose MultiJobProc: size : 10
[09/13/21 06:16:10] I verbose MultiJobProc: [0/9] in.i: 0
[09/13/21 06:16:10] I verbose MultiJobProc: [0/9] out.outfile: /home/pwwang/github/pipen/pipen-0_results/MultiJobProc/0/0.txt
[09/13/21 06:16:10] I verbose MultiJobProc: [0/9] out.outfile: /home/pwwang/github/pipen/Pipen-output/MultiJobProc/0/0.txt
[09/13/21 06:16:16] I verbose MultiJobProc: Time elapsed: 00:00:06.139s
[09/13/21 06:16:16] I main
```
Expand All @@ -287,7 +287,7 @@ To "de-cache" the jobs:
[09/13/21 06:18:18] I main ║ # procs = 1 ║
[09/13/21 06:18:18] I main ║ plugins = ['main', 'notifyplugin-0.0.0'] ║
[09/13/21 06:18:18] I main ║ profile = default ║
[09/13/21 06:18:18] I main ║ outdir = pipen-0_results ║
[09/13/21 06:18:18] I main ║ outdir = Pipen-output ║
[09/13/21 06:18:18] I main ║ cache = True ║
[09/13/21 06:18:18] I main ║ dirsig = 1 ║
[09/13/21 06:18:18] I main ║ error_strategy = ignore ║
Expand Down Expand Up @@ -338,7 +338,7 @@ To "de-cache" the jobs:
[09/13/21 06:19:45] I main ║ # procs = 1 ║
[09/13/21 06:19:45] I main ║ plugins = ['main', 'verbose-0.0.1'] ║
[09/13/21 06:19:45] I main ║ profile = default ║
[09/13/21 06:19:45] I main ║ outdir = pipen-0_results ║
[09/13/21 06:19:45] I main ║ outdir = Pipen-output ║
[09/13/21 06:19:46] I main ║ cache = True ║
[09/13/21 06:19:46] I main ║ dirsig = 1 ║
[09/13/21 06:19:46] I main ║ error_strategy = ignore ║
Expand Down Expand Up @@ -366,7 +366,7 @@ To "de-cache" the jobs:
[09/13/21 06:19:46] I verbose PythonScriptProc: size: 1
[09/13/21 06:19:46] I verbose PythonScriptProc: [0/0] in.a: 1
[09/13/21 06:19:46] I verbose PythonScriptProc: [0/0] out.outfile:
/home/pwwang/github/pipen/pipen-0_results/PythonScriptProc/1.txt
/home/pwwang/github/pipen/Pipen-output/PythonScriptProc/1.txt
[09/13/21 06:19:48] I verbose PythonScriptProc: Time elapsed: 00:00:02.031s
[09/13/21 06:19:48] I main
```
Expand All @@ -390,7 +390,7 @@ To "de-cache" the jobs:
[09/13/21 06:20:38] I main ║ # procs = 1 ║
[09/13/21 06:20:38] I main ║ plugins = ['main', 'verbose-0.0.1'] ║
[09/13/21 06:20:38] I main ║ profile = default ║
[09/13/21 06:20:38] I main ║ outdir = pipen-0_results ║
[09/13/21 06:20:38] I main ║ outdir = Pipen-output ║
[09/13/21 06:20:38] I main ║ cache = True ║
[09/13/21 06:20:38] I main ║ dirsig = 1 ║
[09/13/21 06:20:38] I main ║ error_strategy = ignore ║
Expand Down
5 changes: 3 additions & 2 deletions pipen/pipen.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ def __init__(
desc or self.__class__.desc or desc_from_docstring(self.__class__)
)
self.outdir = Path(
outdir or self.__class__.outdir or f"./{self.name}_results"
outdir or self.__class__.outdir or f"./{self.name}-output"
).resolve()
self.workdir: Path = None
self.profile: str = "default"
Expand Down Expand Up @@ -172,7 +172,7 @@ async def async_run(self, profile: str = "default") -> bool:
"""
self.profile = profile
self.workdir = Path(self.config.workdir) / self.name
self.workdir.mkdir(parents=True, exist_ok=True)
# self.workdir.mkdir(parents=True, exist_ok=True)

succeeded = True
await self._init()
Expand Down Expand Up @@ -367,6 +367,7 @@ async def _init(self) -> None:
# configs from files and CONFIG are loaded
# allow plugins to change the default configs
await plugin.hooks.on_init(self)
self.workdir.mkdir(parents=True, exist_ok=True)
# Then load the extra configurations passed from __init__(**kwargs)
# Make sure dict options get inherited
self.config.template_opts.update(self._kwargs.pop("template_opts", {}))
Expand Down
2 changes: 1 addition & 1 deletion pipen/version.py
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
"""Provide version of pipen"""

__version__ = "0.9.5"
__version__ = "0.9.6"
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "poetry.masonry.api"

[tool.poetry]
name = "pipen"
version = "0.9.5"
version = "0.9.6"
description = "A pipeline framework for python"
authors = [ "pwwang <[email protected]>",]
license = "MIT"
Expand Down

0 comments on commit 25c3075

Please sign in to comment.