Skip to content

Commit afdc77b

Browse files
freddiev4freddiev4
committed
Initial commit
0 parents  commit afdc77b

File tree

11 files changed

+585
-0
lines changed

11 files changed

+585
-0
lines changed

‎.github/workflows/publish-package.yaml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
name: Publish package to PyPI
2+
3+
on:
4+
release:
5+
types: [published]
6+
7+
jobs:
8+
pypi:
9+
runs-on: ubuntu-latest
10+
steps:
11+
- name: Checkout
12+
uses: actions/checkout@v3
13+
with:
14+
fetch-depth: 0
15+
- run: python3 -m pip install --upgrade build && python3 -m build
16+
17+
- name: Publish package
18+
uses: pypa/gh-action-pypi-publish@release/v1
19+
with:
20+
password: ${{ secrets.PYPI_API_TOKEN }}

‎.github/workflows/run-tests.yaml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
name: Run Python unit tests
2+
3+
on:
4+
push:
5+
branches: [ "main" ]
6+
pull_request:
7+
branches: [ "main" ]
8+
9+
jobs:
10+
build:
11+
12+
runs-on: ubuntu-latest
13+
strategy:
14+
matrix:
15+
python-version: ["3.8", "3.9", "3.10"]
16+
17+
steps:
18+
- uses: actions/checkout@v3
19+
- name: Set up Python ${{ matrix.python-version }}
20+
uses: actions/setup-python@v4
21+
with:
22+
python-version: ${{ matrix.python-version }}
23+
24+
- name: Install pytest
25+
run: |
26+
pip install -U pip
27+
pip install pytest
28+
- name: Run tests with pytest
29+
run: |
30+
pytest

‎.gitignore‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
.idea/
2+
__pycache__/
3+
4+
*.pyc
5+
*.swp
6+
*.DS_Store
7+
*.egg-info
8+
9+
*.env

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2023 Freddie Vargus
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
# tinypipeline
2+
3+
## Overview
4+
5+
`tinypipeline` is a tiny mlops library that provides a simple framework for organizing your machine learning pipeline code into a series of steps. It does not handle networking, I/O, or compute resources. You do the rest in your pipeline steps.
6+
7+
## Installation
8+
9+
```
10+
$ pip install tinypipeline
11+
```
12+
13+
## Usage
14+
15+
`tinypipeline` exposes two main objects:
16+
- `pipeline`: a decorator for defining your pipeline. Returns a `Pipeline` instance.
17+
- `step`: a function that is used to define individual pipeline steps. Returns a `Step` instance.
18+
19+
Each object requires you provide a `name`, `version`, and `description` to explicitly define what pipeline you're creating.
20+
21+
The `Pipeline` object that is returned from the decorator has a single method: `run()`.
22+
23+
## API
24+
25+
If you'd like to use this package, you can follow the `example.py` below:
26+
27+
```python
28+
from tinypipeline import pipeline, step
29+
30+
31+
def step_fn_one():
32+
print("Step function one")
33+
34+
def step_fn_two():
35+
print("Step function two")
36+
37+
@pipeline(
38+
name='test-pipeline',
39+
version='0.0.1',
40+
description='a test tinypipeline',
41+
)
42+
def pipe():
43+
step_one = step(
44+
callable=step_fn_one,
45+
name='step_one',
46+
version='0.0.1',
47+
description='first step',
48+
)
49+
step_two = step(
50+
name='step_two',
51+
version='0.0.1',
52+
description='second step',
53+
callable=step_fn_two,
54+
)
55+
return [step_one, step_two]
56+
57+
pipe = pipe()
58+
pipe.run()
59+
```
60+
61+
**Output**:
62+
63+
You can run the `example.py` like so:
64+
65+
```console
66+
$ python example.py
67+
+-------------------------------------------------------------------+
68+
| Running pipeline: Pipeline(name='test-pipeline', version='0.0.1') |
69+
+-------------------------------------------------------------------+
70+
71+
Running step [step_one]...
72+
Step function one
73+
Step [step_one] completed in 0.000356 seconds
74+
75+
Running step [step_two]...
76+
Step function two
77+
Step [step_two] completed in 0.000317 seconds
78+
```
79+
80+
## Running tests
81+
82+
Tests are run using pytest. To run the tests you can do:
83+
84+
```console
85+
$ pip install pytest
86+
$ pytest
87+
```
88+
89+
## Limitations
90+
91+
- Currently `tinypipeline` only supports linear pipelines, and doesn't support full dependency graphs

‎pyproject.toml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
[build-system]
2+
requires = ["setuptools", "setuptools-scm"]
3+
build-backend = "setuptools.build_meta"
4+
5+
[project]
6+
name = "tinypipeline"
7+
version = "0.1.0"
8+
authors = [
9+
{name = "Freddie Vargus", email = "[email protected]"},
10+
]
11+
description = "A tiny mlops library for building machine learning pipelines on your local machine."
12+
readme = "README.md"
13+
requires-python = ">=3.8"
14+
keywords = ["mlops", "machine learning", "pipelines"]
15+
license = {text = "MIT"}
16+
classifiers = [
17+
"Programming Language :: Python :: 3",
18+
"License :: OSI Approved :: MIT License",
19+
]

‎tests/__init__.py‎

Whitespace-only changes.

‎tests/test_pipeline.py‎

Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,193 @@
1+
from unittest.mock import Mock
2+
3+
import pytest
4+
5+
from tinypipeline import pipeline, step
6+
7+
8+
def test_pipeline_completion():
9+
"""
10+
Test that the pipeline runs all the steps in the correct order.
11+
"""
12+
mock_step_1 = Mock()
13+
mock_step_2 = Mock()
14+
mock_step_3 = Mock()
15+
16+
mock_step_1.return_value = "step 1"
17+
mock_step_2.return_value = "step 2"
18+
mock_step_3.return_value = "step 3"
19+
20+
@pipeline(
21+
name="test_pipeline",
22+
description="Test pipeline",
23+
version="1.0.0",
24+
)
25+
def test_pipeline():
26+
step_one = step(
27+
callable=mock_step_1,
28+
name="step_one",
29+
description="Step one",
30+
version="1.0.0",
31+
)
32+
33+
step_two = step(
34+
callable=mock_step_2,
35+
name="step_two",
36+
description="Step two",
37+
version="1.0.0",
38+
)
39+
40+
step_three = step(
41+
callable=mock_step_3,
42+
name="step_three",
43+
description="Step three",
44+
version="1.0.0",
45+
)
46+
47+
return [step_one, step_two, step_three]
48+
49+
pipe = test_pipeline()
50+
pipe.run()
51+
52+
assert len(pipe.steps) == 3
53+
mock_step_1.assert_called_once()
54+
mock_step_2.assert_called_once()
55+
mock_step_3.assert_called_once()
56+
57+
58+
def test_pipeline_failure_no_function_passed():
59+
"""
60+
Test that the pipeline fails with a TypeError if no function is passed
61+
to the method.
62+
63+
Also check that the error message is correct.
64+
"""
65+
data = ""
66+
with pytest.raises(TypeError) as context:
67+
pipe = pipeline(
68+
name="test_pipeline",
69+
description="Test pipeline",
70+
version="1.0.0",
71+
)
72+
pipe(data)().run()
73+
74+
assert (
75+
f"The pipeline decorator only accepts functions. Passed {type(data)}"
76+
== str(context.value)
77+
)
78+
79+
def test_pipeline_failure_invalid_step():
80+
"""
81+
Test that the pipeline fails with a TypeError if there is no instance of Step,
82+
in a list of pipeline steps.
83+
84+
Also check that the error message is correct.
85+
"""
86+
87+
@pipeline(
88+
name="test_pipeline",
89+
description="Test pipeline",
90+
version="1.0.0",
91+
)
92+
def test_pipeline():
93+
return ["step_one"]
94+
95+
with pytest.raises(TypeError) as context:
96+
pipe = test_pipeline()
97+
pipe.run()
98+
99+
assert (
100+
"Not a valid step. Consider using the step() method to create steps for your pipeline."
101+
== str(context.value)
102+
)
103+
104+
def test_pipeline_failure_no_steps_list():
105+
"""
106+
Test that the pipeline fails with a TypeError if there is no list of steps.
107+
108+
Also check that the error message is correct.
109+
"""
110+
111+
@pipeline(
112+
name="test_pipeline",
113+
description="Test pipeline",
114+
version="1.0.0",
115+
)
116+
def test_pipeline():
117+
return "step_one"
118+
119+
with pytest.raises(TypeError) as context:
120+
pipe = test_pipeline()
121+
pipe.run()
122+
123+
assert "Pipeline steps must be in a list." == str(context.value)
124+
125+
def test_pipeline_failure_no_steps_to_run():
126+
"""
127+
Test that the pipeline fails with an Exception if there are no steps to run.
128+
129+
Also check that the error message is correct.
130+
"""
131+
132+
@pipeline(
133+
name="test_pipeline",
134+
description="Test pipeline",
135+
version="1.0.0",
136+
)
137+
def test_pipeline():
138+
return []
139+
140+
with pytest.raises(Exception) as context:
141+
pipe = test_pipeline()
142+
pipe.run()
143+
144+
assert f"Pipeline {pipe.name} has no steps to run." == str(context.value)
145+
146+
def test_pipeline_failure_exception_in_step():
147+
"""
148+
Test that the pipeline fails with an Exception if there is an exception in one of the steps.
149+
150+
Also check that the error message is correct.
151+
"""
152+
mock_step_1 = Mock()
153+
mock_step_2 = Mock()
154+
mock_step_3 = Mock()
155+
156+
mock_step_1.return_value = "step 1"
157+
mock_step_2.side_effect = Exception("Something went wrong")
158+
mock_step_3.return_value = "step 3"
159+
160+
@pipeline(
161+
name="test_pipeline",
162+
description="Test pipeline",
163+
version="1.0.0",
164+
)
165+
def test_pipeline():
166+
step_one = step(
167+
callable=mock_step_1,
168+
name="step_one",
169+
description="Step one",
170+
version="1.0.0",
171+
)
172+
173+
step_two = step(
174+
callable=mock_step_2,
175+
name="step_two",
176+
description="Step two",
177+
version="1.0.0",
178+
)
179+
180+
step_three = step(
181+
callable=mock_step_3,
182+
name="step_three",
183+
description="Step three",
184+
version="1.0.0",
185+
)
186+
187+
return [step_one, step_two, step_three]
188+
189+
with pytest.raises(Exception) as context:
190+
pipe = test_pipeline()
191+
pipe.run()
192+
193+
assert str(mock_step_2.side_effect) == str(context.value)

‎tinypipeline/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
from .pipeline import pipeline
2+
from .step import step

0 commit comments

Comments
 (0)