Skip to content

Commit ec4b4a9

Browse files
vsochvsoch
committed
init: original creation of flux compspec module
Problem: we want to use the rpc to create a node list. Solution: create a compspec Python plugin. Note that this is potentially deviant in behavior because I was not able to use the RPC to generate a full graph, but just the shorthand. I tried to account for all cases - either finding the shorthand and expanding it, finding the v1 graph and using it, either converting to v2 JGF or returning as is under direction of the user. I do not have need for this right now but wanted to create it in case future folks do. Signed-off-by: vsoch <[email protected]>
0 parents  commit ec4b4a9

19 files changed

+617
-0
lines changed

‎.devcontainer/Dockerfile

+27
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
FROM fluxrm/flux-sched:jammy
2+
3+
LABEL maintainer="Vanessasaurus <@vsoch>"
4+
5+
# Match the default user id for a single system so we aren't root
6+
ARG USERNAME=vscode
7+
ARG USER_UID=1000
8+
ARG USER_GID=1000
9+
ENV USERNAME=${USERNAME}
10+
ENV USER_UID=${USER_UID}
11+
ENV USER_GID=${USER_GID}
12+
USER root
13+
14+
# extra interactive utilities
15+
RUN apt-get update \
16+
&& apt-get -qq install -y --no-install-recommends \
17+
vim less sudo python3-pip
18+
19+
# Ensure regular python is visible
20+
RUN ln -s /usr/bin/python3 /usr/bin/python
21+
22+
# Add the group and user that match our ids
23+
RUN groupadd -g ${USER_GID} ${USERNAME} && \
24+
adduser --disabled-password --uid ${USER_UID} --gid ${USER_GID} --gecos "" ${USERNAME} && \
25+
echo "${USERNAME} ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers
26+
USER $USERNAME
27+
ENV PATH=/home/vscode/.local/bin:$PATH

‎.devcontainer/devcontainer.json

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
{
2+
"name": "Compspec Flux Development Environment",
3+
"dockerFile": "Dockerfile",
4+
"context": "../",
5+
6+
"customizations": {
7+
"vscode": {
8+
"settings": {
9+
"terminal.integrated.defaultProfile.linux": "bash"
10+
},
11+
"extensions": [
12+
"ms-vscode.cmake-tools"
13+
]
14+
}
15+
},
16+
"postStartCommand": "git config --global --add safe.directory /workspaces/compspec-ior"
17+
}

‎.gitignore

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
.eggs
2+
compspec_flux.egg-info
3+
*.so
4+
build
5+
__pycache__
6+
env
7+
dist

‎.pre-commit-config.yaml

+31
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
repos:
2+
- repo: https://github.com/pre-commit/pre-commit-hooks
3+
rev: v4.3.0
4+
hooks:
5+
- id: check-added-large-files
6+
- id: check-case-conflict
7+
- id: check-docstring-first
8+
- id: end-of-file-fixer
9+
- id: trailing-whitespace
10+
- id: mixed-line-ending
11+
12+
- repo: local
13+
hooks:
14+
- id: black
15+
name: black
16+
language: python
17+
types: [python]
18+
entry: black
19+
20+
- id: isort
21+
name: isort
22+
args: [--filter-files]
23+
language: python
24+
types: [python]
25+
entry: isort
26+
27+
- id: flake8
28+
name: flake8
29+
language: python
30+
types: [python]
31+
entry: flake8

‎CHANGELOG.md

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# CHANGELOG
2+
3+
This is a manually generated log to track changes to the repository for each release.
4+
Each section should include general headers such as **Implemented enhancements**
5+
and **Merged pull requests**. Critical items to know are:
6+
7+
- renamed commands
8+
- deprecated / removed commands
9+
- changed defaults
10+
- backward incompatible changes (recipe file format? image file format?)
11+
- migration guidance (how to convert images?)
12+
- changed behaviour (recipe sections work differently)
13+
14+
The versions coincide with releases on pip. Only major versions will be released as tags on Github.
15+
16+
## [0.0.x](https://github.com/compspec/compspec-flux/tree/main) (0.0.x)
17+
- Initial creation of project skeleton (0.0.0)

‎COPYRIGHT

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
Intellectual Property Notice
2+
----------------------------
3+
4+
HPCIC DevTools is licensed under the MIT license (LICENSE).
5+
6+
Copyrights and patents in this project are retained by
7+
contributors. No copyright assignment is required to contribute to
8+
HPCIC DevTools.
9+
10+
SPDX usage
11+
------------
12+
13+
Individual files contain SPDX tags instead of the full license text.
14+
This enables machine processing of license information based on the SPDX
15+
License Identifiers that are available here: https://spdx.org/licenses/

‎LICENSE

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2024 LLNS, LLC and other HPCIC DevTools Developers.
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎MANIFEST.in

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
include README.md LICENSE
2+
recursive-include compspec_ior *
3+
prune OLD*
4+
prune env*
5+
global-exclude .env
6+
global-exclude *.py[co]
7+
recursive-exclude .git *
8+
global-exclude __pycache__

‎NOTICE

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
This work was produced under the auspices of the U.S. Department of
2+
Energy by Lawrence Livermore National Laboratory under Contract
3+
DE-AC52-07NA27344.
4+
5+
This work was prepared as an account of work sponsored by an agency of
6+
the United States Government. Neither the United States Government nor
7+
Lawrence Livermore National Security, LLC, nor any of their employees
8+
makes any warranty, expressed or implied, or assumes any legal liability
9+
or responsibility for the accuracy, completeness, or usefulness of any
10+
information, apparatus, product, or process disclosed, or represents that
11+
its use would not infringe privately owned rights.
12+
13+
Reference herein to any specific commercial product, process, or service
14+
by trade name, trademark, manufacturer, or otherwise does not necessarily
15+
constitute or imply its endorsement, recommendation, or favoring by the
16+
United States Government or Lawrence Livermore National Security, LLC.
17+
18+
The views and opinions of authors expressed herein do not necessarily
19+
state or reflect those of the United States Government or Lawrence
20+
Livermore National Security, LLC, and shall not be used for advertising
21+
or product endorsement purposes.

‎README.md

+100
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# Compspec Flux
2+
3+
<p align="center">
4+
<img height="300" src="https://raw.githubusercontent.com/compspec/spec/main/img/compspec-circle.png">
5+
</p>
6+
7+
[![PyPI version](https://badge.fury.io/py/compspec-flux.svg)](https://badge.fury.io/py/compspec-flux)
8+
9+
A compspec (Composition spec) is a specification and model for comparing things. Compspec Flux
10+
provides a simple means to extract metadata about a running flux instance, namely the graph of nodes.
11+
Other links of interest include:
12+
13+
- [Compspec](https://github.com/compspec/compspec): the Python library that discovers and loads this plugin.
14+
- [Compatibility](https://github.com/compspec/spec/tree/main/compatibility): of container images and applications to a host environment.
15+
- [Compspec Go](https://github.com/compspec/compspec-go): the Go library that retrieves artifacts and makes graphs for image selection and scheduling.
16+
17+
## Usage
18+
19+
Install compspec and the plugin here:
20+
21+
```bash
22+
pip install compspec
23+
pip install compspec-flux
24+
```
25+
26+
Then (alongside a Flux instance) run the plugin. Note that the VSCode setup [provided here](.devcontainer) will provide this for you.
27+
You can use defaults, or add any parameters to the plugin after the plugin name "flux" Here is how to start a flux instance:
28+
29+
```bash
30+
flux start --test-size=4
31+
```
32+
33+
And here is how to print to the terminal:
34+
35+
```bash
36+
compspec extract flux
37+
```
38+
39+
<details>
40+
41+
<summary>Flux output</summary>
42+
43+
```console
44+
{
45+
"0": {
46+
"id": 0,
47+
"metadata": {
48+
"type": "cluster",
49+
"basename": "cluster",
50+
"name": "cluster0",
51+
"uniq_id": 0,
52+
"containment": {
53+
"paths": "/cluster0"
54+
},
55+
"size": 1,
56+
"unit": "",
57+
"rank": 0,
58+
"exclusive": false
59+
}
60+
},
61+
"1": {
62+
"id": 1,
63+
"metadata": {
64+
"type": "node",
65+
"basename": "node",
66+
"name": "node1",
67+
"uniq_id": 1,
68+
"containment": {
69+
"paths": "/cluster0/node1"
70+
},
71+
"size": 1,
72+
"unit": "",
73+
"rank": 0,
74+
"exclusive": false
75+
}
76+
}
77+
}
78+
```
79+
80+
</details>
81+
82+
And how to save to file
83+
84+
```bash
85+
compspec extract --outfile cluster-resources.json flux
86+
```
87+
88+
89+
## License
90+
91+
HPCIC DevTools is distributed under the terms of the MIT license.
92+
All new contributions must be made under this license.
93+
94+
See [LICENSE](https://github.com/converged-computing/cloud-select/blob/main/LICENSE),
95+
[COPYRIGHT](https://github.com/converged-computing/cloud-select/blob/main/COPYRIGHT), and
96+
[NOTICE](https://github.com/converged-computing/cloud-select/blob/main/NOTICE) for details.
97+
98+
SPDX-License-Identifier: (MIT)
99+
100+
LLNL-CODE- 842614

‎compspec_flux/__init__.py

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
from .plugin import Plugin
2+
from .version import __version__

‎compspec_flux/defaults.py

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# These are required to be given to compspec
2+
spec_version = "0.0.0"
3+
namespace = "io.compspec.ior"
4+
plugin_type = "generic"
5+
description = "Flux resources extractor"

‎compspec_flux/nodelist.py

+40
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
import re
2+
3+
4+
# we assume index parts are sane
5+
def expand_nodeids(nodeids):
6+
"""
7+
Given a list of node ids, expand.
8+
9+
This expects only very simple directives.
10+
"""
11+
for bracket in ["[", "]"]:
12+
nodeids = nodeids.replace(bracket, "")
13+
parts = list(set(nodeids.split(",")))
14+
expanded = []
15+
for part in parts:
16+
if "-" in part:
17+
start, end = part.split("-", 2)
18+
nodeset = range(int(start), int(end) + 1)
19+
expanded += nodeset
20+
else:
21+
expanded.append(int(part))
22+
return sorted(expanded)
23+
24+
25+
def parse_nodelist(nodelist):
26+
"""
27+
Parse a nodelist into an actual node list.
28+
"""
29+
# index pattern
30+
regex = re.compile(r"(?P<full>(?P<prefix>\w+)(?P<ids>\d+|\[[\d\,\-]+\]),?)")
31+
nodes = []
32+
for nodestr in nodelist:
33+
match = regex.match(nodestr)
34+
if not match:
35+
return
36+
match = match.groupdict()
37+
nodeids = expand_nodeids(match["ids"])
38+
prefix = match["prefix"]
39+
nodes += [f"{prefix}{n}" for n in nodeids]
40+
return nodes

0 commit comments

Comments
 (0)