python: add `QueueList` class for access to queue config, limits, status and resource counts #6723

grondo · 2025-03-21T16:38:26Z

Problem: There's currently no easy way to access Flux queue information from the Python API, and that information is currently distributed between different parts the config, various job-manager RPCs, and resource configuration.

This PR adds a new flux.queue.QueueList class which fetches information for all currently configured queues in one go. It then presents as a list of QueueInfo objects for each queue, which offer access to status, limits, and even resources counts on a per queue basis. This class is then used to replace similar code in flux queue list.

In a follow on PR, flux queue list will be extended to allow access to the resource stats.

Problem: There's a typo in the vendored Python modules README.md, the 'tomli' package is misspelled. Fix the typo.

Problem: Python dataclasses would be useful, but they were not introduced until Python 3.7 and Flux still has to support 3.6. Add the dataclassees backport to 3.6 from: https://github.com/ericvsmith/dataclasses

Problem: Since the dataclasses source is added to flux.utils directly, importing the `dataclass` decorator requires an overly long path. Add an __init__.py to the dataclasses source directory so that dataclass can be imported from flux.utils.dataclasses.

Problem: The dataclasses backport is not included in `make dist` or `make install`. Add the directory to Makefile.am so it is.

Problem: The typos checker detects typos in a vendored python module. Ignore the module in .typos.toml.

Problem: The SchedResourceList initializer assumes it will only be passed a dictionary of dictionaries, each of which needs to be passed to the ResourceSet() constructor. This assumption makes it awkward to create a SchedResourceList from another instance of the same class. Check to see if the each value is already a ResourceSet in the `resp` parameter to the SchedResourceList initializer, and only convert to a ResourceSet if necessary.

Problem: There's no easy way to create a copy of a SchedResourceList object using a constraint, e.g. to return only the reosurces in a given queue. Add a copy_constraint() method to the SchedResourceList class for this purpose.

Problem: There's no user friendly method in Python to fetch the list of configured queues along with information about them, including name (if named queues are used), status, limits and defaults, and associated resources. Add a QueueList class which can fetch information about all queues using a combination of the configuration, queue statuses, and resource list. It presents as a list of QueueInfo objects, each of which contains attributes describing the above data.

Problem: There's no tests of the Python QueueList class. Add t/python/t0034-queuelist.py for this purpose.

Problem: The `--from-stdin` option in `flux queue list` is not used. Drop it.

Problem: Much of the code in `flux queue list` duplicates functionality in the flux.queue.QueueList class. Update `flux queue list` to use functionality from the flux.queue.QueueList class.

codecov · 2025-03-21T22:22:44Z

Codecov Report

Attention: Patch coverage is 98.76543% with 2 lines in your changes missing coverage. Please review.

Project coverage is 83.84%. Comparing base (a718ef5) to head (a2e3c8d).

Files with missing lines	Patch %	Lines
src/bindings/python/flux/queue.py	98.33%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6723      +/-   ##
==========================================
+ Coverage   83.80%   83.84%   +0.04%     
==========================================
  Files         534      535       +1     
  Lines       89164    89195      +31     
==========================================
+ Hits        74720    74783      +63     
+ Misses      14444    14412      -32

Files with missing lines	Coverage Δ
src/bindings/python/flux/resource/list.py	`92.75% <100.00%> (+0.81%)`	⬆️
src/cmd/flux-queue.py	`94.89% <100.00%> (+0.29%)`	⬆️
src/bindings/python/flux/queue.py	`98.33% <98.33%> (ø)`

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jameshcorbett

A bunch of small comments but generally looks great!

jameshcorbett · 2025-03-28T16:22:58Z

src/bindings/python/flux/resource/list.py

@@ -83,6 +83,12 @@ def free(self):
        res.state = "free"


typo in commit message: resources

jameshcorbett · 2025-03-28T16:23:52Z

src/bindings/python/flux/resource/list.py

@@ -82,6 +84,7 @@ def free(self):
        return res


+


stray newline?

Hm, I'm assuming black added this, but I can try removing it and see if it complains.

Oh I get it, as @chu11 said, in the wrong commit.

jameshcorbett · 2025-03-28T16:27:40Z

src/cmd/flux-queue.py

-            config = future.get()
-        except Exception as e:
-            LOGGER.warning("Could not get flux config: " + str(e))
+    future = handle.rpc("config.get")


could be handle.conf_get()? Since there's a method for it I think I would prefer that.

Great suggestion! However, this particular commit was updating existing code and I'd prefer not to make two changes at once. Actually, I see there are several uses of handle.rpc("config.get") throughout the code that should all be updated, let's do that in a separate PR.

jameshcorbett · 2025-03-28T16:44:26Z

src/bindings/python/flux/queue.py

+    Attributes:
+        min (:obj:`QueueResourceCounts`): Configured resource minimums.
+        max (:obj:`QueueResourceCounts`): Configured resource maximums.
+            (If no maximum, individual resource count will be ``inf``)


math.inf is not an integer so that violates the type hints for QueueResourceCounts. Maybe typing.Union[int, float]?

Thanks! I'm not all that familiar with Python typing hints. Is there a more generic "Number" type hint?

Ah from PEP 484

when an argument is annotated as having type float, an argument of type int is acceptable;

I'll use float

jameshcorbett · 2025-03-28T16:48:36Z

src/bindings/python/flux/queue.py

+        resources, config = map(
+            lambda x: x.get(),
+            [resource_list(handle), handle.rpc("config.get")],
+        )


resources = resource_list.get() config = handle.conf_get()

?

This should have a comment perhaps, but we don't want to run the RPCs back-to-back. (This code initiates both RPCs before calling get on either (at least I hope that's what it does)). May need to change anyway if we update to handle.conf_get() - though now I'm realizing there's no way to make that call asynchronous, which is why the explicit RPC may be better here 🤔

Yeah that seems reasonable. You could maybe do:

resources = resource_list(handle) config = handle.conf_get() resources = resources.get()

?

jameshcorbett · 2025-03-28T17:01:47Z

src/bindings/python/flux/queue.py

+        If ``queues`` is None, return the whole config, which may be empty
+        if there is no configured queues.
+        """
+        if not queues:


Currently if you pass in an empty list for queues you would get the whole config returned. That doesn't make a ton of sense to me, I think I would probably go for creating an empty object or raising an error.

This is done purposely to easily support callers that are passing argparse args.queue option directly which is of type append or Flux's FilterSetUpdate, where the default (no queue specifically specified) value is an empty list or set.

I will update the documentation accordingly.

jameshcorbett · 2025-03-28T17:03:19Z

src/bindings/python/flux/queue.py

+ >>> batchq = qlist["batch"]
+
+If there is a single anonymous queue, the queue name is the empty string,
+so this queue may be indexed using an empty string as the key:


I think there is something to be said for having the queue name be None in this case rather than the empty string. I don't think it would make the interface any different really, but None is more of what I'd expect in this case, and it would simplify the implementation a bit.

Unfortunately the empty string is required for use in flux queue list, where we don't want to print anything at all for the anonymous queue (we don't want to display queue name "None").

Ah ok makes sense. There could be a check for None in the flux queue list command though right? Up to you though.

jameshcorbett · 2025-03-28T17:06:26Z

src/bindings/python/flux/queue.py

+            if "queues" in config:
+                return config["queues"]
+            return {}


maybe return config.get("queues") instead?

jameshcorbett · 2025-03-28T17:16:48Z

src/bindings/python/flux/queue.py

+                result = default
+        return result
+
+    def policy_system_limit(self, key, default="inf"):


should these policy_ and size_ methods be prefixed with an underscore? They don't seem like they should be part of the interface.

jameshcorbett · 2025-03-28T17:22:57Z

t/python/t0034-queuelist.py

+        """
+        self.fh.rpc("config.load", tomllib.loads(testconf)).get()
+
+        qlist = QueueList(self.fh)


Maybe add a test like QueueList(self.fh, [])?

and also requesting specific queues, QueueList(self.fh, ["batch"], and probably multiple queues ["batch", "debug"]

chu11 · 2025-03-28T16:57:58Z

src/bindings/python/flux/resource/list.py

@@ -82,6 +84,7 @@ def free(self):
        return res


+


errant line introduced? (in python: accept ResourceSet dict in SchedResourceList initializer commit)

chu11 · 2025-03-28T17:27:19Z

src/bindings/python/flux/queue.py

+attributes are also documented for each corresponding class in this module::
+
+  queue.name (str) (empty string if using a single anonymous queue)
+  queue.is_default (str)


is_default -> bool?

chu11 · 2025-03-28T17:28:55Z

src/bindings/python/flux/queue.py

+
+    min: QueueResourceCounts
+    max: QueueResourceCounts
+    duration: str


str -> float?

chu11 · 2025-03-28T17:39:10Z

t/python/t0034-queuelist.py

+        """
+        self.fh.rpc("config.load", tomllib.loads(testconf)).get()
+
+        qlist = QueueList(self.fh)


and also requesting specific queues, QueueList(self.fh, ["batch"], and probably multiple queues ["batch", "debug"]

chu11 · 2025-03-28T17:41:50Z

src/cmd/flux-queue.py

-        return t
+    def __getattr__(self, attr):
+        # Forward most attribute lookups to underlying QueueLimits instance
+        return getattr(self.__limits, attr)


 class QueueInfo:


to avoid confusion with the new QueueInfo class, rename to ....???

grondo added 11 commits March 21, 2025 06:55

python: fix typo in README file

14b4066

Problem: There's a typo in the vendored Python modules README.md, the 'tomli' package is misspelled. Fix the typo.

python: add dataclasses backport in flux.utils.dataclasses

1962ff7

Problem: Python dataclasses would be useful, but they were not introduced until Python 3.7 and Flux still has to support 3.6. Add the dataclassees backport to 3.6 from: https://github.com/ericvsmith/dataclasses

build: include dataclasses source in dist/install

59e76e1

Problem: The dataclasses backport is not included in `make dist` or `make install`. Add the directory to Makefile.am so it is.

typos: ignore typos in vendored dataclasses module

b3f00c4

Problem: The typos checker detects typos in a vendored python module. Ignore the module in .typos.toml.

testsuite: add unit tests for flux.queue.QueueList

9fdae35

Problem: There's no tests of the Python QueueList class. Add t/python/t0034-queuelist.py for this purpose.

flux-queue: drop unused test-only option from flux queue list

eef8151

Problem: The `--from-stdin` option in `flux queue list` is not used. Drop it.

flux-queue: update to use flux.queue.QueueList

a2e3c8d

Problem: Much of the code in `flux queue list` duplicates functionality in the flux.queue.QueueList class. Update `flux queue list` to use functionality from the flux.queue.QueueList class.

grondo force-pushed the qlist-pyclass branch 2 times, most recently from 35fdc4d to a2e3c8d Compare March 21, 2025 21:56

jameshcorbett approved these changes Mar 28, 2025

View reviewed changes

chu11 reviewed Mar 28, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python: add `QueueList` class for access to queue config, limits, status and resource counts #6723

python: add `QueueList` class for access to queue config, limits, status and resource counts #6723

grondo commented Mar 21, 2025

codecov bot commented Mar 21, 2025

jameshcorbett left a comment

jameshcorbett Mar 28, 2025

jameshcorbett Mar 28, 2025

grondo Mar 28, 2025

grondo Mar 28, 2025

jameshcorbett Mar 28, 2025

grondo Mar 28, 2025

jameshcorbett Mar 28, 2025

grondo Mar 28, 2025

grondo Mar 28, 2025

jameshcorbett Mar 28, 2025

grondo Mar 28, 2025

jameshcorbett Mar 28, 2025

jameshcorbett Mar 28, 2025

grondo Mar 28, 2025

jameshcorbett Mar 28, 2025

grondo Mar 28, 2025

jameshcorbett Mar 28, 2025 •

edited

Loading

jameshcorbett Mar 28, 2025

jameshcorbett Mar 28, 2025

jameshcorbett Mar 28, 2025

chu11 Mar 28, 2025

chu11 Mar 28, 2025

chu11 Mar 28, 2025

chu11 Mar 28, 2025

chu11 Mar 28, 2025

chu11 Mar 28, 2025

python: add QueueList class for access to queue config, limits, status and resource counts #6723

Are you sure you want to change the base?

python: add QueueList class for access to queue config, limits, status and resource counts #6723

Conversation

grondo commented Mar 21, 2025

codecov bot commented Mar 21, 2025

Codecov Report

jameshcorbett left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jameshcorbett Mar 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

python: add `QueueList` class for access to queue config, limits, status and resource counts #6723

python: add `QueueList` class for access to queue config, limits, status and resource counts #6723

jameshcorbett Mar 28, 2025 •

edited

Loading