Expose celery queue metrics #1275
Conversation
flower/utils/broker.py
Outdated
| queue_names = set([capp.conf.CELERY_DEFAULT_QUEUE]) | \ | ||
| set([q.name for q in capp.conf.CELERY_QUEUES or [] if q.name]) | ||
|
|
||
| queues = yield broker.queues(sorted(queue_names)) |
There was a problem hiding this comment.
The website hangs a bit sometimes. This might be running in the main thread. Looking for ways to fix it. Just spawn a new Thread?
There was a problem hiding this comment.
Probably fixed with @tornado.concurrent.run_on_executor (https://mike.depalatis.net/blog/tornado-background-tasks.html)
There was a problem hiding this comment.
Fixed for good this time:
- moved update metrics loop to a while loop with asyncio sleep
- fixed Redis broker calls to be run in ThreadPoolExecutor instead of main thread
HTRafal
force-pushed
the
queue_length_metric
branch
3 times, most recently
from
924d2f1
to
1f2f7c0
Compare
HTRafal
changed the title
|
Alternative approach is to keep it simple and expose just the queue lengths without number of online workers. The last commit seems the most hacky. I'm ok with removing it - commit / draft PR without online workers |
HTRafal
force-pushed
the
queue_length_metric
branch
from
d235e49
to
46fa852
Compare
|
While this was open version 2.0 was released and I got a lot of conflicts, so I resolved all of them and squashed everything to a single commit. |
Implements mher#1173 * Quick fix synchronous redis calls. Redis broker calls were synchronous, so blocking the main thread. This moves them to ThreadPoolExecutor * Update workers list regularly for up-to-date metrics Previously self.workers was only updated only on UI events. This is no longer fine as we use the data to produce up-to-date metrics
HTRafal
force-pushed
the
queue_length_metric
branch
from
46fa852
to
18121a1
Compare
|
Hey folks, we've been running a patched |
|
Could this pull request be merged? |
|
@mher can we have this one merged please? |
This was tested by running flower locally and letting it read metrics from a real broker.
This is a draft / proposal. I'm open for suggestions on how to implement it in a better way.
Implements #1173