Schedule bots directly in IntelMQ

[kamil-certat]

Currently, scheduled bots has to be started outside IntelMQ, e.g. using cron. It would be good to introduce some scheduling directly in the IntelMQ, e.g. using https://pypi.org/project/APScheduler/

Two possible architectural solutions:

• bot is loading, but instead of initializing, it's registering the scheduler. The bot process is running the whole time, but doesn't do anything.
• a separated controller is responsible for scheduling and starting bots.

[aaronkaplan]

Not sure if I think this is good. Why?

• leave scheduling to the program which was made for it (cron , etc.) . Do not try to add that complexity to intelmq
• it adds code and complexity

But I see the need for something like that (maybe another type of cron) inside of docker.

[kamil-certat]

I understand your point, but I believe it - when implemented properly - would remove complexity rather than adding it. Leaving such an important feature to be manages outside the IntelMQ adds a significant complexity for users. I think it's reasonable to expect IntelMQ to handle it; on the other hand, we could also leave starting normal bots to user and do not support it through API/CLI - with just saying that it's a complexity. ;)

However, this issue is so far more a reminder - a solution should be carefully designed, keeping the simplicity in mind. You know, I suggested using Python-native solutions, but an alternative would be to e.g. configure underlying program (cron, systemd, etc.), taking away the manual task from user without deep changes.

[arvchristos]

I totally agree that this feature would make user's life much easier (also on the documentation/standardization of IntelMQ deployments). I can see that it will add more code surface to the project but solutions like Celery (pretty mature + compatible with current IntelMQ stack) for scheduling could introduce many new use-cases and features for bots.

This also means that we would not be differentiating set ups according to the existence of systemd or dockerization. Instead there would be a unified approach in terms of deployment and configuration.

[aaronkaplan]

@arvchristos could you maybe sketch out how you would see an IntelMQ + celery integration? Like as in an IEP?

[aaronkaplan]

Maybe I should be a bit more verbose why my original answer was that I was afraid of adding complexity:
because we already have a mechanism to run bots once (one-shot) only. So, if we use that config option + schedule them via cronjob, then I would say that's the simplest way of achieving that. No?

[kamil-certat]

For me, it's not because it only moves the complexity to the user (so it increases the complexity needed to use IntelMQ, and as anything made by users, it's vulnerable to human errors ;)). And in fact, it looks like partially implemented feature (´scheduled´ type of running means nothing for IntelMQ).

I propose a little different solution than celery: for bots with the scheduled running type, add another configuration option with the schedule definition (e.g. in cron format; intelmqctl check should check syntax). Then, let a separate daemon be running in background, read this configuration periodically and ensure execution (e.g. using https://github.com/agronholm/apscheduler). I think it should be rather simple and flexible solution, with a possibility to even replace the scheduler with different solution (if you prefer e.g. generating crontab instead).

[arvchristos]

For me, it's not because it only moves the complexity to the user (so it increases the complexity needed to use IntelMQ, and as anything made by users, it's vulnerable to human errors ;)). And in fact, it looks like partially implemented feature (´scheduled´ type of running means nothing for IntelMQ).

I propose a little different solution than celery: for bots with the scheduled running type, add another configuration option with the schedule definition (e.g. in cron format; intelmqctl check should check syntax). Then, let a separate daemon be running in background, read this configuration periodically and ensure execution (e.g. using https://github.com/agronholm/apscheduler). I think it should be rather simple and flexible solution, with a possibility to even replace the scheduler with different solution (if you prefer e.g. generating crontab instead).

Thank you for the input @kamil-certat . Actually what you describe is more or less what I had in mind with Celery, bots would have a config parameter for scheduled runs. I suggested Celery mainly because it has a programmatic interface for this as well as it is battle tested.

I hope life permits and I can come up with the time soon to write the IEP @aaronkaplan !