Error 404 on Login request

[lucasRivas19]

When I log in, it returns a 404, since it cannot find /intelmq/v1/api/login. When checking the nginx logs, it tells me that it cannot find the var.js file, located in /usr/share/intelmq-manager/html/js/.

Thank you so much. stay tuned

[kamil-certat]

Hey, could you say which version do you use and how you installed it? The lack of var.js should not be a problem - it's necessary rather when you need to customise something, none of my setups uses it.

However, the API is a separated service you need to have running if you want to use API. The DEB package comes with it configured through Apache, so I assume you have either installed it manually or use the Docker setup. Could you clarify it?

[lucasRivas19]

Hello, install manually. Every time I update the server to 22.04, the api stops working. and I do all the steps that are clarified in the doc. If you need more data, I am available. thank you

[sebix]

Hello, install manually.

Please show your current active webserver configuration then, as well as your logs and what steps you did for the installation of intelmq-api

[kamil-certat]

@lucasRivas19 Waiting for webserver config, I have a few additional tips:

For API to work, you have to have a working API service. In newer versions API is served by Gunicorn using FastAPI, in older it was hug run directly by webserver. Please have a look at example service configuration for the current version:

• https://github.com/certtools/intelmq-api/blob/develop/contrib/intelmq-api.service
• https://github.com/certtools/intelmq-api/blob/develop/contrib/intelmq-api.socket
• https://github.com/certtools/intelmq-api/blob/develop/contrib/api-apache.conf (example proxy config for Apache).

If the problem is related to Ubuntu upgrade, please check logs of the API service - I have a few ideas, assuming you use the newest stable IntelMQ:

• old FastAPI package or its dependencies -> apt upgrade should help (but the upgrade for Ubuntu should do it anyway),
• IntelMQ installed in a virtualenv, and the Python version was changed after server Upgrade -> you have to recreate virtualenvs and reinstall IntelMQ packages.

[lucasRivas19]

Good morning, @kamil-certat . thank you very much for the response, the matter was resolved.

Once we set up the API, when I want to run a bot (any one) from the interface, it never runs and it doesn't return anything to me in the logs. Now, when I run it from the console it tells me that it cannot evaluate a type "None". Looking at other issues I saw that they add an if in the corresponding file and by doing that, it doesn't raise the bot either. Can you give me some help?

@sebix

[sebix]

This now looks more like a broken IntelMQ bit configuration file. Can you please run intelmqctl check to test that?

[lucasRivas19]

@sebix Thanks for the prompt response. Here is the output of the intelmqctl check (it already runs intelmqctl update-config).

Reading configuration files.
Checking runtime and pipeline configuration.
Orphaned queues found: 'TPOTParser-Parser-queue-internal', 'JSON-Parser-queue', 'TPOTParser-Parser-queue', 'Elasticsearch-LogexporterUDP-Output-queue', 'Elasticsearch-LogexporterUDP-Output-queue-internal'. Possible leftover from past reconfigurations without cleanup. Have a look at the FAQ at https://docs.intelmq.org/latest/admin/faq/#orphaned-queues
Checking harmonization configuration.
Checking for bots.
Incomplete installation: Bot 'ElasticDeduplicator-Expert' not importable.
Incomplete installation: Bot 'TPOT-Parser' not importable.
Upgrade function v320_update_turris_greylist_url not completed (successfully). Please run 'intelmqctl upgrade-config'.
Upgrade function v322_url_replacement not completed (successfully). Please run 'intelmqctl upgrade-config'.
Upgrade function v322_removed_feeds_and_bots not completed (successfully). Please run 'intelmqctl upgrade-config'.
Some issues have been found, please check the above output.

[sebix]

I guess this is the culprit:

Incomplete installation: Bot 'ElasticDeduplicator-Expert' not importable.
Incomplete installation: Bot 'TPOT-Parser' not importable.

The modules of the configured bots are not correct

[lucasRivas19]

@sebix Good afternoon, I'm getting back to the problem. Could you help correct this: The configured bot modules are not correct? thank you very much. This arose after updating the operating system. It made me reinstall only the intelmq package. Maybe this has something to do with it? Was there a duplicate folder or something similar?

[sebix]

It is very hard to assist as we have very little information about your setup. For example, @kamil-certat already asked you how you installed IntelMQ. We also have no idea what your configuration looks like.

The error message means that the python module, which is the bot itself, cannot be found. The module of ElasticDeduplicator-Expert is what you can find under module: of the bot's configuration. You can also test this yourself by executing: python3 -c "import $botmodule"

[lucasRivas19]

@sebix

Good morning, sorry. I missed informing about the installation. install intelmq with pip. What other configuration do you need?

[kamil-certat]

@lucasRivas19 Please try the manual import as asked by Sebix. This may give us an information what is wrong. The module path you will find in the configuration in the IntelMQ Manager or in the runtime config file (/opt/intelmq/etc/runtime.yaml, ev. /etc/intelmq/runtime.yaml).

This arose after updating the operating system. It made me reinstall only the intelmq package.

Some bots may require reinstalling additional dependencies, but the error should happen after importing. Based on issues from your screenshots, it looks like entry points for bot modules weren't installed correctly or are inaccessible from the IntelMQ. As I see, you have installed IntelMQ globally, so it's strange.

Please try following commands and show us results:

python3 -c "import intelmq.bots.experts.deduplicator.expert"
intelmq.bots.experts.deduplicator.expert -h
which intelmq.bots.experts.deduplicator.expert
head /usr/local/bin/intelmq.bots.experts.deduplicator.expert # (or any other path resulted from the previus command)

Which system do you use? Ubuntu 22.04, based on the Python version?

[lucasRivas19]

@sebix @kamil-certat Here I detail the output of the requested commands. And yes, I am working with Ubuntu 22.04.

[kamil-certat]

Hmm, it looks correct. Could you please have a look at your ElasticDeduplicator-Expert and TPOT-Parser, and which modules you see there? And try to import them?

Could you also share how you set up the API? Does it use a systemd service, or did you do it differently?

[lucasRivas19]

@sebix @kamil-certat
This is what I know matters:
from intelmq.lib.bot import Bot
import json
from intelmq.lib import utilities
from elasticsearch import Elasticsearch
from elasticsearch_dsl import search, UpdateByQuery
import base64
import pytz
from datetime import datetime

Regarding the api, it is not running as a service, there is a file in apache2, called api-apache, where I understand it is running.

[kamil-certat]

Okay, so it sounds quite clear - the ElasticDeduplicator-Expert and TPOT-Parser use your own bots, right? They don't use any built-in bot? Have you re-installed / copied them after upgrading your OS, together with all dependencies?

Ubuntu installed the new version of Python. It usually works so, that every version has its own directory for packages (in addition to a generic one, but this is used almost exclusively for packages installed through system package manager, e.g. apt). This is why you needed to reinstall intelmq - new Python version wasn't able to find the package installed in the previous one. In addition, please note that your bot should be callable from the shell (=have properly configured console_script entry). Please note that starting IntelMQ 3.3.0 there is an official mechanism for extending IntelMQ with your bots - refer to https://docs.intelmq.org/latest/dev/extensions-packages/ if you have any trouble.

If you need to restore the code from the previous installation, it should be still available in the directory like /usr/local/lib/python3.8/dist-packages/....

The /usr/local/lib/python3.10/dist-packages/ is your current installation path for packages installed via pip.