Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better error messages for missing authentication #238

Open
ghost opened this issue Mar 1, 2021 · 34 comments
Open

Better error messages for missing authentication #238

ghost opened this issue Mar 1, 2021 · 34 comments

Comments

@ghost
Copy link

ghost commented Mar 1, 2021

If login is required, but the user is not logged in, the error messages are not good.

On Monitor:

11:21 AM Error loading botnet status: {"errors": {"Authentication Required": "Please provide valid Token verification credentials"}} Unauthorized

On configuration and management page, plus the same of monitor:

11:20 AM Get an error error, Unauthorized when trying to obtain config file properly
http://example.com/intelmq/v1/api/config?file=runtime
11:20 AM Error incomplete botnet: {"errors": {"Authentication Required": "Please provide valid Token verification credentials"}} Unauthorized

On check page: pop-up with error getting check command output
Ob about: pop-up with Error getting debugging information. Do you have IntelMQ >= 2.2.0?

@oscarliz
Copy link

10:45 AM 24× Error loading bot queues information: {"errors": {"Invalid Authentication": "Provided Token verification credentials were invalid"}} Unauthorized

10:44 AM Get an error error, Unauthorized when trying to obtain config file properly
http://172.29.26.232/intelmq/v1/api/config?file=positions
.

10:44 AM Get an error error, Unauthorized when trying to obtain config file properly
http://172.29.26.232/intelmq/v1/api/config?file=pipeline
.

10:44 AM Get an error error, Unauthorized when trying to obtain config file properly
http://172.29.26.232/intelmq/v1/api/config?file=runtime
.

10:44 AM Get an error error, Unauthorized when trying to obtain config file properly
http://172.29.26.232/intelmq/v1/api/config?file=defaults
.

10:44 AM Get an error error, Unauthorized when trying to obtain config file properly
http://172.29.26.232/intelmq/v1/api/config?file=bots
.

@aragorne007
Copy link

Hello,

Sorry to up this topic but I've the same issue.
Any help ?

Thank in advance :)

@sebix
Copy link
Member

sebix commented Aug 13, 2022

For any help more information is required: What error messages do you get at what action and which preconditions (e.g. logged in), how did you install and set up IntelMQ, which operating system are you using etc.

@oencarnacion
Copy link

oencarnacion commented Aug 13, 2022

@aragorne007 It could be that it is redis that is giving you a problem solution:

chown -R redis:redis /var/log/redis
chmod -R u+rwX,g+rwX,u+rx /var/log/redis
chmod +r /etc/redis/redis.conf
systemctl start redis

@aragorne007
Copy link

@oencarnacion
I tried but it doesn't work.

@sebix
I installed intelmq, intelmq-manager and intelmq-api.
After that, I've created an account with the cmd "intelmq-api-adduser".

When I'm log in and when I go to any menu, I've 2 kinds of error :

First One (Configuration & Monitor) :

22:27:55 369× Error loading bot queues information: {"errors": {"Invalid Authentication": "Provided Token verification credentials were invalid"}} Unauthorized

22:21:53 Get an error error, Unauthorized when trying to obtain config file properly
http://192.168.0.21/intelmq/v1/api/positions
.

22:21:53 Get an error error, Unauthorized when trying to obtain config file properly
http://192.168.0.21/intelmq/v1/api/runtime
.

22:21:53 Get an error error, Unauthorized when trying to obtain config file properly
http://192.168.0.21/intelmq/v1/api/bots
.

The second one (Check) :

error getting check command output

I'm sure that it's a permission issue but I'm not sure to understand which folder need to have the right access.
Or maybe it's a file configuration error but I think that I've well followed the Installation guide line

Thank for your help :)

@aragorne007
Copy link

If it can help, the result of "intelmqctl debug"

Paths:
HARMONIZATION_CONF_FILE: '/etc/intelmq/harmonization.conf'
RUNTIME_CONF_FILE: '/etc/intelmq/runtime.yaml'
VAR_RUN_PATH: '/var/run/intelmq/'
STATE_FILE_PATH: '/var/lib/intelmq/state.json'
DEFAULT_LOGGING_PATH: '/var/log/intelmq/'
file: '/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py'
CONFIG_DIR: '/etc/intelmq/'
ROOT_DIR: '/'
Environment variables:
INTELMQ_ROOT_DIR: None
INTELMQ_PATHS_NO_OPT: None
INTELMQ_PATHS_OPT: None
INTELMQ_MANAGER_CONTROLLER_CMD: None
PATH: '/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games'

@sebix
Copy link
Member

sebix commented Aug 13, 2022

I assume you installed the deb/rpm packages? Which operating system?

When I'm log in and when I go to any menu, I've 2 kinds of error :

Is the login successful (Is the username shown instead of the login-button after the login?)
I assume it is, because then the error message would be different.

What you are seeing must be some glitch in the authentication between the manager and the API.

See also https://intelmq.readthedocs.io/en/latest/user/intelmq-api.html#access-denied-authentication-required-please-provide-valid-token-verification-credentials (but that does not really apply as you are using the API indirectly with the Manager). The Manager needs to pass the token to the API in its calls.

If you are up for some debugging, check what you see in the HTTP request headers of the API calls (using the debug tools of the browser).

@aragorne007
Copy link

@sebix

The documentation isn't enough complete I think :(

I'm not enough documented to try some debug with HTTP request

In "check" panel, I've this error :
error getting check command output

In "about" panel, I've 2 others error messages :
Error getting debugging information. Do you have IntelMQ >= 2.2.0?

error getting version

I don't know if it can help you or give you some clue ?

@sebix
Copy link
Member

sebix commented Aug 14, 2022

In order to reconstruct your error, it is still important to know if you installed the deb/rpm packages or via Pip or from git, and which operating system you are using?

The documentation isn't enough complete I think :(

Yes, I know. That section is not meant for the error you are experiencing, I linked it because it is related.

I'm not enough documented to try some debug with HTTP request

Which browser are you using? I can guide you.

In "check" panel, I've this error : error getting check command output

In "about" panel, I've 2 others error messages : Error getting debugging information. Do you have IntelMQ >= 2.2.0?

error getting version

All that indicates an error with the authentication, but no trace on the reasons.

I don't know if it can help you or give you some clue ?

I have no clue what the error is, either, but I am offering some of my spare time so we can debug it together.

@oencarnacion
Copy link

oencarnacion commented Aug 14, 2022

Hi @aragorne007 confirm me that in this /var/lib/ path you have a folder called dbconfig-common

@aragorne007
Copy link

Hello @sebix ;

I installed the package from pip3 install.

I'm using Google Chrome :)

Yes, It's what I expect, an authentication issue but I didn't find any log to prove that.

Hello @oencarnacion,

Yes, and "intelmq" user is the owner BUT I think that I've manually created this folder because it wasn't create during the installation.
That can helping you ?

Thank a lot for you help guys :)

@oencarnacion
Copy link

@aragorne007 The solution to your problem is to go to the /var/lib/ path, unzip the zip that I supply, give it read and write permission and then proceed to create the user and if you want, restart the server just in case and that's it, it should work for you. intelmq bug both for installation of binaries or through python.

You know @sebix :)

dbconfig-common.zip

@aragorne007
Copy link

@oencarnacion
Thank a lot for your help :)

I've move your folder to my IntelMQ server and, to be sure, add chmod 766 :P
But, when I try to create a new IntelMQ account, I've this error :

Loading config from /etc/intelmq/api-config.json
Traceback (most recent call last):
File "/usr/local/bin/intelmq-api-adduser", line 27, in
session_store = intelmq_api.session.SessionStore(str(api_config.session_store), api_config.session_duration)
File "/usr/local/lib/python3.9/dist-packages/intelmq_api/session.py", line 86, in init
self.init_sqlite_db()
File "/usr/local/lib/python3.9/dist-packages/intelmq_api/session.py", line 100, in init_sqlite_db
with self.connect() as con:
File "/usr/local/lib/python3.9/dist-packages/intelmq_api/session.py", line 91, in connect
return sqlite3.connect(self.dbname, check_same_thread=False,
sqlite3.OperationalError: unable to open database file

I guess, it's a new permission issue :P

@aragorne007
Copy link

What is the service used by intelmq ?
To know which service restart to apply new modification

@aragorne007
Copy link

Ohh no, I find the issue :

cat /etc/intelmq/api-config.json
{
"intelmq_ctl_cmd": ["sudo", "-u", "intelmq", "intelmqctl"],
"allowed_path": "/var/lib/intelmq/bots",
"session_store": "/var/lib/dbconfig-common/sqlite3/intelmq-api/intelmqapi",
"session_duration": 86400,
"allow_origins": ["*"]
}

The session_store folder doesn't exists

/var/lib/dbconfig-common/
└── dbconfig-common
├── config
├── intelmq-api.conf
└── sqlite3
└── intelmq-api

Can I create the folder myself ?

@oencarnacion
Copy link

yes

@sebix
Copy link
Member

sebix commented Aug 17, 2022

I installed the package from pip3 install.

OK, then it's clear that some manual work needs to be done. Installing via pip is more difficult than the automated deb/rpm package installation.

Permissions 766 don't sound right at all. Make the file and the directory containing the file writable for the webserver, only the webserver.

What is the service used by intelmq ?
To know which service restart to apply new modification

The API is run by apache -> sudo systemctl restart apache2

@aragorne007
Copy link

I'm dumb ^^
The folder wasn't /var/lib/dbconfig-common/sqlite3/intelmq-api/intelmqapi but /var/lib/dbconfig-common/dbconfig-common/sqlite3/intelmq-api/intelmqapi.

So twice dbconfig-common in depth ^^
I fixed that.

I created the new user with intelmq-api-adduser succesfully but now, in the intelmq-manager panel, I can't login.
I've this error :

Login failed with unknown reason. Please report this bug.

Redis and Apache2 has ben restarted

@sebix
Copy link
Member

sebix commented Aug 17, 2022

What does the apache error log say?

@aragorne007
Copy link

[Wed Aug 17 22:32:41.602163 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501] mod_wsgi (pid=941): Exception occurred processing WSGI script '/usr/lib/python3/dist-packages/intelmq_api/intelmq-api.wsgi'., referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602322 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501] Traceback (most recent call last):, referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602385 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501]   File "/usr/lib/python3/dist-packages/intelmq_api/intelmq-api.wsgi", line 12, in application, referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602390 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501]     return __hug_wsgi__(environ, start_response), referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602399 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501]   File "/usr/local/lib/python3.9/dist-packages/hug/api.py", line 500, in api_auto_instantiate, referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602403 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501]     module.__hug_wsgi__ = module.__hug__.http.server(), referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602412 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501]   File "/usr/local/lib/python3.9/dist-packages/hug/api.py", line 371, in server, referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602416 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501]     self.api._ensure_started(), referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602425 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501]   File "/usr/local/lib/python3.9/dist-packages/hug/api.py", line 638, in _ensure_started, referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602429 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501]     startup_handler(self), referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602437 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501]   File "/usr/local/lib/python3.9/dist-packages/intelmq_api/serve.py", line 46, in setup, referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602441 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501]     intelmq_api.api.initialize_api(api_config), referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602450 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501]   File "/usr/local/lib/python3.9/dist-packages/intelmq_api/api.py", line 82, in initialize_api, referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602454 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501]     session_store = session.SessionStore(str(session_file),, referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602463 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501]   File "/usr/local/lib/python3.9/dist-packages/intelmq_api/session.py", line 86, in __init__, referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602467 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501]     self.init_sqlite_db(), referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602476 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501]   File "/usr/local/lib/python3.9/dist-packages/intelmq_api/session.py", line 100, in init_sqlite_db, referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602480 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501]     with self.connect() as con:, referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602488 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501]   File "/usr/local/lib/python3.9/dist-packages/intelmq_api/session.py", line 91, in connect, referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602492 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501]     return sqlite3.connect(self.dbname, check_same_thread=False,, referer: http://192.168.0.21/intelmq-manager/
[Wed Aug 17 22:32:41.602512 2022] [wsgi:error] [pid 941:tid 140627228423936] [client 192.168.0.144:59501] sqlite3.OperationalError: unable to open database file, referer: http://192.168.0.21/intelmq-manager/

The log for 1 attemp

It look like another permission missing

@sebix
Copy link
Member

sebix commented Aug 17, 2022

is the session database and the directory containing the file writable for the webserver? Please show jq .session_store /etc/intelmq/api-config.json | xargs ls -l and jq .session_store /etc/intelmq/api-config.json | xargs dirname | xargs ls -ld

@aragorne007
Copy link

root@HostIntell:/var/lib# jq .session_store /etc/intelmq/api-config.json | xargs ls -l
-rw-r--r-- 1 root root 24576 Aug 17 22:30 /var/lib/dbconfig-common/sqlite3/intelmq-api/intelmqapi
root@HostIntell:/var/lib# jq .session_store /etc/intelmq/api-config.json | xargs dirname | xargs ls -ld
drwxrw-rw- 2 intelmq intelmq 4096 Aug 17 22:30 /var/lib/dbconfig-common/sqlite3/intelmq-api
root@HostIntell:/var/lib#

"root" for the first one. I'll change that.
Can you confirm that the user need to be used is "intelmq" ?
Because it's the user I used eveywhere in the system (not the user used in the webGUI )

@sebix
Copy link
Member

sebix commented Aug 17, 2022

Can you confirm that the user need to be used is "intelmq" ?

I assume the user your webserver is using, is not intelmq, but rather apache, www-data or similar. Use that one.

@aragorne007
Copy link

Can you confirm that the user need to be used is "intelmq" ?

I assume the user your webserver is using, is not intelmq, but rather apache, www-data or similar. Use that one.

Ohhh, It's maybe that the main issue.
Every intelmq folder is "own" by "intelmq" user and not apache or www-data

@aragorne007
Copy link

Do you have an exhaustive list of all folder that I need to change the owner from "intelmq" to "www-data" ?

www-data is the user mention and existing in the /etc/passwd file. "Apache" is not existing

@sebix
Copy link
Member

sebix commented Aug 17, 2022

Did you actually run intelmqsetup after the installation?

@aragorne007
Copy link

No, because this command isn't existing.
Maybe another issue with the pip3 install ?

I performed "intelmqctl check" before and "No Issues Found" diplayed.

@sebix
Copy link
Member

sebix commented Aug 17, 2022

Did you use https://intelmq.readthedocs.io/en/maintenance/user/installation.html#pypi or a different installation method?

@aragorne007
Copy link

Yes, I used this one.

I tried a new time :

root@HostIntell:/var/lib# sudo -i
root@HostIntell:~# pip3 install intelmq
Requirement already satisfied: intelmq in /usr/lib/python3/dist-packages (3.0.2)
root@HostIntell:~# useradd -d /opt/intelmq -U -s /bin/bash intelmq
useradd: user 'intelmq' already exists
root@HostIntell:~# sudo intelmqsetup
sudo: intelmqsetup: command not found
root@HostIntell:~#

@aragorne007
Copy link

I tried also that, but not really relevant :

root@HostIntell:/var/lib# apt install intelmqsetup
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package intelmqsetup
root@HostIntell:/var/lib# apt search intelmqsetup
Sorting... Done
Full Text Search... Done
root@HostIntell:/var/lib# apt search intelmq
Sorting... Done
Full Text Search... Done
intelmq/unknown,now 3.0.2-1 all [installed]
  Solution for IT security teams for collecting and processing security feeds

intelmq-api/unknown,now 3.0.1-1 all [installed,automatic]
  HUG based API for the intelmq project

intelmq-manager/unknown,now 3.0.1-1 all [installed]
  Graphical interface to manage configurations for the IntelMQ framework.

@sebix
Copy link
Member

sebix commented Aug 17, 2022

OK, that looks like you have mixed different installation methods. You installed intelmq with deb packages and pip. The outcome is naturally a mishmash. Please stick to one method.

@aragorne007
Copy link

Mhmm maybe because I tried some things when it doesn't work properly the first time.
I'll delete the VM and create a new one to have a fully fresh new VM to work :)

Thank a lot for your help !!
Could you let this topic open if I've another issue related to this same error ?

@sebix
Copy link
Member

sebix commented Aug 17, 2022

Could you let this topic open if I've another issue related to this same error ?

Please just create a discussion or ask for help on the mailing list to keep the existing issues on-topic. Thanks.

@aragorne007
Copy link

Ok ok :)

Thank you very much for the time spent to help me :)
It really appreciated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants