Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ldap user search very slow #13

Open
dschick opened this issue Mar 1, 2019 · 17 comments
Open

ldap user search very slow #13

dschick opened this issue Mar 1, 2019 · 17 comments

Comments

@dschick
Copy link

dschick commented Mar 1, 2019

Hi,

my colleagues are little complaining about the fact, that the user search via ldap is pretty slow, even if one type a complete and correct user name.
I've checked with ldapsearch in the terminal, that the ldap connection itself, seems to be fast.

Is there anyway how to tackle this problem?

Best

Daniel
@johanfforsberg
Copy link
Contributor

Hi Daniel,

I see, unfortunately I haven't currently got access to a LDAP server to test this against, although it was initially tested against a real one. Is it slower than when running without LDAP settings?

There is "throttling" of the search in the GUI, and it could be it's a little too much. It's set to not send search queries to the backend more often than every 0.5 seconds. This was intended to prevent overloading the LDAP, but perhaps I was being too careful. It would certainly also be possible to cache the results in the backend, since LDAP info rarely changes. If you're able to experiment with the frontend, you could maybe try changing "frontend/src/entryeditor.js", in the section

        this.slowFetchUserSuggestions = debounce(
            this.fetchUserSuggestions.bind(this),
            500
        );

you might want to experiment with changing the number 500 to something smaller, or even 0 and see what happens. It might send quite a lot of queries (one per key stroke I think) to the LDAP server though. If you try this, please report your results so we can change the setting.

Thanks for reporting this! BTW, did you try updating to the latest master? A change was merged to remember the last author list used, which might also make the situation less annoying in some cases.

@johanfforsberg
Copy link
Contributor

To test the speed of the backend querying LDAP, you can also try something like

time curl http://<elogyhost>/api/users/?search=daniel

and see how long it takes. If it takes more than an instant, the problem is somewhere between the backend and the LDAP server.

Anyway, I've set up a LDAP server now so I can test, it does not seem very slow but the experience can definitely be improved :)

@dschick
Copy link
Author

dschick commented Mar 11, 2019

Thanks for your comments.
First of all the "remember-author" feature will pretty much solve most of our issues.

However, when I try the curl cmd I get a rather slow response

time curl http://elogy/api/users/?search=mylogin
{
    "users": [
        {
            "login": "mylogin",
            "name": "D......",
            "email": "D......"
        }
    ]
}

real	0m4,052s
user	0m0,012s
sys	0m0,011s

while an ldapsearch return quasi instantaneously from the same terminal.
Changing frontend throtteling seem not to be the issue here.

@johanfforsberg
Copy link
Contributor

johanfforsberg commented Mar 11, 2019 via email

@dschick
Copy link
Author

dschick commented Mar 11, 2019

hm, doing the ldap search directly in python is also pretty fast:
con.search_s("ou=people,o=server,dc=de", ldap.SCOPE_SUBTREE, filterstr="(|(cn=*sch*)(uid=*sch*))", attrlist=["uid", "cn", "mail"])

ähm, is the python lib doing a new binding on every search?
It does not seem to keep the binding to the ldap alive, right?
Maybe this is the problem? and also in case the server is throttling after to many rebindings

@dschick
Copy link
Author

dschick commented Mar 11, 2019

doint 100 rebinds and searches with python ldap takes 1.8sec and is perfectly linear with the number of attempts, so a single search takes 0.018sec.

@johanfforsberg
Copy link
Contributor

johanfforsberg commented Mar 11, 2019 via email

@dschick
Copy link
Author

dschick commented Mar 11, 2019 via email

@dschick
Copy link
Author

dschick commented Mar 11, 2019 via email

@dschick
Copy link
Author

dschick commented Mar 11, 2019 via email

@johanfforsberg
Copy link
Contributor

I did test with docker, but only against my own test LDAP server with a few users. I'll do some more testing.

It might be that the search query we're using is too slow. I believe elogy being used with a real LDAP server at MAXIV, and as far as I can remember performance was OK when I was testing. But I suppose the performance could vary between server implementations, and maybe with database size.

The search string is e.g. (|(cn=*daniel*)(uid=*daniel*)), do you mind trying it with your ldap client?

Regarding local install, did you try the make install, make run way? If it does not work, I'd like to know what error you get. It will require python3.5 or later, though.

@dschick
Copy link
Author

dschick commented Mar 12, 2019

Hej,

I actually used the (|(cn=*daniel*)(uid=*daniel*)) query in my test with python from the elogy host and it was pretty fast.

I tried already the make install in my virtual machine for testing and it worked without problems.
However, I need to do a proper config, because I could not reach the webinterface so far.
Do I need to edit the config.py files for that?

For testing on my production machine, I would like to do also a clean uninstall of the local install. is that possible?

@dschick
Copy link
Author

dschick commented Mar 12, 2019

python2.7 vs python 3.6 does not make a difference

@johanfforsberg
Copy link
Contributor

OK, so slow query is also eliminated... 🤔

The make scripts will start the backend on port 8888 and the frontend proxy on port 3000, if you run it with make -j2 run (that enables make to run both processes in parallel). So to check out the interface, go to port 3000 on your VM, and to talk directly to to the api you can use either port. You may have to make some changes to config.py though, at least to get LDAP to work. Just put in the LDAP variables you use in the docker installation.

Regarding uninstall, make install should not install anything outside the git checkout directory, and does not even need to be done as a super user. All python deps are installed in a virtualenv in backend/env and the npm stuff is in frontend/node_modules. Uninstalling elogy is just a matter of removing the git checkout.

The only thing you may have to do on the host is to install some of the dependencies in the underlying OS, in particular the ldap libs. E.g. sudo apt-get install libldap2-dev libsasl2-dev (on ubuntu).

@dschick
Copy link
Author

dschick commented Mar 13, 2019

now I am completely confused.
I just tried the ldap search from my elogy installation from within my virtual machine (only for testing) and it is pretty quick (instantaneous).

time curl http://<elogyhost>/api/users/?search=daniel

takes only 0.083sec instead of 4s on my production server.
Both systems are running now with docker with the latest source from github.
I did the same setup on both systems and also the OS of the virtual machine and the production server is the same.

The only obvious difference is, that both machines are in different networks.
However, the ldap search directly in python is a bit slower on the virtual machine than on the production server.

@dschick
Copy link
Author

dschick commented Mar 13, 2019

We are currently also checking our infrastructure, since there could be a problem with one of our switches.

@johanfforsberg
Copy link
Contributor

Very confusing indeed, but I think we have to conclude that this is likely not directly caused by Elogy. However it might still be worth thinking about some form of caching of the search results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@johanfforsberg @dschick