replace Requests http client by Niquests #664
Conversation
Niquests is a simple, yet elegant, HTTP library. It is a drop-in replacement for Requests, which is under feature freeze. HTTP/1.1, HTTP/2, and HTTP/3 supported. Both sync and async interfaces are available.
|
Sorry, but from my perspective there are no benefits.There are no features missing in requests for the purposes of this library. In addition a lot of risk would be introduced by switching to something less widely used than requests which has been stable for years and is PSF supported. |
|
Also note that this broke half of all tests, seems it's not just a drop-in |
|
You came back to take a deeper lock.
That I may understand. But it is not entirely true. I would not propose this if your community wouldn't benefit from it. Then, you should know that YT admins may, without notice, kill HTTP/1 requests to make this lib useless. So, HTTP/2, and HTTP/3 would be nice to gain more confidence on that topic.
That I've heard many times. This thinking is tautological. Let's summarize what you said: Old + Big usage = Better ? So, if given a choice (for free) between a "Toyota Corolla from 2003" and a "Tesla Model X from 2023" you would pick the first by that logic. You would also choose Google Chrome over others without question. (and so on..) Ok, so there I am not saying that Niquests is bug-free, but it's in a pretty good place right now. Progress can't happen otherwise.
You closed the PR in minutes, so I didn't get the chance of looking further. If you're curious, re-open and re-rerun them. hint: regards, |
|
Thank you for the eloquent and detailed response, appreciate it. To address your points:
I will keep your project in mind, but at this point it is too early for me to make a decision that is not prompted by a need Edit: I will reopen the PR for now but that does not mean this will be merged anytime soon |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #664 +/- ##
==========================================
- Coverage 94.99% 94.96% -0.04%
==========================================
Files 38 38
Lines 2278 2283 +5
==========================================
+ Hits 2164 2168 +4
- Misses 114 115 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
You are not alone on this take. But unfortunately, no other way emerge to solve this. Basically here, you'll need to duplicate code and all the way down from the http calls up to the user front call, you'll need to convert async/await(able). I've already contributed to that goal at https://github.com/grafana-toolbox/grafana-client/blob/main/script/generate_async.py
There is no official statement (figure...) but all over internet, the number of reports about bans start to grow. You have yourself some complaint about it. Not only Google, but major provider like Cloudflare use the way you connect to them (TLS layer, http headers, protocol used, cipher, etc...) to determine how likely you are NOT a traditional browser. (see https://niquests.readthedocs.io/en/latest/user/advanced.html#tls-fingerprint-like-ja3 for more info) This is how they target and act upon library/client like yours. I am extremely confident that they will kill HTTP/1 soon. Some services (major) in my home country started to progressively kill HTTP/1, this was the starting point of the development for Niquests.
To you, yes, no breaking. But for other major libraries out there, there are major breakage happening while they are in maintenance mode only. And there's still major fixes awaited by the community, sometimes for months or years. You are right, this happen to be a minor issue. The Google QUIC implementation seems to be aggressive toward keepalive/ACKs.
Don't wait for complete "out-of-order" situation. Maybe try to be proactive about this one. Keep in mind that Requests stated that the project is already half-dead.
I understand completely. Take your time. Until then, Regards, |
Could you point out the mentioned fix?
Could you link the source for this please? Additionally, since your project claims performance benefits (up to 3x faster than requests), I had a look at the test runtime. It seems worse than with requests:
Granted that most of the runtime is to blame on server-side processing. Nevertheless it still seems like a significant performance regression at first glance. Any clue? |
Of course. See jawah/urllib3.future#161 and jawah/urllib3.future#162
There is more, but, you will excuse me to not collect the hundreds of demonstration for it.
Short story: no multiplexing used, only pure synchronous. re-do QUIC handshake hundred of times using Post-Quantum Security, don't reuse session, etc... To witness the speed up in CI, that's tricky, because you'll need to instantiate once the niquests.Session() and leverage pytest concurrency, then you'll see improvement for sure, up to 2 times faster. (x3 times is reserved to async or manual multiplexing aka. lazy responses) To be clear, our claim about 3x faster is accompanied with a reproducer / specific scenario. If you use HTTP/2+ as you use HTTP/1, it will be slower because you don't leverage his potential. On the latest run, we get:
so, not so significant after all. The failure here seems unrelated to us:
regards, |
|
It seems that we are facing more ban from YT. The phenomena is getting more and more important. maybe it worth to pursue that PR and getting HTTP/3 mainstream. |
This PR effectively replace the http client Requests for Niquests.
Niquests is a drop-in replacement for Requests that is no longer under feature freeze.
This new client support HTTP/2, and HTTP/3 by default and offers both sync and async interfaces. It supports all the latest shiny features you would expect from an http client.
If you were interested on merging, I will be interested to propose a followup PR that will propose a script that generate
the async part of ytmusicapi automatically. If you want to.
disclaimer: I maintain Niquests.