Resolved: DNScrypt's queries randomly fail with the error RESPONSE_ERROR. #1884
Replies: 16 comments 40 replies
-
|
Same issue here ! so frustrating |
Beta Was this translation helpful? Give feedback.
-
|
Same problem here since around the same time. Based on this thread it seems Cloudflare are either mishandling or deliberately rate limiting signed or encrypted DNS queries. Looks like we might have to file an issue with Cloudflare themselves. |
Beta Was this translation helpful? Give feedback.
-
|
Update: apparently you can no longer file a Cloudflare ticket with a free account, so I created a thread at their community about this. Please chime in there if this issue is also affecting you. |
Beta Was this translation helpful? Give feedback.
-
|
The cloudflare entry is registered as DoH so you'd need to have this in the main config file You can put the sdns string into this web page to see how it's setup https://dnscrypt.info/stamps I see Google is also showing as DoH - their stamps from the file are: [cloudflare] with stamp [sdns://AgcAAAAAAAAABzEuMC4wLjEAEmRucy5jbG91ZGZsYXJlLmNvbQovZG5zLXF1ZXJ5] Can you check your main config file to see if |
Beta Was this translation helpful? Give feedback.
-
|
Will give it a shot later |
Beta Was this translation helpful? Give feedback.
-
|
I'm not so sure this will cause the random failures actually.... |
Beta Was this translation helpful? Give feedback.
-
Yep, it's been set all this time. |
Beta Was this translation helpful? Give feedback.
-
@ianbashford Wait, are you saying both Google and Cloudflare DNS have this issue? I've noticed it with Cloudflare only but that's because Cloudflare is my only upstream DNS. If Google DNS is also affected then that would seem to be indicative either of a FWIW, I've also pinged Cloudflare support on Twitter about this. If anyone wants to RT or reply to that tweet to get some traction on this, it would be much appreciated. >2 months of breakage no es bueno. |
Beta Was this translation helpful? Give feedback.
-
|
Anybody having any luck with this at all? I don't see any further replies, so if anyone managed to resolve the problem kindly share the solution with the rest of us. UPDATE at this point I've even tried replacing the standalone At the same time, NextDNS DoH works on my phone on the same home network. This leads me to believe the DNS stamps used by |
Beta Was this translation helpful? Give feedback.
-
|
query.log.gz
Output: In total I got 26 'NETWORK_ERROR' written into the logs ( which tallies with the lost queries ) I'll happily keep these running if we can get any more useful input from them, but I don't see much... If you're seeing much higher numbers than I'm seeing in this test, could it be something to do with some locations? Since cloudflare run an anycast address, if we're in different geographic locations, we'll be hitting different cloudflare datacentres... Any luck from twitter ? -- |
Beta Was this translation helpful? Give feedback.
-
|
I'm definitely getting more errors than that. I'll run the same tests and report back the results tomorrow |
Beta Was this translation helpful? Give feedback.
-
|
Just got home, setting up the test now. In order to get the domain names from your cut -f 3-4 query.log | uniq > domain_names.txtWhen I was starting the dnscrypt-proxy service, I got the following error: With the following config: listen_addresses = ['127.0.0.1:53']
server_names = ['cloudflare']
doh_servers = true
[query_log]
file = '/var/log/dnscrypt-proxy/query.log'
[nx_log]
file = '/var/log/dnscrypt-proxy/nx.log'
[sources]
[sources.'public-resolvers']
url = 'https://download.dnscrypt.info/resolvers-list/v2/public-resolvers.md'
cache_file = '/var/cache/dnscrypt-proxy/public-resolvers.md'
minisign_key = 'redacted'
refresh_delay = 72
prefix = ''Forgive me if my config is wrong, this is my second time setting up DNSCrypt |
Beta Was this translation helpful? Give feedback.
-
I'm strongly inclined to say this is (1 of) our root problem(s). I was wondering if the lists weren't downloading at all but I couldn't think of any way to verify. That said, I couldn't get |
Beta Was this translation helpful? Give feedback.
-
|
Ah, I forgot to replace the minisign_key |
Beta Was this translation helpful? Give feedback.
-
|
@OoLunar something like this should work
-d == file to read Obvs the ip address and port ( -s 127.0.0.1 -p 5353 ) you make as appropriate for your system |
Beta Was this translation helpful? Give feedback.
-
|
I tried From forum convos with Pi-hole devs and mods it seems they've largely migrated to Anyway, I deployed @ianbashford I'm on MetroNet in SE Iowa, USA. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks -- so am I right in thinking that's not encrypted upstream unless you apply additional steps? |
Beta Was this translation helpful? Give feedback.
-
|
@ianbashford Depends on how you define "encrypted". Because I have DNSSEC enabled in Pi-hole, the replies are cryptographically signed. However as far as I can tell the queries
In other words, your entire DNS query activity is no longer fed directly to any single entity, which makes gaining a master view of the former considerably more difficult for a privacy attacker.
There's also a slight (depending on whether the user thinks they can achieve better uptime with their own hardware than a company with global datacenters and an army of engineers 😜) reliability benefit as the user no longer has to worry about upstream quirks (such as the one on this thread) or outages. |
Beta Was this translation helpful? Give feedback.
-
|
Alternate view:
|
Beta Was this translation helpful? Give feedback.
-
|
@jedisct1 I agree. However, to quote a lead software developer at my 2nd job out of grad school: a decent idea that works is better than an excellent idea that doesn't. Pi-hole works just fine with either Cloudflare directly upstream or |
Beta Was this translation helpful? Give feedback.
-
I'm using DNSCrypt directly with a Wireguard VPN, on VoidLinux. These requests are failing on both my VPN clients and my server |
Beta Was this translation helpful? Give feedback.
-
|
@OoLunar Are you also in the the US or elsewhere? Also, no pihole here... |
Beta Was this translation helpful? Give feedback.
-
|
@OoLunar follow up - perhaps a better question. Where does the wireguard termination think it is? |
Beta Was this translation helpful? Give feedback.
-
My server is in St. Louis, Missouri. My VPN client is in Texas. I use Cloudflare on all possible devices, but only notice the issue when using DNSCrypt.
What do you mean by "termination"? |
Beta Was this translation helpful? Give feedback.
-
|
You answered it for me -- so I have an openvpn provider, but it can terminate in the US/Australia/Germany etc. and make it look like I'm located there. |
Beta Was this translation helpful? Give feedback.
-
|
Let's double check everything here:
Are all of these correct? |
Beta Was this translation helpful? Give feedback.
-
However, as Ian also pointed out, the problem seems also to vary by the user's geographic location. For him, for example, the failure rate is too low to be disruptive, while for me stuff like, for example, It could be that the major DoH providers (except NextDNS?) are using Cloudflare's load balancing system so as to not reinvent the wheel. Assuming this load balancing system is in the cloud, it would explain why my switching from Cloudflare to Quad9 resulted in the same problems. All that said, here's an interesting tidbit from Cloudflare's own documentation on Rate Limiting:
The writeup doesn't define "high", BUT my Pi-Hole dashboard says it handles ~115K queries/day (11 PCs, multiple smart TVs, phones, consoles, etc.). IIRC things functioned just fine back when I was below 100K. For reference, NextDNS starts charging at ~10K queries/day. Does anyone else experiencing this problem know how many queries per day they're sending upstream? TBH if that's the issue then |
Beta Was this translation helpful? Give feedback.
-
No -- they're both DoH stamps, not dnscrypt, that's all I said |
Beta Was this translation helpful? Give feedback.
-
|
@OoLunar - I don't see this issue ( but don't use pihole ) |
Beta Was this translation helpful? Give feedback.
-
My apologies. Previous comment edited accordingly. |
Beta Was this translation helpful? Give feedback.
-
|
Further question -- I see you have quite a few AAAA records in your initial report -- I don't have IPv6 here, so all my queries are IPv4. Are you on native IPv6? Is it possible all of your queries are being made over IPv6 ? Update on the testing: I've setup a VPN to Chicago, IL (that's the closest termination point from my VPN provider I could find to MIssouri ) I've kept the rate high, so that 5 queries per second ( each query is for a different hostname so no repetition ) Looks like I'm being served from cloudflare's server in 'ORD', which is O'Hare: |
Beta Was this translation helpful? Give feedback.
-
|
Well, that's honestly a good question. My VPS provider claims they have ipv6 support, and I can retrieve my ipv6 address, and even have software listen on it. However, I can't seem to make any requests with it. What's even worse is that none of my devices seem to support ipv6, even with the VPN off. When I try to do any of the following, I get a "Network is unreachable" error: For now I'll add the |
Beta Was this translation helpful? Give feedback.
-
|
OK folks, over 2 months later (LOL) this issue is finally getting Cloudfare staff attention. If you're still having problems, please chime in at this Cloudflare Community thread. |
Beta Was this translation helpful? Give feedback.
-
|
Not really sure what's laughable about a 2 month wait time, especially when the post wasn't properly tagged. I'm excited to see what the underlying issue is, if there is any. |
Beta Was this translation helpful? Give feedback.
-
|
Well I'd also tried to file a support ticket, but apparently only paid customers can do that now. I also apologized for the wrong tag in the CC thread. 🤷♂️ Folks can argue about tags and other technicalities all they want, but at the end of the day if a user's problems aren't being addressed it's very likely they'll move on to something else ... which I have. I may not be a paying Cloudflare customer but there's absolutely no doubt even anonymized DNS traffic data is useful to them or they wouldn't be offering if for free. I'm still here trying to connect those who use |
Beta Was this translation helpful? Give feedback.
-
|
Wow -- great response now they've got to it
|
Beta Was this translation helpful? Give feedback.
-
|
@ianbashford I agree I still think As for the various methods of DNS protection available, per this opinion of an Infrastructure Security Architect who also teaches grad level sysadmin and programming courses at Stevens Institute of Technology:
DNSSEC is going to be good enough for me for now. ^1 This is not because I don't trust Google but because I don't think Google's business is organized around being as resistant to state actor inquiry as Cloudflare is |
Beta Was this translation helpful? Give feedback.
-
|
@ianbashford, glad to hear that our issue was upstream and not with DNSCrypt itself. I'll still be running the tests, but it's safe to close the issue. Thank you, @ianbashford and @jdrch for your support |
Beta Was this translation helpful? Give feedback.
-
You're always welcome. Community isn't just a word 😉 |
Beta Was this translation helpful? Give feedback.
-
|
You're welcome and I'm glad we seem to have a solution |
Beta Was this translation helpful? Give feedback.
-
Happy Holidays to you too and stay safe! |
Beta Was this translation helpful? Give feedback.
-
|
FWIW, the test to the US cloudflare has completed -- although since I won't be exiting from the (hopefully) now fixed IP block I doubt they're worth much. 5 queries / second for 24 hours == 432k queries -- total lost was 336 ( 0.08% ) |
Beta Was this translation helpful? Give feedback.
-
|
it happen with doh cloudflare gateway but its fine via Dot so much reply with respon 'REFUSED' |
Beta Was this translation helpful? Give feedback.
-
Thanks for the useful information about DoT! It appears Cloudflare's abuse protection system is DNS encryption protocol-specific. I don't experience any rate limiting querying them directly using DNSSEC. I suppose Cloudflare's DoH rate limit is related to the fact that DoH was intended to be application specific, but as deployed by most If you read this very well written summary of various DNS protection methods you'll see DoT was apparently not designed to be application specific as DoH was. ^1 One could argue this discrepancy between design and use is somewhat self-inflicted as Cloudflare enables the latter via their 1st party package ;) |
Beta Was this translation helpful? Give feedback.
-
|
Hey @ianbashford, unsure if I should open up a new discussion or continue on this one, but since this one is already "open", I'll just ask here. As per my comment over in the CF thread, I was going to continue the testing. Day before yesterday I flunked up the command and tried making DNS requests on the wrong port, which led to a bunch of timeout errors. Today I completed the correct command, however now I have to create my own source using the CF stamp provided to us. Now, I read through the wiki and got myself a provider file as shown: # dns-cloudflare-resolver
Supports one source only: cloudflare
--
## cloudflare
Cloudflare, but instead of using dns.cloudflare.com we're using cloudflare-dns.com
https://community.cloudflare.com/t/doh-dnscrypt-queries-randomly-fail-with-the-error-response-error-solved/334624/8?u=oolunar
sdns://AgcAAAAAAAAABzEuMC4wLjEAEmNsb3VkZmxhcmUtZG5zLmNvbQovZG5zLXF1ZXJ5Following the link provided to the minisign website, I ran the following commands to get myself a sudo xbps-install minisign # Void Linux, a rolling release distro. Installed version 0.10
minisign -G # Generates the key pair
minisign -Sm dns-cloudflare-resolver.mdThe output of Here's my listen_addresses = ['127.0.0.1:53']
server_names = ['cloudflare']
ipv4_servers = true
ipv6_servers = false
dnscrypt_servers = true
doh_servers = true
odoh_servers = false
require_dnssec = true
require_nolog = true
require_nofilter = true
[query_log]
file = '/var/log/dnscrypt-proxy/query.log'
format = 'tsv'
[nx_log]
file = '/var/log/dnscrypt-proxy/nx.log'
format = 'tsv'
[sources]
[sources.'cloudflare']
urls = ['https://files.forsaken-borders.net/dns-cloudflare-resolver.md']
cache_file = '/var/cache/dnscrypt-proxy/dnscrypt-resolver.md'
minisign_key = 'RWQFkh3QI+pwFA3sgq9pN8Q/CUvKBie8uTwINo+tpAm/G2bPhwH/AdZR' # This was grabbed from the minisign.pub generated from `minisign -G`However, when I start dnscrypt, I get the following error: (Ignore the
|
Beta Was this translation helpful? Give feedback.
-
|
You must add that stamp as static if You want to try that new stamp until get update as public resolver change server_names |
Beta Was this translation helpful? Give feedback.
-
|
Oh! That makes so much more sense. Did I miss a section of the wiki that mentions this, or was it not on there at all? |
Beta Was this translation helpful? Give feedback.
-
|
This describes it |
Beta Was this translation helpful? Give feedback.
-
|
Just finding this thread.. Was it actually stated that there was a possible problem between dnscrypt-proxy and Cloudflare.. and keeping Cloudflare was more important than dnscrypt-proxy? Quad9 is a better alternative than Cloudflare.. and Frank pays better attention to us than Cloudflare.. Maybe just me.. #ThanksFrank |
Beta Was this translation helpful? Give feedback.
-
AFAICT Quad9 lacks user forums and a ticket system, so reporting problems is difficult. I do this instead. |
Beta Was this translation helpful? Give feedback.
-
DNScrypt's queries randomly fail with the error
RESPONSE_ERROR.Log:
Config:
Beta Was this translation helpful? Give feedback.
All reactions