GeoIP only loading the first zone - no errors

[a1466d44-d3dc-4c0b-90c7-315b088731d7]

Hi

I have an issue with my PowerDNS configuration, and/or understanding of how the geoip module should work. (Maybe it is just some issue with understanding the documentation?)

Lets start with the used versions:
PowerDNS 4.9.7 on Ubuntu 22.04.5 LTS
Backend gsqlite3 and geoip

Relevant configuration

# grep geoip /etc/powerdns/pdns.d/pdns.local.conf
launch=gsqlite3:SRV,geoip:GEO
geoip-GEO-zones-file=/var/db/powerdns/geoipdbzones.yml
geoip-GEO-database-files=/usr/share/GeoIP/GeoIP.dat

# cat /var/db/powerdns/geoipdbzones.yml
# https://doc.powerdns.com/authoritative/backends/geoip.html
zones_dir: /var/db/powerdns/geo-zones/
mapping_lookup_formats: ['%cc-%re', '%cc']

# ls /var/db/powerdns/geo-zones/
01_subsub.sub.cloud.example.org.yml  geo2.example.org.ym

# cat /var/db/powerdns/geo-zones/01_subsub.sub.cloud.example.org.yml 
# https://doc.powerdns.com/authoritative/backends/geoip.html
zone:
  domain: subsub.sub.cloud.example.org
  ttl: 30
  records:
    subsub.sub.cloud.example.org:
      - soa: ns1.example.org hostmaster.example.org 2025081301 7200 3600 1209600 3600
      - ns:
        content: ns1.example.org
        ttl: 600
      - mx: 10 mx.example.org
    ip.subsub.sub.cloud.example.org:
      - txt: "your ip is %ip in %cn-%co "
    aut.service.subsub.sub.cloud.example.org:
      - cname: lb2.cloud.example.org
    at.service.subsub.sub.cloud.example.org:
      - cname: lb2.cloud.example.org
    nl.service.subsub.sub.cloud.example.org:
      - cname: lb.cloud.example.org
    as.service.subsub.sub.cloud.example.org:
      - cname: lb.cloud.example.org
    us.service.subsub.sub.cloud.example.org:
      - cname: lb.cloud.example.org
    usa.service.subsub.sub.cloud.example.org:
      - cname: lb.cloud.example.org
    irn.service.subsub.sub.cloud.example.org:
      - cname: lb.cloud.example.org
    internal.service.subsub.sub.cloud.example.org:
      - cname: lb2.cloud.example.org
  services:
    subsub.sub.cloud.example.org:
      default: [ '%co.%cn.service.subsub.sub.cloud.example.org', '%co.service.subsub.sub.cloud.example.org', '%cn.service.subsub.sub.cloud.example.org', 'at.service.subsub.sub.cloud.example.org' ]
      10.0.0.0/8: 'internal.service.subsub.sub.cloud.example.org'

# cat ls /var/db/powerdns/geo-zones/geo.example.org.yml 
cat: ls: Datei oder Verzeichnis nicht gefunden
# https://doc.powerdns.com/authoritative/backends/geoip.html
zone:
  domain: geo.example.org
  ttl: 30
  records:
    geo.example.org:
      - soa: ns1.example.org hostmaster.example.org 2025081301 7200 3600 1209600 3600
      - ns:
        content: ns1.example.org
        ttl: 600
      - mx: 10 mx.example.org
    ip.geo.example.org:
      - txt: "your ip is %ip in %cn-%co "
    aut.service.geo.example.org:
      - a: 1.1.1.2
    at.service.geo.example.org:
      - a: 1.1.1.112
    nl.service.geo.example.org:
      - a: 1.1.1.113
    us.service.geo.example.org:
      - a: 2.2.2.2
    usa.service.geo.example.org:
      - a: 2.2.2.3
    internal.service.geo.example.org:
      - a: 10.0.0.1
  services:
    geo.example.org:
      default: [ '%co.%cn.service.geo.example.org', '%co.service.geo.example.org', '%cn.service.geo.example.org', 'at.service.geo.example.org' ]
      10.0.0.0/8: 'internal.service.geo.example.org'

# pdnsutil list-zone example.org | egrep "NS"
subsub.sub.cloud.example.org          600     IN      NS      ns1.example.org.
geo.example.org   600     IN      NS      ns1.example.org.

But when I try out my configuration i get

# host -4 -t txt ip.subsub.sub.cloud.example.org                                                                                                                                                                                                                      
ip.subsub.sub.cloud.example.org descriptive text "your ip is 141.101.104.49 in eu-nld "
#
# host -4 -t txt geo.example.org                                                                                                                                                                                                                                            
geo.example.org has no TXT record

The Funny thing is: when i switch the filenames to make the other zone get loaded first, it works the other way around.

so the mail issue for me is: Only the FIRST Zone gets loaded and is working!
Please, someone, point out my error. I already lost a day in debugging without issue as the geoip backend does basically no debug logging
(i am only a few more hours away from trying to compile powerdns myself :D

PS.: I also added errors in the files to verify that they even get read, and powerdns properly threw an error as expected. so the Files ARE being read.

[Habbie]

what does pdnsutil list-all-zones say?

do both zones work if you don't load the sqlite backend?

[a1466d44-d3dc-4c0b-90c7-315b088731d7]

Hi and thanks for the very fast response

Yes they turn up

# pdnsutil list-all-zones | grep example.org
example.org
subsub.sub.cloud.example.org
geo.example.org

[Habbie]

Note that I asked two questions.

[a1466d44-d3dc-4c0b-90c7-315b088731d7]

ohh, god damit... sry

Yes the behavior is exactly the same.
Also (as expected) only the geoip zones show up when querying for list-all-zones

[a1466d44-d3dc-4c0b-90c7-315b088731d7]

Any new ideas on that?
I am unfortunately out of ideas and luck :(

[miodvallat]

I have been unable to reproduce. Using your two yml files, and 4.9.7, pdns_server answers for both subsub.sub.cloud.example.org and geo.example.org. Can you share your complete pdns.conf (except for secrets), in case this is caused by a particular setting you are using but my test setup doesn't?

[a1466d44-d3dc-4c0b-90c7-315b088731d7]

Thank you for trying
I will, but only after the 2nd of September (vacation...;)

[a1466d44-d3dc-4c0b-90c7-315b088731d7]

@miodvallat
Here is my currently running config (I only changed sensible (api-key) and identifiable (allow-dnsupdate-from, local-address) stuff)

The /var/db/powerdns/geoipdbzones.yml can be seen above
And yes I changed the geoip-GEO-database-files as I have found out that we have a payed subscription in our company

# pdns_control current-config | egrep -v '^($|#)'
allow-dnsupdate-from=127.0.0.0/8,::1,198.51.100.1
api=yes
api-key=ofcourseichangedthis
default-ttl=600
disable-syslog=
dnsupdate=yes
domain-metadata-cache-ttl=60
expand-alias=yes
geoip-GEO-database-files=/var/lib/GeoIP/GeoIP2-City.mmdb,/var/lib/GeoIP/GeoIP2-Country.mmdb
geoip-GEO-zones-file=/var/db/powerdns/geoipdbzones.yml
gsqlite3-SRV-database=/var/db/powerdns/powerdns.db
gsqlite3-SRV-dnssec=1
include-dir=/etc/powerdns/pdns.d
launch=gsqlite3:SRV,geoip:GEO
local-address=198.51.100.1,a000::1
log-dns-details=yes
log-dns-queries=yes
log-timestamp=no
logging-facility=0
outgoing-axfr-expand-alias=yes
primary=yes
receiver-threads=2
secondary=yes
server-id=ns1
setgid=pdns
setuid=pdns
tcp-control-address=127.0.0.1
tcp-control-range=127.0.0.0/8, ::1/128, fe80::/10
version-string=anonymous
webserver=yes
webserver-allow-from=127.0.0.1
write-pid=no
xfr-cycle-interval=15

Our server is hosting 111 Zones

# pdnsutil list-all-zones | wc -l
111

And the geoip zones are all present in the list-all-zones output, with and without the gsqlit3 backend.

PS.: I just discovered that I have another github account on my home machine (of course i do...)!?

[a1466d44-d3dc-4c0b-90c7-315b088731d7]

(This text is written as i tested along, so its like a timeline...)

At the moment I have three(!) possible outcomes for a query directly to the server:

1. It works and i get a response (desired)
2. It does not work and i get something in the range of "... hast no TXT Record" (Zone file not loaded/working)
3. It does not work and i just get an empty response (???)

PS.: I got two zone to work, but not a third one (all copy pasted!)
the two working geoip zones are in different master(?) Zones all hosted the same server.
Is that my underlying issue?
Some wrong configuration on the specific Zone I am trying to use?

All three zones get recognized by the powerdns service and are loaded without any error

I looks like this in the Logs (the first query did NOT worked, the second worked)

Sep 05 09:30:37 ns1 pdns_server[1622659]: Remote 1.2.3.4 wants 'ip.geo2.example.org|TXT', do = 0, bufsize = 512: packetcache MISS
Sep 05 09:30:40 ns1 pdns_server[1622659]: Remote 1.2.3.4 wants 'ip.geo.example.org|TXT', do = 0, bufsize = 512: packetcache MISS

NOW I RENAMED the the geoip zone yml file from 03_something.yml to 05_something.yml in order to get it loaded AFTER the 04_something.yml where geo.example.org is configured
AND SUDDENLY BOTH ZONES WORK! (systemd unit restart invoked)
I am out of Ideas where I might have have an issue, as i cant seem to pinpoint my exact issue....

I'd really appreciate any idea one might give me

[a1466d44-d3dc-4c0b-90c7-315b088731d7]

Ok, Now i have managed to figure our MY problem. (still no solution, but at least the Title is now definitely wrong. geoip is working as expected!)

My issue is that in my zone the specific entry is not working.

I have the following geoip config right now:

eu01234.subsub.sub.example.org
v2eu01234.subsub.sub.example.org
v3eu01234.subsub.sub.example.org

in the example.org zone
*.subsub.sub.example.org is a CNAME
the three geoip zones have their own NS records.

The current result:

eu01234.subsub.sub.example.org <- Works
v2eu01234.subsub.sub.example.org <- empty reply
v3eu01234.subsub.sub.example.org <- Works

any ma question: Why is the middle one not working???

[a1466d44-d3dc-4c0b-90c7-315b088731d7]

Ok, I can now finally rest (even when I don't find the complete answer...)

My solution: I copied the working geoip zone file over the broken one, renamed everything and it magically worked...

Can i see any differences in the file?
no

So the real Issue was: PEBKAC

Sorry for all the messages.
I'll try to figure out what the underlying issue was. (I have kept the not working files with me)
But I do not want to get into this atm...

I'm just happy that it works now :)