route53: hosted-zone-records

[jblosser]

I just found this repo today and I'm thrilled, but I'm going to need more route53 support since we do a lot of that. So, here's the first draft at a command to display all record sets for a route53 zone, using jq for parsing since the default awscli text output for this is a lot more difficult to handle. Note that some awscli help text still claims "list-resource-record-sets" only provides 100 recordsets at a time, but I don't see this behavior from the current endpoint. I get all results in one response, tested up to over 10,000 records in a single zone.

I'm not really assuming this will be acceptable without more changes so I welcome feedback.
Particularly:

• Is using jq ok? There is precedent, but not necessarily this detailed.
• Is the sample output ok for the docs? I tried to show a few record types, but don't have a real ALIAS from your zone to show, and I'm not sure what the precedent is for a fake data format.

Last, AWS ALIAS records make anything like this a bit tricky but I went with what's usually the useful info and a fake TTL (similar to what cli53 does). It might need more detail later for action commands vs this basic query.

[mbailey]

I just found this repo today and I'm thrilled

Thanks! I was thrilled to read what you created.

but I'm going to need more route53 support since we do a lot of that.

It looks like you've grokked BMA and solved your problem very nicely.

TLDR The conversation is about to get philosophical but please don't let this hold you back
from continuing to create more commands. I keep a file called lib/mike-functions where
some new functions I'm working on are incubated (and often live out their lives). This this provides a safe place to keep them with close to zero fear of merge conflicts in future.

BMA is aimed at letting you move fast and sometimes a slow moving Master actually
helps with that. PEP20 does a good job
of describing my philosophy.

You've raised some good questions:

* Is using jq ok? There is precedent, but not necessarily this detailed.

BMA only uses jq for three commands for the --sort-keys functionality.
Personally I would prefer to only use jp in the project to keep things simpler.

However, someone mentioned a good list of jp limitations in the HN thread yesterday.

I think the relevant limitation for this case is:

• can't reference parents when doing iteration. Why? All options for iteration, [* ] and map, all use the iterated item as the context for any expression. There's no opportunity to get any other values in. May be possible for a fixed set of lengths. Something akin to the following (except there is no syntax for switching or if statements):

  switch (length):
  case 1: [expression[0]]
  case 2: [expression[1], expression[1]]
  case 3: [expression[0], expression[1], expression[2]]

The options for BMA (in my order of preference) seem to be:

1. Remove the limitations in jq that prevent us using it.
   JMESPath was(?) created by the AWSCLI team which I think that's an awesome gift. Improving it would be giving back something (but also involve effort from multiple parties).

2. Use jq but only when necessary (which may require some expert arbitration)
   This seems like the easiest option but will require more effort both to manage and to read.

3. Don't do things can't be done without jq (with 3 grandfathered exceptions)
   "Special cases aren't special enough to break the rules."
   The Zen of Python

So, here's the first draft at a command to display all record sets for a route53 zone, using jq for parsing since the default awscli text output for this is a lot more difficult to handle. Note that some awscli help text still claims "list-resource-record-sets" only provides 100 recordsets at a time, but I don't see this behavior from the current endpoint. I get all results in one response, tested up to over 10,000 records in a single zone.

* Is the sample output ok for the docs? I tried to show a few record types, but don't have a real ALIAS from your zone to show, and I'm not sure what the precedent is for a fake data format.

What did the ALIAS look like for your zone?

Last, AWS ALIAS records make anything like this a bit tricky but I went with what's usually the useful info and a fake TTL (similar to what cli53 does). It might need more detail later for action commands vs this basic query.

I noticed that the hosted-zone-ns-records command also faked the TTL. This was sitting in lib/mike-functions for over a year as a deliberated whether it would be of enough value to people to include it and the faked TTL of 300 suited my needs at work. I use it to bulk import into our DNS Hosts web console.

What are your use cases? List, create, update, delete records? I'd encourage you to go ahead and create them if the ROI for you justifies it. Perhaps the functionality will influence BMA to accept some more jq but even if it doesn't, you've got what you need.

[danbst]

@mbailey isn't jq listed as a dependency already?

Unfortunately, I didn't notice this PR, so I wrote my custom implementation... I'll put the important bit here then:

  echo "$result" | jq -r '.ResourceRecordSets[] | [ .Name, .Type, (.AliasTarget ?| .DNSName), ([ .ResourceRecords[].Value ] ?| join(",") ) ] | @tsv' \
         | sed 's/\\\\052/*/g' | column -t -T 3 -T 4 -T 5  | LC_ALL=C sort | less -SMK

• .ResourceRecords may have multiple values
• sed 's/\\\\052/*/g' is required to correctly translate *.example.com DNS names
• sort just makes output more deterministic. LC_ALL=C removes this "feature" to place "foobar" and "_foobar" close to each other, so order is ASCII
• I pipe it to column -t and less -S for a nicer display by default

[jblosser]

Just a note I haven't forgotten this. I have been trying to figure out a way to do this with jmespath and also how to cover more of the route53 cases than just the bind-common ones, but yeah, the limitations of jmespath vs the lack of normalization of the different extensions to the records may be too much to overcome.

[nitrocode]

Good start!

If jq is a dependency, it seems like it could be OK to use in our functions. It would give a lot more flexibility when jmespath is lacking.

However, I've been using this locally using only jmespath

aws route53 list-resource-record-sets \
  --hosted-zone-id $(aws route53 list-hosted-zones --query 'HostedZones[?Name==`domain.com.`].Id' --output text | cut -d'/' -f3) \
  --output text \
  --query 'ResourceRecordSets[].[Name, TTL, Type, to_string(ResourceRecords[].Value)]' |
  column -s$'\t' -t

and it gives me the following output

internal.snip.com.               172800  NS     ["ns-snip.awsdns-00.co.uk.","ns-snip.awsdns-00.com.","ns-snip.awsdns-00.org.","ns-snip.awsdns-00.net."]
internal.snip.com.               900     SOA    ["ns-snip.awsdns-00.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400"]
snip.internal.snip.com.          300     A      ["snip"]
snip.internal.snip.com.          60      CNAME  ["snip.cluster-snip.us-west-2.rds.amazonaws.com"]

The only annoying thing is that the to_string() function is a poor replacement for a join(). I'd rather see a comma separated string returned but this looks good enough for now.

What do you think @mbailey ?

[nitrocode]

Just so you folks know, I'm completely okay with this PR as-is.

Any chance we can get this merged soon ? 😄 🙏

cc: @mbailey