feat(hyperpod-eks): Local Zone deployment support for Terraform modules

[aravneelaws]

Purpose

Adds optional, backward-compatible support for deploying SageMaker HyperPod EKS
clusters into an AWS Local Zone using the existing hyperpod-eks-tf Terraform
stack. Local Zone workers previously could not be targeted (the AZ-discovery
filter excludes opt-in zones), and even when forced in, all egress hairpinned
through the parent Region via the regional NAT gateway. This PR closes both gaps
with inputs that default to today's standard-AZ behavior when unset.

Changes

• Private subnet AZ pinning (private_subnet module + root): new
  private_subnet_availability_zone_ids pins worker subnets to explicit AZ IDs
  (1:1 with private_subnet_cidrs), bypassing the
  opt-in-status = "opt-in-not-required" discovery filter that excludes Local
  Zones. Default [] preserves auto-discovery.
• FSx placement (root + fsx_lustre wiring): new fsx_availability_zone_id
  lets FSx be placed in a specific AZ (e.g. a parent AZ) when the instance
  group's subnet is in a Local Zone where FSx or a given tier is not offered.
  Default "" keeps FSx co-located with the first instance group's subnet.
• Local-Zone-local NAT egress (vpc + private_subnet modules + root):
  three new inputs (local_zone_egress_zone_ids,
  local_zone_public_subnet_cidrs, local_zone_network_border_groups) create
  one LZ-local NAT gateway per listed zone with a NetworkBorderGroup-scoped
  EIP. The vpc module emits nat_gateway_ids_by_zone_id; the private_subnet
  module routes matching subnets to their LZ NAT via lookup(...), falling back
  to the regional NAT for unmapped AZs. Wired to private_subnet only — not
  eks_cluster, since EKS control-plane subnets cannot live in a Local Zone.
• Docs: new "Local Zone Deployment" section in
  1.architectures/7.sagemaker-hyperpod-eks/terraform-modules/README.md
  (variable table, egress/NAT explanation, example custom.tfvars, FSx-in-LZ
  guidance, and the LZ opt-in prerequisite).

All new inputs default to empty/standard-AZ behavior: existing tfvars files
that never reference them produce an identical terraform plan.

Test Plan

Validated with a full end-to-end Local Zone HyperPod EKS deployment using the
companion Local Zone quickstart
(a single local-zone.tfvars applied against this stack; no forked Terraform).
The cluster came up with worker nodes in the Local Zone and control-plane
subnets in parent AZs, and egress used the LZ-local NAT gateway.

This configuration has also been verified in a production deployment by a
customer using this template.

Environment:

• AWS Service: SageMaker HyperPod (EKS-orchestrated)
• Instance type: ml.c6i.2xlarge (Local Zone worker instance group)
• Number of nodes: 1 (worker instance group) + EKS control plane in parent AZs

Test commands:

# Local Zone opt-in (verify opted-in before deploy)
aws ec2 modify-availability-zone-group --group-name us-west-2-phx-2a --opt-in-status opted-in

# From hyperpod-eks-tf/
terraform init
terraform fmt -recursive -check
terraform validate
terraform plan  -var-file=/path/to/local-zone.tfvars
terraform apply -var-file=/path/to/local-zone.tfvars

# Verify workers landed in the Local Zone
kubectl get nodes --show-labels
aws sagemaker list-cluster-nodes --cluster-name "$(terraform output -raw hyperpod_cluster_name)"

Test Results

• terraform fmt -recursive -check passes clean; terraform validate succeeds.
• Backward compatibility: a plan with a standard (non-LZ) tfvars is unchanged
  versus main — the new inputs are inert when unset.
• End-to-end LZ deployment: HyperPod worker nodes registered and became
  Schedulable in the Local Zone; EKS control-plane ENIs stayed in parent AZs.
• Verified in a production deployment by a customer using this template.
• With the LZ-local NAT enabled, worker egress stays in-zone instead of
  hairpinning to the parent Region, materially improving first-hop latency and
  internet throughput for Local Zone workers (origin/CDN-anchored services
  benefit less).

Directory Structure

N/A — this PR updates existing Terraform modules under
1.architectures/7.sagemaker-hyperpod-eks/terraform-modules/ and does not add a
3.test_cases/ entry.

Checklist

• I have read the contributing guidelines.
• I am working against the latest main branch.
• I have searched existing open and recently merged PRs to confirm this is not a duplicate.
• The contribution is self-contained with documentation and scripts.
• External dependencies are pinned to a specific version or tag (no latest).
• A README is included or updated with prerequisites, instructions, and known issues.
• New test cases follow the expected directory structure.

[KeitaW]

Batch 1: Local Zone Feature Availability

[KeitaW]

NAT gateway in a Local Zone is a per-zone capability, and the docs read as though it is universal

This is my main suggestion, and it is a docs change rather than a code one, because the code is
right.

The AWS Local Zones features page
carries NAT Gateway as a per-zone column in its service matrix, not as a general Local Zones
capability. Reading that column across the 39 Local Zone rows on the page as of 2026-08-14, two are
marked as supporting it: us-west-2-phx-2a and us-west-2-lax-1a. Those are your test zone and the
zone of your A/B measurement, so your validation is sound. The other 37 zones are not marked.

I confirmed the negative side by applying this PR's own vpc and private_subnet modules against
us-east-1-atl-2a:

Error: creating EC2 NAT Gateway: ... api error NotAvailableInZone:
Nat Gateway is not available in this availability zone

  with module.vpc.aws_nat_gateway.lz_nat[0],
  on .../modules/vpc/main.tf line 177, in resource "aws_nat_gateway" "lz_nat":

Two details worth folding into the docs. First, the LZ public subnet, the border-group EIP, the LZ
route table and its association all create successfully before the NAT is refused, so the operator
is left with a partially applied stack and has to re-run after removing the variables. Second, the
API rejects on zone capability before it evaluates the EIP's border group (I got the identical
error with a correctly border-group-scoped EIP and with a region-scoped one), so a user in an
unsupported zone gets no signal about the border group they were just told to configure.

Would you consider a sentence in this section saying NAT gateway availability varies by Local Zone,
pointing at the NAT Gateway column of the features page, quoting that error, and noting that
leaving the three local_zone_* variables unset (regional NAT, accepting the hairpin) is the
correct configuration in the other zones? Everything else in this PR works in any Local Zone; only
this one feature is narrow.

Verified live 2026-08-14: applied this PR's modules against both us-east-1-atl-2a (refused, as
quoted above) and us-west-2-lax-1a (succeeded end to end).

[KeitaW]

Batch 2: Silent Failure and Wasted Spend

[KeitaW]

Duplicate AZ IDs collapse in both zipmap outputs, stranding a paid NAT gateway

The validation blocks check list lengths against each other, which is genuinely more than most
PRs do, but not uniqueness. Resources are created once per list entry
(modules/vpc/main.tf:144), while zipmap keeps only the last value for a repeated key. Verified
on Terraform 1.14.3:

zipmap(["az1","az1","az2"], ["natA","natB","natC"])
=> { "az1" = "natB", "az2" = "natC" }

So a duplicated entry in local_zone_egress_zone_ids creates two LZ public subnets, two
border-group EIPs and two NAT gateways, of which only the second is reachable by any route. The
first bills at roughly 0.045 USD per hour plus data processing forever, carrying no traffic, and
nothing in the plan says so.

The same shape applies to private_subnet_availability_zone_ids, because
modules/private_subnet/outputs.tf:26-31 builds az_to_subnet_map with zipmap over the same
zone IDs: duplicates create all the subnets but expose only the last one per AZ, so HyperPod and
FSx cannot select the earlier one.

A distinct() check alongside the existing length validations covers both:

Suggested change

	default = []
	}
	default = []
	validation {
	condition = length(var.local_zone_egress_zone_ids) == length(distinct(var.local_zone_egress_zone_ids))
	error_message = "local_zone_egress_zone_ids must not contain duplicate zone IDs."
	}
	}

The same distinct() guard on private_subnet_availability_zone_ids would close the
az_to_subnet_map half of this.

[KeitaW]

Two individually valid but mismatched lists degrade silently to the regional NAT

Credit first, because this is handled better than I expected: a typo'd AZ ID fails loudly, in
either list. A bad value in private_subnet_availability_zone_ids breaks aws_subnet.private and
a bad one in local_zone_egress_zone_ids breaks aws_subnet.lz_public, both at apply.

The case that stays silent is a mismatch between two lists that are each individually valid. With
private_subnet_availability_zone_ids = ["usw2-phx2-az1"] and
local_zone_egress_zone_ids = ["usw2-lax1-az1"], everything applies cleanly: an LZ NAT is built in
Los Angeles, the Phoenix worker subnet takes the var.nat_gateway_id fallback on this line, the
cluster comes up healthy, the latency feature you paid for is simply absent, and the Los Angeles
NAT bills without carrying traffic. The same orphaned-NAT outcome occurs with
create_vpc_module = true and create_private_subnet_module = false, since the private-subnet
module is the only stock consumer of the map.

Either of these would close it, and the second is nearly free:

1. A root-level check block asserting every entry of local_zone_egress_zone_ids also appears in
   private_subnet_availability_zone_ids.
2. Re-export the resolved mapping. modules/vpc/outputs.tf:46 and :56 already emit
   nat_gateway_ids_by_zone_id and lz_nat_eips_by_zone_id, but the root outputs.tf is unchanged
   in this PR, so neither map reaches the operator. Surfacing them (or better, a per-subnet
   "this subnet's NAT" map) makes the wiring visible after apply.

[KeitaW]

closed_network = true with Local Zone inputs fails with a zipmap error that names neither

modules/vpc/main.tf:140 gates lz_egress_enabled on !var.closed_network && length(...) > 0, so
with closed_network = true the resource count drops to 0 while the zone-ID list stays populated.
Both outputs then zip a non-empty key list against an empty splat. Confirmed on 1.14.3:

Call to function "zipmap" failed: number of keys (1) does not match number of values (0).

That is reachable for anyone toggling an otherwise-configured deployment into closed-network mode,
and the message mentions neither closed_network nor the variable they set. Guarding both outputs
on the same local that gates the resources keeps it consistent:

Suggested change

	value = zipmap(var.local_zone_egress_zone_ids, aws_nat_gateway.lz_nat[*].id)
	value = local.lz_egress_enabled ? zipmap(var.local_zone_egress_zone_ids, aws_nat_gateway.lz_nat[*].id) : {}

The same guard is worth applying to lz_nat_eips_by_zone_id on line 58.

[KeitaW]

Local Zone egress inputs are silently ignored in a BYO-VPC deployment

You already know this and the code comment says so, so this is really a docs note. With
create_vpc_module = false this line replaces the map with {}, and matching private subnets fall
back to the regional NAT at modules/private_subnet/main.tf:72. The apply succeeds. But the README
states unconditionally that setting the three variables creates Local Zone NAT gateways, which is
not true on the BYO-VPC path. One clause in the README ("when this module manages the VPC") would
match the docs to the code comment.

[KeitaW]

Batch 3: Documentation Consistency

Worker subnet sizing is worth a sentence, since the whole point is workers in one Local Zone

A Local Zone deployment usually means a single worker subnet. On high-NIC-count instances, VPC CNI
warm-ENI and warm-IP allocation consume addresses faster than operators expect, and a /22 yields
roughly 1019 usable addresses. Your example uses a /16, so the example is fine; it is the readers
who shrink it who get bitten. A sizing note plus a pointer at WARM_ENI_TARGET / WARM_IP_TARGET
would fit well next to the subnet variables.

[KeitaW]

modify-availability-zone-group is passed the zone name instead of the zone group

--group-name takes the Local Zone's GroupName, which is the zone name minus the trailing zone
letter. As written the command targets us-west-2-phx-2a and will not match a group.

Suggested change

	--group-name us-west-2-phx-2a --opt-in-status opted-in
	--group-name us-west-2-phx-2 --opt-in-status opted-in

This is the same transform the PR already documents for local_zone_network_border_groups, and
GroupName and NetworkBorderGroup are in fact the same string. Verified live 2026-08-14: in
us-east-1, describe-availability-zones reports GroupName: us-east-1-atl-2 for zone
us-east-1-atl-2a, and in us-west-2 it reports GroupName: us-west-2-lax-1 for
us-west-2-lax-1a. I did not query us-west-2-phx-2a directly, so that specific string follows
the rule rather than a direct observation.

[KeitaW]

The lookup command just above does not return the value the opt-in command needs

Related to the above, and the reason the wrong identifier is easy to reach for: this query projects
[ZoneName,ZoneId,ParentZoneName], so it never shows GroupName, which is exactly the value the
next command consumes. It also omits OptInStatus, even though the following step tells the
operator to verify the zone reports opted-in.

Suggested change

	--query "AvailabilityZones[?ZoneType=='local-zone'].[ZoneName,ZoneId,ParentZoneName]" \
	--query "AvailabilityZones[?ZoneType=='local-zone'].[ZoneName,ZoneId,GroupName,NetworkBorderGroup,OptInStatus]" \

That one line then hands the reader every string this section asks them to use: the AZ ID for
private_subnet_availability_zone_ids, the group name for the opt-in call, the network border
group for local_zone_network_border_groups, and the status to verify.

[KeitaW]

The FSx cross-zone escape hatch fails against the example custom.tfvars as written

The advice here is right, but a reader following the example one section above cannot act on it.
That example defines a single private subnet, in usw2-phx2-az1. fsx_availability_zone_id is
resolved by a direct map index at main.tf:79 (local.az_to_subnet_map[var.fsx_availability_zone_id]),
and az_to_subnet_map is built only from the private subnets that exist. Setting it to a parent-AZ
ID in that configuration therefore fails the plan rather than relocating FSx. Reproduced the
failure shape on 1.14.3:

Error: Invalid index
The given key does not identify an element in this collection value.

Could this section say that using fsx_availability_zone_id also requires adding a parent-AZ entry
to private_subnet_cidrs and private_subnet_availability_zone_ids, and ideally show the
two-subnet form of the example?

While you are in this section, one performance sentence would help the audience: FSx in a parent AZ
means every read crosses the Local Zone to Region link, which the Local Zones user guide documents
at an MTU of 1300 bytes for most Local Zones, 8801 bytes for us-east-1-atl-2a / us-west-2-phx-2a
and several others, and 9001 bytes for us-west-2-lax-1a / 1b. For a training cluster that is a
throughput decision, not only a placement one.

Source: Local Zones user guide, How Local Zones work.

[KeitaW]

Batch 4: Network Boundary

[KeitaW]

The Local Zone public subnet auto-assigns public IPv4 addresses although only the NAT lives there

Raising this as a nit rather than a defect, because it mirrors the existing regional public subnets
(modules/vpc/main.tf:41 and :56 both set it), so it is consistent with the module's own
precedent and clearly deliberate.

Still: this subnet exists only to host the NAT gateway, which uses its own explicitly allocated
EIP and does not need subnet-level auto-assignment. With map_public_ip_on_launch = true and the
internet gateway default route at modules/vpc/main.tf:198, anything accidentally launched into it
lands directly on an internet-routed boundary with a public address. Setting it to false costs
nothing here and narrows the blast radius of a misplaced instance.

Suggested change

	map_public_ip_on_launch = true
	map_public_ip_on_launch = false

[KeitaW]

The code comment predicting the EIP border-group error asserts a string AWS does not return

Small accuracy point on the comment above aws_eip.lz_nat. It predicts AWS will reject with "EIP
is not associated with the border group of the subnet". In an unsupported zone that is not what
comes back, because CreateNatGateway rejects on zone capability first (I got NotAvailableInZone
with both a region-scoped and a border-group-scoped EIP). In a supported zone I could not produce
it either, since the correct border group succeeds. The requirement the comment describes is
real and worth keeping; it is only the quoted error string that is unverified.

Suggested change

	# attach to a NAT gateway in an LZ subnet. AWS will reject the association
	# with "EIP is not associated with the border group of the subnet".
	# attach to a NAT gateway in an LZ subnet; AWS rejects the association when the
	# EIP is not in the subnet's network border group.

[KeitaW]

Batch 5: Things That Look Great

I tested this PR rather than only reading it, and it holds up. Concretely, on
us-west-2-lax-1a (already opted in on my account, and one of the two NAT-capable zones), applying
this PR's vpc and private_subnet modules gave Apply complete! Resources: 19 added, and then:

• the LZ NAT gateway reached available in a subnet whose zone really is us-west-2-lax-1a, while
  the regional NAT sat in us-west-2a, exactly as designed;
• its EIP carried NetworkBorderGroup: us-west-2-lax-1 and attached without complaint;
• the worker subnet's route table had 0.0.0.0/0 pointing at the Local Zone NAT, so the
  lookup(...) mapping resolves correctly end to end;
• an instance launched into the Local Zone private subnet reported
  EGRESS_IP=15.254.9.47, which is precisely the LZ NAT's EIP, with first hop
  10.198.20.35 at 0.089 ms.

That last number is worth calling out: your inline comment claims "Traceroute hop 1: 23.8 ms ->
0.095 ms" and I independently measured 0.089 ms. Performance claims in PRs frequently do not
replicate. Yours did.

The backward-compatibility story is also verified rather than asserted. I generated the same
configuration twice, once sourcing the modules from main and once from this branch, with every
new variable unset, and diffed the planned resource sets: identical, 19 resources each, no drift.
terraform fmt -recursive -check and terraform validate both pass on the branch as you say.

Three design decisions I want to credit specifically, because they are the ones people usually get
wrong:

1. The AZ-ID versus zone-name discipline. Zone IDs (usw2-phx2-az1) for placement, zone names
   (us-west-2-phx-2) for the network border group, kept straight throughout the variables, the
   docs and the example. That distinction trips up most Local Zone code.
2. Making the border group a required input rather than deriving it. The cross-variable
   validation blocks mean you cannot enable the feature and forget it, and the comment explaining
   why it is passed explicitly rather than suffix-stripped is the kind of note that saves the
   next reader an hour.
3. Not wiring the LZ NAT into eks_cluster. This matches the EKS guide exactly: "you must not
   specify Local Zone subnets when you create your cluster. However, you can have worker nodes in
   multiple Local Zones connected to the same cluster." Deliberately scoping the map to
   private_subnet is the correct call and the comment says so.

The test plan naming a real zone, a real instance type and a customer deployment is more than most
infrastructure PRs carry, and it made this much faster to verify.

[KeitaW]

Nice work! Few minor comments.