Skip to content

Migrate the WranglesPY CI test image to Python 3.13 - #1141

Merged
ebhills merged 3 commits into
mainfrom
codex/python313-container-migration
Aug 27, 2026
Merged

ebhills merged 3 commits into
mainfrom
codex/python313-container-migration

Conversation

@ebhills

@ebhills ebhills commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Linked issue

Related to #1131.

This PR delivers the WranglesPY CI test-image portion of that issue. It no longer uses Closes because the production Python migration belongs to wrangleworks/Lambda-Recipes and is not completed here.

What changes

  • Moves both stages of the repository-root CI test image to python:3.13-slim-bookworm.
  • Adds test-image-only pins for pandas==2.3.3 and numpy==2.4.6. The reusable package contract remains pandas>=2.0,<3.0, so Pandas 3 is still prohibited.
  • Requires binary wheels for every test-image dependency and removes the NumPy 1.24 source-build path, legacy pip option, and compiler packages.
  • Makes Botocore/Pandas package-data cleanup independent of the Python minor version while preserving the S3-only Botocore model.
  • Adds a credential-free smoke check covering constraint-derived versions, package metadata, imports, NumPy/Pandas/PyArrow behavior, S3 model loading without network access, package-data trimming, and compiler absence.
  • Loads and executes the exact pull-request image in CI without publishing it or supplying repository secrets. That image runs the smoke check, pip check, pip freeze, and both local recipe samples.
  • Avoids duplicating the full credential-safe suite inside every PR image; the existing host matrix already runs it. The post-merge full in-image test and promote-by-digest sequence remains intact.
  • Documents in README, AGENTS, CI labels, and repository guidance that this is a test image—not the deployed recipe runtime.

Production runtime ownership

The production execute-recipe AWS Lambda image is built, pushed to ECR, and deployed by wrangleworks/Lambda-Recipes.

A review of its current main found that production is not yet on Python 3.13:

  • Lambda-Recipes dockerfile uses public.ecr.aws/lambda/python:3.12 and has a hard-coded /var/lang/lib/python3.12/site-packages patch path.
  • CI, DEV deployment, and PROD deployment also run their test stages on the Lambda Python 3.12 image.
  • No open Lambda-Recipes issue or PR for Python 3.13 was found.

Migrating that runtime requires separate Lambda-Recipes work. Merging this PR does not change the deployed Lambda Python version.

How it was verified

CI run 108 passed in full on commit 7eb7cd64:

  • Full existing pytest matrix passed on Ubuntu and Windows with Python 3.11 and 3.13.
  • All four package-install checks and schema generation passed.
  • The Linux/amd64 CI test image built successfully with every dependency restricted to a binary wheel.
  • Exact-image runtime: Python 3.13.1, Wrangles 1.19.0, NumPy 2.4.6, pandas 2.3.3, PyArrow 25.0.1, Polars 1.33.0, NumExpr 2.14.2, Boto3 1.43.81, and Botocore 1.43.81.
  • Constraint-derived version checks, installed metadata pandas<3.0 validation, S3-only Botocore data, pandas-test removal, compiler absence, data round trip, and no-network S3 model construction passed.
  • Exact-image python -m pip check reported no broken requirements; pip freeze was captured in the job log.
  • Both local recipe samples passed inside the exact image.

Local/offline validation:

  • Python 3.13.1 focused smoke tests: 14 passed.
  • Python 3.13.1 credential-safe suite: 1,548 passed, 112 deselected.
  • The two warnings are existing pandas fillna downcasting FutureWarning instances in wrangles/connectors/file.py on current main; this PR does not suppress them.
  • python -m pip check: no broken requirements.
  • Python compilation, workflow YAML parsing, line-ending, static terminology, ancestry, scope, and git diff --check validation passed.
  • The branch is zero commits behind current origin/main.

Not yet claimed:

  • The pull-request image is intentionally not published.
  • Post-merge immutable test-image publication, credentialed tests inside that pushed image, remote recipe execution, and digest promotion cannot occur until merge.
  • No Python 3.13 production Lambda build or deployment has occurred.

Compatibility and risk

  • Public package metadata is unchanged: pandas remains >=2.0,<3.0; Python 3.11 and 3.12 package support are not removed.
  • The CI test image deliberately pins pandas 2.3.3 and NumPy 2.4.6. Pandas 3 remains separate work.
  • Existing versioned GHCR tags, Lambda-Recipes, ECR images, Lambda functions, and pinned downstream consumers are untouched.
  • Requiring wheels for all test-image dependencies makes future Python/wheel gaps fail during image construction instead of falling through to an absent compiler.
  • Rollback is a revert of the two test-image commits; production Lambda rollback is outside this repository.

Ready-for-review checklist

  • One human delivery owner is assigned
  • The linked issue and corrected repository ownership are documented
  • The branch is current with main and has no unrelated diff
  • Focused and credential-safe local tests pass
  • New or changed behavior has direct test coverage
  • Documentation/configuration is updated
  • The PR description reflects the branch's current scope and latest validation
  • Current-commit CI passes
  • Reviewers verify the fixes, resolve the blocking conversations, and submit a fresh approval

@ebhills ebhills self-assigned this Aug 24, 2026
Comment thread dockerfile
Comment thread scripts/container_smoke.py
Comment thread tests/test_container_smoke.py
Comment thread scripts/container_smoke.py
Comment thread scripts/test-container-local.sh Outdated
@ebhills
ebhills requested a review from lmolotii August 26, 2026 19:04
@ebhills
ebhills marked this pull request as ready for review August 26, 2026 19:07

@lmolotii lmolotii left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some changes needs to be reviewed.

Comment thread .github/workflows/ci.yml Outdated
Comment thread .github/workflows/ci.yml Outdated
Comment thread .github/copilot-instructions.md Outdated
Comment thread .github/workflows/ci.yml Outdated
Comment thread .github/workflows/ci.yml
Comment thread constraints/container-python313.txt Outdated
Comment thread scripts/container_smoke.py Outdated
@ebhills ebhills changed the title Migrate production container to Python 3.13 Migrate the WranglesPY CI test image to Python 3.13 Aug 27, 2026
@ebhills
ebhills merged commit 4f48393 into main Aug 27, 2026
13 checks passed
@ebhills
ebhills deleted the codex/python313-container-migration branch August 27, 2026 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants