Migrate the WranglesPY CI test image to Python 3.13

.github/copilot-instructions.md

@@ -14,13 +14,13 @@
 ## Tech Stack
 
 - **Python:** 3.11, 3.12, 3.13 (multi-version support)
-- **Core Dependencies:** pandas (>=2.0), numpy, polars (1.33.0), pyyaml
+- **Core Dependencies:** pandas (>=2.0,<3.0), numpy, polars (1.33.0), pyyaml
 - **Database Connectors:** sqlalchemy, pymssql, psycopg2-binary, pymysql, pymongo
 - **Cloud/External:** boto3 (AWS S3), simple-salesforce, fabric (SFTP)
 - **Data Formats:** openpyxl (Excel), xlsxwriter
 - **AI/ML:** OpenAI integration, Hugging Face models
 - **Testing:** pytest (9.0.2), pytest-mock, lorem (test data generation)
-- **Containerization:** Production Docker image uses Python 3.11-slim-bookworm; development container uses Python 3.13-bookworm
+- **Containerization:** The WranglesPY CI test image uses Python 3.13-slim-bookworm with pandas 2.3.3 and NumPy 2.4.6; the development container uses Python 3.13-bookworm. The deployed recipe runtime image is owned by `wrangleworks/Lambda-Recipes`.
 
 ## Project Structure
 
@@ -50,8 +50,8 @@ WranglesPY/
 │   └── recipe_base_schema.json
 ├── setup.py                    # Package setup and release version
 ├── requirements.txt            # Production dependencies
-├── dockerfile                  # Multi-stage Docker build
-├── main.py                     # Container entry point
+├── dockerfile                  # Multi-stage CI test-image build
+├── main.py                     # CI test-image entry point
 └── .github/workflows/          # CI/CD pipelines
 ```
 
@@ -125,14 +125,18 @@ pip install .
 wrangles.recipe tests/samples/generate-data.wrgl.yml
 ```
 
-### Docker Build
-The Dockerfile uses multi-stage builds for optimization:
-1. Compile stage: Installs build dependencies and packages
+### CI Test Image Build
+The repository-root Dockerfile builds a GHCR test image used to validate the
+installable package and recipes. It is not the deployed production runtime;
+`wrangleworks/Lambda-Recipes` owns the AWS Lambda image and its Python version.
+
+The test-image Dockerfile uses multi-stage builds for optimization:
+1. Dependency stage: Installs binary wheels under the tracked test-container constraint; no compiler toolchain is installed
 2. Build stage: Copies only necessary files (~400MB final image)
 3. Special optimizations: Removes unused botocore AWS service definitions, pandas test data
 
 ```bash
-docker build -t wrangles:latest .
+docker build -t wrangles:test .
 ```
 
 ## Coding Guidelines
@@ -249,12 +253,12 @@ deployments from an explicit `main` SHA and removing the remaining legacy paths.
 ### Merging never releases anything
 
 Merging to `main` runs CI and, under the current transitional workflow,
-publishes a tested container image. It does not publish a Python package and
+publishes a tested GHCR CI image. It does not publish a Python package and
 does not deploy to any environment.
 
 | Action | Result |
 | --- | --- |
-| Merge to `main` | tests, then `:latest` image promoted |
+| Merge to `main` | tests, then GHCR test image `:latest` promoted |
 | Manually dispatch `deploy-dev.yml` from `main` | `<version>rcN` to CodeArtifact, then DEV deploy in Lambda-Recipes |
 | Push a matching `v*` tag from `main` | `:<version>` image, then CodeArtifact, then PyPI |
 
@@ -280,7 +284,8 @@ to define.
   - Pytest on Ubuntu + Windows across Python 3.11 + 3.13 for `main` PRs
   - Test pip installation
   - Generate and test JSON schema
-  - Build the Docker image, pushed on merges to `main` under the new policy
+  - Build the CI test image and, on PRs, run smoke checks and local recipes against that exact image
+  - Push the image on merges to `main` under the new policy
   - Run container tests, then promote the mutable tag
 - **deploy-dev.yml** (*Deploy Dev*)**:** manually dispatch from `main`. The
   workflow still accepts `dev` temporarily; do not use that path for new work.
@@ -293,14 +298,15 @@ to define.
 1. **pytest:** Run test suite across OS/Python matrix
 2. **test-pip-install:** Verify package installs correctly
 3. **test-generate-schema:** Generate JSON schema from code
-4. **build:** Create Docker image and push to GitHub Container Registry
-5. **test-container:** Validate Docker image with full test suite
+4. **build:** Create the CI test image and push it to GitHub Container Registry
+5. **test-container:** Validate the CI test image with the full test suite
 6. **promote-image:** Retag the tested image; `dev` support is transitional and
    `latest` handling will be hardened under issue #1117
 
-Mutable tags are only moved after `test-container` passes, and package
-publication is gated on the container, so the wheel and the image cannot
-diverge. See `docs/release-lifecycle.md` for the current release direction.
+Mutable GHCR test-image tags are only moved after `test-container` passes, and
+package publication is gated on that validation. Production Lambda image
+publication and deployment remain the responsibility of `Lambda-Recipes`. See
+`docs/release-lifecycle.md` for the current release direction.
 
 ## Known Issues & Workarounds
 
@@ -322,9 +328,11 @@ brew install freetds
 ### Pandas Performance Warnings
 Performance warnings from pandas are suppressed in `recipe.py` as they appear during recipe execution without actual performance impact. This is a known issue being monitored.
 
-### Docker Image Size Optimization
+### CI Test Image Size Optimization
 - Botocore data reduced to S3-only (removes ~300MB)
 - Pandas test data removed from final image
+- Test-image data stack constrained to pandas 2.3.3 and NumPy 2.4.6 while the reusable package continues to allow pandas 2.x
+- All test-image dependencies install from binary wheels; compiler tools are absent from the runtime image
 - Uses slim Debian base image for minimal footprint
 
 ## Common Commands
@@ -342,8 +350,8 @@ pytest -v tests/test_wrangles.py
 # Generate schema
 cd schema && python generate_recipe_schema.py
 
-# Build Docker image
-docker build -t wrangles:latest .
+# Build CI test image
+docker build -t wrangles:test .
 
 # Run recipe locally
 wrangles.recipe tests/samples/recipe-basic.wrgl.yml

.github/workflows/ci.yml

@@ -61,9 +61,9 @@ jobs:
             fi
           fi
 
-          # Pushes land on an immutable per-commit tag. The mutable release tag
+          # Pushes land on an immutable per-commit tag. The mutable test-image tag
           # (dev / latest) is only moved onto that image once test-container
-          # passes, so consumers never resolve it to an untested build.
+          # passes, so validation jobs never resolve it to an untested build.
           if [[ "$PUSH_IMAGE" == "true" ]]; then
             BUILD_TAG="sha-${GITHUB_SHA}"
           else
@@ -188,17 +188,35 @@ jobs:
         with:
           images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
 
-      # Build and push Docker image with Buildx (build only, never push, on PRs)
+      # Build and push the CI test image with Buildx (build only, never push, on PRs)
       # https://github.com/docker/build-push-action
-      - name: Build and push Docker image
+      - name: Build and push CI test image
         id: build
         uses: docker/build-push-action@v7
         with:
           context: .
+          load: ${{ needs.config.outputs.push_image != 'true' }}
           push: ${{ needs.config.outputs.push_image == 'true' }}
           tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ needs.config.outputs.build_tag }}
           labels: ${{ steps.meta.outputs.labels }}
 
+      - name: Validate pull request test image
+        if: needs.config.outputs.push_image != 'true'
+        env:
+          IMAGE: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ needs.config.outputs.build_tag }}
+        run: |
+          docker run --rm \
+            --volume "${GITHUB_WORKSPACE}:/workspace:ro" \
+            "$IMAGE" \
+            python /workspace/scripts/container_smoke.py
+          docker run --rm "$IMAGE" python -m pip check
+          docker run --rm "$IMAGE" python -m pip freeze
+          docker run --rm \
+            --volume "${GITHUB_WORKSPACE}:/workspace:ro" \
+            --workdir /workspace/tests/samples \
+            "$IMAGE" \
+            sh -c 'wrangles.recipe recipe-basic.wrgl.yml && wrangles.recipe recipe_custom_function.wrgl.yml -f custom_functions.py'
+
   test-container:
     runs-on: ubuntu-latest
     if: needs.config.outputs.push_image == 'true'
@@ -221,6 +239,12 @@ jobs:
       - name: Remove wrangles folder
         run: rm -r wrangles
 
+      - name: Validate CI test runtime
+        run: |
+          python scripts/container_smoke.py
+          python -m pip check
+          python -m pip freeze
+
       - name: Install Test Dependencies
         run: |
           python -m pip install --upgrade pip
@@ -238,11 +262,11 @@ jobs:
           && wrangles.recipe recipe-basic.wrgl.yml
           && wrangles.recipe recipe_custom_function.wrgl.yml -f custom_functions.py
 
-  # Moves :dev / :latest onto the image that test-container just validated.
+  # Moves :dev / :latest onto the CI test image that test-container validated.
   # Retagging by digest guarantees the promoted image is the tested one and not
   # a rebuild, so no consumer of a mutable tag ever sees an untested image.
   promote-image:
-    name: Promote Image
+    name: Promote CI Test Image
     runs-on: ubuntu-latest
     if: needs.config.outputs.push_image == 'true'
     needs: [config, build, test-container]

AGENTS.md

@@ -19,6 +19,16 @@
   After every legacy change is recovered, superseded, or intentionally
   abandoned, archive or delete the legacy branch.
 
+## Container ownership
+
+- The repository-root `dockerfile` builds the WranglesPY CI test image published
+  to `ghcr.io/wrangleworks/wrangles`. Do not describe it as the production
+  runtime image.
+- The deployed `execute-recipe` AWS Lambda image is built and deployed from the
+  `wrangleworks/Lambda-Recipes` repository. Its `dockerfile` controls the
+  production Python version; verify that repository before making production
+  runtime claims.
+
 ## Code Review Rules
 
 ### Make the required action explicit

README.md

@@ -36,6 +36,20 @@ the intentionally live database, AWS, WrangleWorks, AI, and search-provider
 checks. The complete credentialed suite remains a CI validation and a local
 dependency/import pass does not claim live-service validation.
 
+## Container and deployment ownership
+
+The repository-root `dockerfile` builds the WranglesPY **CI test image** published
+to `ghcr.io/wrangleworks/wrangles`. It validates the installable package and
+recipe behavior in a slim Linux environment; it is not the deployed recipe
+runtime.
+
+The production `execute-recipe` AWS Lambda image is built, published to ECR, and
+deployed from the
+[`wrangleworks/Lambda-Recipes`](https://github.com/wrangleworks/Lambda-Recipes)
+repository. Its `dockerfile` is the source of truth for the production Python
+version. Changing the Python version in this repository's test image does not
+migrate the production Lambda runtime.
+
 ## What are Wrangles?
 
 Wrangles are a set of modular transformations for data cleaning and enrichment. Each Wrangle is optimized for a particular job, many of which are backed by sophisticated machine learning models.

constraints/test-container-python313.txt

@@ -0,0 +1,5 @@
+# Reproducible data-stack pins for the Python 3.13 CI test image.
+# These exact test-runtime pins do not replace the reusable package
+# compatibility ranges declared in requirements.txt.
+numpy==2.4.6
+pandas==2.3.3

dockerfile

@@ -1,36 +1,60 @@
-FROM python:3.11-slim-bookworm AS compile-image
+# syntax=docker/dockerfile:1
+
+FROM python:3.13-slim-bookworm AS dependency-image
 
 # Copy package
 COPY . /pkg
 
-# Install compile requirements
-RUN apt-get update \
-    && apt-get install -y build-essential gcc \
-    gfortran python3-dev \
-    --no-install-recommends
-
 # Create a virtual env
 RUN python -m venv /opt/venv
 ENV PATH="/opt/venv/bin:$PATH"
 
-# Install package + dependencies
-RUN pip install --no-cache-dir wheel
-# Special install for numpy to reduce size
-RUN CFLAGS="-g0 -Wl,--strip-all" pip install --no-cache-dir --compile --global-option=build_ext numpy==1.24.3
-# Regular install (without cache) for everything else
-RUN pip install --no-cache-dir /pkg
+# Install the package with the CI test-image data-stack constraint. Every
+# dependency must resolve to a wheel because the image has no compiler.
+RUN python -m pip install \
+    --no-cache-dir \
+    --only-binary=:all: \
+    --constraint /pkg/constraints/test-container-python313.txt \
+    /pkg packaging
+
+# Retain only the Botocore data used by the S3 connector and remove Pandas test
+# data. Resolve installed-package locations instead of embedding a Python minor.
+RUN python - <<'PY'
+from pathlib import Path
+import shutil
+
+import botocore
+import pandas
+
+botocore_data = Path(botocore.__file__).resolve().parent / "data"
+keep = {
+    "s3",
+    "_retry.json",
+    "endpoints.json",
+    "partitions.json",
+    "sdk-default-configuration.json",
+}
+for path in botocore_data.iterdir():
+    if path.name in keep:
+        continue
+    if path.is_dir():
+        shutil.rmtree(path)
+    else:
+        path.unlink()
 
-# Botocore contains lots of definitions for all AWS services. We are only using S3. Remove all other files to save space
-RUN cd /opt/venv/lib/python3.11/site-packages/botocore/data && cp -r s3 _retry.json endpoints.json partitions.json sdk-default-configuration.json /tmp/
-RUN rm -r /opt/venv/lib/python3.11/site-packages/botocore/data/*
-RUN cp -r /tmp/s3 /tmp/_retry.json /tmp/endpoints.json /tmp/partitions.json /tmp/sdk-default-configuration.json /opt/venv/lib/python3.11/site-packages/botocore/data
+pandas_tests = Path(pandas.__file__).resolve().parent / "tests"
+if pandas_tests.exists():
+    shutil.rmtree(pandas_tests)
+PY
 
-# Pandas contains a lot of unnecessary test data that we won't use
-RUN rm -r /opt/venv/lib/python3.11/site-packages/pandas/tests/*
+# Fail the build before the runtime stage if versions, metadata, package-data
+# trimming, imports, or the credential-free data/S3 checks are incorrect.
+RUN python -m pip check \
+    && python /pkg/scripts/container_smoke.py
 
 # Create build image
-FROM python:3.11-slim-bookworm AS build-image
-COPY --from=compile-image /opt/venv /opt/venv
+FROM python:3.13-slim-bookworm AS build-image
+COPY --from=dependency-image /opt/venv /opt/venv
 
 LABEL maintainer="WrangleWorks"
 ENV PATH="/opt/venv/bin:$PATH"
@@ -39,4 +63,4 @@ RUN mkdir /app
 COPY main.py /app/
 WORKDIR /app/
 
-CMD python main.py
+CMD python main.py

pytest-local.ini

@@ -2,6 +2,7 @@
 testpaths =
     tests/test_ai_cache.py
     tests/test_ai_definition.py
+    tests/test_container_smoke.py
     tests/test_data.py
     tests/test_dataframe.py
     tests/test_openai_extract_ai.py