Repository navigation
Bug: Sam build extremely slow with image based lambdas #8504
Description
Bug: Sam build extremely slow with image based lambdas
Description:
Sam build extremely slow
Steps to reproduce:
Sam build --debug
Command hangs at
2025-12-13 12:33:18,659 | Set global container socket path: container_socket_path=
Observed result:
sam build DatasetFunction --debug
2025-12-13 12:24:32,513 | Config file location: [reducted]/samconfig.toml
2025-12-13 12:24:32,519 | Loading configuration values from [default.['build'].parameters] (env.command_name.section) in config file at '[reducted]/samconfig.toml'...
2025-12-13 12:24:32,521 | Configuration values successfully loaded.
2025-12-13 12:24:32,523 | Configuration values are: {'stack_name': 'tiledb-api', 'capabilities': ['CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND'], 'region': 'us-east-1', 'output_template_file': './.aws-sam/build/template.yaml', 'resolve_image_repos': True, 'parameter_overrides':
['DbName=appdata', 'JobQueue=ec2-job-queue', 'VPCEndpointId=[reducted]', 'ElasticHostUrl=[reducted]', 'MongoSecretName=[reducted]',
'CertificateArn=[reducted]', 'HostedZoneId=[reducted]', 'BaseBucketName=[reducted]', 'SecurityGroupIds=[reducted]', 'SubnetIds=[reducted]', 'Environment=prod', 'GitRev=s'], 'tags':
['owner=roya@cgen.com', 'map-migrated=[reducted]', 'env=prod'], 'use_container': True}
2025-12-13 12:24:32,608 | Using SAM Template at [reducted]/template.yaml
2025-12-13 12:24:32,691 | Using config file: samconfig.toml, config environment: default
2025-12-13 12:24:32,692 | Expand command line arguments to:
2025-12-13 12:24:32,693 | --template_file=[reducted]/template.yaml --resource_logical_id=DatasetFunction --use_container --mount_with=READ --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache --parameter_overrides={'DbName': 'appdata', 'JobQueue': 'ec2-job-queue',
'VPCEndpointId': '[reducted]', 'ElasticHostUrl': '[reducted]', 'MongoSecretName': '[reducted]', 'CertificateArn': '[reducted]', 'HostedZoneId':
'[reducted]', 'BaseBucketName': '[reducted]', 'SecurityGroupIds': '[reducted]', 'SubnetIds': '[reducted]', 'Environment': 'prod', 'GitRev': 's'}
2025-12-13 12:24:32,735 | 'build' command is called
2025-12-13 12:24:32,737 | Starting Build inside a container
2025-12-13 12:24:32,821 | Collected default values for parameters: {'GitRev': '', 'Environment': 'test'}
2025-12-13 12:24:32,894 | There is no customer defined id or cdk path defined for resource APIGWIndexFunction, so we will use the resource logical id as the resource id
2025-12-13 12:24:32,895 | There is no customer defined id or cdk path defined for resource DatasetFunction, so we will use the resource logical id as the resource id
2025-12-13 12:24:32,897 | There is no customer defined id or cdk path defined for resource MapGenesFunction, so we will use the resource logical id as the resource id
2025-12-13 12:24:32,898 | There is no customer defined id or cdk path defined for resource GetExpressionDataFunction, so we will use the resource logical id as the resource id
2025-12-13 12:24:32,899 | There is no customer defined id or cdk path defined for resource MsAuthorizer, so we will use the resource logical id as the resource id
2025-12-13 12:24:32,901 | There is no customer defined id or cdk path defined for resource MyRestApi, so we will use the resource logical id as the resource id
2025-12-13 12:24:32,902 | There is no customer defined id or cdk path defined for resource BatchECR, so we will use the resource logical id as the resource id
2025-12-13 12:24:32,903 | There is no customer defined id or cdk path defined for resource PostNewDatasetBatchJobRole, so we will use the resource logical id as the resource id
2025-12-13 12:24:32,905 | There is no customer defined id or cdk path defined for resource PostNewDatasetBatchJobDefinition, so we will use the resource logical id as the resource id
2025-12-13 12:24:32,915 | Unable to resolve property Image: {'Fn::Join': [':', [{'Fn::GetAtt': ['BatchECR', 'RepositoryUri']}, 'latest']]}. Leaving as is.
2025-12-13 12:24:32,917 | 0 stacks found in the template
2025-12-13 12:24:32,918 | Collected default values for parameters: {'GitRev': '', 'Environment': 'test'}
2025-12-13 12:24:32,986 | There is no customer defined id or cdk path defined for resource APIGWIndexFunction, so we will use the resource logical id as the resource id
2025-12-13 12:24:32,987 | There is no customer defined id or cdk path defined for resource DatasetFunction, so we will use the resource logical id as the resource id
2025-12-13 12:24:32,989 | There is no customer defined id or cdk path defined for resource MapGenesFunction, so we will use the resource logical id as the resource id
2025-12-13 12:24:32,990 | There is no customer defined id or cdk path defined for resource GetExpressionDataFunction, so we will use the resource logical id as the resource id
2025-12-13 12:24:32,991 | There is no customer defined id or cdk path defined for resource MsAuthorizer, so we will use the resource logical id as the resource id
2025-12-13 12:24:32,993 | There is no customer defined id or cdk path defined for resource MyRestApi, so we will use the resource logical id as the resource id
2025-12-13 12:24:32,994 | There is no customer defined id or cdk path defined for resource BatchECR, so we will use the resource logical id as the resource id
2025-12-13 12:24:32,995 | There is no customer defined id or cdk path defined for resource PostNewDatasetBatchJobRole, so we will use the resource logical id as the resource id
2025-12-13 12:24:32,997 | There is no customer defined id or cdk path defined for resource PostNewDatasetBatchJobDefinition, so we will use the resource logical id as the resource id
2025-12-13 12:24:33,005 | Unable to resolve property Image: {'Fn::Join': [':', [{'Fn::GetAtt': ['BatchECR', 'RepositoryUri']}, 'latest']]}. Leaving as is.
2025-12-13 12:24:33,007 | 9 resources found in the stack
2025-12-13 12:24:33,008 | Found Serverless function with name='APIGWIndexFunction' and CodeUri='./functions/APIGWIndexFunction'
2025-12-13 12:24:33,009 | --base-dir is not presented, adjusting uri ./functions/APIGWIndexFunction relative to [reducted]/template.yaml
2025-12-13 12:24:33,011 | Found Serverless function with name='DatasetFunction' and ImageUri='None'
2025-12-13 12:24:33,012 | --base-dir is not presented, adjusting uri ./functions/DatasetFunction relative to [reducted]/template.yaml
2025-12-13 12:24:33,013 | --base-dir is not presented, adjusting uri . relative to [reducted]/template.yaml
2025-12-13 12:24:33,014 | Found Serverless function with name='MapGenesFunction' and ImageUri='None'
2025-12-13 12:24:33,016 | --base-dir is not presented, adjusting uri ./functions/MapGenesFunction relative to [reducted]/template.yaml
2025-12-13 12:24:33,017 | --base-dir is not presented, adjusting uri . relative to [reducted]/template.yaml
2025-12-13 12:24:33,018 | Found Serverless function with name='GetExpressionDataFunction' and ImageUri='None'
2025-12-13 12:24:33,019 | --base-dir is not presented, adjusting uri ./functions/GetExpressionDataFunction relative to [reducted]/template.yaml
2025-12-13 12:24:33,021 | --base-dir is not presented, adjusting uri . relative to [reducted]/template.yaml
2025-12-13 12:24:33,022 | Found Serverless function with name='MsAuthorizer' and ImageUri='None'
2025-12-13 12:24:33,023 | --base-dir is not presented, adjusting uri ./functions/MsAuthorizer relative to [reducted]/template.yaml
2025-12-13 12:24:33,024 | --base-dir is not presented, adjusting uri . relative to [reducted]/template.yaml
2025-12-13 12:24:33,027 | ContainerClientFactory.create_client() called
2025-12-13 12:24:33,029 | Admin preference: None
2025-12-13 12:24:33,031 | Using auto-detected client creation
2025-12-13 12:24:33,032 | Trying Docker client creation
2025-12-13 12:24:33,033 | Creating Docker container client from environment variable.
2025-12-13 12:24:33,034 | Creating container client with parameters: {'version': '1.35'}
2025-12-13 12:24:33,036 | DockerContainerClient created successfully
2025-12-13 12:24:33,040 | Using Docker as Container Engine.
2025-12-13 12:24:33,042 | Set global container socket path: container_socket_path=
2025-12-13 12:24:33,123 | 9 resources found in the stack
2025-12-13 12:24:33,125 | Found Serverless function with name='APIGWIndexFunction' and CodeUri='./functions/APIGWIndexFunction'
2025-12-13 12:24:33,126 | Found Serverless function with name='DatasetFunction' and ImageUri='None'
2025-12-13 12:24:33,127 | Found Serverless function with name='MapGenesFunction' and ImageUri='None'
2025-12-13 12:24:33,129 | Found Serverless function with name='GetExpressionDataFunction' and ImageUri='None'
2025-12-13 12:24:33,130 | Found Serverless function with name='MsAuthorizer' and ImageUri='None'
2025-12-13 12:24:33,132 | Error occurred while trying to track an event: Event 'BuildFunctionRuntime' does not accept value 'None'.
2025-12-13 12:24:33,133 | Instantiating build definitions
2025-12-13 12:24:33,157 | Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(None, [reducted], Image, , 428387b4-63a6-4939-965b-9eea6724454c, {'Dockerfile': 'Dockerfile', 'DockerContext':
'[reducted]/functions/DatasetFunction'}, {}, x86_64, []), Function: Function({'function_id': 'DatasetFunction', 'name': 'DatasetFunction', 'functionname': 'DatasetFunction', 'runtime': None, 'memory': 2048, 'timeout': 60, 'handler': None, 'imageuri': None, 'packagetype':
'Image', 'imageconfig': None, 'codeuri': '[reducted]', 'environment': {'Variables': {'DbName': 'appdata', 'JobQueue': 'ec2-job-queue', 'JobDefinitionArn': 'PostNewDatasetBatchJobDefinition', 'ElasticHostUrl': '[reducted]',
'MongoSecretName': '[reducted]', 'POWERTOOLS_SERVICE_NAME': 'local', 'StackName': 'local', 'StripPrefixes': '[ "/api/v1/tiledb","/api/test/tiledb" ]', 'POWERTOOLS_LOG_LEVEL': 'INFO', 'Environment': 'prod', 'GitRev': 's'}}, 'rolearn': None, 'layers': [], 'events':
{'DatasetDeleteEvent': {'Type': 'Api', 'Properties': {'Auth': {'Authorizer': 'AWS_IAM', 'InvokeRole': 'NONE'}, 'Path': '/datasets/{datasetid}', 'Method': 'delete', 'RestApiId': 'MyRestApi'}}, 'PostDataSetEvent': {'Type': 'Api', 'Properties': {'Path': '/datasets', 'Method': 'post', 'RestApiId':
'MyRestApi'}}, 'GetDatasetidEvent': {'Type': 'Api', 'Properties': {'Path': '/datasets/{datasetid}', 'Method': 'get', 'RestApiId': 'MyRestApi', 'Auth': {'Authorizer': 'MyLambdaRequestAuth'}}}, 'GetVarData': {'Type': 'Api', 'Properties': {'Path': '/datasets/{datasetid}/var/data/{varid}', 'Method': 'get',
'RestApiId': 'MyRestApi', 'Auth': {'Authorizer': 'MyLambdaRequestAuth'}}}, 'ListDatasetsEvent': {'Type': 'Api', 'Properties': {'RequestParameters': [{'method.request.querystring.dataset_type': {'Required': False}}, {'method.request.querystring.expand': {'Required': False}},
{'method.request.querystring.annotations': {'Required': False}}, {'method.request.querystring.offset': {'Required': False}}, {'method.request.querystring.limit': {'Required': False}}, {'method.request.querystring.cnt': {'Required': False}}, {'method.request.querystring.organism': {'Required': False}},
{'method.request.querystring.clinical': {'Required': False}}], 'Path': '/datasets', 'Method': 'get', 'RestApiId': 'MyRestApi', 'Auth': {'Authorizer': 'MyLambdaRequestAuth'}}}, 'GetAnnotationsEvent': {'Type': 'Api', 'Properties': {'Path': '/datasets/{datasetid}/obs/annotations', 'Method': 'get', 'RestApiId':
'MyRestApi'}}, 'GetAnnotationValuesEvent': {'Type': 'Api', 'Properties': {'Path': '/datasets/{datasetid}/obs/annotations/{annotationid}', 'RequestParameters': [{'method.request.path.datasetid': {'Required': True, 'Caching': True}}, {'method.request.path.annotationid': {'Required': True, 'Caching': True}}],
'Method': 'get', 'RestApiId': 'MyRestApi'}}, 'GetVarEvent': {'Type': 'Api', 'Properties': {'Path': '/datasets/{datasetid}/var/ids', 'Method': 'get', 'RequestParameters': [{'method.request.path.datasetid': {'Required': True, 'Caching': True}}, {'method.request.querystring.expand': {'Caching': True}},
{'method.request.querystring.offset': {'Caching': True}}, {'method.request.querystring.limit': {'Caching': True}}, {'method.request.querystring.cnt': {'Caching': True}}], 'RestApiId': 'MyRestApi'}}, 'PatchObsSpec': {'Type': 'Api', 'Properties': {'Path': '/datasets/{datasetid}/obs/spec', 'Method': 'patch',
'RequestParameters': [{'method.request.path.datasetid': {'Required': True}}], 'RestApiId': 'MyRestApi'}}, 'GetObsSpec': {'Type': 'Api', 'Properties': {'Path': '/datasets/{datasetid}/obs/spec', 'Method': 'get', 'RequestParameters': [{'method.request.path.datasetid': {'Required': True}}], 'RestApiId':
'MyRestApi'}}}, 'metadata': {'Dockerfile': 'Dockerfile', 'DockerContext': '[reducted]/functions/DatasetFunction', 'SamResourceId': 'DatasetFunction'}, 'inlinecode': None, 'codesign_config_arn': None, 'architectures': None, 'function_url_config': None,
'function_build_info': <FunctionBuildInfo.BuildableImage: ('BuildableImage', 'Regular IMAGE function which can be build with SAM CLI')>, 'stack_path': '', 'runtime_management_config': None, 'logging_config': None, 'tenancy_config': None, 'durable_config': None, 'capacity_provider_config': None,
'publish_to_latest_published': None}))
2025-12-13 12:24:33,177 | Building codeuri: [reducted] runtime: None architecture: x86_64 functions: DatasetFunction
2025-12-13 12:24:33,178 | Building to following folder [reducted]/.aws-sam/build/DatasetFunction
2025-12-13 12:24:33,179 | Building image for DatasetFunction function
2025-12-13 12:24:33,181 | Setting DockerBuildArgs for DatasetFunction function
2025-12-13 12:24:33,182 | ContainerClientFactory.create_client() called
2025-12-13 12:24:33,183 | Admin preference: None
2025-12-13 12:24:33,184 | Using auto-detected client creation
2025-12-13 12:24:33,185 | Trying Docker client creation
2025-12-13 12:24:33,186 | Creating Docker container client from environment variable.
2025-12-13 12:24:33,188 | Creating container client with parameters: {'version': '1.35'}
2025-12-13 12:24:33,189 | DockerContainerClient created successfully
2025-12-13 12:24:33,193 | Using Docker as Container Engine.
2025-12-13 12:24:33,194 | Set global container socket path: container_socket_path=
2025-12-13 12:26:50,838 | <Image: 'datasetfunction:latest'> image is built for DatasetFunction function
Step 1/14 : FROM public.ecr.aws/lambda/python:3.13 AS builder
---> 442ab9e91c0e
Step 2/14 : RUN dnf update && dnf install -y tar gzip
---> Using cache
---> 0d68fe29e022
Step 3/14 : RUN curl -LsSf https://astral.sh/uv/install.sh | sh && /root/.local/bin/uv --version
---> Using cache
---> b9c0e0f815d1
Step 4/14 : COPY pyproject.toml .
---> Using cache
---> cc5464fbe76f
Step 5/14 : COPY uv.lock .
---> Using cache
---> 1bc13b2a638e
Step 6/14 : RUN /root/.local/bin/uv export --no-dev --no-editable -o requirements.txt
---> Using cache
---> 04a9eb33a66d
Step 7/14 : RUN pip install -r requirements.txt --target "${LAMBDA_TASK_ROOT}"
---> Using cache
---> 27bfcfaf8582
Step 8/14 : FROM public.ecr.aws/lambda/python:3.13
---> 442ab9e91c0e
Step 9/14 : COPY --from=builder ${LAMBDA_TASK_ROOT} ${LAMBDA_TASK_ROOT}/packages
---> Using cache
---> 038d2de46b4f
Step 10/14 : COPY . ${LAMBDA_TASK_ROOT}/app
---> 89a4f88a9442
Step 11/14 : ENV PYTHONPATH="${LAMBDA_TASK_ROOT}/packages:${PYTHONPATH}"
---> Running in 4cc780225d3a
---> Removed intermediate container 4cc780225d3a
---> e5ac14ae4fa0
Step 12/14 : RUN chmod 777 -R ${LAMBDA_TASK_ROOT}/app
---> Running in 02ffaa8bde27
---> Removed intermediate container 02ffaa8bde27
---> 6b65efb5d126
Step 13/14 : WORKDIR ${LAMBDA_TASK_ROOT}/app
---> Running in 912f05cc652f
---> Removed intermediate container 912f05cc652f
---> 2df85575dcf9
Step 14/14 : CMD ["app.handler"]
---> Running in 452a289a1ad9
---> Removed intermediate container 452a289a1ad9
---> 61742a6ade9f
Successfully built 61742a6ade9f
Successfully tagged datasetfunction:latest
2025-12-13 12:26:50,840 | There is no customer defined id or cdk path defined for resource APIGWIndexFunction, so we will use the resource logical id as the resource id
2025-12-13 12:26:50,842 | There is no customer defined id or cdk path defined for resource DatasetFunction, so we will use the resource logical id as the resource id
2025-12-13 12:26:50,844 | There is no customer defined id or cdk path defined for resource MapGenesFunction, so we will use the resource logical id as the resource id
2025-12-13 12:26:50,845 | There is no customer defined id or cdk path defined for resource GetExpressionDataFunction, so we will use the resource logical id as the resource id
2025-12-13 12:26:50,847 | There is no customer defined id or cdk path defined for resource MsAuthorizer, so we will use the resource logical id as the resource id
2025-12-13 12:26:50,848 | There is no customer defined id or cdk path defined for resource MyRestApi, so we will use the resource logical id as the resource id
2025-12-13 12:26:50,850 | There is no customer defined id or cdk path defined for resource BatchECR, so we will use the resource logical id as the resource id
2025-12-13 12:26:50,851 | There is no customer defined id or cdk path defined for resource PostNewDatasetBatchJobRole, so we will use the resource logical id as the resource id
2025-12-13 12:26:50,853 | There is no customer defined id or cdk path defined for resource PostNewDatasetBatchJobDefinition, so we will use the resource logical id as the resource id
2025-12-13 12:26:50,854 | 9 resources found in the stack
2025-12-13 12:26:50,855 | Found Serverless function with name='APIGWIndexFunction' and CodeUri='./functions/APIGWIndexFunction'
2025-12-13 12:26:50,857 | Found Serverless function with name='DatasetFunction' and ImageUri='None'
2025-12-13 12:26:50,858 | Found Serverless function with name='MapGenesFunction' and ImageUri='None'
2025-12-13 12:26:50,859 | Found Serverless function with name='GetExpressionDataFunction' and ImageUri='None'
2025-12-13 12:26:50,861 | Found Serverless function with name='MsAuthorizer' and ImageUri='None'
Build Succeeded
Built Artifacts : .aws-sam/build
Built Template : .aws-sam/build/template.yaml
Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided
2025-12-13 12:26:50,976 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2025-12-13 12:26:50,989 | No container socket path in global storage, falling back to DOCKER_HOST detection:
2025-12-13 12:26:50,991 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2025-12-13 12:26:50,992 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '7c82e05d-f6a7-4a24-8f20-5048dcaa7d9a', 'installationId': '15267a1b-9410-4824-b902-cf40d4dc0605', 'sessionId': 'db7d927c-4f00-4464-861f-8de0872e9d99', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.10',
'samcliVersion': '1.150.1', 'osPlatform': 'Linux', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': 'us-east-1', 'commandName': 'sam build', 'metricSpecificAttributes': {'projectType': 'CFN', 'gitOrigin': None, 'projectName':
'c1452eead9970c350eca9a2338aae504ec9179dcde1521ca3c58ba1d6e8484b8', 'initialCommit': None, 'containerEngine': 'docker-default', 'adminContainerPreference': None, 'functionFieldsUsed': {'codeuri': 5, 'environment': 5, 'events': 4}}, 'duration': 138284, 'exitReason': 'success', 'exitCode': 0}}]}
2025-12-13 12:26:50,992 | Unable to find Click Context for getting session_id.
2025-12-13 12:26:51,000 | No container socket path in global storage, falling back to DOCKER_HOST detection:
2025-12-13 12:26:51,001 | Sending Telemetry: {'metrics': [{'events': {'requestId': '89f12036-940c-42ec-838a-db232510acfd', 'installationId': '15267a1b-9410-4824-b902-cf40d4dc0605', 'sessionId': 'db7d927c-4f00-4464-861f-8de0872e9d99', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.10',
'samcliVersion': '1.150.1', 'osPlatform': 'Linux', 'commandName': 'sam build', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '45ce6ee1c72344d2b19d88b1fa0e97ec', 'time_stamp': '2025-12-13 12:24:32.513551+00', 'exception_name': None},
{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '7a01c80ffd104e59a5087c4555a440e7', 'time_stamp': '2025-12-13 12:24:32.690920+00', 'exception_name': None}], 'containerEngine': 'docker-default'}}}]}
2025-12-13 12:26:51,243 | Telemetry response: 200
2025-12-13 12:26:51,247 | Telemetry response: 200
Expected result:
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
- OS:
Linux ip-172-31-39-52 6.8.0-1043-aws #45~22.04.1-Ubuntu SMP Wed Nov 12 16:16:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux sam --version:SAM CLI, version 1.150.1- AWS region: us-east-1
{
"version": "1.150.1",
"system": {
"python": "3.11.10",
"os": "Linux-6.8.0-1043-aws-x86_64-with-glibc2.35"
},
"additional_dependencies": {
"container_engine": "Docker(v28.0.2)",
"aws_cdk": "2.1010.0 (build 6b421db)",
"terraform": "1.11.2"
},
"available_beta_feature_env_vars": [
"SAM_CLI_BETA_FEATURES",
"SAM_CLI_BETA_BUILD_PERFORMANCE",
"SAM_CLI_BETA_TERRAFORM_SUPPORT",
"SAM_CLI_BETA_PACKAGE_PERFORMANCE",
"SAM_CLI_BETA_RUST_CARGO_LAMBDA"
]
}
Add --debug flag to command you are running
Activity
github-actions commentedon Dec 13, 2025
Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.