Skip to content

Commit 22e1e3f

Browse files
authored
feat(extras): reject an oversized MicroVM input (#833)
microvm and vm.invoke now reject a job input whose JSON is over MAX_JOB_INPUT_BYTES (6 MiB minus 8 KiB) with ValueError, before any durable operation starts. The check measures the input with dumps, the encoder of the job body. The fixed fields of the body take less than 3 KiB, so an accepted input always fits in the 6 MiB body. The worker no longer handles Expect: 100-continue or reads and drops an oversized body. It answers 400 without reading the body. Every SDK now rejects an oversized input before it sends it: the JS SDK does so since aws/aws-durable-execution-sdk-js#1088.
1 parent 38c1333 commit 22e1e3f

12 files changed

Lines changed: 314 additions & 304 deletions

File tree

‎packages/aws-durable-execution-sdk-python-extras/README.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -123,7 +123,7 @@ except MicrovmError:
123123
... # Any other MicroVM failure.
124124
```
125125

126-
The class is the same on the first run and on every replay, so code that branches on it stays deterministic. A `microvm` call inside your own child context, or inside a `map` or `parallel` item, reaches the code around that context as the SDK's `ChildContextError`. Its `__cause__` is the MicroVM error, so check `isinstance(error.__cause__, MicrovmTimeoutError)` there. An invalid name or config raises the SDK's `ValidationError`, and an input that is not JSON-serializable raises `TypeError`, before any durable operation starts. `Duration` raises the same `ValidationError` for a negative value. The operation does not check `image_identifier` and `execution_role_arn` before it starts. boto3 checks them against the API model before it sends `RunMicrovm`, and the service checks them again. So an invalid value fails the launch step at once with `MicrovmLaunchError`, after the callback step.
126+
The class is the same on the first run and on every replay, so code that branches on it stays deterministic. A `microvm` call inside your own child context, or inside a `map` or `parallel` item, reaches the code around that context as the SDK's `ChildContextError`. Its `__cause__` is the MicroVM error, so check `isinstance(error.__cause__, MicrovmTimeoutError)` there. The operation raises three errors before any durable operation starts. An invalid name or config raises the SDK's `ValidationError`. An input that is not JSON-serializable raises `TypeError`. An input over `MAX_JOB_INPUT_BYTES` raises `ValueError`. `Duration` raises the same `ValidationError` for a negative value. The operation does not check `image_identifier` and `execution_role_arn` before it starts. boto3 checks them against the API model before it sends `RunMicrovm`, and the service checks them again. So an invalid value fails the launch step at once with `MicrovmLaunchError`, after the callback step.
127127

128128
A failed AWS call keeps its request ID in the error message, in the form `<Operation> request ID: <id>.`, for example `RunMicrovm request ID: <id>.`. A support case can quote it. This applies to `RunMicrovm`, `GetMicrovm`, `ResumeMicrovm`, `CreateMicrovmAuthToken`, and `TerminateMicrovm`. The SDK checkpoints the message of a failed step, and not its `__cause__`. So the request ID is in the message, the handler sees it on the first run and on replay, and the execution history keeps it. A failed `TerminateMicrovm` call does not fail the operation. So a `TerminateMicrovm` request ID appears only in the terminate log entry and in the terminate step record. A message has no request ID when the answer carries none, for example after a connection failure. A message also has no request ID when the error is not the answer of the Lambda MicroVMs call. For example, a `client` with role-assuming credentials calls STS `AssumeRole` inside each call, and an STS error carries the request ID of STS. The operation recognizes such an error by its `operation_name`. The step raises a copy of the botocore error, of the same class, with the same `response`, and with the request ID in its message. So `default_microvm_retry_strategy` decides as before. A custom `retry_strategy` receives the error that botocore raised, without the request ID. So a strategy that matches the message, such as `create_retry_strategy(RetryStrategyConfig(retryable_errors=[...]))`, decides on the message of the service. A `ResourceNotFoundException` from `GetMicrovm`, `ResumeMicrovm`, or `CreateMicrovmAuthToken` is the exception. It means that the MicroVM no longer exists. So the operation raises `MicrovmStateError` instead, and the strategy receives that error. Its `__cause__` is the botocore error. An error that the operation creates itself, such as `MicrovmStateError` or "RunMicrovm returned no microvmId", has the request ID in its message when the strategy receives it. A custom client can raise one error object for several calls. Each step message then names only its own call, and the object keeps its message.
129129

@@ -146,6 +146,7 @@ The 10-second reserve keeps time for the step to record its failure. A Lambda ti
146146

147147
- `timeout` must be between 1 second and 8 hours, because a MicroVM lives at most 8 hours. The MicroVM's `maximumDurationInSeconds` is `timeout` plus 5 minutes, at most 8 hours. So a timeout above 7 hours 55 minutes leaves the MicroVM less than 5 minutes beyond the callback's timeout.
148148
- `job_input` is sent as JSON. So the MicroVM receives what `json.dumps` writes: a tuple arrives as a list, and a dict key that is not a string arrives as a string.
149+
- The worker accepts a job request body of at most 6 MiB. The body holds the JSON of `job_input` and a few job fields. The job fields take less than 3 KiB. So `job_input` can be at most `MAX_JOB_INPUT_BYTES`, 6 MiB minus 8 KiB, as UTF-8 JSON. A larger input raises `ValueError` from `microvm` or `vm.invoke` before any durable operation starts.
149150
- The MicroVM gets the `INTERNET_EGRESS` connector, because it needs egress to reach the Lambda callback APIs. For ingress, it gets `NO_INGRESS` with `run` hook delivery and `ALL_INGRESS` with HTTP delivery. If you set `ingress_network_connectors`, allow ingress for every job that goes over HTTP. A `microvm` job goes over HTTP when its `run` hook payload is over 4,096 code points, or when its config sets `request.path`. Every session job goes over HTTP. An empty tuple sends no connector.
150151
- `heartbeat_timeout` starts when the callback is created, before the launch. The worker sends its first heartbeat when the job arrives. So the first period must cover the launch, the boot, and, for HTTP delivery, the request step, each with its retries. A throttled launch can take several minutes, and the request step retries for up to `retry_window` in each attempt. With a shorter heartbeat timeout, the operation fails with `MicrovmTimeoutError` before the MicroVM sends its first heartbeat.
151152
- The request step needs at least 11 seconds left in the invocation when it starts. It keeps a 10-second reserve before the invocation times out, and it starts a call only with at least 1 second left before the reserve. The launch step can run earlier in the same invocation, so the function timeout must cover it too. A function timeout of `retry_window` plus 11 seconds, 71 seconds by default, leaves the first retry tier about its whole window. With less than 11 seconds left, every attempt fails with `MicrovmEndpointUnavailableError`, and the message of the final `MicrovmDeliveryError` names the function timeout. The Lambda default timeout is 3 seconds. With it, `microvm` runs small jobs in the `run` hook and fails every job that goes over HTTP: a job too large for the hook, and a job with `request.path`.
@@ -240,7 +241,7 @@ A failed `vm.invoke` raises its MicroVM error inside the handler. The handler ca
240241
1. A MicroVM error that the handler raises itself reaches the caller with its own class, such as `MicrovmJobFailedError`.
241242
2. Any other error reaches the caller as the SDK's `ChildContextError`, as from `run_in_child_context`. This includes a MicroVM error that leaves a child context that the handler created: that child context records `ChildContextError` as its error type. The caller then gets the MicroVM error's message, but not its class. Inside the handler, the `ChildContextError` of that child context has the MicroVM error as its `__cause__`. An SDK error that leaves the handler, such as the `StepError` of a failed step, also reaches the caller as `ChildContextError`, with the SDK error as its `__cause__`. This rule matches `run_in_child_context`. The JavaScript operation differs here: it passes an SDK error to the caller with its own class.
242243

243-
An invalid job raises `ValidationError` or `TypeError` from `vm.invoke`, before the job's durable operations start. The handler can catch it too. A call without the context, `vm.invoke("build", job_input=..., options=...)`, raises Python's `TypeError`. A first argument that is not a durable context, as in `vm.invoke("build", "build", ...)`, raises `AttributeError`.
244+
An invalid job raises `ValidationError`, `TypeError`, or `ValueError` from `vm.invoke`, before the job's durable operations start. The handler can catch it too. A call without the context, `vm.invoke("build", job_input=..., options=...)`, raises Python's `TypeError`. A first argument that is not a durable context, as in `vm.invoke("build", "build", ...)`, raises `AttributeError`.
244245

245246
The handler's return value is checkpointed as the session's result. So it must be serializable by the SDK, and it counts toward the checkpoint size limit.
246247

@@ -299,9 +300,9 @@ The MicroVM operations handle the network, the service, and the worker lifecycle
299300
### Job inputs
300301

301302
1. Pass a JSON-serializable input with finite numbers. Any other input raises `TypeError` before any durable operation starts. See [Errors](#errors).
302-
2. Keep the serialized job request at or under 6 MiB. The job request holds the input and a few job fields. Pass larger data by reference, for example as an S3 key.
303-
- The Python worker reads a job request body of at most 6 MiB. It answers a larger body with HTTP 400.
304-
- The operation does not check the size before it sends the job. So a larger body reaches the worker, and the job fails with `MicrovmDeliveryError`. It fails at once when the 400 reaches the operation. It fails only after the retries of both tiers when the connection is reset instead, and the MicroVM is billed meanwhile. [What the worker does](../aws-durable-execution-sdk-python-microvm-worker/README.md#what-the-worker-does) says when each case happens.
303+
2. Keep the job input's JSON at or under `MAX_JOB_INPUT_BYTES`, 6 MiB minus 8 KiB, in UTF-8. Pass larger data by reference, for example as an S3 key.
304+
- The Python worker reads a job request body of at most 6 MiB. The body holds the input's JSON and a few job fields.
305+
- `microvm` and `vm.invoke` measure the input's JSON before any durable operation starts. A larger input raises `ValueError`. So `microvm` launches no MicroVM for it.
305306
3. Expect a job to go over HTTP in three cases:
306307
- The `run` hook payload of a `microvm` job is over 4,096 code points. The payload holds the input, the callback ID, and a few other fields.
307308
- The `microvm` config sets `request.path`. The job then goes over HTTP for any input size.

‎packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/microvm/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
MicrovmTimeoutError,
2626
)
2727
from aws_durable_execution_sdk_python_extras.microvm.job import (
28+
MAX_JOB_INPUT_BYTES,
2829
MAX_RUN_HOOK_PAYLOAD_LENGTH,
2930
)
3031
from aws_durable_execution_sdk_python_extras.microvm.operation import microvm
@@ -49,6 +50,7 @@
4950
"DEFAULT_AUTO_SUSPEND_IDLE_SECONDS",
5051
"DEFAULT_MICROVM_JOB_PATH",
5152
"DEFAULT_MICROVM_PORT",
53+
"MAX_JOB_INPUT_BYTES",
5254
"MAX_MICROVM_DURATION_SECONDS",
5355
"MAX_RUN_HOOK_PAYLOAD_LENGTH",
5456
"MicrovmBaseConfig",

‎packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/microvm/job.py‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,27 @@
2929
counts what the service counts.
3030
"""
3131

32+
MAX_JOB_INPUT_BYTES = 6 * 1024 * 1024 - 8 * 1024
33+
"""The largest job input, in UTF-8 bytes of its JSON, that ``microvm`` and
34+
``vm.invoke`` accept.
35+
36+
1. The worker accepts a job request body of at most 6 MiB.
37+
2. The body holds the input as the same JSON that :func:`dumps` writes.
38+
3. The body also holds fixed fields: the version, the Region, the MicroVM
39+
ID, the callback ID, and the heartbeat timeout. Their names and the JSON
40+
punctuation take 92 bytes.
41+
4. The botocore ``lambda-microvms`` model allows a MicroVM ID of 1 to 256
42+
characters, with no pattern. JSON writes a character in at most 6 bytes,
43+
as a ``\\u00XX`` escape. So the MicroVM ID takes at most 1,536 bytes.
44+
5. The Lambda model allows a callback ID of at most 1,024 characters of
45+
``[A-Za-z0-9+/]+={0,2}``. JSON writes each of these characters in 1 byte.
46+
So the callback ID takes at most 1,024 bytes.
47+
6. The Region is short ASCII. The heartbeat timeout is a few digits.
48+
7. So the fixed fields take less than 3 KiB. So 8 KiB covers them.
49+
50+
So an input whose JSON has at most 6 MiB minus 8 KiB fits in the body.
51+
"""
52+
3253

3354
@dataclass(frozen=True, kw_only=True)
3455
class RunHookPayload:

‎packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/microvm/operation.py‎

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@
66
from __future__ import annotations
77

88
import hashlib
9-
import json
109
import os
1110
import re
1211
import threading
@@ -57,8 +56,10 @@
5756
MicrovmTimeoutError,
5857
)
5958
from aws_durable_execution_sdk_python_extras.microvm.job import (
59+
MAX_JOB_INPUT_BYTES,
6060
JobRequest,
6161
RunHookPayload,
62+
dumps,
6263
fits_run_hook,
6364
run_hook_payload_without_job,
6465
)
@@ -258,6 +259,9 @@ def microvm(
258259
value.
259260
TypeError: When ``job_input`` is not JSON-serializable. No durable
260261
operation starts.
262+
ValueError: When the JSON of ``job_input`` is over
263+
:data:`MAX_JOB_INPUT_BYTES` UTF-8 bytes. No durable operation
264+
starts.
261265
MicrovmLaunchError: When ``RunMicrovm`` fails after all retries, or
262266
rejects the config, for example an empty ``image_identifier``.
263267
MicrovmNotRunningError: When ``CreateMicrovmAuthToken`` reports that
@@ -356,16 +360,29 @@ def _validate_heartbeat(name: str, heartbeat_timeout: Duration | None) -> int |
356360

357361

358362
def _validate_input(name: str, job_input: Any) -> None:
359-
"""Check that the job input is JSON-serializable.
363+
"""Check that the job input is JSON-serializable, and fits in a job body.
364+
365+
The size is the UTF-8 length of the JSON that the job body holds. So it
366+
is measured on the output of :func:`dumps`.
360367
361368
Raises:
362-
TypeError: When it is not.
369+
TypeError: When it is not JSON-serializable.
370+
ValueError: When its JSON is over :data:`MAX_JOB_INPUT_BYTES` bytes.
363371
"""
364372
try:
365-
json.dumps(job_input, allow_nan=False)
373+
text = dumps(job_input)
366374
except (TypeError, ValueError) as error:
367375
msg = f'MicroVM "{name}": the job input must be JSON-serializable: {error}'
368376
raise TypeError(msg) from error
377+
size = len(text.encode())
378+
if size > MAX_JOB_INPUT_BYTES:
379+
msg = (
380+
f'MicroVM "{name}": the job input is {size} bytes as JSON. The limit '
381+
f"is {MAX_JOB_INPUT_BYTES} bytes, because the worker accepts a job "
382+
"body of at most 6 MiB. Pass larger data by reference, for example "
383+
"as an S3 key."
384+
)
385+
raise ValueError(msg)
369386

370387

371388
def _validate_region(context: DurableContext, name: str) -> tuple[str, str]:
@@ -720,6 +737,7 @@ def create(
720737
ValidationError: When ``name`` or ``config`` is not valid, or the
721738
Region cannot be determined.
722739
TypeError: When ``job_input`` is not JSON-serializable.
740+
ValueError: When the JSON of ``job_input`` is too large.
723741
"""
724742
timeout_seconds = _validate_base(name, config.timeout)
725743
heartbeat_timeout_seconds = _validate_heartbeat(name, config.heartbeat_timeout)

‎packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/microvm/session.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -250,6 +250,8 @@ def invoke(
250250
AttributeError: When ``context`` is not a durable context, for
251251
example a job name passed as the first argument. Python
252252
raises it at ``context.run_in_child_context``.
253+
ValueError: When the JSON of ``job_input`` is over
254+
``MAX_JOB_INPUT_BYTES`` UTF-8 bytes.
253255
ValidationError: When ``name`` or ``options`` is not valid.
254256
MicrovmNotRunningError: When the session's MicroVM has ended. The
255257
session reports it when it checks the MicroVM state before

‎packages/aws-durable-execution-sdk-python-extras/tests/microvm/operation_test.py‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@
3636
from botocore.stub import ANY, Stubber
3737

3838
from aws_durable_execution_sdk_python_extras.microvm import (
39+
MAX_JOB_INPUT_BYTES,
3940
MAX_RUN_HOOK_PAYLOAD_LENGTH,
4041
MicrovmConfig,
4142
MicrovmError,
@@ -366,6 +367,33 @@ def run_microvm(self, **request: Any) -> dict[str, Any]:
366367
assert "RunMicrovm returned no microvmId" in error["message"]
367368

368369

370+
def test_rejects_a_job_input_over_the_size_limit_before_the_launch(
371+
run: Any, fake_client: Any
372+
) -> None:
373+
client = fake_client()
374+
375+
@durable_execution
376+
def handler(event: Any, context: DurableContext) -> Any:
377+
# The JSON of the input is the limit plus 1 byte, with the 2 quotes.
378+
job_input = "x" * (MAX_JOB_INPUT_BYTES - 1)
379+
try:
380+
microvm(context, "build", job_input=job_input, config=config_with(client))
381+
except ValueError as error:
382+
return str(error)
383+
return None
384+
385+
result = run(handler, {})
386+
387+
assert outcome(result) == (
388+
f'MicroVM "build": the job input is {MAX_JOB_INPUT_BYTES + 1} bytes as '
389+
f"JSON. The limit is {MAX_JOB_INPUT_BYTES} bytes, because the worker "
390+
"accepts a job body of at most 6 MiB. Pass larger data by reference, "
391+
"for example as an S3 key."
392+
)
393+
assert client.calls == []
394+
assert result.operations == []
395+
396+
369397
def test_sends_a_job_too_large_for_the_run_hook_over_http(
370398
run: Any, fake_client: Any, fake_endpoint: Any
371399
) -> None:

‎packages/aws-durable-execution-sdk-python-extras/tests/microvm/session_test.py‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@
3838

3939
from aws_durable_execution_sdk_python_extras.microvm import (
4040
DEFAULT_MICROVM_JOB_PATH,
41+
MAX_JOB_INPUT_BYTES,
4142
MicrovmError,
4243
MicrovmInvokeOptions,
4344
MicrovmJobFailedError,
@@ -405,6 +406,68 @@ def body(vm: MicrovmSession, ctx: DurableContext, event: Any) -> Any:
405406
]
406407

407408

409+
def test_rejects_a_job_input_over_the_size_limit_before_the_job_starts(
410+
run: Any, fake_client: Any, fake_endpoint: Any
411+
) -> None:
412+
endpoint = fake_endpoint()
413+
client = fake_client(endpoint=endpoint.url)
414+
415+
def body(vm: MicrovmSession, ctx: DurableContext, event: Any) -> Any:
416+
# The JSON of the input is the limit plus 1 byte, with the 2 quotes.
417+
job_input = "x" * (MAX_JOB_INPUT_BYTES - 1)
418+
try:
419+
vm.invoke(ctx, "build", job_input=job_input, options=job())
420+
except ValueError as error:
421+
return str(error)
422+
return None
423+
424+
result = run(session_handler(session_config(client), body))
425+
426+
assert outcome(result) == {
427+
"result": (
428+
f'MicroVM "build": the job input is {MAX_JOB_INPUT_BYTES + 1} bytes '
429+
f"as JSON. The limit is {MAX_JOB_INPUT_BYTES} bytes, because the "
430+
"worker accepts a job body of at most 6 MiB. Pass larger data by "
431+
"reference, for example as an S3 key."
432+
)
433+
}
434+
# Only the session's own launch ran. No job operation started, and no
435+
# request reached the MicroVM.
436+
assert len(client.requests("run_microvm")) == 1
437+
assert client.requests("create_microvm_auth_token") == []
438+
assert endpoint.requests == []
439+
session = result.get_context("pipeline")
440+
assert [operation.name for operation in session.child_operations] == [
441+
"pipeline.session",
442+
"pipeline.launch",
443+
"pipeline.terminate",
444+
]
445+
446+
447+
def test_sends_a_job_input_at_the_size_limit(
448+
run: Any, fake_client: Any, fake_endpoint: Any
449+
) -> None:
450+
endpoint = fake_endpoint()
451+
client = fake_client(endpoint=endpoint.url)
452+
# The JSON of the input is the limit, with the 2 quotes. Each "é" takes
453+
# 2 bytes.
454+
job_input = "é" * ((MAX_JOB_INPUT_BYTES - 2) // 2)
455+
456+
def body(vm: MicrovmSession, ctx: DurableContext, event: Any) -> Any:
457+
return vm.invoke(ctx, "build", job_input=job_input, options=job())
458+
459+
result = run(
460+
session_handler(session_config(client), body),
461+
None,
462+
[("build", succeed_with("done"))],
463+
)
464+
465+
assert outcome(result) == {"result": "done"}
466+
[posted] = endpoint.requests
467+
assert posted["body"]["input"] == job_input
468+
assert int(posted["headers"]["content-length"]) <= 6 * 1024 * 1024
469+
470+
408471
def test_retries_a_throttled_launch_with_the_same_client_token(
409472
run: Any, fake_client: Any, fake_endpoint: Any, make_error: Any
410473
) -> None:

0 commit comments

Comments
 (0)