This was first raised by @kikiscodedeliveryservice in Discussion #1514
Description
When the quarantine directory reaches MAX_QUARANTINE_SIZE_GB, PayloadExtractionJob.run() breaks out of the download loop immediately:
if self._quarantine_usage_bytes() >= max_size_bytes:
self.log.warning("Quarantine directory has reached the limit. Stopping downloads.")
breakNo metadata-only record is created for the remaining payloads in the batch. Because the /recent endpoint uses strictly sequential, non-overlapping time windows based on EXTRACTION_INTERVAL, each payload appears in exactly one time window. If that window is skipped due to the quota, the automated pipeline will never see those payloads again.
There is no backfill, no retry path, and no manual recovery mechanism.
Expected behavior
When the quota is reached, the job should still persist metadata-only stubs (payload_file=None) for all remaining payloads in the current batch.
Next step
A follow-on improvement (separate issue) would be a management command or periodic task that sweeps HoneypotPayload.objects.filter(payload_file__isnull=True, locator__gt="") and retries downloads automatically when quota is available.
This was first raised by @kikiscodedeliveryservice in Discussion #1514
Description
When the quarantine directory reaches
MAX_QUARANTINE_SIZE_GB,PayloadExtractionJob.run()breaks out of the download loop immediately:No metadata-only record is created for the remaining payloads in the batch. Because the
/recentendpoint uses strictly sequential, non-overlapping time windows based onEXTRACTION_INTERVAL, each payload appears in exactly one time window. If that window is skipped due to the quota, the automated pipeline will never see those payloads again.There is no backfill, no retry path, and no manual recovery mechanism.
Expected behavior
When the quota is reached, the job should still persist metadata-only stubs (
payload_file=None) for all remaining payloads in the current batch.Next step
A follow-on improvement (separate issue) would be a management command or periodic task that sweeps
HoneypotPayload.objects.filter(payload_file__isnull=True, locator__gt="")and retries downloads automatically when quota is available.