Team/Alpha: goboxd Stage 1 - #7
Open
Eyuvaraj wants to merge 99 commits into
Open
Conversation
- Added Runner struct to manage job submissions with bounded concurrency. - Implemented job execution in a sandboxed environment using nsjail. - Introduced limits management for resource constraints during execution. - Created workspace management for isolated job execution directories. - Added validation for filenames, flags, and source sizes to ensure security. - Implemented status parsing for build and run outcomes. - Developed integration tests for various programming languages including Python, C, C++, Java, and more. - Added load testing scripts to evaluate performance under concurrent requests. - Included installation scripts for required languages (GCC, Java, Node.js, Python, Verilog).
- Introduced Swagger YAML file for API documentation, detailing endpoints, request/response structures, and examples. - Updated Go modules to include necessary dependencies for Swagger support. - Enhanced existing handler functions with Swagger annotations for better documentation. - Modified request and response types to include JSON tags and examples for improved clarity. - Added new types for health check responses and service statistics to align with the new API documentation.
Bug fixes: - sandbox: ParseBuildStatus matched any nsjail log line containing "nsjail" causing all compile failures to report internal_error; now matches [E][ prefix (nsjail error-level lines only) so compiler failures correctly report failed - runner: compareOutput checked sandbox errors only after output mismatch; sandbox-level failures (TLE/OOM/signal) now take precedence, and wrong-output is returned explicitly rather than falling through ParseRunStatus - registry: All() iterated map keys (non-deterministic order); now backed by an ordered slice populated at load time, giving stable YAML insertion order - handler: body limit excluded expected_stdout — unbounded client input; now accounts for MaxTests × 2 × MaxStdinBytes + framing overhead - server: WriteTimeout hardcoded 120 s; now derived via Registry.MaxJobDuration so long-running test suites are not killed mid-stream Security: - sandbox: remove --disable_clone_newnet so jail processes get their own network namespace and cannot reach the host network - validate: add ExpectedSize and Limits checks — clients could previously inject unlimited expected_stdout or escalate resource limits beyond defaults New features: - registry: ProbeCache — /readyz and /info now served from a 30 s TTL cache refreshed in the background; no child processes spawned per HTTP call - logctx: new package for passing per-request execution fields (language, exec_status, build_duration_ms, tests_total, tests_accepted) from handler to structured-logger middleware; logged on every /run request - registry: MaxJobDuration method for dynamic WriteTimeout computation Languages: - configs: add ruby, lua, rust, kotlin, ocaml to language registry - Dockerfile: install toolchains and smoke-test all bonus languages at build - scripts/lang_install: add per-language install helpers for each bonus runtime Tests: - integration: add TestTimeExceeded, TestRuntimeError, TestSourceTooLarge, TestExpectedTooLarge, TestInvalidLimits, TestReadyz, TestInfo - sandbox: add regression test for ParseBuildStatus with real nsjail info log Repo: - external/nsjail: pin nsjail 3.4 as git submodule (eliminates network fetch at image build time; Dockerfile now uses COPY instead of git clone) - README: rewrite — remove decorations, add chi rationale, Makefile-first flow
- Bump pinned nsjail submodule from 3.4 to 3.6 and update all version references across docs and Swagger examples - Add Kafel deny-list seccomp policy via --seccomp_string; blocks ptrace, kexec, io_uring, bpf, userfaultfd, and other sandbox-escape primitives while keeping all 12 language runtimes functional - Enable cgroup v2 resource enforcement (--detect_cgroupv2, --cgroup_mem_max, --cgroup_pids_max) alongside existing rlimit guards; fixes unreliable OOM detection on kernels without cgroup v1 memory controller - Add --rlimit_cpu as a secondary CPU time cap alongside --time_limit - Improve ParseRunStatus to detect cgroup v2 OOM kills (memory.max + SIGKILL pattern) and add corresponding test cases - Fix tools image to install golangci-lint v2 (module path changed to golangci-lint/v2/cmd/golangci-lint); resolves "config file for v2 with v1 binary" lint failure
golangci-lint v2 requires Go >= 1.25 (v2.12.2 errors with GOTOOLCHAIN=local on 1.23). The runtime image uses Debian so this only affects the build stage.
Use the current stable release (1.26.3) instead of 1.25. The golang:1.26-bookworm tag tracks the latest 1.26.x patch automatically.
…ntation for limit overrides
…iene - Add Go (13th language) to configs/languages.yaml with GO111MODULE=off, CGO_ENABLED=0, GOPATH=/, GOCACHE=/.cache/go-build env vars required for building single-file Go programs without a go.mod inside the chroot jail - Add Env []string field to LanguageDef and wire it through sandbox.RunConfig so per-language environment variables are injected as --env args to nsjail - Add scripts/lang_install/go.sh and golang-go to Dockerfile runtime stage - Add TestGoHelloWorld integration test; fix TestMain startup wait loop - Inject Env into both compile and runTest RunConfig structs in job.go - Fix stale nsjail invocation example in architecture.md - Update languages.md: Go in bonus table, env field docs, PHP add-language example - Remove stale .gitkeep placeholders from cmd/goboxd/, docs/, internal/, tests/ - Improve .gitignore: cover .DS_Store, .claude/, IDE dirs, build output binaries
…nguages
Wire up the integration test harness and fix five bugs that prevented
compiled-language execution from working correctly.
**docker-compose / infra**
- Add GOBOXD_URL=http://goboxd:8080 to the tools service so integration
tests reach the goboxd container instead of localhost (themselves)
- Add depends_on with condition: service_healthy so the tools container
only starts once goboxd is accepting requests
- Add a wget-based /healthz healthcheck to the goboxd service
- Add wget to the runtime Docker image (needed by the healthcheck)
**registry/probe.go — two probe bugs**
- ProbeNsjail: was running `nsjail --version`, which exits 255 (nsjail
has no --version flag), causing the probe to always report ok=false
and /readyz to return 503. Now uses os.Stat + an --help exec check
(exit 255 from --help is an ExitError, not a real exec failure).
- ProbeLanguage: was probing lang.Run.Cmd for compiled languages, which
is the per-job artifact path (/solution) and never exists at probe
time. Now probes lang.Build.Cmd (the compiler binary) instead.
**runner/job.go — internal_error on C/C++/Rust/Go**
- buildBindMounts called addIfNotCovered(lang.Run.Cmd) for compiled
languages where Run.Cmd="/solution". filepath.Dir("/solution")="/"
was added to the bind-mount set, causing nsjail to attempt mounting
"/" over the chroot root, producing an [E][ log line and
ParseBuildStatus returning internal_error. Skip the run-cmd mount
entirely for compiled languages; the artifact lives inside the
workspace (the chroot) and is not a host path.
**sandbox/nsjail.go — JVM cannot start on ARM64**
- RLIMIT_AS (virtual address space) floor was 512 MiB. On ARM64 the
JVM pre-allocates ~1 GiB of virtual memory for compressed class
space alone, so both Java (javac build phase) and Kotlin (java -jar
run phase) crashed with "Could not allocate compressed class space".
Raise the floor from 512 MiB to 4096 MiB. RLIMIT_AS limits virtual
address reservations, not physical RAM; cgroup memory.max remains the
real RSS enforcement mechanism.
All 28 integration tests now pass (make integration: ok 5.664s).
…y tracking - Add a self-contained embedded playground SPA at `/playground/` for interactive testing - Mount Swagger UI at `/docs/` and transition API documentation to swagger.yaml spec - Implement peak memory tracking via cgroup v2 `memory.peak` - Update benchmarks.md with load test results, security.md with seccomp/cgroup updates, and clean up deprecated docs (api.md, loopholes.md) - Ensure load_test.sh resolves hey/k6 from GOPATH if not on standard PATH
…round Remove omitempty from TestResult.MemoryPeakKB so it is always present in JSON output as the spec example shows. Fix a broken loopholes.md reference in architecture.md (file does not exist; correct link is security.md). Update playground SPA with cleaner styling.
…and README Adds missing unit tests for config, logctx, playground handler, and stats packages. Updates validate/request to fix edge cases surfaced by the new tests. Polishes README to under 60 lines, expands prompts.md AI log, and refreshes architecture/languages/security docs.
- tests: add TestInvalidJSON, TestInvalidTestCount, TestStdinTooLarge, TestMissingSourceFilename integration tests - fix: map SIGXCPU signal → time_exceeded in sandbox status parser - feat: animated neon gradient shimmer on figlet idle art and team badge (background-clip:text gradient + badge border glow, subtle palette) - handler: expand validation and error coverage in run.go / types.go - sandbox: nsjail.go improvements; pass --build-arg COMMIT to Docker - docs: restructure benchmarks.md, update Swagger (docs.go / swagger.*) - scripts: expand load_test.sh scenarios
…r unregistered languages
- handler/health: build typed response structs (HealthzResponse, ReadyzResponse, InfoResponse) instead of hand-rolled untyped maps, matching the Swagger schema - handler/run: extract resolveFilename helper, removing duplicated source/artifact filename validation; replace ad-hoc strQ with strconv.Quote - sandbox/nsjail: strconv over fmt.Sprintf for argv ints, errors.As for ExitError, parse memory.peak with strconv.ParseInt, collapse redundant bind-mount branch - registry/probe: errors.As for ExitError, strings.Cut in firstLine No behavior change; gofmt, go vet, and all unit tests pass.
Remove swagger UI serving (httpSwagger route, docs import, godoc annotation blocks in handlers). Strip example:/enums: struct tags from types.go. Prune comments that restate the code; keep WHY-comments for non-obvious behaviour (nsjail flags, seccomp policy, cgroup v2 OOM detection, etc.). Use for-range-N semaphore init (Go 1.22 idiom). Clean up doc.go.
Team/alpha
- validateLang now rejects unknown {{...}} placeholders in args,
zero/negative limits, missing source_filename, and invalid
filename strategy values — failures surface at load time, not runtime
- startup logs Warn for each failed language probe and Error+exits
if nsjail itself is unavailable
- 7 new registry tests covering each new validation path
…trings lua5.4 uses -v and iverilog uses -V instead of --version. Without probe_args, /readyz was returning the error message as the version string (ok:true but version showed "unrecognized option").
Team/Alpha: goboxd Stage 1
…g Swift installation script to remove unnecessary tools
…ions and update Swift installation script to remove unused libraries
…2GB)
- Add docs/loadtest/ with full load test artefacts
- results.csv: 10 steps (5–400 req/s), 30s each, 10s timeout
- breaking-point.png: error rate vs offered RPS, break marked at 5 req/s
- latency.png: p50/p95/p99 vs offered RPS
- load-test.sh: reproducible vegeta script (brew install vegeta)
- plot.py: matplotlib graph generator from CSV
- run-request.json + target.txt: MemoryHog.java POST /run payload
- report-{N}.json: raw vegeta JSON per step (proof)
- docker-compose.override.yml: enforce 2 vCPU / 2 GB RAM limit
Breaking point: 5 req/s (18% errors), caused by memory exhaustion.
Each JVM needs ~200-230 MB; 2 GB cap allows only ~8-10 concurrent JVMs.
MemoryHog holds 150 MB resident for 1s; queue backs up past 10s timeout.
Service degraded gracefully: continued processing ~2 jobs/slot, no crash,
full recovery after load dropped.
…tage3-loadtest) - load-test.sh: auto-generate target.txt at runtime with the script's own resolved SCRIPT_DIR absolute path instead of relying on a pre-baked hardcoded /Users/einstein/... path — fixes reproducibility on any clone - load-test.sh: fix FAIL_COUNT float/int comparison bug — jq can return 27.0 (float); bash [[ -gt ]] is integer-only and would abort the script under set -euo pipefail; use | floor | round in jq + (( )) arithmetic - load-test.sh: separate [[ -z ]] and (( FAIL_COUNT > 0 )) conditions so arithmetic expansion is handled safely regardless of jq version - Makefile: fix 'make load' target to point at docs/loadtest/load-test.sh (was pointing at scripts/load_test.sh which does not exist) - docs/loadtest/README.md: correct error-type classification — raw vegeta reports show status code 0 (connection reset by peer, EOF, one brief 'connection refused' at 75 rps), not clean HTTP 503; update degradation narrative, result table header, and failure timeline accordingly - docs/loadtest/README.md: add 'make load' shortcut and clarify target.txt is auto-generated so no manual path editing is required - docs/loadtest/target.txt: replace hardcoded absolute path with explanatory comment; the real target.txt is now written by the script at runtime - docs/benchmarks.md: add Stage 3 MemoryHog section with summary table and links to docs/loadtest/ so evaluators find the results from the standard benchmark doc
Brings docs/loadtest/ (results, graphs, script, report JSONs), docker-compose.override.yml (2 vCPU / 2 GB constraints), updated Makefile load target, and benchmarks.md Stage 3 section from the team/Alpha-stage3-loadtest branch into team/Alpha-stage3. Breaking point: 5 req/s. Root cause: 2-slot semaphore + ~200 MB per JVM causes queue pile-up and client-side 10 s timeouts.
Author
|
Hi, I wanted to follow up on the Stage 1 review and share a bit of context. After Stage 1 results came in, I spent the days leading up to competition day working through the feedback. The gaps flagged were code quality/SDLC, communication, and no CI, so I went and addressed those properly: added a full GitHub Actions pipeline, got the integration suite to 50 pass / 0 fail, wrote out the missing docs (architecture, security with per-hole file:line, benchmarks with real numbers), added load shedding, real cgroup v2 memory tracking, tighter seccomp rules, and more. That work is all sitting on On competition day we were told to use the last commit before June 2 for Stage 2 and 3. I thought the improvements between stages would carry forward. So all the work from those days didn't end up counting for the evaluation, which was a bit tough to take given how much time went into it. Today's additions (three languages, evaluation docs, MemoryHog load test) were all done on top of the June 2 base as instructed. No Go code was touched, just YAML, scripts, and docs. Those are documented in the PR description above. I also applied all the same Stage 2 and 3 tasks simultaneously to If there's any way to consider Thanks for the competition and for the detailed Stage 1 feedback. It pointed me in the right direction. |
…hog workflow docker-compose.override.yml was a local load-test artifact pinning the container to 2 vCPU / 2 GB for the Stage 3 constrained-environment run. Results are committed; the override file does not belong in the repo as it silently caps any local `docker compose up` without warning. The memhog-loadtest workflow push trigger fired an expensive 40-min run on every commit to team/Alpha and team/Alpha-stage3. Testing is done -- switching to workflow_dispatch only so the workflow runs on demand. The "Ensure override" step is simplified to always write the file since it will never be present in the checkout anymore.
Author
|
@jkmadathil will zip and share the AI logs json after 3-4 days. |
Author
Sharing the ZIP file containing my Claude Code sessions: I've also used Antigravity and Copilot, but this ZIP file does not contain logs from those two coding agents. |
Closed
Eyuvaraj
force-pushed
the
team/Alpha
branch
2 times, most recently
from
June 24, 2026 13:17
38973af to
3eef651
Compare
…ken langs - Add .dockerignore: exclude docs/ (44MB) and .git/ (5.7MB) from build context - Dockerfile: add curl to runtime base deps for healthcheck - docker-compose: restart: unless-stopped, container mem limit (8g), simplify healthcheck to curl, add tuning comments for MAX_CONCURRENT_JOBS/MAX_QUEUE_DEPTH - Makefile: inject VERSION via git describe into /info; add make up/down/logs/ clean/test-local targets - Disable kotlin: Debian bookworm apt ships 1.3.31 (2019); modern syntax fails - Disable assembly: 4/5 stage2 failures due to nsjail mount namespace link issues - Docs: remove kotlin/swift/assembly from README and docs/languages.md; add prolog; annotate csharp Mono version (C# 6.0 support only)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Team
Eyuvaraj D - solo - Team Alpha
Framework
chiis used as the HTTP router because it wraps plainnet/httphandlers with no custom context model, keeping middleware explicit and adding zero abstraction overhead.How to Run
Requires Docker with Compose v2. The container must run with
--privileged(nsjail uses Linux namespaces). nsjail 3.4 is a git submodule compiled from source inside the image — no prebuilt binary.make build— build the Docker image (~5 min cold, ~1 min warm)make run— start the service on:8080make test— unit tests, no Docker neededmake integration— end-to-end tests (requiresmake runin a separate terminal)make lint— golangci-lintmake load— load benchmarks (requiresheyork6:go install github.com/rakyll/hey@latest)Security Holes Closed
All 7 holes from the reference implementation are closed.
Path traversal via filename —
validate.Filename()enforcesfilepath.Base(n) == n,[a-zA-Z0-9._-]+only, no leading dot, max 64 chars, called before any path join. (run.go:335)Shell invocation — No
exec.Command("sh", "-c", ...)anywhere. Workspaces useos.MkdirTempandos.RemoveAll; every external program is a pure[]stringargv viaexec.CommandContext. (workspace.go:20, nsjail.go)Compiler-flag injection —
validate.Flags()checks every client-supplied flag against a per-languageflag_allowlistin the YAML. Unlisted flags return400 invalid_flag. (run.go:79, run.go:88)Unbounded request sizes —
http.MaxBytesReaderat the HTTP layer, plusvalidate.SourceSize,validate.StdinSize, andvalidate.ExpectedSizeper field. (middleware.go:16, run.go:59)Workspace collisions under load —
os.MkdirTemp(jailDir, "goboxd-*")creates a unique directory atomically. No counter, no retry. (workspace.go:20)Unbounded child output —
io.LimitReader(stdoutPipe, max+1)caps captured stdout. Overflow drains toio.Discardand appends[output truncated]. (nsjail.go:164)Stale jail directories —
defer ws.Cleanup()on every exit path.SweepOrphansat startup removes orphans from prior crashes. (runner.go:93, main.go:30)A Kafel seccomp deny-list additionally blocks
ptrace,bpf,init_module,kexec_load,mount, and 15 others viaKILL_PROCESS. (nsjail.go:41, full table in docs/security.md)Languages Supported
In scope (7):
py3,bash,js,c,cpp,java,verilogAdditional (4, all pass
/readyz):ruby,lua,ocaml,goEach language is a single YAML block in
configs/languages.yaml. Language toolchains are dynamically installed via independent scripts inscripts/lang_install/*.shexecuted in theDockerfile— adding a new language requires zeroDockerfileor Go code edits.Benchmarks
Results from a clean Docker container (
make build && make run) on the measurement host (MacBook Air Apple M4, 10-core CPU, 16 GB RAM).MAX_CONCURRENT_JOBS= 10. Load tool:hey. Full details in docs/benchmarks.md.Stage 2 — New Languages
Three languages added on competition day via YAML + Dockerfile only, no Go code changed:
csharpprologassemblyFull end-to-end evaluation with payloads and responses:
stage2-evaluation/EVALUATION.mdStage 3 — MemoryHog Load Test
Ran 10 steps (5 → 400 req/s, 30 s each) against the service pinned to 2 vCPU / 2 GB RAM.
Breaking point: 5 req/s. The concurrency semaphore defaults to 2 slots (= cgroup CPU count). Each JVM holds ~200 MB resident for 1 s, so incoming requests queue up and hit the 10 s client timeout before a slot frees. At ≥75 req/s, TCP resets join the timeouts as the accept backlog saturates. The service stayed healthy throughout and recovered cleanly once load dropped.
docs/loadtest/README.mddocs/loadtest/results.csvdocs/loadtest/breaking-point.pngdocs/loadtest/latency.png