Skip to content

Commit 9dd5f74

Browse files
authored
[dev] Introduce script and CI step using trivy to scan and enforce 0 CRITICAL in images (#20712)
* [scripts] Introduce trivy-scan-images.sh Tool: gitpod/catfood.gitpod.cloud * [trivy] Fitting trivyignore.yaml Tool: gitpod/catfood.gitpod.cloud * [trivy] Add scan and enforcement of "CRITICAL" vulns at build time Tool: gitpod/catfood.gitpod.cloud * Fix base repo ref Tool: gitpod/catfood.gitpod.cloud * Replace docker run by oci-tool fetch file Tool: gitpod/catfood.gitpod.cloud
1 parent 40c5b7c commit 9dd5f74

File tree

4 files changed

+352
-1
lines changed

4 files changed

+352
-1
lines changed

.github/workflows/build.yml

+27-1
Original file line numberDiff line numberDiff line change
@@ -303,9 +303,33 @@ jobs:
303303
GITHUB_EMAIL: [email protected]
304304
VERSION: ${{ needs.configuration.outputs.version }}
305305

306+
trivy-scan:
307+
name: "Scan Images for Vulnerabilities"
308+
needs:
309+
- configuration
310+
- build-gitpod
311+
- create-runner
312+
runs-on: ${{ needs.create-runner.outputs.label }}
313+
container:
314+
image: eu.gcr.io/gitpod-core-dev/dev/dev-environment:main-gha.30393
315+
steps:
316+
- uses: actions/checkout@v4
317+
- name: Setup Environment
318+
uses: ./.github/actions/setup-environment
319+
with:
320+
identity_provider: ${{ github.ref == 'refs/heads/main' && secrets.CORE_DEV_PROVIDER || secrets.DEV_PREVIEW_PROVIDER }}
321+
service_account: ${{ github.ref == 'refs/heads/main' && secrets.CORE_DEV_SA || secrets.DEV_PREVIEW_SA }}
322+
leeway_segment_key: ${{ secrets.LEEWAY_SEGMENT_KEY }}
323+
- name: Scan Images for Vulnerabilities
324+
shell: bash
325+
run: |
326+
INSTALLER_IMAGE_BASE_REPO=${{ needs.configuration.outputs.image_repo_base }}
327+
./scripts/trivy/trivy-scan-images.sh ${{ needs.configuration.outputs.version }} CRITICAL
328+
exit $?
329+
306330
install-app:
307331
runs-on: ${{ needs.create-runner.outputs.label }}
308-
needs: [ configuration, build-gitpod, create-runner ]
332+
needs: [ configuration, build-gitpod, trivy-scan, create-runner ]
309333
if: ${{ needs.configuration.outputs.is_main_branch == 'true' }}
310334
strategy:
311335
fail-fast: false
@@ -343,6 +367,7 @@ jobs:
343367
- configuration
344368
- build-previewctl
345369
- build-gitpod
370+
- trivy-scan
346371
- infrastructure
347372
- create-runner
348373
runs-on: ${{ needs.create-runner.outputs.label }}
@@ -490,6 +515,7 @@ jobs:
490515
- build-previewctl
491516
- infrastructure
492517
- build-gitpod
518+
- trivy-scan
493519
- install-app
494520
- install
495521
- monitoring
+56
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
apiVersion: v1
2+
domain: example.com
3+
authProviders: []
4+
blockNewUsers:
5+
enabled: false
6+
passlist: []
7+
certificate:
8+
kind: secret
9+
name: https-certificates
10+
containerRegistry:
11+
enableAdditionalECRAuth: false
12+
inCluster: false
13+
privateBaseImageAllowList: []
14+
subassemblyBucket: ""
15+
external:
16+
url: "registry.example.com"
17+
certificate:
18+
kind: secret
19+
name: registry-certificate
20+
database:
21+
inCluster: false
22+
external:
23+
certificate:
24+
kind: secret
25+
name: database-certificate
26+
disableDefinitelyGp: true
27+
kind: Full
28+
metadata:
29+
region: local
30+
shortname: default
31+
objectStorage:
32+
inCluster: false
33+
resources:
34+
requests:
35+
memory: 2Gi
36+
s3:
37+
endpoint: "s3.example.com"
38+
bucket: "gitpod-storage"
39+
credentials:
40+
kind: secret
41+
name: object-storage-credentials
42+
observability:
43+
logLevel: info
44+
openVSX:
45+
url: https://open-vsx.org
46+
repository: example.org
47+
workspace:
48+
maxLifetime: 36h0m0s
49+
resources:
50+
requests:
51+
cpu: "1"
52+
memory: 2Gi
53+
runtime:
54+
containerdRuntimeDir: /var/lib/containerd/io.containerd.runtime.v2.task/k8s.io
55+
containerdSocketDir: /run/containerd
56+
fsShiftMethod: shiftfs

scripts/trivy/trivy-scan-images.sh

+184
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
#!/bin/bash
2+
# Copyright (c) 2025 Gitpod GmbH. All rights reserved.
3+
# Licensed under the GNU Affero General Public License (AGPL).
4+
# See License.AGPL.txt in the project root for license information.
5+
6+
set -euo pipefail
7+
8+
# Check if VERSION and FAIL_ON are provided
9+
if [[ $# -lt 2 ]]; then
10+
echo "Usage: $0 VERSION FAIL_ON [TRIVY_ARGS...]"
11+
echo " VERSION: The version to scan (e.g., main-gha.32006)"
12+
echo " FAIL_ON: Severity threshold to fail on (empty, HIGH, or CRITICAL)"
13+
echo " TRIVY_ARGS: Additional arguments to pass to Trivy"
14+
echo "Example: $0 main-gha.32006 HIGH"
15+
exit 1
16+
fi
17+
18+
INSTALLER_IMAGE_BASE_REPO="${INSTALLER_IMAGE_BASE_REPO:-eu.gcr.io/gitpod-dev-artifact}"
19+
20+
# Extract VERSION and FAIL_ON from arguments and remove them from args list
21+
VERSION="$1"
22+
FAIL_ON="$2"
23+
shift 2
24+
25+
# Validate FAIL_ON value
26+
if [[ -n "$FAIL_ON" ]] && [[ "$FAIL_ON" != "HIGH" ]] && [[ "$FAIL_ON" != "CRITICAL" ]]; then
27+
echo "Error: FAIL_ON must be either empty, 'HIGH', or 'CRITICAL'"
28+
exit 1
29+
fi
30+
31+
32+
if ! command -v jq &> /dev/null; then
33+
echo "jq not found. Please install jq to continue."
34+
exit 1
35+
fi
36+
37+
# Set up working directory
38+
SCAN_DIR=$(mktemp -d -t trivy-scan-XXXXXX)
39+
echo "Working directory: $SCAN_DIR"
40+
41+
# Directory where this script is located
42+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
43+
INSTALLER_CONFIG_PATH="$SCRIPT_DIR/scan-installer-config.yaml"
44+
TRIVYIGNORE_PATH="$SCRIPT_DIR/trivyignore.yaml"
45+
46+
# Ensure Trivy is installed
47+
TRIVY_CMD="trivy"
48+
if ! command -v "$TRIVY_CMD" &> /dev/null; then
49+
echo "Trivy not found. Installing..."
50+
mkdir -p "$SCAN_DIR/bin"
51+
TRIVY_CMD="$SCAN_DIR/bin/trivy"
52+
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b "$SCAN_DIR/bin"
53+
fi
54+
55+
OCI_TOOL_CMD="oci-tool"
56+
OCI_TOOL_VERSION="0.2.0"
57+
if ! command -v "$OCI_TOOL_CMD" &> /dev/null; then
58+
mkdir -p "$SCAN_DIR/bin"
59+
OCI_TOOL_CMD="$SCAN_DIR/bin/oci-tool"
60+
curl -fsSL https://github.com/csweichel/oci-tool/releases/download/v${OCI_TOOL_VERSION}/oci-tool_${OCI_TOOL_VERSION}_linux_amd64.tar.gz | tar xz -C "$(dirname "$OCI_TOOL_CMD")" && chmod +x "$OCI_TOOL_CMD"
61+
fi
62+
63+
echo "=== Gathering list of all images for $VERSION"
64+
65+
# Extract installer binary from installer image
66+
INSTALLER_IMAGE="$INSTALLER_IMAGE_BASE_REPO/build/installer:${VERSION}"
67+
INSTALLER="$SCAN_DIR/installer"
68+
"$OCI_TOOL_CMD" fetch file -o "$INSTALLER" --platform=linux-amd64 "${INSTALLER_IMAGE}" app/installer
69+
echo ""
70+
chmod +x "$INSTALLER"
71+
72+
# Run the installer docker image to get the list of images
73+
"$INSTALLER" mirror list -c "$INSTALLER_CONFIG_PATH" > "$SCAN_DIR/mirror.json"
74+
75+
# Extract original image references
76+
jq -r '.[].original' "$SCAN_DIR/mirror.json" > "$SCAN_DIR/images.txt"
77+
78+
# Remove empty lines
79+
sed -i '/^\s*$/d' "$SCAN_DIR/images.txt"
80+
81+
# Filter out specific image patterns
82+
echo "=== Filtered out images:"
83+
TOTAL_BEFORE=$(wc -l < "$SCAN_DIR/images.txt")
84+
85+
# Apply all filters at once using extended regex
86+
grep -v -E "/build/ide/|/gitpod/workspace-|/library/mysql|/library/redis|/cloudsql-docker/gce-proxy" "$SCAN_DIR/images.txt" > "$SCAN_DIR/filtered_images.txt"
87+
88+
TOTAL_AFTER=$(wc -l < "$SCAN_DIR/filtered_images.txt")
89+
FILTERED=$((TOTAL_BEFORE - TOTAL_AFTER))
90+
91+
echo " Total filtered: $FILTERED"
92+
93+
# Use filtered list for scanning
94+
mv "$SCAN_DIR/filtered_images.txt" "$SCAN_DIR/images.txt"
95+
96+
# Count total images
97+
TOTAL_IMAGES=$(wc -l < "$SCAN_DIR/images.txt")
98+
echo "=== Found $TOTAL_IMAGES images to scan"
99+
100+
# Create results directory
101+
RESULT_FILE="$SCAN_DIR/result.jsonl"
102+
103+
# Scan all images with Trivy
104+
COUNTER=0
105+
FAILED=0
106+
while IFS= read -r IMAGE_REF; do
107+
((COUNTER=COUNTER+1))
108+
109+
echo "= Scanning $IMAGE_REF [$COUNTER / $TOTAL_IMAGES]"
110+
111+
# Run Trivy on the image
112+
scan_time=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
113+
set +e
114+
trivy_output=$("$TRIVY_CMD" image "$IMAGE_REF" --ignorefile "$TRIVYIGNORE_PATH" --scanners vuln --format json "$@" | jq -c)
115+
scan_status=$?
116+
117+
# Create a JSON object for the current scan
118+
if [ $scan_status -eq 0 ]; then
119+
# Check if trivy_output is valid JSON
120+
if echo "$trivy_output" | jq empty > /dev/null 2>&1; then
121+
# Direct approach - create the combined JSON object using jq directly
122+
jq -c --arg image "$IMAGE_REF" --arg scan_time "$scan_time" \
123+
'. + {image: $image, scan_time: $scan_time}' <<< "$trivy_output" | jq >> "$RESULT_FILE"
124+
else
125+
# If trivy output is not valid JSON, treat as error
126+
echo "Warning: Trivy returned invalid JSON for $IMAGE_REF"
127+
jq -n --arg image "$IMAGE_REF" \
128+
--arg scan_time "$scan_time" \
129+
--arg error "Invalid JSON output from Trivy" \
130+
--arg details "$trivy_output" \
131+
'{image: $image, scan_time: $scan_time, error: $error, error_details: $details}' | jq >> "$RESULT_FILE"
132+
((FAILED=FAILED+1))
133+
fi
134+
135+
else
136+
# For error cases, create a simple JSON object
137+
jq -n --arg image "$IMAGE_REF" \
138+
--arg scan_time "$scan_time" \
139+
--arg error "Trivy scan failed" \
140+
--arg details "$trivy_output" \
141+
'{image: $image, scan_time: $scan_time, error: $error, error_details: $details}' >> "$RESULT_FILE"
142+
((FAILED=FAILED+1))
143+
fi
144+
set -e
145+
146+
echo ""
147+
done < "$SCAN_DIR/images.txt"
148+
149+
# Generate summary report
150+
echo "=== Scan Summary ==="
151+
echo "Scan directory: $SCAN_DIR"
152+
echo "Results file: $RESULT_FILE"
153+
echo "Total ignored images: $FILTERED"
154+
echo "Total scanned images: $TOTAL_IMAGES"
155+
echo "Failed scans: $FAILED"
156+
echo "Triviy binary: $TRIVY_CMD"
157+
echo "Triviy version: $($TRIVY_CMD version)"
158+
echo ""
159+
160+
# Count vulnerabilities by severity
161+
echo "=== Vulnerability Summary ==="
162+
CRITICAL="$(jq -r 'if .Results != null then [.Results[].Vulnerabilities // [] | .[] | select(.Severity == "CRITICAL")] | length else 0 end' "$RESULT_FILE" 2>/dev/null | awk '{sum+=$1} END {print sum}')"
163+
HIGH="$(jq -r 'if .Results != null then [.Results[].Vulnerabilities // [] | .[] | select(.Severity == "HIGH")] | length else 0 end' "$RESULT_FILE" 2>/dev/null | awk '{sum+=$1} END {print sum}')"
164+
echo "CRITICAL: $CRITICAL"
165+
echo "HIGH: $HIGH"
166+
echo ""
167+
168+
echo "=== Scan completed ==="
169+
if [[ $FAILED -gt 0 ]]; then
170+
echo "ERROR: $FAILED scans failed"
171+
exit 1
172+
fi
173+
174+
# Check if we should fail based on vulnerability counts
175+
if [[ "$FAIL_ON" == "CRITICAL" ]] && [[ $CRITICAL -gt 0 ]]; then
176+
echo "FAIL: Found $CRITICAL CRITICAL vulnerabilities, and FAIL_ON=CRITICAL was specified"
177+
exit 1
178+
elif [[ "$FAIL_ON" == "HIGH" ]] && [[ $((CRITICAL + HIGH)) -gt 0 ]]; then
179+
echo "FAIL: Found $CRITICAL CRITICAL and $HIGH HIGH vulnerabilities, and FAIL_ON=HIGH was specified"
180+
exit 1
181+
fi
182+
183+
echo "0 $FAIL_ON or higher vulnerabilities found."
184+
exit 0

scripts/trivy/trivyignore.yaml

+85
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
vulnerabilities:
2+
# Typescript / typeorm
3+
- id: CVE-2022-33171
4+
paths:
5+
- "app/node_modules/typeorm/package.json"
6+
statement: |
7+
This vulnerability in TypeORM's findOne / findOneOrFail functions can improperly interpret a crafted JSON object
8+
and concatenate it into raw SQL, potentially allowing SQL injection attacks.
9+
10+
In Gitpod’s usage, TypeORM is not exposed to arbitrary user input. For example, DB migrations run preset queries;
11+
the server/bridge code does not hand raw JSON from external sources to findOne. Therefore, there is no path for
12+
injecting malicious JSON into a query, rendering the vulnerability non-exploitable.
13+
14+
# image-builder-bob
15+
- id: CVE-2024-32002
16+
statement: |
17+
A vulnerability allowing remote code execution when cloning a malicious repository (often leveraging submodules
18+
and symlinks on case-insensitive filesystems) that can place malicious scripts in .git/hooks/. On Windows/macOS
19+
(case-insensitive FS), a malicious repo can use symlinks or tricky paths to cause Git to write hook scripts in
20+
the main .git directory during clone, which Git then executes automatically. This results in code execution on
21+
the system performing the clone.
22+
23+
The standard exploit path involving .Git vs. .git doesn’t work on Linux. Also, Gitpod’s builder typically
24+
fetches known repositories or user-provided repos in a controlled environment. Even if a user cloned a hostile
25+
repo in their own workspace, they’d compromise only themselves (in a sandboxed container). No broader exploit
26+
of the underlying host is possible.
27+
28+
- id: CVE-2024-23652
29+
paths:
30+
- "usr/bin/buildctl"
31+
- "usr/bin/buildkitd"
32+
statement: |
33+
A vulnerability where the RUN --mount cleanup logic could remove directories outside of the build context,
34+
leading to potential arbitrary file deletion on the host if BuildKit is used with untrusted Dockerfiles.
35+
36+
Gitpod runs BuildKit inside containers in a Kubernetes environment. Even if a malicious Dockerfile tries to
37+
exploit this, it can only affect the container’s filesystem (which the user already controls). There’s no path to
38+
escalate beyond the container to the node’s host filesystem, so this has no material security impact in Gitpod’s
39+
isolated build setup.
40+
41+
- id: CVE-2024-23653
42+
paths:
43+
- "usr/bin/buildctl"
44+
- "usr/bin/buildkitd"
45+
statement: |
46+
A flaw allowing a build container to run with elevated privileges without the required security.insecure
47+
entitlement. This can grant a malicious Dockerfile or BuildKit client more privileges than intended during the
48+
build.
49+
50+
In Gitpod, BuildKit operates under tight Kubernetes constraints. Even if a user tries to request privileged mode,
51+
it won’t escalate to the actual host or break out of the container. The user is effectively “attacking” their
52+
own build container. They already have control there, so there’s no additional privilege escalation beyond their
53+
existing user container in Gitpod’s architecture.
54+
55+
56+
- id: CVE-2024-45337
57+
paths:
58+
- "usr/bin/buildctl"
59+
- "usr/bin/buildkitd"
60+
- "usr/bin/kube-rbac-proxy"
61+
statement: |
62+
An SSH authentication bypass issue if PublicKeyCallback is used incorrectly in Go’s SSH server library. Misuse
63+
can lead to validating the wrong public key and granting access under the wrong identity.
64+
65+
Neither BuildKit nor kube-rbac-proxy runs an SSH server that uses PublicKeyCallback for authentication.
66+
Kube-rbac-proxy does HTTP-based RBAC checks. BuildKit may include x/crypto/ssh but does not expose an SSH server
67+
requiring PublicKeyCallback. Hence, there’s no exploit path for this bug in these components.
68+
69+
- id: CVE-2024-24790
70+
statement: |
71+
A logic flaw in Go’s net/netip packages causing misclassification of certain IPv4-mapped IPv6 addresses (e.g., failing to mark them as loopback/private). Could lead to security checks that rely on IP classification (like IsPrivate, IsLoopback) being bypassed.
72+
73+
In these images (Cloud SQL Proxy, bob-runc, BuildKit, kube-rbac-proxy), there’s no code path that enforces security decisions using IsLoopback/IsPrivate from Go’s netip. They either accept connections in a controlled environment (Cloud SQL Proxy -> GCP) or use different authentication mechanisms (kube-rbac-proxy). Hence, no external attack can exploit this misclassification.
74+
75+
- id: CVE-2024-45491
76+
statement: |
77+
Integer overflow in dtdCopy when dealing with a large number of default attributes in a DTD, causing potential out-of-bounds write or memory corruption in Expat on 32-bit systems.
78+
79+
The container runs on a 64-bit Alpine Linux environment, not a 32-bit build, so the overflow is not triggered the same way. Moreover, there is no component automatically parsing untrusted XML with Expat. Hence, no practical exploitation path in Gitpod’s usage.
80+
81+
- id: CVE-2024-45492
82+
statement: |
83+
Similar integer overflow in the nextScaffoldPart function of Expat, triggered by enormous or deeply nested content models in the DTD. Can lead to denial of service or potential corruption.
84+
85+
As with CVE-2024-45491, the environment is 64-bit Alpine, and no untrusted XML parsing occurs by default. Thus, the bug cannot be reached in a way that leads to exploitation.

0 commit comments

Comments
 (0)