Skip to content

Commit aacb1f3

Browse files
committed
refactor: allow evals to take longer than 10min
Some evals may take longer than 10min, especially when configured to run with more repair attempts and when operating on more brittle code. We shouldn't intervene here and guard simply on the individual actions with default timeouts from the local executor.
1 parent 1ad1cba commit aacb1f3

File tree

1 file changed

+4
-2
lines changed

1 file changed

+4
-2
lines changed

runner/orchestration/generate.ts

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -117,8 +117,10 @@ export async function generateCodeAndAssess(options: AssessmentConfig): Promise<
117117
workerConcurrencyQueue,
118118
progress,
119119
),
120-
// 10min max per app evaluation. We just want to make sure it never gets stuck.
121-
10,
120+
// 30min max per app evaluation. We just want to make sure it never gets stuck.
121+
// Note that this timeout is expected to never be hit as individual action timeouts
122+
// should fire first. E.g. local executor build or test timeouts.
123+
30,
122124
);
123125
return results;
124126
} catch (e: unknown) {

0 commit comments

Comments
 (0)