Experimental: Testing pipeline Performance Optimizations#2927
Experimental: Testing pipeline Performance Optimizations#2927kevin-hashimoto merged 8 commits intodevelopfrom
Conversation
Backend Test Results862 tests 862 ✅ 9m 11s ⏱️ Results for commit 1a08d3e. ♻️ This comment has been updated with latest results. |
Frontend Test Results 1 files 187 suites 1m 37s ⏱️ Results for commit 1a08d3e. ♻️ This comment has been updated with latest results. |
kevin-hashimoto
force-pushed
the
experimental/kevin-2821
branch
from
328a268 to
1ece447
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
- Add backend test sharding with pytest-split (50 shards, ~17 tests each) - Add frontend test sharding with Vitest (50 shards) - Implement shared setup jobs to reduce duplication - Add proper XML test result merging with junitparser - Optimize artifact handling and dependency caching - Update backend dependencies to include pytest-split Performance improvements: - Backend tests: Parallel execution across 50 shards - Frontend tests: Parallel execution across 50 shards - Reduced CI time through efficient test distribution
kevin-hashimoto
force-pushed
the
experimental/kevin-2821
branch
from
2c80ad8 to
0c8512a
Compare
…t setup - Add materialized view refresh to pytest seeder after data seeding - Ensure mv_transaction_aggregate and mv_credit_ledger are populated with test data - Fix credit_ledger test that was failing in CI but passing locally - Update CI workflow for improved test reliability
kevin-hashimoto
force-pushed
the
experimental/kevin-2821
branch
from
f3789cc to
d1887c0
Compare
kevin-hashimoto
changed the title
- Implement test sharding with 45 backend and 32 frontend shards for parallel execution - Add comprehensive caching strategies for Poetry, npm, and pytest timing data - Optimize database operations with template-based restoration and concurrent processing - Enhance frontend pipeline with node_modules caching and artifact compression - Improve test execution with optimized Vitest configuration and memory management - Add graceful error handling and fallback mechanisms for cache misses - Reduce overall pipeline execution time by ~60% through parallelization and caching
kevin-hashimoto
force-pushed
the
experimental/kevin-2821
branch
from
9853894 to
f716f0e
Compare
…in permissions Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
kevin-hashimoto
requested review from
AlexZorkin,
hamed-valiollahi and
prv-proton
kevin-hashimoto
changed the title
|
This looks great just one question: Are there issues with max number of concurrent jobs with so many running at once? |
I found that 45 for the backend and 32 for the frontend are the sweet spots, there seems to be some diminishing returns on how long the result merging takes and how tests are distributed amongst shards past those numbers. We shouldnt run into any issues with our amount of concurrent jobs (for enterprise accounts, we get 500 concurrent jobs, matrixes/shards get 256 concurrent jobs). Currently we're running the backend sharding with file based time distribution, we can speed this up with test based time distribution when we split out the unit tests and integration tests (i think). using test based distribution will balance the amount of tests per shard better so we may be able to reduce the shard numbers here. |
Awesome thanks for doing the deep dive on this. Looks good to get this into the pipeline 👍 |
2 participants
🚀 Optimize CI/CD Pipeline Performance
Summary
This PR implements comprehensive optimizations to our testing pipeline, reducing execution time by 63% from 7.5 minutes to 2.75 minutes through intelligent parallelization, caching strategies, and
resource optimization.
Key Improvements
🔄 Test Sharding & Parallelization
💾 Enhanced Caching Strategy
⚡ Performance Optimizations
🛠️ Infrastructure Improvements
Impact
Technical Details
Testing
---Before: 7.5 minutes total pipeline timeAfter: 2.75 minutes total pipeline timeImprovement: 63% reduction in execution time ⚡
NOTE: we will need to change our branch rules to check for 'Publish Backend Test Results' and 'Publish Frontend Test Results'