Repository navigation
feat: hidden-active recovery benchmark, EF=4.0 at k=5, 75 tests - #3
Closed
cschanhniem wants to merge 2 commits into
Closed
cschanhniem wants to merge 2 commits into
cschanhniem wants to merge 2 commits into
Conversation
…tion - Add hydrophobic_moment() to physchem.py using Eisenberg (1984) consensus scale at 100°/residue helical projection; literature-cited correlate of AMP activity - Expand activity_likeness_score() to incorporate amphipathicity (15% weight) with reduced charge/hydrophobicity weights to keep total at 1.0 - Add recall_at_k(), random_recall_at_k(), enrichment_factor(), benchmark_summary() to benchmark/evaluate.py with honest disclaimer in every output - Add 'openamp-foundry bench baseline' CLI subcommand for pipeline vs random recall - Add 'make bench-baseline' Makefile target - 20 new tests: amphipathicity feature, hydrophobic moment edge cases, recall@k boundary conditions, enrichment factor, benchmark summary structure
- Add examples/benchmark/mixed_candidates.csv (20 sequences: 5 known-active AMPs + 15 non-AMP control sequences) for proper enrichment benchmarking - Add examples/benchmark/active_labels.csv (5 known-active IDs matching above) - Add make bench-hidden-active target using bench baseline CLI - 12 new tests in test_hidden_active_recovery.py: - all positives rank in top half - recall@5 = 1.0 (perfect recovery) - enrichment factor >= 2.0 at k=5 (actual EF=4.0) - pipeline verdict correctly says 'outperforms random' - negatives score lower than positives on average - CLI integration test for bench baseline command - benchmark data integrity checks - Pipeline achieves EF=4.0 at k=5: all 5 known AMPs recovered in top 5 of 20 vs 25% expected from random — meets Phase 2 criterion from AGENTS.md
Collaborator
Author
|
Superseded by PR #11 (feat/integrate-all-phases), which merges all Phase 2 + Phase 3 work into a single consolidation PR with 251 tests passing. |
8 tasks done
cschanhniem
added a commit
that referenced
this pull request
Jun 28, 2026
* feat: elastase resistance + aggregation propensity scoring Two new computational features to reduce wet-lab failure rate: 1. Elastase resistance (GAP #3 from audit): - physchem.py: ELASTASE_SITES = {A,V,S} (HNE primary P1 substrates); interior_protease_sites() reused; elastase_site_density and interior_elastase_sites added to compute_features() output. - stability.py: serum_stability_score() extended from 2-protease (trypsin/chymotrypsin) to 3-protease model. Weighted sum with trypsin:2 > chymotrypsin:1 > elastase:0.5. Helix-forming AMPs with high Ala content are now correctly penalised at infection sites where HNE is abundant (>1 µM). Denominator 3.5 = sum of weights; backward-compatible (missing elastase key → 0.0). - Literature: Bieth (1986); Doherty et al. (1991 Biochemistry). - 12 new tests in test_elastase_stability.py. 2. Aggregation propensity (GAP #1 from audit): - physchem.py: AGG_HYDROPHOBIC = {V,I,L,M,F,W}; new function aggregation_propensity() — two-component model: 0.7 × interior_run_risk (run ≥ 4 → ramp 0→1 over 5 residues) + 0.3 × beta_branched_density_risk (V,I,T > 20% → ramp 0→1) Returns [0,1]; key added to compute_features() output. - synthesis.py: synthesis_feasibility_score() now includes: if agg > 0: score -= min(agg * 0.25, 0.20) Max penalty = 0.20 (capped). Backward-compat (missing key → 0). - Literature: Quittot et al. (2017 Protein Sci); Wurth et al. (2006 J Mol Biol). - 22 new tests in test_aggregation_propensity.py. Impact: AUROC=0.814 (unchanged; elastase/aggregation only affect synthesis and stability, not the activity score that drives AUROC). Total test count: 1122. * fix: address code review HIGH issues for PR #49 - physchem.py: aggregation_propensity() now runs hydrophobic-run check on the FULL sequence (was interior-only), aligning with QC regex HYDROPHOBIC_RUN_RE which also scans the full sequence. Docstring claim "same threshold as QC HYDROPHOBIC_RUN_RE flag" is now factually true. Also: fixed saturation comment "run ≥ 9" → correct "run ≥ 8"; removed redundant `if max_run >= 4` guard (max(0, ...) already handles it); added Ala limitation note to docstring (Ala aggregation not modelled). - test_aggregation_propensity.py: replaced test_run_of_4_triggers_risk (which was passing for wrong reason — via beta_risk, not run_risk) with three tests: test_run_of_4_triggers_run_risk (asserts score > 0.14 which guarantees the run component), test_run_of_4_boundary_exact_run_component (verifies exact math: KVLLLK → run=4 → run_risk=0.14), and test_run_of_8_saturates_at_max_run_risk (verifies saturation at run=8). - Makefile: update test count to 1124.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the Phase 2 key requirement from AGENTS.md: "Known active AMPs hidden from training appear disproportionately in top-ranked candidates."
examples/benchmark/mixed_candidates.csv— 20 sequences: 5 known-active AMPs (typical cationic, amphipathic sequences) mixed with 15 non-active controls (low-charge, repetitive, no amphipathic character)examples/benchmark/active_labels.csv— 5 positive IDs matching the mixed poolmake bench-hidden-activeruns the benchmark and writesoutputs/bench_hidden_active_report.jsonResults
All 5 known-active AMPs rank in positions 1–5 of 20. The pipeline achieves 4× enrichment vs random at k=5, meeting the Phase 2 criterion.
Honest caveats (stated in every output)
Test plan
make test— 75 tests passmake demo— pipeline runs end-to-endmake bench-hidden-active— EF=4.0 confirmed