fix(llmobs): remove version setting in langchain auto prompt template #14937
Conversation
jsimpher
changed the title
|
|
jsimpher
added
the
changelog/no-changelog
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 240 ± 6 ms. The average import time from base is: 240 ± 4 ms. The import time difference between this PR and base is: -0.4 ± 0.2 ms. The difference is not statistically significant (z = -1.66). Import time breakdownThe following import paths have shrunk:
|
Performance SLOsComparing candidate jsimpher/prompts-v1-langchain-auto-user-version-undefined (58ced0e) with baseline main (18a5e9a) 📈 Performance Regressions (1 suite)📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 4.338µs (SLO: <10.000µs 📉 -56.6%) vs baseline: +0.3% Memory: ✅ 37.670MB (SLO: <39.000MB -3.4%) vs baseline: +5.0% ✅ ospathbasename_noaspectTime: ✅ 1.088µs (SLO: <10.000µs 📉 -89.1%) vs baseline: -0.2% Memory: ✅ 37.729MB (SLO: <39.000MB -3.3%) vs baseline: +4.9% ✅ ospathjoin_aspectTime: ✅ 6.847µs (SLO: <10.000µs 📉 -31.5%) vs baseline: 📈 +10.8% Memory: ✅ 37.631MB (SLO: <39.000MB -3.5%) vs baseline: +4.6% ✅ ospathjoin_noaspectTime: ✅ 2.317µs (SLO: <10.000µs 📉 -76.8%) vs baseline: +0.7% Memory: ✅ 37.709MB (SLO: <39.000MB -3.3%) vs baseline: +4.8% ✅ ospathnormcase_aspectTime: ✅ 3.601µs (SLO: <10.000µs 📉 -64.0%) vs baseline: +1.5% Memory: ✅ 37.690MB (SLO: <39.000MB -3.4%) vs baseline: +4.8% ✅ ospathnormcase_noaspectTime: ✅ 0.584µs (SLO: <10.000µs 📉 -94.2%) vs baseline: +1.1% Memory: ✅ 37.690MB (SLO: <39.000MB -3.4%) vs baseline: +4.9% ✅ ospathsplit_aspectTime: ✅ 4.951µs (SLO: <10.000µs 📉 -50.5%) vs baseline: -0.5% Memory: ✅ 37.709MB (SLO: <39.000MB -3.3%) vs baseline: +5.0% ✅ ospathsplit_noaspectTime: ✅ 1.599µs (SLO: <10.000µs 📉 -84.0%) vs baseline: -0.5% Memory: ✅ 37.690MB (SLO: <39.000MB -3.4%) vs baseline: +5.0% ✅ ospathsplitdrive_aspectTime: ✅ 3.746µs (SLO: <10.000µs 📉 -62.5%) vs baseline: -0.5% Memory: ✅ 37.650MB (SLO: <39.000MB -3.5%) vs baseline: +4.9% ✅ ospathsplitdrive_noaspectTime: ✅ 0.711µs (SLO: <10.000µs 📉 -92.9%) vs baseline: +1.6% Memory: ✅ 37.749MB (SLO: <39.000MB -3.2%) vs baseline: +5.0% ✅ ospathsplitext_aspectTime: ✅ 4.643µs (SLO: <10.000µs 📉 -53.6%) vs baseline: +0.3% Memory: ✅ 37.690MB (SLO: <39.000MB -3.4%) vs baseline: +4.9% ✅ ospathsplitext_noaspectTime: ✅ 1.392µs (SLO: <10.000µs 📉 -86.1%) vs baseline: +0.3% Memory: ✅ 37.631MB (SLO: <39.000MB -3.5%) vs baseline: +4.7% 🟡 Near SLO Breach (4 suites)🟡 djangosimple - 30/30✅ appsecTime: ✅ 20.490ms (SLO: <22.300ms -8.1%) vs baseline: ~same Memory: ✅ 65.437MB (SLO: <67.000MB -2.3%) vs baseline: +4.8% ✅ exception-replay-enabledTime: ✅ 1.348ms (SLO: <1.450ms -7.0%) vs baseline: +0.7% Memory: ✅ 64.502MB (SLO: <67.000MB -3.7%) vs baseline: +4.7% ✅ iastTime: ✅ 20.479ms (SLO: <22.250ms -8.0%) vs baseline: ~same Memory: ✅ 65.470MB (SLO: <67.000MB -2.3%) vs baseline: +4.8% ✅ profilerTime: ✅ 15.187ms (SLO: <16.550ms -8.2%) vs baseline: -0.4% Memory: ✅ 53.828MB (SLO: <54.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ resource-renamingTime: ✅ 20.529ms (SLO: <21.750ms -5.6%) vs baseline: -0.2% Memory: ✅ 65.359MB (SLO: <67.000MB -2.4%) vs baseline: +4.7% ✅ span-code-originTime: ✅ 25.368ms (SLO: <28.200ms 📉 -10.0%) vs baseline: -0.1% Memory: ✅ 67.676MB (SLO: <69.500MB -2.6%) vs baseline: +5.0% ✅ tracerTime: ✅ 20.412ms (SLO: <21.750ms -6.2%) vs baseline: -0.2% Memory: ✅ 65.447MB (SLO: <67.000MB -2.3%) vs baseline: +4.8% ✅ tracer-and-profilerTime: ✅ 21.988ms (SLO: <23.500ms -6.4%) vs baseline: -0.5% Memory: ✅ 66.623MB (SLO: <67.500MB 🟡 -1.3%) vs baseline: +4.9% ✅ tracer-dont-create-db-spansTime: ✅ 19.215ms (SLO: <21.500ms 📉 -10.6%) vs baseline: -0.8% Memory: ✅ 65.449MB (SLO: <66.000MB 🟡 -0.8%) vs baseline: +4.7% ✅ tracer-minimalTime: ✅ 16.582ms (SLO: <17.500ms -5.2%) vs baseline: ~same Memory: ✅ 65.428MB (SLO: <66.000MB 🟡 -0.9%) vs baseline: +4.9% ✅ tracer-nativeTime: ✅ 20.486ms (SLO: <21.750ms -5.8%) vs baseline: -0.1% Memory: ✅ 71.316MB (SLO: <72.500MB 🟡 -1.6%) vs baseline: +4.8% ✅ tracer-no-cachesTime: ✅ 18.359ms (SLO: <19.650ms -6.6%) vs baseline: -0.3% Memory: ✅ 65.410MB (SLO: <67.000MB -2.4%) vs baseline: +4.7% ✅ tracer-no-databasesTime: ✅ 18.734ms (SLO: <20.100ms -6.8%) vs baseline: -0.3% Memory: ✅ 65.454MB (SLO: <67.000MB -2.3%) vs baseline: +4.9% ✅ tracer-no-middlewareTime: ✅ 20.170ms (SLO: <21.500ms -6.2%) vs baseline: ~same Memory: ✅ 65.431MB (SLO: <67.000MB -2.3%) vs baseline: +4.7% ✅ tracer-no-templatesTime: ✅ 20.214ms (SLO: <22.000ms -8.1%) vs baseline: -0.4% Memory: ✅ 65.513MB (SLO: <67.000MB -2.2%) vs baseline: +4.9% 🟡 errortrackingdjangosimple - 6/6✅ errortracking-enabled-allTime: ✅ 18.198ms (SLO: <19.850ms -8.3%) vs baseline: +0.8% Memory: ✅ 65.235MB (SLO: <66.500MB 🟡 -1.9%) vs baseline: +4.9% ✅ errortracking-enabled-userTime: ✅ 18.084ms (SLO: <19.400ms -6.8%) vs baseline: -0.2% Memory: ✅ 65.176MB (SLO: <66.500MB 🟡 -2.0%) vs baseline: +4.8% ✅ tracer-enabledTime: ✅ 18.221ms (SLO: <19.450ms -6.3%) vs baseline: +0.8% Memory: ✅ 65.313MB (SLO: <66.500MB 🟡 -1.8%) vs baseline: +4.9% 🟡 flasksimple - 18/18✅ appsec-getTime: ✅ 4.580ms (SLO: <4.750ms -3.6%) vs baseline: -0.4% Memory: ✅ 61.912MB (SLO: <65.000MB -4.8%) vs baseline: +4.8% ✅ appsec-postTime: ✅ 6.601ms (SLO: <6.750ms -2.2%) vs baseline: ~same Memory: ✅ 61.991MB (SLO: <65.000MB -4.6%) vs baseline: +4.9% ✅ appsec-telemetryTime: ✅ 4.594ms (SLO: <4.750ms -3.3%) vs baseline: +0.3% Memory: ✅ 62.010MB (SLO: <65.000MB -4.6%) vs baseline: +4.9% ✅ debuggerTime: ✅ 1.856ms (SLO: <2.000ms -7.2%) vs baseline: -0.1% Memory: ✅ 45.436MB (SLO: <47.000MB -3.3%) vs baseline: +4.8% ✅ iast-getTime: ✅ 1.861ms (SLO: <2.000ms -6.9%) vs baseline: +0.2% Memory: ✅ 42.408MB (SLO: <49.000MB 📉 -13.5%) vs baseline: +4.8% ✅ profilerTime: ✅ 1.911ms (SLO: <2.100ms -9.0%) vs baseline: +0.2% Memory: ✅ 46.478MB (SLO: <47.000MB 🟡 -1.1%) vs baseline: +4.9% ✅ resource-renamingTime: ✅ 3.364ms (SLO: <3.650ms -7.8%) vs baseline: ~same Memory: ✅ 52.219MB (SLO: <53.500MB -2.4%) vs baseline: +4.7% ✅ tracerTime: ✅ 3.357ms (SLO: <3.650ms -8.0%) vs baseline: +0.2% Memory: ✅ 52.278MB (SLO: <53.500MB -2.3%) vs baseline: +4.9% ✅ tracer-nativeTime: ✅ 3.358ms (SLO: <3.650ms -8.0%) vs baseline: +0.2% Memory: ✅ 58.211MB (SLO: <60.000MB -3.0%) vs baseline: +4.8% 🟡 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 2.971µs (SLO: <20.000µs 📉 -85.1%) vs baseline: +1.2% Memory: ✅ 32.067MB (SLO: <34.000MB -5.7%) vs baseline: +4.7% ✅ 1-count-metrics-100-timesTime: ✅ 205.264µs (SLO: <220.000µs -6.7%) vs baseline: +1.7% Memory: ✅ 32.145MB (SLO: <34.000MB -5.5%) vs baseline: +5.0% ✅ 1-distribution-metric-1-timesTime: ✅ 3.374µs (SLO: <20.000µs 📉 -83.1%) vs baseline: +2.4% Memory: ✅ 32.126MB (SLO: <34.000MB -5.5%) vs baseline: +4.8% ✅ 1-distribution-metrics-100-timesTime: ✅ 216.441µs (SLO: <220.000µs 🟡 -1.6%) vs baseline: +1.7% Memory: ✅ 32.106MB (SLO: <34.000MB -5.6%) vs baseline: +4.7% ✅ 1-gauge-metric-1-timesTime: ✅ 2.174µs (SLO: <20.000µs 📉 -89.1%) vs baseline: -0.7% Memory: ✅ 32.145MB (SLO: <34.000MB -5.5%) vs baseline: +4.7% ✅ 1-gauge-metrics-100-timesTime: ✅ 136.857µs (SLO: <150.000µs -8.8%) vs baseline: -0.3% Memory: ✅ 32.086MB (SLO: <34.000MB -5.6%) vs baseline: +4.7% ✅ 1-rate-metric-1-timesTime: ✅ 3.100µs (SLO: <20.000µs 📉 -84.5%) vs baseline: +1.0% Memory: ✅ 32.126MB (SLO: <34.000MB -5.5%) vs baseline: +4.9% ✅ 1-rate-metrics-100-timesTime: ✅ 217.266µs (SLO: <250.000µs 📉 -13.1%) vs baseline: +1.0% Memory: ✅ 32.067MB (SLO: <34.000MB -5.7%) vs baseline: +4.7% ✅ 100-count-metrics-100-timesTime: ✅ 20.302ms (SLO: <22.000ms -7.7%) vs baseline: -0.8% Memory: ✅ 32.145MB (SLO: <34.000MB -5.5%) vs baseline: +4.9% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.248ms (SLO: <2.300ms -2.3%) vs baseline: -0.8% Memory: ✅ 32.126MB (SLO: <34.000MB -5.5%) vs baseline: +4.9% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.418ms (SLO: <1.550ms -8.5%) vs baseline: +1.5% Memory: ✅ 32.165MB (SLO: <34.000MB -5.4%) vs baseline: +4.9% ✅ 100-rate-metrics-100-timesTime: ✅ 2.228ms (SLO: <2.550ms 📉 -12.6%) vs baseline: +0.9% Memory: ✅ 32.106MB (SLO: <34.000MB -5.6%) vs baseline: +4.7% ✅ flush-1-metricTime: ✅ 4.900µs (SLO: <20.000µs 📉 -75.5%) vs baseline: +9.5% Memory: ✅ 32.145MB (SLO: <34.000MB -5.5%) vs baseline: +5.1% ✅ flush-100-metricsTime: ✅ 175.037µs (SLO: <250.000µs 📉 -30.0%) vs baseline: ~same Memory: ✅ 32.204MB (SLO: <34.000MB -5.3%) vs baseline: +5.0% ✅ flush-1000-metricsTime: ✅ 2.117ms (SLO: <2.500ms 📉 -15.3%) vs baseline: -0.5% Memory: ✅ 32.952MB (SLO: <34.500MB -4.5%) vs baseline: +4.9%
|
jsimpher
force-pushed
the
jsimpher/prompts-v1-langchain-auto-user-version-undefined
branch
from
3a20b27 to
4ff86d7
Compare
jsimpher
changed the title
jsimpher
removed
the
changelog/no-changelog
jsimpher
deleted the
jsimpher/prompts-v1-langchain-auto-user-version-undefined
branch
…#14937) ## Description Remove setting version from auto instrumented prompt templates. We adjusted the way we interpret explicitly set versions. Auto instrumentation should not set one. ## Testing <!-- Describe your testing strategy or note what tests are included --> ## Risks <!-- Note any risks associated with this change, or "None" if no risks --> ## Additional Notes <!-- Any other information that would be helpful for reviewers --> (cherry picked from commit 8b7fe13)
|
The backport to To backport manually, run these commands in your terminal: # Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-3.17 3.17
# Navigate to the new working tree
cd .worktrees/backport-3.17
# Create a new branch
git switch --create backport-14937-to-3.17
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 8b7fe138dc2c50197d81128bef6821529cd66916
# Push it to GitHub
git push --set-upstream origin backport-14937-to-3.17
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-3.17Then, create a pull request where the |
…#14937) ## Description Remove setting version from auto instrumented prompt templates. We adjusted the way we interpret explicitly set versions. Auto instrumentation should not set one. ## Testing <!-- Describe your testing strategy or note what tests are included --> ## Risks <!-- Note any risks associated with this change, or "None" if no risks --> ## Additional Notes <!-- Any other information that would be helpful for reviewers --> (cherry picked from commit 8b7fe13)
…#14937) ## Description Remove setting version from auto instrumented prompt templates. We adjusted the way we interpret explicitly set versions. Auto instrumentation should not set one. ## Testing <!-- Describe your testing strategy or note what tests are included --> ## Risks <!-- Note any risks associated with this change, or "None" if no risks --> ## Additional Notes <!-- Any other information that would be helpful for reviewers --> (cherry picked from commit 8b7fe13)
2 participants
Description
Remove setting version from auto instrumented prompt templates.
We adjusted the way we interpret explicitly set versions. Auto instrumentation should not set one.
Testing
Risks
Additional Notes