Skip to content

Commit 7faa5db

Browse files
committed
Guard settings rails from commerce overreads
1 parent 6ac6f41 commit 7faa5db

File tree

4 files changed

+56
-4
lines changed

4 files changed

+56
-4
lines changed

docs/AI_DESIGN_SYSTEM_ROADMAP.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,7 @@ Current status:
119119
- local semantic cue families now also separate editor/canvas, catalog/store, profile/community, activity/feed, document/review, map/navigation, media/player, capture/scan, schedule/calendar, commerce/checkout, secure-access, support/help, and rewards/loyalty screenshots from those broader buckets, reducing more note/filename-driven non-OCR false positives without bundling built-in model dependencies
120120
- screenshot analysis now persists semantic-family labels plus confidence for those local cue families, and stronger raster-only ambiguity guards reject weak special-family matches when generic settings/control evidence dominates
121121
- screenshot analysis now also persists semantic-family rationale, competing family candidates, and review-needed flags so weak local matches are easier to inspect before they steer planning
122-
- raster-family structural scoring now also helps more distinctive non-OCR profile/document/editor/catalog/map/media/capture layouts plus support/help-style list flows resolve into the right semantic family without OCR text, surfaces checkout-style summary/price-rail rows as stronger reviewed commerce contenders, gives secure-access screens more weight when focused auth-gate structure aligns with explicit security cues, and stronger settings/paywall ambiguity guards still block more weak profile/reward/commerce/security family overreads from filenames or terse notes alone
122+
- raster-family structural scoring now also helps more distinctive non-OCR profile/document/editor/catalog/map/media/capture layouts plus support/help-style list flows resolve into the right semantic family without OCR text, surfaces checkout-style summary/price-rail rows as stronger reviewed commerce contenders, gives secure-access screens more weight when focused auth-gate structure aligns with explicit security cues, and stronger settings/paywall ambiguity guards still block more weak profile/reward/commerce/security family overreads from filenames, terse notes, or checkout-like settings/account rails alone
123123
- preview sessions now support manual semantic-family override/reset with session persistence, surfaced rationale/ambiguity diagnostics, and bulk family-review actions for flagged screenshots, so borderline local screenshot classification can be corrected faster during review without adding a built-in model dependency
124124
- reviewed semantic-family state can now also feed forward into deterministic replanning/materialization through `appframe_plan_variant_set` reviewed-analysis input and `appframe_rebuild_autopilot_session_from_review`, so saved review metadata is no longer preview-only
125125
- variant plans now emit explicit per-concept frame strategies plus per-screen/per-frame crop plans, including text-occupied-region avoidance and focal-point anchoring
@@ -223,7 +223,7 @@ This roadmap now has three concrete pieces underway:
223223
- screenshot understanding now also uses OCR/layout semantics to better read onboarding, paywall, settings, chat, workflow/discovery, and data-heavy dashboard/reporting screenshots instead of treating them as mostly generic dense screens
224224
- screenshot understanding now also derives raster-only occupied-region and semantic layout cues so non-OCR screenshots can still drive crop avoidance, copy direction, and role-aware planning deterministically
225225
- non-OCR local cues now also include stronger discovery/template-browse and workflow/action phrase banks plus ambiguity guards against settings/reporting false positives
226-
- local cue-family understanding now also detects editor/canvas, catalog/store, profile/community, activity/feed, document/review, map/navigation, media/player, capture/scan, schedule/calendar, commerce/checkout, secure-access, support/help, and rewards/loyalty screenshots so those flows stop collapsing into generic settings/detail/reporting/editor/media/paywall reads as often, and secure-access scoring now responds more strongly when explicit auth cues line up with focused centered-panel/CTA structure
226+
- local cue-family understanding now also detects editor/canvas, catalog/store, profile/community, activity/feed, document/review, map/navigation, media/player, capture/scan, schedule/calendar, commerce/checkout, secure-access, support/help, and rewards/loyalty screenshots so those flows stop collapsing into generic settings/detail/reporting/editor/media/paywall reads as often; secure-access scoring now responds more strongly when explicit auth cues line up with focused centered-panel/CTA structure, and settings/account evidence now blocks checkout-like right-rail commerce reads when explicit checkout cues are absent
227227
- individual composition breadth: dynamic individual concepts now materialize multi-device compositions, supporting screenshots, loupes, overlays, and palette-aware backgrounds instead of staying inside a single-device layout
228228
- recipe-specific composition/background reactions now respond more explicitly to onboarding, paywall, settings, chat, reporting, workflow, discovery, editor, profile, catalog, activity, document, map, media, capture, schedule, commerce, security, support, and reward-style screens instead of only generic frame/crop signals
229229
- panoramic planning/materialization now also emits tool-ribbon, profile-spotlight, activity-wave, folio-stack, browse-strip, route-arc, playback-marquee, capture-focus, timeline-band, checkout-lane, trust-shield, support-beacon, and reward-ribbon treatments plus background accents and family-aware pacing when those local cue families are present, so the strip reacts more clearly than generic proof/decorative systems alone

docs/NEXT_STEPS.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ AppFrame now has an initial autopilot pipeline implemented:
6161
- screenshot analysis now also broadens deterministic non-OCR understanding for workflow/action and discovery/browse screens, and local ambiguity guards reduce some raster-only false positives where wide panels previously collapsed into settings or reporting
6262
- screenshot analysis now also uses stronger local cue families for editor/canvas, catalog/store, profile/community, activity/feed, document/review, map/navigation, media/player, capture/scan, schedule/calendar, commerce/checkout, secure-access, support/help, and rewards/loyalty screenshots, widening deterministic understanding and cutting more raster-only settings/reporting/editor/media/paywall false positives without bundling built-in model dependencies
6363
- screenshot analysis now persists explicit semantic-family metadata plus confidence for those local cue families, and stricter raster-only ambiguity guards keep generic settings/control surfaces from over-triggering special family treatments off weak filename/note cues alone
64-
- screenshot analysis now also uses broader raster-family structural scoring for more distinctive non-OCR profile/document/editor/catalog/map/media/capture-like layouts, now gives support/help-style list flows stronger raster corroboration, surfaces checkout-style summary/price-rail rows as stronger reviewed commerce contenders, gives secure-access screens more weight when explicit auth cues align with focused centered-panel/CTA structure, and keeps checkout/paywall/settings ambiguity conservative so weak profile/reward/commerce/security hints do not override generic control or premium CTA structure as often
64+
- screenshot analysis now also uses broader raster-family structural scoring for more distinctive non-OCR profile/document/editor/catalog/map/media/capture-like layouts, now gives support/help-style list flows stronger raster corroboration, surfaces checkout-style summary/price-rail rows as stronger reviewed commerce contenders, gives secure-access screens more weight when explicit auth cues align with focused centered-panel/CTA structure, and keeps checkout/paywall/settings ambiguity conservative so weak profile/reward/commerce/security hints or checkout-like settings rails do not override generic control or premium CTA structure as often
6565
- dynamic individual planning/materialization now reacts more explicitly to onboarding, paywall, settings, chat, and reporting-style screens with role-aware composition/background strategies instead of only generic frameStrategy/cropPlan behavior
6666
- dynamic individual planning/materialization now also reacts more explicitly to workflow and discovery screens with dedicated composition/background treatments instead of folding them back into generic proof-grid behavior
6767
- dynamic planning/materialization now also reacts more explicitly to editor/profile/catalog/activity/document/map/media/capture/schedule/commerce/security/support/reward-style screens with dedicated individual backgrounds, copy guidance, family-aware panoramic pacing, and richer panoramic tool-ribbon / profile-spotlight / activity-wave / folio-stack / browse-strip / route-arc / playback-marquee / capture-focus / timeline-band / checkout-lane / trust-shield / support-beacon / reward-ribbon treatments
@@ -264,7 +264,7 @@ Recommended order:
264264
- [x] data-heavy dashboard/reporting
265265
- [x] workflow/action
266266
- [x] discovery/browse/template-library
267-
Status: OCR/layout semantics now improve these cases when text enrichment is available, raster/non-OCR deterministic logic now covers the same families plus workflow/discovery and local editor/profile/catalog/activity/document/map/media/capture/schedule/commerce/security/support/reward cue families, raster-family scoring now helps more structurally distinctive non-OCR profile/document/editor/catalog/map/media/capture screens plus support/help-style list flows resolve into the right semantic family even with generic filenames, checkout-style summary/price-rail rows now surface stronger reviewed commerce contenders, secure-access scoring now gets stronger when explicit auth/security cues align with focused auth-gate structure, and ambiguity guards still cut more settings/reporting/profile/reward/commerce/media/paywall/security false positives; broader scene-graph understanding is still incomplete.
267+
Status: OCR/layout semantics now improve these cases when text enrichment is available, raster/non-OCR deterministic logic now covers the same families plus workflow/discovery and local editor/profile/catalog/activity/document/map/media/capture/schedule/commerce/security/support/reward cue families, raster-family scoring now helps more structurally distinctive non-OCR profile/document/editor/catalog/map/media/capture screens plus support/help-style list flows resolve into the right semantic family even with generic filenames, checkout-style summary/price-rail rows now surface stronger reviewed commerce contenders, secure-access scoring now gets stronger when explicit auth/security cues align with focused auth-gate structure, and ambiguity guards still cut more settings/reporting/profile/reward/commerce/media/paywall/security false positives including settings/account screens with checkout-like right-rail structure; broader scene-graph understanding is still incomplete.
268268
- [x] Add raster-only occupied-region heuristics when OCR/text enrichment is absent.
269269
Status: screenshot analysis now infers top/bottom/left/right/center occupancy from local PNG structure, and those regions feed crop avoidance plus copy/planning guidance even without OCR sidecars.
270270
- [x] Add screenshot ordering inference from filenames, timestamps, and roles.

packages/mcp-server/src/tools/design-planning.test.ts

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -713,6 +713,41 @@ describe('design planning helpers', () => {
713713
expect(commerceAnalysis?.semanticFlavorAlternatives?.some((entry) => entry.flavor === 'commerce')).toBe(true);
714714
});
715715

716+
it('keeps settings toggle rails from auto-resolving as commerce without explicit checkout cues', async () => {
717+
const billingSettingsPath = await makePngFile('billing-settings.png', 120, 200, (x, y) => {
718+
if (y < 10) return [255, 251, 235, 255];
719+
if (x > 18 && x < 102 && y > 20 && y < 70) return [254, 215, 170, 255];
720+
if (
721+
((x > 18 && x < 90) && (y > 86 && y < 102))
722+
|| ((x > 18 && x < 90) && (y > 110 && y < 126))
723+
) {
724+
return [148, 163, 184, 255];
725+
}
726+
if (
727+
((x > 96 && x < 110) && (y > 86 && y < 102))
728+
|| ((x > 96 && x < 110) && (y > 110 && y < 126))
729+
) {
730+
return [30, 64, 175, 255];
731+
}
732+
if (x > 28 && x < 92 && y > 150 && y < 172) return [37, 99, 235, 255];
733+
return [255, 247, 237, 255];
734+
});
735+
736+
const analysis = await analyzeScreenshotSet([
737+
{
738+
path: billingSettingsPath,
739+
note: 'Billing settings, privacy controls, and notification toggles',
740+
},
741+
]);
742+
743+
const settingsAnalysis = analysis.find((entry) => entry.path === billingSettingsPath);
744+
expect(settingsAnalysis?.role).toBe('settings');
745+
expect(settingsAnalysis?.semanticFlavor).toBeUndefined();
746+
expect(settingsAnalysis?.semanticFlavorNeedsReview).toBe(true);
747+
expect(settingsAnalysis?.semanticFlavorAlternatives?.some((entry) => entry.flavor === 'commerce')).toBe(true);
748+
expect(settingsAnalysis?.semanticFlavorReason?.some((line) => line.includes('Settings/account evidence'))).toBe(true);
749+
});
750+
716751
it('rejects weak profile and reward family matches when generic settings structure dominates', async () => {
717752
const profileSettingsPath = await makePngFile('screen-4.png', 120, 200, (x, y) => {
718753
if (y < 28) return [248, 250, 252, 255];

packages/mcp-server/src/tools/design-planning.ts

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1038,6 +1038,23 @@ function inferSemanticFlavorDetails(args: {
10381038
};
10391039
}
10401040

1041+
if (
1042+
!hasTextInsights
1043+
&& args.role === 'settings'
1044+
&& (best.flavor === 'commerce' || (commerceScore > 0 && commerceScore >= best.score - 2))
1045+
&& signals.commerceSignalCount === 0
1046+
&& signals.settingsConflictCount >= 1
1047+
) {
1048+
return {
1049+
reason: [
1050+
...reason,
1051+
'Settings/account evidence outweighs checkout-like right-rail structure, so this stays generic until reviewed.',
1052+
],
1053+
alternatives,
1054+
needsReview: true,
1055+
};
1056+
}
1057+
10411058
if (
10421059
!hasTextInsights
10431060
&& args.role === 'paywall'

0 commit comments

Comments
 (0)