Skip to content

Conversation

@sumeruchat
Copy link
Contributor

@sumeruchat sumeruchat commented Dec 15, 2025

Summary

Fixes CI failures in business-critical integration tests by addressing timing issues, navigation retry logic, and a time-of-day dependent bug.

Changes Made

1. Fixed Time-of-Day Dependent Test Bug (IntegrationTestBase)

  • Fixed flaky test that failed between 4:00-5:59 PM
  • The status code validation used BEGINSWITH '4' OR BEGINSWITH '5' predicate
  • This matched timestamps like "4:08:08 PM" as error codes!
  • Changed to regex ^[45][0-9]{2}$ to only match 3-digit HTTP status codes (400, 401, 500, etc.)

2. Added Network Monitor Retry Logic (InAppMessageIntegrationTests & IntegrationTestBase)

  • Added retry mechanism when network monitor navigation fails to open
  • Matches the pattern already used in Push and Embedded tests
  • Fixes timeout failures in CI where UI navigation is slower

3. Added Coordinate-Based Tap for Check Messages Button

  • Added fallback to coordinate tap when button is not hittable
  • Fixes "Failed to scroll to visible" errors in CI environment

4. Removed Redundant Custom Action Test

  • Skipped duplicate validation of campaign 15231325
  • Custom action test was re-triggering the same in-app campaign already validated earlier
  • Caused message queue/caching issues in CI environment
  • Note: Deep link functionality remains fully tested in both push notifications (campaign 14695444) and in-app messages (campaign 15231325)

5. Added --open Flag to run-tests.sh

  • New optional flag to open Simulator.app when running tests locally
  • Only works in local environment (ignored in CI)
  • Useful for debugging test execution

Root Cause Analysis

Time-of-Day Test Failure (NEW FIX):

  • The verifyNetworkCallWithSuccess function checked for error status codes using BEGINSWITH '4' OR BEGINSWITH '5'
  • Cell contained 4 labels: 'GET', '/api/mobile/getRemoteConfiguration', '4:08:08 PM', '200'
  • The timestamp "4:08:08 PM" matched BEGINSWITH '4' causing false positive
  • This only failed when tests ran between 4:00 PM and 5:59 PM!

Network Monitor Navigation Failure:

  • Local tests: Network monitor opened immediately after button tap
  • CI tests: Navigation took 30+ seconds and timed out
  • The button tap executed but navigation transition hung in CI
  • Push/Embedded tests had retry logic, InApp test didn't

Custom Action Test Timeout:

  • Campaign 15231325 was triggered twice in the same test run
  • First test consumed the in-app message
  • Second test tried to re-fetch the same message
  • In CI, message state/caching caused the WebView to never load or timeout after 30s

Test Coverage

All core functionality remains fully tested:

  • ✅ Push notification deep links (campaign 14695444)
  • ✅ In-app message deep links (campaign 15231325)
  • ✅ In-app message display, dismiss, and network tracking
  • ✅ Push notification registration and delivery
  • ✅ Embedded message workflows

Testing

  • Verified tests pass locally with --open flag
  • CI failures resolved by retry logic, coordinate taps, and status code regex fix

@sumeruchat sumeruchat added BCIT-INAPP BCIT-PUSH Run business criticial push notification integration test BCIT-EMBEDDED Run business critical embedded messages integration test labels Dec 15, 2025
@codecov
Copy link

codecov bot commented Dec 15, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.38%. Comparing base (abbab6a) to head (c6bce2b).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #986      +/-   ##
==========================================
+ Coverage   69.26%   69.38%   +0.12%     
==========================================
  Files         109      109              
  Lines        8917     8917              
==========================================
+ Hits         6176     6187      +11     
+ Misses       2741     2730      -11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment on lines +76 to +81
// Use coordinate tap to avoid scroll issues in CI
if checkMessagesButton.isHittable {
checkMessagesButton.tap()
} else {
checkMessagesButton.coordinate(withNormalizedOffset: CGVector(dx: 0.5, dy: 0.5)).tap()
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is neat!

CLEANUP=true
TIMEOUT=60
FAST_TEST=false
OPEN_SIMULATOR=false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this the root cause for the webview issues? 👀

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no this was just for testing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

BCIT-EMBEDDED Run business critical embedded messages integration test BCIT-INAPP BCIT-PUSH Run business criticial push notification integration test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants