[QA-1126] Adding sanity check against real devices for Prod builds #5526

ifernandezdiaz · 2025-07-14T20:11:20Z

🎟️ Tracking

📔 Objective

This PR streamlines and enhances our CI/CD pipeline by allowing the execution of sanity checks on real devices in our main build workflow.
It includes all necessary setup, secret retrieval, and artifact management to support E2E testing against real devices using SauceLabs.

📸 Screenshots

⏰ Reminders before review

Contributor guidelines followed
All formatters and local linters executed and passed
Written new unit and / or integration tests where applicable
Protected functional changes with optionality (feature flags)
Used internationalization (i18n) for all UI strings
CI builds passed
Communicated to DevOps any deployment requirements
Updated any necessary documentation (Confluence, contributing docs) or informed the documentation team

🦮 Reviewer guidelines

👍 (:+1:) or similar for great changes
📝 (:memo:) or ℹ️ (:information_source:) for notes or general info
❓ (:question:) for questions
🤔 (:thinking:) or 💭 (:thought_balloon:) for more open inquiry that's not quite a confirmed issue and could potentially benefit from discussion
🎨 (:art:) for suggestions / improvements
❌ (:x:) or ⚠️ (:warning:) for more significant problems or concerns needing attention
🌱 (:seedling:) or ♻️ (:recycle:) for future improvements or indications of technical debt
⛏ (:pick:) for minor or nitpick changes

This reverts commit 4b2f0da.

codecov · 2025-07-18T14:47:45Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.40%. Comparing base (22114d5) to head (0aafc52).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5526   +/-   ##
=======================================
  Coverage   84.40%   84.40%           
=======================================
  Files         719      719           
  Lines       54764    54765    +1     
  Branches     7528     7528           
=======================================
+ Hits        46226    46227    +1     
  Misses       5897     5897           
  Partials     2641     2641

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

SaintPatrck

Can the androidTest files be put in com.x8bit.bitwarden.* packages? I.e., com.x8bit.bitwarden.data.TestData.kt and com.x8bit.bitwarden.e2e.*.

app/build.gradle.kts

SaintPatrck · 2025-07-22T17:24:26Z

app/build.gradle.kts

+    implementation(libs.androidx.uiautomator)
+    implementation(libs.androidx.espresso.core)
+    implementation(libs.androidx.junit.ktx)
+    implementation(libs.androidx.ui.test.junit4.android)


Can we remove these as implementation dependencies?

For some reason, the build fails if I remove those dependencies 🤔

I think he want's them to be an androidTestImplementation to avoid having these test dependencies in the non-test builds

Yeah, but those deps are already present as androidTestImplementation in lines 302-306.
Apart from that, we need those deps in non-test builds since we aim to test the release build on real devices

🤔 We do not want to ship with unnecessary dependencies like this. Is there a specific error you are seeing?

I couldn't find a way to build the required testApp without those deps 🫤

Maybe we need a new variant for this.

app/build.gradle.kts

.github/workflows/build.yml

app/src/androidTest/kotlin/e2e/pageObjects/Page.kt

app/src/androidTest/kotlin/e2e/pageObjects/login/EnvironmentSettingsPage.kt

app/src/androidTest/kotlin/e2e/pageObjects/login/LoginPage.kt

gradle/libs.versions.toml

david-livefront · 2025-08-04T18:03:26Z

app/proguard-rules.pro

+# Keep Kotlin standard library classes
+-keep class kotlin.** { *; }
+-keep class kotlinx.** { *; }
+-keep class kotlin.io.** { *; }


Why do we need this?

Are we obfuscating test code?

SaintPatrck · 2025-10-07T19:04:43Z

.github/workflows/build.yml

+      # TODO: test if bundle exec fastlane assembleTestApk works and replace this step
+      # - name: Sign and rename test APK
+      #   if: ${{ (matrix.variant == 'prod') && (matrix.artifact == 'apk') }}
+      #   env:
+      #     _TEST_APK_PATH: app/build/outputs/apk/androidTest/standard/release/com.x8bit.bitwarden-standard-release-androidTest.apk
+      #     _TEST_APK_SIGNED_PATH: app/build/outputs/apk/androidTest/standard/release/com.x8bit.bitwarden-test.apk
+      #     _PLAY_KEYSTORE_PASSWORD: ${{ steps.get-kv-secrets.outputs.PLAY-KEYSTORE-PASSWORD }}
+      #     _PLAY_KEYSTORE_ALIAS: ${{ steps.get-kv-secrets.outputs.PLAY-KEYSTORE-ALIAS }}
+      #   run: |
+      #     $ANDROID_SDK_ROOT/build-tools/34.0.0/apksigner sign \
+      #       --ks keystores/app_play-keystore.jks \
+      #       --ks-key-alias bitwarden \
+      #       --ks-pass pass:$_PLAY_KEYSTORE_PASSWORD \
+      #       --key-pass pass:$_PLAY_KEYSTORE_PASSWORD \
+      #       $_TEST_APK_PATH
+      #     mv $_TEST_APK_PATH $_TEST_APK_SIGNED_PATH
+


Is this TODO and commented code still needed? If so, can we associate it to a ticket so it isn't forgotten?

SaintPatrck · 2025-10-07T19:16:42Z

.github/workflows/build.yml

+  test-device:
+    name: Test device
+    needs: publish_playstore
+    uses: bitwarden/android/.github/workflows/test-device.yml@QA-1126b/adding-native-sanity-test #TODO replace branch with main before merging


This TODO needs to be addressed.

SaintPatrck · 2025-10-07T19:21:56Z

.github/workflows/test-device.yml

 permissions:
  contents: read
+  actions: read
+  checks: write


checks: write is only needed to create or update status checks. Neither are happening directly in the job, that I can see. I don't think this permission is needed.

ifernandezdiaz added 30 commits June 9, 2025 16:06

Creating base e2e test for Real Device validations

815c739

Creating base e2e test for Real Device validations

ed5b752

Implementing page object pattern

5b012e1

Using Compose Testing + Espresso

a6ef2ea

Merge branch 'main' into QA-1126b/adding-native-sanity-test

047b762

Updating code to read credentials from an external json file

2af019e

Adding missing steps to test-device workflow

6181378

Merge branch 'main' into QA-1126b/adding-native-sanity-test

772245c

Adding missing steps to retrieve secrets from Azure

a2ed706

Enabling release test build type

2856545

Merge branch 'main' into QA-1126b/adding-native-sanity-test

58c98fa

Pulling SM creds from Azure

a44a8dc

Updating Azure creds to pull SM secrets

b0fb071

Merge branch 'main' into QA-1126b/adding-native-sanity-test

26cb4ce

Adding testData file

a371efb

Installing saucectl via npm

b8172f9

Pulling SauceLabs creds from GH secrets

4b2f0da

Revert "Pulling SauceLabs creds from GH secrets"

6b2ac29

This reverts commit 4b2f0da.

Fixing SauceLabs creds retrieval

22644ab

Fixing testApp filename

5fc3c02

Signing testApp

e9afd92

Signing testApp

a6bd9b6

Signing testApp

c6f8fe2

Fixing importing issues during test execution

9a724d8

Updating the way we read json data

a5721f3

Merge branch 'main' into QA-1126b/adding-native-sanity-test

e20c814

Adding steps to enable screen recording

7bedb1f

Removing steps to test faster on Device Farm

85f0bd9

Increasing timeout

d3d02f4

Decreasing timeout

cab4461

ifernandezdiaz force-pushed the QA-1126b/adding-native-sanity-test branch from c6058a3 to 62b9088 Compare July 15, 2025 15:47

ifernandezdiaz and others added 2 commits July 18, 2025 11:09

fixing static analisys issues

7a98555

Merge branch 'main' into QA-1126b/adding-native-sanity-test

c4ebde7

ifernandezdiaz added 2 commits July 21, 2025 15:19

Merge branch 'main' into QA-1126b/adding-native-sanity-test

be325eb

Merge branch 'main' into QA-1126b/adding-native-sanity-test

b6e90e4

SaintPatrck reviewed Jul 22, 2025

View reviewed changes

ifernandezdiaz and others added 3 commits July 22, 2025 16:52

Adding suggestions

b1a74c6

Adding package structure to androidTest folders

a6efb31

Merge branch 'main' into QA-1126b/adding-native-sanity-test

b74d3b1

ifernandezdiaz requested a review from SaintPatrck July 25, 2025 12:05

Merge branch 'main' into QA-1126b/adding-native-sanity-test

ab481d2

vvolkgang force-pushed the QA-1126b/adding-native-sanity-test branch from 1f61ab4 to a4b464c Compare July 31, 2025 16:31

Refactor - move test apk assembly to build job

f803752

vvolkgang force-pushed the QA-1126b/adding-native-sanity-test branch from a4b464c to f803752 Compare July 31, 2025 16:32

Adding missing permissions to test-device workflow call

bb8dda4

ifernandezdiaz force-pushed the QA-1126b/adding-native-sanity-test branch from 7d32d32 to bb8dda4 Compare August 4, 2025 17:48

david-livefront reviewed Aug 4, 2025

View reviewed changes

ifernandezdiaz added 2 commits August 8, 2025 09:58

Moving test-app creation to fastlane

12edccc

Updating branch

15b5b86

ifernandezdiaz force-pushed the QA-1126b/adding-native-sanity-test branch from eabd248 to 2cb6a6a Compare August 8, 2025 15:52

Fixing build issues

da9b60f

ifernandezdiaz force-pushed the QA-1126b/adding-native-sanity-test branch from 2cb6a6a to da9b60f Compare August 8, 2025 16:57

Merge branch 'main' into QA-1126b/adding-native-sanity-test

9bf3d1e

vvolkgang marked this pull request as draft September 4, 2025 16:48

Updating branch

0aafc52

SaintPatrck reviewed Oct 7, 2025

View reviewed changes

ifernandezdiaz marked this pull request as ready for review October 7, 2025 19:07

SaintPatrck reviewed Oct 7, 2025

View reviewed changes

[QA-1126] Adding sanity check against real devices for Prod builds #5526

Are you sure you want to change the base?

[QA-1126] Adding sanity check against real devices for Prod builds #5526

Uh oh!

Conversation

ifernandezdiaz commented Jul 14, 2025

🎟️ Tracking

📔 Objective

📸 Screenshots

⏰ Reminders before review

🦮 Reviewer guidelines

Uh oh!

codecov bot commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

SaintPatrck left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

david-livefront Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov bot commented Jul 18, 2025 •

edited

Loading

david-livefront Aug 4, 2025 •

edited

Loading