Skip to content

iOS media handling changes #42

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 17 commits into
base: main
Choose a base branch
from
Open

Conversation

Miki-Session
Copy link
Collaborator

@Miki-Session Miki-Session commented May 9, 2025

  • Media sending functions generally push first and expect the file to be present
  • Adds custom util to push to an iOS Simulator's Downloads directory on disk
  • Adds custom partial image matching function to tap a media thumbnail based on similarity to the original image
  • Completes the changes brought in by Android 1.23.0 changes #40

// Checking Sent status on both platforms
await this.waitForTextElementToBePresent({
...new OutgoingMessageStatusSent(this).build(),
maxWait: 50000,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

50s?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be 20s now

// Simulators are cleared on start of a test run to minimize device bloat
// This has to be done before starting them because you can't erase booted simulators
// If a simulator is already running this fails silently
await runScriptAndLog(`xcrun simctl erase ${udid}`);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for many tests/simulators do we need to erase the data? Just wondering if there is a better way to do this, more reliably.
As in, we always push the file/media from the first device for each tests that need it, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, I think this may need to be removed. The enormous time bloat is far outweighing the problem of having a flaky locator for media.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm working on a different approach that could allow us to not rely on any set dates for accessibility IDs (which was the whole point of this refactor)
If that works out (🤞🏻) we might get away with having a simple cronjob on the CI that periodically wipes all iOS simulators, just to minimize device bloat.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed this, we no longer wipe iOS simulators on start

// Simulators are cleared on start of a test run to minimize device bloat
// This has to be done before starting them because you can't erase booted simulators
// If a simulator is already running this fails silently
await runScriptAndLog(`xcrun simctl erase ${udid}`);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, I think this may need to be removed. The enormous time bloat is far outweighing the problem of having a flaky locator for media.

}
await alice1.onIOS().clickOnByAccessibilityID('Photo, 25 March, 11:09 am', 1000);
await matchAndTapImage(
alice1.onIOS(),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this onIOS for?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, should we move this function to the DeviceWrapper class directly?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The partial image matching only needs to be done on iOS. Android can be wiped at runtime so we're still tapping the one and only image that's just been pushed.

I added the matchAndTapImage to the DeviceWrapper

@@ -20,6 +20,7 @@
"devDependencies": {
"@appium/execute-driver-plugin": "^3.0.1",
"@appium/images-plugin": "^3.0.17",
"@appium/opencv": "^3.0.9",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we cannot do the same with the image matching that playwright offers?
I guess not because

  • this will not just look for a buffer that looks like another one,
  • but will actually try to find in a buffer, a part that looks like another one. Is that it?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes this is partial image matching

const center = {
x: rect.x + Math.floor(rect.width / 2),
y: rect.y + Math.floor(rect.height / 2),
confidence: score,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the confidence field can be removed, right?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, removed

@Miki-Session Miki-Session changed the title iOS media handling changes WIP: iOS media handling changes May 30, 2025
@Miki-Session Miki-Session mentioned this pull request Jun 4, 2025
@Miki-Session Miki-Session dismissed burtonemily’s stale review June 16, 2025 03:29

github don't make me cry

@Miki-Session Miki-Session requested a review from Bilb June 16, 2025 03:29
@Miki-Session Miki-Session changed the title WIP: iOS media handling changes iOS media handling changes Jun 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants