Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add bidirectional clipboard for iOS Simulator, support multiline paste, show toast when copying/pasting #936

Merged
merged 6 commits into from
Feb 12, 2025
Merged

Add bidirectional clipboard for iOS Simulator, support multiline paste, show toast when copying/pasting #936

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4f2828f
support bidirectional keyboard on iOS, fix cmd+v on Android
balins Feb 6, 2025
39b7592
handle multiline pastes
balins Feb 10, 2025
3087701
show notification on copy/paste
balins Feb 11, 2025
9d2b9c6
update simulator server to handle multiline paste
balins Feb 11, 2025
16873fa
move `showToast` to `utils`
balins Feb 12, 2025
12c02d7
show "copied" toast despite Android clipboard is automatically synced
balins Feb 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions packages/vscode-extension/src/common/Project.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,7 @@ export interface ProjectInterface {
dispatchTouches(touches: Array<TouchPoint>, type: "Up" | "Move" | "Down"): Promise<void>;
dispatchKeyPress(keyCode: number, direction: "Up" | "Down"): Promise<void>;
dispatchPaste(text: string): Promise<void>;
dispatchCopy(): Promise<void>;
inspectElementAt(
xRatio: number,
yRatio: number,
Expand Down
6 changes: 5 additions & 1 deletion packages/vscode-extension/src/devices/AndroidEmulatorDevice.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -565,7 +565,11 @@ export class AndroidEmulatorDevice extends DeviceBase {
}

async sendBiometricAuthorization(isMatch: boolean) {
// TO DO: implement android biometric authorization
// TODO: implement android biometric authorization
}

async getClipboard() {
// No need to copy clipboard, Android Emulator syncs it for us whenever a user clicks on 'Copy'
}
}

Expand Down
40 changes: 37 additions & 3 deletions packages/vscode-extension/src/devices/DeviceBase.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,24 @@ import { AppPermissionType, DeviceSettings, TouchPoint } from "../common/Project
import { DeviceInfo, DevicePlatform } from "../common/DeviceManager";
import { tryAcquiringLock } from "../utilities/common";

const LEFT_META_HID_CODE = 0xe3;
const RIGHT_META_HID_CODE = 0xe7;
const V_KEY_HID_CODE = 0x19;
const C_KEY_HID_CODE = 0x06;

export abstract class DeviceBase implements Disposable {
protected preview: Preview | undefined;
private previewStartPromise: Promise<string> | undefined;
private acquired = false;
private pressingLeftMetaKey = false;
private pressingRightMetaKey = false;

abstract get lockFilePath(): string;

abstract bootDevice(deviceSettings: DeviceSettings): Promise<void>;
abstract changeSettings(settings: DeviceSettings): Promise<boolean>;
abstract sendBiometricAuthorization(isMatch: boolean): Promise<void>;
abstract getClipboard(): Promise<string | void>;
abstract installApp(build: BuildResult, forceReinstall: boolean): Promise<void>;
abstract launchApp(build: BuildResult, metroPort: number, devtoolsPort: number): Promise<void>;
abstract makePreview(): Preview;
Expand Down Expand Up @@ -96,11 +104,37 @@ export abstract class DeviceBase implements Disposable {
}

public sendKey(keyCode: number, direction: "Up" | "Down") {
this.preview?.sendKey(keyCode, direction);
// iOS simulator has a buggy behavior when sending cmd+V key combination.
// It sometimes triggers paste action but with a very low success rate.
// Other times it kicks in before the pasteboard is filled with the content
// therefore pasting the previously copied content instead.
// As a temporary workaround, we disable sending cmd+V as key combination
// entirely to prevent this buggy behavior. Users can still paste content
// using the context menu method as they'd do on an iOS device.
// This is not an ideal workaround as people may still trigger cmd+v by
// pressing V first and then cmd, but it is good enough to filter out
// the majority of the noisy behavior since typically you press cmd first.
// Similarly, when pasting into Android Emulator, cmd+V has results in a
// side effect of typing the letter "v" into the text field (the same
// applies to cmd+C).
if (keyCode === LEFT_META_HID_CODE) {
this.pressingLeftMetaKey = direction === "Down";
} else if (keyCode === RIGHT_META_HID_CODE) {
this.pressingRightMetaKey = direction === "Down";
}

if (
(this.pressingLeftMetaKey || this.pressingRightMetaKey) &&
(keyCode === C_KEY_HID_CODE || keyCode === V_KEY_HID_CODE)
) {
// ignore sending C and V when meta key is pressed
} else {
this.preview?.sendKey(keyCode, direction);
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why, did you move this code to deviceBase I understand that is does not break android, as you pointed out in your comment, but it is still an ios specific workaround is it not?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It actually applies to Android as well, otherwise it result in typing 'ccccc' or 'vvvvv' when trying to copy/paste to Android. And since it is a workaround for iOS and Android problems as well, I've elevated it higher

}

public async sendPaste(text: string) {
return this.preview?.sendPaste(text);
public async sendClipboard(text: string) {
return this.preview?.sendClipboard(text);
}

async startPreview() {
Expand Down
44 changes: 10 additions & 34 deletions packages/vscode-extension/src/devices/IosSimulatorDevice.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,6 @@ import { AppPermissionType, DeviceSettings, Locale } from "../common/Project";
import { EXPO_GO_BUNDLE_ID, fetchExpoLaunchDeeplink } from "../builders/expoGo";
import { IOSBuildResult } from "../builders/buildIOS";

const LEFT_META_HID_CODE = 0xe3;
const RIGHT_META_HID_CODE = 0xe7;
const V_KEY_HID_CODE = 0x19;

interface SimulatorInfo {
availability?: string;
state?: string;
Expand Down Expand Up @@ -218,44 +214,24 @@ export class IosSimulatorDevice extends DeviceBase {
]);
}

private pressingLeftMetaKey = false;
private pressingRightMetaKey = false;

public sendKey(keyCode: number, direction: "Up" | "Down"): void {
// iOS simulator has a buggy behavior when sending cmd+V key combination.
// It sometimes triggers paste action but with a very low success rate.
// Other times it kicks in before the pasteboard is filled with the content
// therefore pasting the previously compied content instead.
// As a temporary workaround, we disable sending cmd+V as key combination
// entirely to prevent this buggy behavior. Users can still paste content
// using the context menu method as they'd do on an iOS device.
// This is not an ideal workaround as people may still trigger cmd+v by
// pressing V first and then cmd, but it is good enough to filter out
// the majority of the noisy behavior since typically you press cmd first.
if (keyCode === LEFT_META_HID_CODE) {
this.pressingLeftMetaKey = direction === "Down";
} else if (keyCode === RIGHT_META_HID_CODE) {
this.pressingRightMetaKey = direction === "Down";
}
if ((this.pressingLeftMetaKey || this.pressingRightMetaKey) && keyCode === V_KEY_HID_CODE) {
// ignore sending V when meta key is pressed
} else {
this.preview?.sendKey(keyCode, direction);
}
public async sendClipboard(text: string) {
const deviceSetLocation = getOrCreateDeviceSet(this.deviceUDID);
await exec("xcrun", ["simctl", "--set", deviceSetLocation, "pbcopy", this.deviceUDID], {
input: text,
});
}

public async sendPaste(text: string) {
public async getClipboard() {
const deviceSetLocation = getOrCreateDeviceSet(this.deviceUDID);
const subprocess = exec("xcrun", [
const { stdout } = await exec("xcrun", [
"simctl",
"--set",
deviceSetLocation,
"pbcopy",
"pbpaste",
this.deviceUDID,
]);
subprocess.stdin?.write(text);
subprocess.stdin?.end();
await subprocess;

return stdout;
}

private async changeLocale(newLocale: Locale): Promise<boolean> {
Expand Down
6 changes: 4 additions & 2 deletions packages/vscode-extension/src/devices/preview.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,9 @@ export class Preview implements Disposable {
this.subprocess?.stdin?.write(`key ${direction} ${keyCode}\n`);
}

public async sendPaste(text: string) {
this.subprocess?.stdin?.write(`paste ${text}\n`);
public sendClipboard(text: string) {
// This is bad, but prevents simulator server going crazy with multiline pastes
// If we want to support multiline pastes we need to change the communication protocol
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what options do we have in regard to other protocols?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can either switch protocol completely, like move to GRPC or stay with communication via stdin-stdout, but add some markers for multiline commands, like this:

paste >>>STARTSIMSERVERPASTEhere is the
actual
multiline content
pasted by
the user<<<ENDSIMSERVERPASTE

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened a PR in simulator-server to handle this properly software-mansion-labs/simulator-server#222

this.subprocess?.stdin?.write(`paste ${text.replace(/(?:\r\n|\r|\n)/g, " ")}\n`);
}
}
8 changes: 6 additions & 2 deletions packages/vscode-extension/src/project/deviceSession.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -310,8 +310,12 @@ export class DeviceSession implements Disposable {
this.device.sendKey(keyCode, direction);
}

public sendPaste(text: string) {
return this.device.sendPaste(text);
public sendClipboard(text: string) {
return this.device.sendClipboard(text);
}

public async getClipboard() {
return this.device.getClipboard();
}

public inspectElementAt(
Expand Down
11 changes: 9 additions & 2 deletions packages/vscode-extension/src/project/project.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { EventEmitter } from "stream";
import os from "os";
import { Disposable, commands, workspace, window, DebugSessionCustomEvent } from "vscode";
import { env, Disposable, commands, workspace, window, DebugSessionCustomEvent } from "vscode";
import _ from "lodash";
import stripAnsi from "strip-ansi";
import { minimatch } from "minimatch";
Expand Down Expand Up @@ -253,7 +253,14 @@ export class Project
//#endregion

async dispatchPaste(text: string) {
await this.deviceSession?.sendPaste(text);
await this.deviceSession?.sendClipboard(text);
}

async dispatchCopy() {
const text = await this.deviceSession?.getClipboard();
if (text) {
env.clipboard.writeText(text);
}
}

onBundleError(): void {
Expand Down
11 changes: 8 additions & 3 deletions packages/vscode-extension/src/webview/components/Preview.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -347,19 +347,24 @@ function Preview({
}, []);

useEffect(() => {
function dispatchPaste(e: ClipboardEvent) {
function synchronizeClipboard(e: ClipboardEvent) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should inform user somehow that thay recived a clipboard?

Like in the example video:

Screen.Recording.2025-02-10.at.09.40.52.mov

I know that usually cmd+c does not trigger such a microinteraction, but this is a very specific case in witch user does not copy highlithed text but rather the contents of a clipboard on a simulated device.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PS. I don't necessary think it needs to be part of this PR

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me, a small modal "device clipboard copied to host"/"host clipboard pasted to device" appearing on copy/paste would be sufficient

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a toast notification for copy/paste

copypaste.mov

if (document.activeElement === wrapperDivRef.current) {
e.preventDefault();

const text = e.clipboardData?.getData("text");
if (text) {
project.dispatchPaste(text);
} else {
project.dispatchCopy();
}
}
}
addEventListener("paste", dispatchPaste);

addEventListener("paste", synchronizeClipboard);
addEventListener("copy", synchronizeClipboard);
return () => {
removeEventListener("paste", dispatchPaste);
removeEventListener("paste", synchronizeClipboard);
removeEventListener("copy", synchronizeClipboard);
};
}, [project]);

Expand Down