Tablet and Phone Interface Improvements

[hello-vinitha]

Description

This PR makes a series of changes to improve the tablet and phone view for the interface made in the temp/deployment branch. The tablet view should appear as a scaled down view from the desktop version and the mobile interface has a different layout to make the best use of space.

Currently there are some known issues that need be resolved:

• In the tablet customization mode some components disappear when another component is added or a component is moved.
• Button pad buttons sometimes overflow out of the panel depending on the size of the screen.

Testing procedure

• Test that components can be successfully added, removed and rearranged (desktop, tablet)
• Verify mobile responsiveness on different screen sizes (tablet & mobile)
• Verify the correct buttons are being highlighted blue/green when the joint is moving
• Verify the correct buttons are highlighted red when the respective joint is at it's limit
• Verify click-to-pregrasp works on (desktop & tablet)
• Verify that cameras and buttons are sized appropriately for the screen (i.e. buttons/camera are not overflowing or too small, etc.)
• Verify that all action modes work across desktop and mobile

Before opening a pull request

From the top-level of this repository, run:

• pre-commit run --all-files

To merge

• Squash & Merge

[mintyfreshv]

Device: phone interface

Issue: the play button and the controls under recordings. Please see screen capture.

[hello-vinitha]

To Do List based on @mintyfreshv's feedback:

• Movement recorder bugs with recording and deletion (@hello-vinitha)
• Do not allow page scrolling in x & y direction on tablet - this should solve both the joystick and customization menu issue (@hello-william)
• Fix placement of camera pan/tilt buttons when buttons underneath camera view are disabled -- see comment (@hello-william)
• Follow up with @mintyfreshv to investigate if click-to-pregrasp issue is due to not being able to solve IK or input issue on tablet vs. desktop. If due to former, open issue to track separately (@hello-vinitha)
• Fix z-index of button controls on mobile -- see comment (@hello-william)

[hello-william]

great job with this testing @mintyfreshv!

[hello-william]

hi @mintyfreshv i just pushed a fix for the unexpected scrolling that was happening on ipad

can u do a git pull and ./launch_interface.sh to verify the fix works?

[mintyfreshv]

(100% Zoom on Chrome)
(50% Zoom on Chrome)

Device: Desktop

The aspect ratio of the interface is off today. The camera views are too small at 100%. The views get better at 50%, but everything else becomes smaller.

[mintyfreshv]

ScreenRecording_06-12-2025.08-07-13_1.mov

Device: iPad

Issue: button pad overlay on interface is not working.

[mintyfreshv]

Device: iPad

Issue: as seen in the picture, when only the real sense camera view is present on the interface, the camera view is stretched out along with the button pad controls.

Although unlike our meeting with Ashley from ALS Northwest, the camera view here isn’t immensely as wide as when we were with her. However, the botton pads surely look funky.

[hello-william]

@hello-vinitha @mintyfreshv hmmm i'm kinda wondering what's the most efficient way to squash these bugs....... are you two going to meet IRL any time soon?

you can pull me on a call, and we can block out 2-3hrs do a bug squashing marathon... you would need your devices/workstation/#3002 there so we can do everything on the spot

[hello-vinitha]

Device: iPad

Issue: as seen in the picture, when only the real sense camera view is present on the interface, the camera view is stretched out along with the button pad controls.

Although unlike our meeting with Ashley from ALS Northwest, the camera view here isn’t immensely as wide as when we were with her. However, the botton pads surely look funky.

I was able to replicate this on my IPad but not on chrome dev tools @hello-william. I agree it might be easier for the three of us to meet to squash these bugs. I will coordinate.

[hello-vinitha]

2025-06-12.11-55-02.mp4

Issue: Elements disappear in the when buttons are moved in customization mode.
Device: Desktop
Browser: Chrome
Robot: 3030

[hello-vinitha]

• Fix aspect ratio for camera views on tablet (@hello-william)
  • Follow up with @mintyfreshv to investigate if Tablet and Phone Interface Improvements #135 (comment) is due to not being able to solve IK or input issue on tablet vs. desktop. If due to former, open issue to track separately (@hello-vinitha)
• Fix elements disappearing when rearranging components in customization mode (@hello-vinitha)
• Make sure onLeave function is disabled for click-click on tablet/mobile (@hello-vinitha)
• Change onClick event handler to onPointerDown (@hello-vinitha @hello-william)
• Fix z-index of button controls on mobile -- Tablet and Phone Interface Improvements #135 (comment) (@hello-william)