Skip to content

Tablet and Phone Interface Improvements #135

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 74 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
e77b1c6
made button grid accessible for voice control
hello-vinitha May 23, 2024
cf148ac
created custom button grid for voice control
hello-vinitha May 23, 2024
3371ea7
made default action mode step actions
hello-vinitha May 23, 2024
37363b1
merged master
hello-vinitha Sep 3, 2024
cd2d350
Merge branch 'master' into feature/voice
hello-vinitha Sep 26, 2024
e9d8d07
bugfixes and scrolling
hello-vinitha Oct 2, 2024
e9a66e7
updats to ensure that browser view persists on tablets
hello-vinitha Oct 31, 2024
077ac3a
added initial voice commands
hello-vinitha Oct 31, 2024
d4eb5f4
update package dependencies
hello-vinitha Nov 1, 2024
25fdd14
handle click-click edge cases; added more functions
hello-vinitha Nov 2, 2024
8aed6d2
remove voice control from customization panel
hello-vinitha Nov 2, 2024
18278bf
added use of FastSAM to detect objects and auto select grasping point
hello-vinitha Oct 31, 2024
4f21fd7
updats to ensure that browser view persists on tablets
hello-vinitha Oct 31, 2024
8c94fd1
handle click-click edge cases; added more functions
hello-vinitha Nov 2, 2024
d875e4b
Added voice control for selecting detected object
hello-vinitha Nov 2, 2024
8ced5fd
added voice functionality for pregrasp
hello-vinitha Nov 2, 2024
da8ebf3
resolve merge conflicts
hello-vinitha Nov 2, 2024
01dae25
minor fixes for tablet voice recognition
hello-vinitha Nov 2, 2024
1174de4
change voice input to lowercase; remove collision detection for arm a…
hello-vinitha Nov 4, 2024
a8ba72d
improving voice control
hello-vinitha Nov 5, 2024
a918fae
added more functions to voice
hello-vinitha Nov 5, 2024
b05a18e
updates from home deployment
hello-vinitha Nov 14, 2024
30206bd
added buttons for feeding mode
hello-vinitha Nov 20, 2024
fa1be70
upated buttons
hello-vinitha Nov 20, 2024
ff278b7
updated buttons and battery guage
hello-vinitha Dec 16, 2024
91cb517
resizing for chrome
hello-vinitha Dec 17, 2024
e47f0d5
improved accessibility for voice control and chrome
hello-vinitha Dec 18, 2024
dc5ee38
improvements for phone accessibility
hello-vinitha Jan 16, 2025
503c230
mobile homing
hello-vinitha Feb 3, 2025
488d556
Added quick actions to recordings; added map to mobile
hello-vinitha Feb 4, 2025
8b0fc97
updates for mobile compatibility
hello-vinitha Feb 4, 2025
5515f94
Merge branch 'temp/deployment' of https://github.com/hello-robot/stre…
hello-vinitha Feb 4, 2025
26d3fac
button size adjustment for mobile
hello-vinitha Feb 7, 2025
59c10d0
telemetry logging
hello-vinitha Feb 24, 2025
fb432d3
log home and map button clicks for telemetry
hello-vinitha Feb 24, 2025
43bd920
added modal to enable recording
hello-vinitha Feb 26, 2025
d0a1b65
Merge remote-tracking branch 'origin/master' into temp/deployment
hello-vinitha Feb 26, 2025
77ae27e
fixed merge conflict
hello-vinitha Feb 26, 2025
04efbff
Merge branch 'temp/deployment' of https://github.com/hello-robot/stre…
hello-vinitha Feb 26, 2025
766fb23
Merge branch 'temp/deployment' into feature/mobile
hello-vinitha Apr 30, 2025
e93c7bf
removed unused buttons
hello-vinitha Apr 30, 2025
d4600d5
tweak a number of things to ensure that buttongrid fits
wle8300 May 16, 2025
618ef8b
minor visual tweaks
wle8300 May 19, 2025
9b2aca7
disable :hover styles for devices that dont have hover state
wle8300 May 27, 2025
7ab8608
Merge branch 'master' of https://github.com/hello-robot/stretch_web_t…
hello-vinitha May 29, 2025
000944f
fixed click-to-pregrasp to handle mouse and touch events
hello-vinitha May 29, 2025
b6fdfbd
Merge branch 'feature/mobile' of https://github.com/hello-robot/stret…
hello-vinitha May 29, 2025
70932f1
running pre-commit
hello-vinitha May 29, 2025
4bfe901
hotfix issue with resizing viewport
wle8300 May 29, 2025
638ba57
fix eslint error
wle8300 May 29, 2025
2521cc9
hotfix: eol error
wle8300 May 29, 2025
2d69718
hotfix: ipad cropping runstop button
wle8300 May 29, 2025
0e6c67d
hide unused button; resize customization menu for mobile
hello-vinitha May 29, 2025
552db68
hide scrollbar when no overflow
hello-vinitha May 29, 2025
415e2b4
ran pre-commit
hello-vinitha May 29, 2025
46670bb
display buttons under camera views by default
hello-vinitha May 29, 2025
fcfa71a
lock scrolling on <body> to prevent ios from allow errant scrolling g…
wle8300 Jun 10, 2025
b302e7c
remove ability to disable "Display Buttons"
wle8300 Jun 10, 2025
49a85ae
make the "Select Object" cursor a bit larger, and change the colors t…
wle8300 Jun 10, 2025
a5b17cf
fixed bug in retreiving stored movement
hello-vinitha Jun 12, 2025
3fb3a2c
fix movement deletion bug
hello-vinitha Jun 12, 2025
f8d981b
changed onClick to onPointerDown to handle both click and touch
hello-vinitha Jun 12, 2025
c70d0e3
remove scroll functionality on smaller devices
hello-vinitha Jun 12, 2025
d878ce1
changed onclick to onpointerdown
hello-vinitha Jun 12, 2025
3c313b6
hotfix for "Base control buttons appearing on "Movement Recorder" tab…
wle8300 Jun 13, 2025
e019a98
hotfix: preserve aspect-ratio for <video>
wle8300 Jun 13, 2025
4dc064e
update the "Select Object" crosshair cursor
wle8300 Jun 13, 2025
68ad836
move runstop+battery to top navbar; added CSS :hover states to every …
wle8300 Jun 16, 2025
48de878
onLeave is not called on mobile devices in click click or step action…
hello-vinitha Jun 18, 2025
5ae3ec5
patching with vinitha's commit: onLeave is not called on mobile devic…
wle8300 Jun 18, 2025
70683aa
changed onClick to onPointerDown
hello-vinitha Jun 18, 2025
f4e10c7
Merge branch 'feature/mobile' of https://github.com/hello-robot/stret…
hello-vinitha Jun 18, 2025
fdc5436
ran pre-commit
hello-vinitha Jun 18, 2025
d48906b
Merge branch 'master' into feature/mobile
hello-vinitha Jun 18, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .prettierrc
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,6 @@
"printWidth": 80,
"trailingComma": "es5",
"semi": true,
"endOfLine": "lf"
"endOfLine": "lf",
"insertFinalNewline": true
}
5 changes: 4 additions & 1 deletion .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
{
"editor.formatOnSave": true,
"editor.defaultFormatter": "esbenp.prettier-vscode",
"prettier.requireConfig": true
"prettier.requireConfig": true,
"[css]": {
"editor.defaultFormatter": "esbenp.prettier-vscode"
}
}
4 changes: 2 additions & 2 deletions documentation/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -372,7 +372,7 @@ export const HomeTheRobot = (props: { hideLabels: boolean }) => {
<React.Fragment>
<div id="home-the-robot-container">
<p>Robot is not homed. Please drive the robot to a safe position and press the home button.</p>
<button onClick={() => { functions.Home(); }}>
<button onPointerDown={() => { functions.Home(); }}>
<span hidden={props.hideLabels}>Home</span>
<HomeIcon />
</button>
Expand All @@ -382,7 +382,7 @@ export const HomeTheRobot = (props: { hideLabels: boolean }) => {
};
```

Notice that we import the instance of `homeTheRobotFunctionProvider` that was created in [`index.tsx`](#indextsx-1). Further notice that we define the set of functions required for the component in an enum called `HomeTheRobotFunction`. This enables us to map the button's `onClick` callback to the `HomeTheRobotFunction.Home` method. The function provider uses the same enum to map a method that gets called when the button is clicked. The function provider is defined in [`HomeTheRobotFunctionProvider`](#function_providers).
Notice that we import the instance of `homeTheRobotFunctionProvider` that was created in [`index.tsx`](#indextsx-1). Further notice that we define the set of functions required for the component in an enum called `HomeTheRobotFunction`. This enables us to map the button's `onPointerDown` callback to the `HomeTheRobotFunction.Home` method. The function provider uses the same enum to map a method that gets called when the button is clicked. The function provider is defined in [`HomeTheRobotFunctionProvider`](#function_providers).

We dress up the HTML in this component with a stylesheet defined in `src/pages/operator/css/HomeTheRobot.css`.

Expand Down
1 change: 1 addition & 0 deletions eslint.config.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ export default defineConfig([
},

rules: {
"eol-last": ["error", "always"],
"prettier/prettier": [
"error",
{},
Expand Down
28 changes: 14 additions & 14 deletions launch/web_interface.launch.py
Original file line number Diff line number Diff line change
Expand Up @@ -487,20 +487,20 @@ def generate_launch_description():
)
ld.add_action(navigation_bringup_launch)

ld.add_action(
ExecuteProcess(
cmd=[
[
FindExecutable(name="ros2"),
" service call ",
"/reinitialize_global_localization ",
"std_srvs/srv/Empty ",
'"{}"',
]
],
shell=True,
),
)
# ld.add_action(
# ExecuteProcess(
# cmd=[
# [
# FindExecutable(name="ros2"),
# " service call ",
# "/reinitialize_global_localization ",
# "std_srvs/srv/Empty ",
# '"{}"',
# ]
# ],
# shell=True,
# ),
# )

ld.add_action(
ExecuteProcess(
Expand Down
2 changes: 1 addition & 1 deletion src/pages/home/tsx/components/ForgotPassword.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ export const ForgotPassword = (props: {
/>
</DialogContent>
<DialogActions sx={{ pb: 3, px: 3 }}>
<Button onClick={props.handleClose}>Cancel</Button>
<Button onPointerDown={props.handleClose}>Cancel</Button>
<Button variant="contained" type="submit">
Continue
</Button>
Expand Down
4 changes: 2 additions & 2 deletions src/pages/home/tsx/components/LoginView.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,7 @@ export const LoginView = (props) => {
<Link
component="button"
type="button"
onClick={handleClickOpen}
onPointerDown={handleClickOpen}
variant="body2"
sx={{ alignSelf: "baseline" }}
>
Expand Down Expand Up @@ -254,7 +254,7 @@ export const LoginView = (props) => {
type="submit"
fullWidth
variant="contained"
onClick={validateInputs}
onPointerDown={validateInputs}
>
Sign in
</Button>
Expand Down
4 changes: 2 additions & 2 deletions src/pages/home/tsx/components/SideBySideView.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ export const SideBySideView = (props) => {
>
Stretch Web Teleop
</Typography>
<Button color="inherit" onClick={handleLogout}>
<Button color="inherit" onPointerDown={handleLogout}>
Logout
</Button>
</Toolbar>
Expand Down Expand Up @@ -80,7 +80,7 @@ export const SideBySideView = (props) => {
>
Stretch Web Teleop
</Typography>
<Button color="inherit" onClick={handleLogout}>
<Button color="inherit" onPointerDown={handleLogout}>
Logout
</Button>
</Toolbar>
Expand Down
37 changes: 37 additions & 0 deletions src/pages/operator/css/BatteryBar.css
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
.bar {
width: 100%;
height: 0.5rem;
background-color: #436fb7;
display: flex;
margin-top: 3px;
}

.barsContainer {
width: 2rem;
height: 4rem;
display: flex;
flex-direction: column-reverse;
align-items: center;
}

.batteryBar {
height: 4rem;
position: absolute;
}

.batteryBarContainer {
align-items: center;
display: flex;
flex-direction: column;
}

@media (max-width: 1200px) {
.barsContainer {
width: 1.5rem;
height: 3rem;
}

.batteryBar {
height: 3rem;
}
}
41 changes: 21 additions & 20 deletions src/pages/operator/css/BatteryGuage.css
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
.batteryGaugeContainer {
align-items: center;
display: flex;
flex-direction: column;
width: 100%;
padding: 1rem 1rem 0.5rem;
}

.bar {
width: 100%;
height: 15px;
Expand All @@ -15,45 +23,38 @@
}

.batteryGauge {
height: 100px;
/* position: absolute; */
padding: 0.9rem 0 0;
width: 100%;
padding: 0px 10px 10px 10px;
height: 50px;
}

.green {
filter: invert(17%) sepia(15%) saturate(6070%) hue-rotate(89deg)
brightness(75%) contrast(108%);
filter: invert(17%) sepia(15%) saturate(6070%) hue-rotate(89deg) brightness(75%) contrast(108%);
}

.yellow-green {
filter: brightness(70%) saturate(100%) invert(0%) sepia(28%) saturate(4155%)
hue-rotate(56deg) contrast(119%);
filter: brightness(70%) saturate(100%) invert(0%) sepia(28%) saturate(4155%) hue-rotate(56deg) contrast(119%);
}

.yellow {
filter: brightness(90%) saturate(100%) invert(0%) sepia(28%) saturate(4155%)
hue-rotate(6deg) contrast(119%);
filter: brightness(90%) saturate(100%) invert(0%) sepia(28%) saturate(4155%) hue-rotate(6deg) contrast(119%);
}

.orange {
filter: brightness(70%) saturate(100%) invert(10%) sepia(28%)
saturate(4155%) hue-rotate(6deg) contrast(119%);
filter: brightness(70%) saturate(100%) invert(10%) sepia(28%) saturate(4155%) hue-rotate(6deg) contrast(119%);
}

.orange-red {
filter: brightness(50%) saturate(100%) invert(10%) sepia(28%)
saturate(4155%) hue-rotate(6deg) contrast(119%);
filter: brightness(50%) saturate(100%) invert(10%) sepia(28%) saturate(4155%) hue-rotate(6deg) contrast(119%);
}

.red {
filter: brightness(30%) saturate(100%) invert(10%) sepia(28%)
saturate(4155%) hue-rotate(346deg) contrast(119%);
filter: brightness(30%) saturate(100%) invert(10%) sepia(28%) saturate(4155%) hue-rotate(346deg) contrast(119%);
}

.batteryGaugeContainer {
align-items: center;
display: flex;
flex-direction: column;
max-height: 90%;
.text-battery-gauge {
padding: 1rem 0 0;
font-size: 0.9rem;
font-weight: 500;
text-align: center;;
}
74 changes: 55 additions & 19 deletions src/pages/operator/css/ButtonGrid.css
Original file line number Diff line number Diff line change
@@ -1,43 +1,77 @@
.button-grid {
flex: 2;
flex: 1;
display: grid;
grid-template-columns: repeat(6, 1fr);
grid-template-rows: repeat(4, auto 1fr);
row-gap: 0.15rem;
column-gap: 0.15rem;
/* grid-template-columns: repeat(4, 1fr); */
/* grid-template-rows: repeat(3, 0fr); */
grid-template-areas:
"header0 header0 header0 header0 header0 header0"
". b0 b1 b2 b3 ."
"header1 header1 header1 header1 header1 header1"
". b4 b5 b6 b7 ."
"header2 header2 header2 header2 header2 header2"
"b8 b9 b10 b11 b12 b13"
"header3 header3 header3 header3 header3 header3"
". . b14 b15 . .";

padding: 0.4rem;
"b0 b0 b0 b0"
"b2 b2 b3 b3"
"b1 b1 b1 b1"
"a0 a0 a0 a0"
"b4 b4 b4 b4"
"b6 b6 b7 b7"
"b5 b5 b5 b5"
"a1 a1 a1 a1"
"b12 b10 b10 b13"
"b12 b14 b15 b13"
"b8 b14 b15 b9"
"b8 b11 b11 b9";
/*
"header0 header0 header0 header0 header0 header0"
"b0 b0 b1 b1 b2 b3"
"header1 header1 header1 header1 header1 header1"
"b4 b5 b6 b6 b7 b7"
"header2 header2 header2 header2 header2 header2"
"b8 b9 b10 b11 b12 b13"
"header3 header3 header3 header3 header3 header3"
"b14 b14 b14 b15 b15 b15" */
padding: 0.6rem;
position: relative;
z-index: 1;
border-radius: var(--radius);
/* border-radius: var(--radius); */
}

@media screen and (orientation: landscape) and (max-device-width: 1200px) {
.button-grid button {
min-width: fit-content;
padding: 0rem 0.5rem 0rem 0.5rem;
}
}


.button-grid::after {
border-radius: var(--radius);
}

.button-grid p {
font-size: x-large;
text-align: center;
margin: 1.5rem 0 0.5rem 0;
margin: 0rem;
}

.button-grid p:first-of-type {
text-align: center;
margin-top: 0.5rem;
margin-bottom: 0.4rem;
}

.button-grid p:last-of-type {
text-align: center;
margin-top: 0.4rem;
}

.button-grid button {
width: 90%;
height: 4rem;
font-size: 0.95rem;
font-weight: 500;
width: 100%;
border-radius: 0px;
min-height: 2.8rem;
height: 100%;
justify-self: center;
align-self: center;
background-color: gray;
color: white;
}

.button-grid button.active {
Expand All @@ -49,10 +83,12 @@
}

.button-grid button.limit {
background-color: red;
background-color: #fc9696;
color: black;
}

.button-grid-bkg-color {
grid-column: 1 / 7;
height: 5rem;
border: 2px solid black;
}
13 changes: 8 additions & 5 deletions src/pages/operator/css/ButtonPad.css
Original file line number Diff line number Diff line change
Expand Up @@ -42,22 +42,24 @@
/* Settings for the buttons on the button pad *********************************/

/*When hovering over a child element of an svg*/
.button-pads path:hover {
fill-opacity: 30%;
@media (hover: hover) {
.button-pads path:hover {
fill-opacity: 30%;
}
}

@media (hover: none) {
.button-pads path.inactive:hover + image {
.button-pads path.inactive:hover+image {
filter: none !important;
}
}

.button-pads image {
pointer-events: none;
touch-action: none;
}

/* Button pad <image> gets saturated while hovering */
.button-pads path.inactive:hover + image {
.button-pads path.inactive:hover+image {
filter: brightness(50%) sepia(100%) saturate(10000%) hue-rotate(194deg);
}

Expand Down Expand Up @@ -98,6 +100,7 @@
/* Disable hover actions */
.button-pads.customizing path {
pointer-events: none;
touch-action: none;
}

.button-pads.selected {
Expand Down
Loading
Loading