Hovering, Dragging and Picking: Deficiencies with the current framework

[viridia]

One of the key goals for the 0.17 release of Bevy (and which I'm currently working on) is to replace the old bevy_ui::Interaction component with a picking-based approach. This seems like a simple task, but unfortunately it surfaces a bunch of weaknesses in the current picking framework.

Take for example a simple widget such as a button. The way buttons typically work is that the "activation" of the button happens on the pointer-up event, rather than on the pointer-down event. This gives the the user the opportunity to change their mind: if they press the mouse button, and then realize they made a mistake, they can move the mouse pointer outside of the button bounds (an action sometimes referred to as "roll-off"), and then release the button, effectively cancelling the action.

bevy_picking already handles this behavior automatically via the Click event: if you press the mouse button while hovering over an entity, and then move the mouse off the entity before releasing the mouse button, no click event is sent.

Except there is a subtle bug: what happens if you click on a child of the button?

This is important because almost all buttons have children: the label of a button is typically a child text entity.

It turns out that if you roll off the child, the click is canceled - even if the mouse is still within the bounds of the button. In other words: if you click on the text label of the button, and (while holding the button down) move outside of the text label, but still within the bounds of the button, no click will occur.

The reason has to do with the way that bevy_picking handles dragging. You see, the Click event is implemented in terms of the drag-and-drop framework: a click event happens when the drag target is the same as the drag source.

However, to really understand this I am going to take a detour to explain how Bevy's drag and drop framework differs from drag and drop on the web.

History of Drag and Drop

Drag and Drop (or DnD) on the web is a bit of a mess: it consists of two completely separate APIs for doing drag operations: the original "drag/drop" events, and the more recent "pointer capture" API.

There are really three different groups of use cases for dragging and dropping:

• Self-dragging represents things like knobs and sliders, where an element is being dragged around to change its position. In this scenario, there's no "drop" action, just dragging.
• Internal DnD includes arranging items such as table rows or columns, where you are dragging elements on a page relative to each other.
• External DnD is for importing files such as photos and documents into a web page by "dropping" them onto the page. (Note that Bevy currently has no support for this.)

The pointer capture mechanism is generally the most useful for the first use case, self-dragging. Basically what this does is initiate a "capture" action such that all pointer movement events are sent directly to the drag source element (instead of being dispatched to whatever element is being hovered). This capture also ensures that the element will receive pointer move events even if the mouse goes outside of the window bounds.

The drag/drop event framework is more suited to the second and third use case: elements receive "drag over" events when a dragged item enters their bounds.

What both of these frameworks have in common is that the programmer gets to choose which element is the drag source: that is, the element which you are dragging "from". In the case of pointer capture, it's the element on which you call the capture method; in the case of drag/drop events, there's a special HTML attribute that you put on an element to indicate that it is draggable (note this is not the same as pickable).

Bevy's drag and drop framework attempts to simplify the mess of HTML by combining both of these models into a hybrid. However, one thing that is missing is the ability to choose the drag source. Instead, the drag source is chosen automatically.

Part of the reason for this decision is that bevy_mod_picking (the predecessor of bevy_picking) was designed in a way to make picking and dragging easy for 3D objects as well as user interfaces.

However, the consequence is that, as in the previous scenario, when you click on the button label, the label, and not the button, becomes the drag source. When you move outside of the label (but still within the button), the button becomes the drag target. Since the drag source != drag target, no click happens.

Possible solutions

bevy_picking doesn't let you opt out of dragging; however it does let you opt out of picking which in effect makes the entity non-draggable. So one solution is to simply make the child non-pickable. In fact, if you ask on the Bevy Discord how to make a button, this is what people will tell you (I've seen this question at least 3 times).

However, I think this is the wrong answer, for a number of reasons.

First, the child of a button might not be a simple text label. Yes, most buttons have a single text entity as their child, but some buttons also have icons. Some buttons have counters or badges. In fact, a button can have an arbitrarily complex hierarchy of children. In order to make the children non-pickable, every entity within that hierarchy has to be made non-pickable.

Also, we're not just talking about buttons here - imagine something like a clickable table cell which contains a chart or a complex formula. There's no upper limit to how complex the descendants of a clickable entity might be.

Secondly, sometimes you really want the children to be pickable. Consider a button with a "funny shape" - one that has knobs or drag handles or other bits sticking outside of the bounds of the button. Ideally, you would want these extensions to expand the hit region of the button, which wouldn't work if these entities were non-pickable.

Thirdly, and somewhat abstractly, there is often a division of labor between the person that created a widget, and the person that created the widget's children. Often times programmers will use widget libraries authored by someone else. These programmers will have complete control over the children, but for the actual button element they will rely on a template. The children of the button will be passed in to the template as a parameter.

I generally think that templates shouldn't impose these kinds of constraints: "you can pass in any entities you want, but they have to be non-pickable" seems like a footgun.

A more robust solution, I think, is to allow the button author to choose the drag source. This could either be done programmatically via an API call (as in the pointer capture API), or declaratively via an opt-in component (as in the drag/drop events model). The nice thing about this solution is that it works regardless of whether the children or pickable or not.

Unfortunately, this means some backwards incompatibility, since currently Bevy users rely on the automatic assignment of the drag source. It might be possible to have both automatic and explicit drag source assignment (basically the explicit assignment would override the automatic choice) but since this would happen in mid-drag, care would need to be taken to avoid a pop or jump in the drag coordinates.

Hovering

Let's say that we want to add a hover highlight to our button.

Part of the motivation for hover highlights is to telegraph to the user what's about to happen: a highlight means that a subsequent click action is going to do something.

But this is only useful if the highlight is truthful: that is, the highlight should accurately reflect what's about to happen. If an element is highlighted when hovered but doesn't respond to a click, or vice-versa, then this is a bug.

Once of the simplest ways to detect whether an entity is being hovered is to check the HoverMap resource: if the entity is in the hover map, then it's being hovered.

But again we have to ask, what happens if you hover over a child of the button? In this case, the element in the hover map will be the button label rather than the button. So a simple equality comparison gives the wrong answer: the button will not have a hover highlight, even though it will receive a click event (because the click event bubbles upward and is received by the button if allowed to propagate).

This is a common problem that Bevy novices run into, going by the comments on Discord.

In order for the hover highlight to be an accurate prediction, it needs to take bubbling into account. So instead of asking, "is the button in the HoverMap", we need to ask: "is the button, or any of its descendants, in the HoverMap". (Note that in order to be in the HoverMap, the descendants by definition need to be pickable).

However, this is a somewhat expensive check, for a couple of reasons: first, because the hover map is recalculated/mutated every frame, so normal Bevy change detection isn't useful here. Secondly, because we need to do an ancestor check (which is not that bad: starting from the hover entity and searching upward for the button is generally much faster than starting from the button and recursing through all descendants).

One way to solve this would be to provide a pair of HoverEnter/HoverLeave events that took ancestry into account. In fact, this is the way mouse in/out events work on the web.

However, handling enter/leave events can get tricky because of bubbling: a parent entity will not only get in and out events for itself, but events for all its children as well. Moving the mouse from one child entity to another means that the parent will get both of the in and out events, even though the mouse hasn't left the parent at all.

(On the web, there are actually two different kinds of mouse enter events: the original in/out which bubble, and the more recent enter/leave, which don't).

Even worse: the current Bevy observer event model doesn't tell you which entity an event came from - the entity field in the event only tells you which entity is currently handling the event. So even if we had enter/leave events, there's no way for the parent to discriminate between the enter/leave events being sent by its children with the enter/leave events for itself - unless we decide to make them non-bubbling.

Part of the reason for having enter/leave events is to be able to update some persistent widget state variable to either show or hide the highlight: but in order to be effective, the accumulated history of enter/leave events must exactly match the future behavior of the next click event.

Rather than try and untangle all of this, I've opted for a different approach to hover states, which doesn't rely on events: the IsHovered(bool) component. This is a component that you can add to an entity, which opts-in to being updated by the Bevy framework whenever the entity is hovered or un-hovered. An element is considered "hovered" if either itself, or any of its descendants, are in the HoverMap. (This is also consistent with the behavior of the CSS :hover pseudo-class.)

The IsHovered component is immutable, and is guaranteed to only be changed when transitioning between states, so you can use regular Bevy change detection or observers to run code when the hover state changes.

That being said, a lot of people seem to like the event-based approach to hovering, so we'll need to sort that all out.

@alice-i-cecile
@cart
@NthTensor
@ickshonpe
@aevyrie

[cart]

External DnD [...] (Note that Bevy currently has no support for this.)

We do have support for this via the FileDragAndDrop event

A more robust solution, I think, is to allow the button author to choose the drag source. This could either be done programmatically via an API call (as in the pointer capture API), or declaratively via an opt-in component

I think a component probably makes sense. However what this looks like largely depends on whether or not we allow multiple pointer captures. On the web, the capture API is element.setPointerCapture(pointerId). This makes sense if multiple elements can capture the same pointer (and the "component equivalent" would be node_entity.insert(CapturedPointer(pointer_id))). But if a pointer can only be captured by one element at a time, I think it makes more sense to do pointer_entity.insert(PointerCapturedBy(node_entity)).

Even worse: the current Bevy observer event model doesn't tell you which entity an event came from - the entity field in the event only tells you which entity is currently handling the event.

Yup we should address this. Seems like @NthTensor has plans for this

That being said, a lot of people seem to like the event-based approach to hovering, so we'll need to sort that all out.

Yeah I think the "hovered" state (as in "entity or descendant is hovered") should be a single concept that has both hovered/unhovered events and opt-in IsHovered component state that reflects that. Accomplishing this seems very possible / not a particularly challenging engineering problem, unless I am missing something.

[viridia]

On the web, the capture API is element.setPointerCapture(pointerId). This makes sense if multiple elements can capture the same pointer

The docs at https://developer.mozilla.org/en-US/docs/Web/API/Element/setPointerCapture are somewhat ambiguous on this point, but my read (and vague memory) is that capture is exclusive: if an element captures pointer X, then any outstanding captures for pointer X by another element is canceled (and produces a corresponding pointercancel event).

What you describe for an opt-in component is somewhat different than what I was suggesting: rather than a dynamic component that is inserted just-in-time, I was contemplating something like the HTML draggable attribute which is inserted ahead of time to opt-in to dragging. This means that when a drag starts, the picking framework looks at the hit entity and then looks for an ancestor that has dragging enabled, which becomes the drag source. This is less flexible, but offers a simpler model for the developer since the component is static.