Enable direct pen and touch to have different touch-action behavior

[RByers]

Today a stylus in Chrome Android behaves as a direct manipulation input device (with the implications on touch-action discussed in #202). This works well in practice except for one special case:

In some situations like note taking applications, you might want the pen to draw by default, while still allowing a finger to scroll. That's not possible to express with touch-action today.

I propose we address this by defining a new pen-action property that is defined to apply ONLY to "direct manipulation pen devices", with (for now) only two possible values:

• auto: use the value of touch-action. I believe this must be the default for web compat reasons (again see Normatively make touch-action apply for direct-manipulation pen devices #202).
• none: disable all pen-based panning (and zooming if it exists), just send events.

For implementation (and specification) complexity reasons I'd prefer to avoid adding other properties like pan-x etc. I think from a UX perspective you're always going to want the pen to behave either identically to the finger, or to draw.

@ChumpChief and @atanassov have told me they're interested in exploring this on Edge.

@NavidZ @mustaqahmed @patrickhlauke WDYT?

[patrickhlauke]

would we consider this for v2 still? concerned about this (and the 2 implementation requirement) potentially prolonguing the path to TR (which we've not embarked on yet).

with that out of the way, this sounds reasonable. it's another "why are we doing this in CSS if it's about behavior?" one, but that boat already sailed with touch-action and the unrelated pointer-events properties I guess...

[scottgonzalez]

Have you considered extending touch-action to allow values based on pointer type?

Perhaps something similar to linear-gradient:

touch-action: "auto", some-other-keyword("none", "pen");

Not sure what would be a good name for some-other-keyword, but the idea is that it would take the touch-action value and then a list of pointer types. Some other syntax could be used as well. I'm just a little nervous about adding a type-specific property when Pointer Events is all about type abstraction.

[RByers]

would we consider this for v2 still?

Nah, not worth delaying v2 for to me. I'd be fine considering it for level 3 (an extension for now).

Have you considered extending touch-action to allow values based on pointer type?

Yeah my concern with that is just around web compat. That rule will fail to parse in browsers that don't support it, so developers will end up having to specify two rules for at least the next 5 years anyway. I think it makes such compat bugs less likely if we just introduce a new property.

I'm just a little nervous about adding a type-specific property when Pointer Events is all about type abstraction.

That's a good point. We should totally think about how it would interact with a new/unexpected pointerType. In the drawing app scenario where you've got the pen doing inking but the finger scrolling, if some other direct manipulation pointer device was used (I have no idea what that would be since the esoteric device are all AFAIK indirect manipulation) should that device do inking or scrolling?

Another way we could address your concern is with a generic property that has a list of pointer types. Eg:

touch-action-pointers: "pen"

The default would be "pen","touch", and it would apply only to direct manipulation pointers of that type (so you couldn't, for example, use this to ask for a mouse device to start scrolling on drag). Any direct manipulation devices not specified here would behave as touch-action: auto.

[mustaqahmed]

I think touch-action-pointers is a cleaner solution.

I have different concern though: any of these ideas (touch-action-pointers or pen-action) adds another dimension to how pens behave, on top of InputDeviceCapabilities.pointerMovementScrolls. I have a feeling that one of them should be redundant when we already have pointerType.

For example, the outcome is not obvious when pen events have pointerMovementScrolls == false but the same events are supposed to pan-y. (This is true for pointerType="touch" even today, but perhaps nobody expects pointerMovementScrolls == false here.)

Am I the only one to find three levels of parameters (type + capabilities + action) too complicated? Any chance we may not need pointerMovementScrolls at all if we have touch-action-pointers?

[patrickhlauke]

Have you considered extending touch-action to allow values based on pointer type?

the danger i see here is that touch-action is becoming a bit of a clown-car of values, personally...

[RByers]

Am I the only one to find three levels of parameters (type + capabilities + action) too complicated? Any chance we may not need pointerMovementScrolls at all if we have touch-action-pointers?

It's definitely complicated but I think they represent different thing. touch-action is an API from the developer to the browser saying where they do and do not want scrolling. pointerMovementScrolls is an API from the browser to the developer saying which devices support touch-action at all. Maybe there's a better name for that? I'm also not convinced pointerMovementScrolls is that valuable - at the PE hackathon people were just concerned about how touch behaves differently than other pointers and not wanting developers to make assumptions about pointerType related to that.

[NekR]

I actually thought about something similar as @scottgonzalez suggested, but kind of in revers:

pointer-action: touch(pan-y), pen(none);  

touch() and pen() functions here are corresponding to supported/possible pointerType. Why it might be good:

• Future compatible: with any future type like dial
• Backwards compatible: the new property. Doesn't break anything, can support touch and mouse
• Allow rich customization for different types of input devices, not only touch or pen specific
• Unknown function are just ignored

[patrickkettner]

I love the concept of allowing for different touch-action like behaviors for each type of pointer, but am opposed to having new *-action values because of the unbounded nature of future pointer types.

This feels like the exact sort of reason @media blocks were invented. What about

@media(pointer-type: touch) {
  #carousel {
    pointer-action: pan-y
  }
}

@media(pointer-type: pen) {
  #carousel {
    pointer-action: none
  }
}

[NekR]

@patrickkettner I thought about this, but we already have @media(pointer: coarse) which was made abstract on purpose, @media(pointer-type: touch) will destroy any purpose of @media(pointer: coarse).

Also problem here is to ably 2 different pointer-actions (one per pen, other per touch) at the same time. I'm not sure how media query would help with that. Do you mean that media query will be automatically applied when one type of a pointer is used? What about 2 at the same time?

[NekR]

If something, I also feel that pointer-action: touch(pan-y), pen(none); syntax doesn't look quite right. Maybe it's possible with CSS grammar to have a better one?

[patrickkettner]

I thought about this, but we already have @​media(pointer: coarse) which was made abstract on purpose

its solving a completely different problem. you could very much want a different *-action of input on something like a trackpad than you would a mouse (e.g. locking panning or scrolling) and both would be fine. Not to mention the idea that other potential pointer types like dials don't fit into either fine or coarse.

media(pointer-type: touch) will destroy any purpose of @​media(pointer: coarse).

firmly disagree. even if it did, a more useful interface shouldn't be stopped because a lesser one already exists.

Also problem here is to ably 2 different pointer-actions (one per pen, other per touch) at the same time.

thats not a problem? you can do that today

@media (min-width: 500px) {
  background: blue
}

@media (width: 500px) {
  background: red
}

both will apply at 500px wide, and if they interfere with one another source order precedence exists.

Do you mean that media query will be automatically applied when one type of a pointer is used?

yep, just like :hover and :selected pseudo selectors, they apply based on state of the element.

What about 2 at the same time?

see above

[NekR]

its solving a completely different problem.

I didn't mean it solves this problem. I mean that pointer-type media query could be started used instead of pointer media query and the idea of pointer media query was to abstract types from concrete, like touch. But yes, pointer media query is unrelated to the problem in this issue. I'm just worried about its future.

thats not a problem? you can do that today

Sorry, I don't get. Having element background property set to red and blue doesn't make it having red and blue at the same time. Same with pointer-type. I doubt that someone will be scrolling and drawing at the same time though.

they apply based on state of the element.

On the whole document then because @media query are global (unless shadow dom?) to the document (just correction, not that I disagree here).

[patrickkettner]

I'm just worried about its future.

Sure, I am just saying that that is tangential to this issue

Sorry, I don't get. Having element background property set to red and blue doesn't make it having red and blue at the same time.

That's exactly my point. If someone sets conflicting pointer-action values, we have known ways of handling the conflicts - source order.

On the whole document then because @media query are global (unless shadow dom?) to the document (just correction, not that I disagree here).

The media block activates when the input type is either pointer 0 or an active pointer on the document. the specific pointer-action, like all CSS properties, apply on a per selector basis

…

On May 9, 2017, at 4:48 PM, Arthur Stolyar ***@***.***> wrote: its solving a completely different problem. I didn't mean it solves this problem. I mean that pointer-type media query could be started used instead of pointer media query and the idea of pointer media query was to abstract types from concrete, like touch. But yes, pointer media query is unrelated to the problem in this issue. I'm just worried about its future. thats not a problem? you can do that today Sorry, I don't get. Having element background property set to red and blue doesn't make it having red and blue at the same time. Same with pointer-type. I doubt that someone will be scrolling and drawing at the same time though. they apply based on state of the element. On the whole document then because @media query are global (unless shadow dom?) to the document (just correction, not that I disagree here). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[patrickkettner]

After speaking with Matt Rakow about this more, due to the number of potential edge cases with my proposal, I withdraw my complaints about pen-action. I am fine with it

[phistuck]

Looks like EdgeHTML 16 shipped pen-action.
https://codepen.io/MicrosoftEdgeDocumentation/pen/jLGZZY

[patrickhlauke]

Discussed at today's PEWG meeting (https://www.w3.org/2019/07/10-pointerevents-minutes.html) ... there's a desire to look at a more input-agnostic model (or at least a more neutral "naming", so we don't end up with lots of different [pointertype]-action css properties every time a new pointer type gets introduced. this feels most in line with the agnostic/generic idea of PE. keeping this open for now though for further discussion.

[BoCupp-Microsoft]

Here's a link to the pen-action explainer that was discussed and resulted in the comment Patrick wrote above.

[BoCupp-Microsoft]

Regarding a more generic proposal in response to the WG feedback, @gked and I played with some syntax and arrived at the same proposal by @NekR above.

pointer-action: pen(none) touch(pan zoom);

We like the future-proofing of this proposal. Here are some additional specifics that I imagined were implied by it:

The type should parse correctly for any string that syntactically could be a valid future pointer-type. The property should also parse correctly for any string that syntactically could be a valid future action.

For example, pointer-action: gaze(pan); could be a valid future pointer-action value and should be allowed by user agents.

The user agent is not obligated to recognize and fulfill any particular action, however it is obligated to limit the actions possible to those that have been explicitly put into the list. For example, pointer-action: pen(none);, not only means that all pen input should generate pointer events, but also means that touch is unaffected as it isn't explicitly listed. It also means that if gaze(pan) is ever supported that the user agent should consider it still enabled.

If the author wants to ensure no actions are possible (the author always wants to see the events), pointer-action: all(none); can be specified.

The touch-action property would become deprecated and equivalent to pointer-action: all(touch-action values here); or pointer-action: touch(values here) pen(values here);

[flackr]

A possibly simpler option that I don't think has been mentioned yet is to declare which devices are direct manipulation (defined in #350) devices and thus produce direct manipulation (i.e. touch) actions. E.g.

direct-manipulation: all | auto | none | [mouse | touch | pen]+

However, I like that the suggestion from @BoCupp-Microsoft also serves to deprecate the use of touch-action and be more fine grained for future actions.