[Async Clipboard API] Use Stream APIs to provide data during write

[snianu]

Copied from w3c/editing#423 (comment).

@saschanaz said:

Hmm, I was not fully understanding this issue back in the meeting, as my focus was on a different issue. To reiterate what @smaug---- said with some code, can this be:

navigator.clipboard.write(new ClipboardItem({
  'text/html': new ReadableStream({
    pull(controller) {
      // This will only be called when the consumer reads it, so this is still about "have callback".
      controller.enqueue(bytes);
    },
    type: "bytes",
  }, { highWaterMark: 0 }),
}));

I think this can coexist with callback, though.

@smaug---- said

Using a stream could let implementations to optimize memory usage, as an example.

I think this could be a good addition in general to the async clipboard APIs, so creating this issue for further discussions.

[saschanaz]

I wonder what the expected use cases look like. If most of the use cases for delayed write includes a time-consuming computation, it would be best for them to use stream to not block the main thread (unless everything happens within a worker) and reduce the delay until the first write.

(It could also be a mitigation for privacy problem for custom formats if the stream consumption immediately starts but with artificial intervals, so that the triggering page have a hard time detecting when the actual paste happens with which format.)

[snianu]

If most of the use cases for delayed write includes a time-consuming computation, it would be best for them to use stream to not block the main thread

Even for formats that are not delayed rendered, this would be a useful addition to the async clipboard API. It would be nice not to force web authors to use delayed clipboard rendering if they want to take advantage of this memory optimization for read/write.

(It could also be a mitigation for privacy problem for custom formats if the stream consumption immediately starts but with artificial intervals, so that the triggering page have a hard time detecting when the actual paste happens with which format.)

Like it was mentioned in this comment, this defeats the purpose of delay rendering the expensive formats as the web authors are forced to trigger the callbacks. We resolved the privacy issue to only support delayed rendering for built-in formats for now, but we will explore options to extend this support to web custom formats with the privacy mitigation in-place in the future.

[saschanaz]

Even for formats that are not delayed rendered, this would be a useful addition to the async clipboard API.

👍

It would be nice not to force web authors to use delayed clipboard rendering if they want to take advantage of this memory optimization for read/write.

My understanding was that delayed rendering won't need special flag, was it? Not sure what you mean by forcing. Authors would just pass readable stream and consuming that would solely depend on user agent implementation, right?

Like it was mentioned in this comment, this defeats the purpose of delay rendering the expensive formats as the web authors are forced to trigger the callbacks. We resolved the privacy issue to only support delayed rendering for built-in formats for now, but we will explore options to extend this support to web custom formats with the privacy mitigation in-place in the future.

Stream sources can produce small chunk for each pull rather than big chunk at once, so I think it doesn't defeat the point as each pull shouldn't take long. 👍 for doing builtin formats so that we can keep discussing, though!

[snianu]

My understanding was that delayed rendering won't need special flag, was it?

I guess it depends on how we implement the stream support, but if a callback is supplied in the ClipboardItem instead of a Promise to Blob/DOMString, then it will be treated as a delayed rendered format. You could extend the callback to use stream instead, but I think it will benefit the web authors even without this callback.

[saschanaz]

Streams wouldn't need callback even with delayed rendering, user agent in that case would just delay consuming it. Requiring steam to be inside callback would be weird as stream itself uses callback already.

[snianu]

Streams wouldn't need callback even with delayed rendering, user agent in that case would just delay consuming it

I'm not sure if I understand this correctly. How would you distinguish between a normal clipboard write operation and delay rendering (not delayed write)? Delayed rendering is tied to the system clipboard. If the system clipboard doesn't call back into the source app because the delay rendered format was never used during paste, then the callback is never triggered. If I understand the stream proposal correctly, then the web authors have to always be prepared with the data to stream it right?

[saschanaz]

If I understand the stream proposal correctly, then the web authors have to always be prepared with the data to stream it right?

A proper stream source would generate data on-demand (pull callback will be called when needed). Technically it can prepare before pull callback via start callback but that's not required and not ideal for a large size data.

[snianu]

pull callback will be called when needed

Does the browser call this callback? If not, then how does the browser indicate to the web authors that they need to return data for the format being requested by the clipboard?

I think the confusing thing for me is the below snippet in the example code:
controller.enqueue(bytes);

How does the web author generate these bytes?

[saschanaz]

Does the browser call this callback?

Yes. stream.getReader().read() (or C++ equivalent of it) will call the callback. You can take look at Chromium Fetch implementation to see how it works there.

How does the web author generate these bytes?

If the data is string then one can use TextEncoderStream to do the work:

let stream = new ReadableStream({
  pull(controller) {
    controller.enqueue("hello");
  }
}, { highWaterMark: 0 });
let encodedStream = stream.pipeThrough(new TextEncoderStream()); // encodes each data chunk on demand
let reader = encodedStream.getReader();
console.log(await reader.read()); // { done: false, value: Uint8Array([104, 101, 108, 108, 111])

If the data is binary, then you already have it.

[snianu]

Adding few people who would be interested in this feature @sanketj @evanstade @inexorabletash @whsieh @annevk