Enhancing Inking on the Web

Today we are excited to announce the preview availability of an enhanced inking experience in the latest Dev Channel builds of Microsoft Edge. We first introduced these enhancements during Build 2021, showcasing how this new web API, which has been implemented upstream in the Chromium open-source project, greatly reduces the latency between the tip of a physical stylus and ink as it’s drawn on the screen.

From our initial experiments on Windows 11 Insider Preview Builds, we’ve measured up to a 240% improvement in latency, which we’ve highlighted in green in the animation below:

We invite you to try this new and improved inking experience on the web by downloading the latest Windows 11 Insider Preview Builds and running the code sample presented later in this blog post on the latest Microsoft Edge Dev Channel.

What’s New?

We’ve introduced a new top level Ink API that enables developers to incorporate the new latency improvements into their web applications. This API is accessible as the ink property on the Navigator global object.

[Exposed=Window]
partial interferace Navigator {
   [SameObject] readonly attribute Ink ink;
};

There are only two steps involved to take advantage of this new API:

1. Acquire an InkPresenter object from the browser.
2. Notify the InkPresenter with the last trusted pointer event from the device.

The following is a simplified code example to show these two steps in practice:

try {
   let canvas = document.querySelector("#canvas");
   let presenter = await navigator.ink.requestPresenter({presentationArea: canvas});

   window.addEventListener('pointermove', function(event) {
      // Gather all of the points that have come from the queue of events.
      let points = event.getCoalescedEvents();

      points.forEach( p => {
         // ... Call renderInkStroke() for your application ...
      });

      //... Render the ink strong belonging to the dispatched event

      // Update the presenter with the last rendered point and give it a style
      presenter.updateInkTrailStartPoint(event, {
         color: "#7851A9",
         diameter: event.pressure * 4
      });
   });
}

Step (1) invokes the requestPresenter function to acquire an InkPresenter object. The returned presenter represents an OS-backed rendering instance or a native browser-backed polyfill instance (more on this later).

Step (2) uses the updateInkTrailStartPoint function to tell the presenter where the last known pointer position is as well as a color and diameter to use in the next few frames until this event fires again.

After this, the OS compositor will handle rendering additional ink strokes on the web applications behalf until the next time a pointermove event is delivered to the application.

See the full Code Example on GitHub.

Technical Details

In Chromium-based browsers today, pen events are first sent to the browser process, which in turn forwards these events to a web application’s JavaScript event loop. The time delay between when the browser process receives these events and when they reach the application can sometimes be significant, depending on the rest of the main thread, resulting in the latency seen when inking.

To improve this, behind the scenes of the InkPresenter implementation on Windows 11, Microsoft Edge is using a new Windows API that will work directly with the operating system’s compositor to draw additional ink strokes outside of Microsoft Edge’s application loop. Thanks to this API, instead of waiting to deliver the event to the web application via JavaScript, we can take these points and provide them to the operating system compositor as soon as we receive them. The compositor can then connect the points with ink strokes and draw these strokes in the next frame that is to be presented to the screen, dramatically reducing latency.

For operating systems like Windows 10 and Linux that don’t have this API, a polyfill implemented directly in Microsoft Edge will take over and provide predictive rendering for ink strokes beyond the last known trusted PointerEvent. This implementation is intended to work similarly to the Windows 11 API – it leverages the points that the browser knows about, along with some predicted points to draw an extension to the application’s ink stroke at the last moment. While the effect isn’t as powerful as the Windows 11 API, it can still provide a much-improved experience for users!

Looking Ahead

As mentioned in the introduction, the work to implement this API was done as an upstream contribution to the Chromium project and is turned on by default for any Chromium-based browsers. We believe in creating the best web platform for everyone and not just users of Microsoft Edge.

Going forward, we’ll be continuing to tune the prediction values provided by the Edge-implemented polyfill for non-Windows 11 installations. We hope to continue to see improvements in latency as we adjust the values and algorithm that the prediction system is using.

To read more about the Ink API and track its progress along the standards track, you can visit the draft specification page on GitHub: Ink API.

We’re always looking for feedback on our APIs. If you have an idea for how we can improve the Ink API further, please open an issue on the ink-enhancement GitHub. For implementation bugs, you can open a new Chromium Bug.
Source: Enhancing Inking on the Web