Feature: live-preview from memory on keystroke

[dvdvgt]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the discussions and believe that my question is not already covered.

Feature Request

Overview

I would like to implement the capability to display a live preview of the current typst document that also changes on every keystroke akin to how live preview works in the web app, or, similarly, in VSCode for markdown documents. This is different from the already existing option to export the PDF on every keystroke because

1. exporting the PDF on every keystroke writes to the disk on every change, which seems wasteful. Whereas live preview, as described here, is to only read the preview image from memory, which is both faster and does not cause wear on the disk.
2. It can be more easily optimized as you could potential only render those images which are currently visible. Also, it should be possible to leverage incremental compilation.

Here's how it could look like and how I implemented it crudely for a proof-of-concept.

Implementation

The way I implemented it for a proof of concept is to expose a command preview that takes a file URI and returns a base64 encoded string containing the rendered document as PNG. Then, the extension issues a executeCommand("typst-lsp.preview", uri) on every change to the currently opened editor window. The string is then wrapped in a image tag <img src="data:image/png;base64,..." /> and displayed in a simple webview panel.

Of course, this is far from the best possible implementation. As I do not have much experience with the LSP or with async in Rust, I would be interested to hear your ideas on how to implement this efficiently. With some mentoring, I am confident that I can pull this off :) I am looking forward to your responses!

[Enter-tainer]

See https://github.com/Enter-tainer/typst-preview-vscode. It supports live preview (in an ugly way), incremental compilation, and update-on-visible functionality. I'm also thinking about how to incorporate this feature into this project!

[Enter-tainer]

incorporate this feature into this project

A key point is how to pass binary data(the framebuffer) between the lsp server and the webview. AFAIK, lsp works on json-rpc, therefore binary data is not possible. Maybe websocket is still needed? I'm not quite sure if this is in typst-lsp's scope.

[dvdvgt]

AFAIK, lsp works on json-rpc, therefore binary data is not possible.

That's true, but we can encode the binary data as a string using base64 and then send it to the webview.

[Enter-tainer]

AFAIK, lsp works on json-rpc, therefore binary data is not possible.

That's true, but we can encode the binary data as a string using base64 and then send it to the webview.

Indeed, but I'm afraid that encoding can hurt performance. The pipeline would be b64 encode -> transmission -> b64 decode.
Also, b64 makes the data larger. I would always prefer raw binary data if possible.

Initially, I did a quick prototype by exporting png files, encoding them to b64 and sending them to frontend. But it's pretty slow. So I currently just send raw framebuffers.

Raw framebuffers can be huge. A single page of document can be as large as 8MB. I tried some tricks to make them smaller(so as to make the whole pipeline faster) but in vain, such as deflate-frame(takes longer time to compress data), rle encoding (decoding is slow in frontend).

[nvarner]

There's a PR that was also experimenting with this (#60). Someone there mentioned Base64 as well, and someone considered putting the Typst compiler into WASM and embedding it into the visualization. I don't like that approach, but if encoding and decoding really are slow, it may be a viable option.

[nvarner]

Using websockets is an interesting idea. I haven't tried serving over two interfaces from the same binary, but if it's possible, that may make the most sense.

[Enter-tainer]

The official web app follows the wasm approach. It doesn't require any kind of data transmission.(So it supposed to be fast, although wasm itself can cause some slowdown.) However, the wasm typst cannot read local fonts, making it totally unusable.😭

[nvarner]

I'm wondering about shared memory now. I worry about portability and fragility (so a websocket/base64 backup would be needed), but that would probably minimize the overhead.

[dmorawetz]

Wouldn't it be possible to output to a tmpfs location, and show from there as a first workaround? #118 would allow to output to /tmp, and pointing an auto-refreshing PDF viewer there would enable a near-real-time preview without touching the disk.

I don't have much experience with this area, just wanted to throw in my thoughts. I also don't know how performant this would be, but probably good enough until a better solution gets implemented.

[Enter-tainer]

I think the bottleneck here is the redundant pipeline: render to pdf -> parse pdf -> render pdf to screen. I don't think tmpfs can resolve this issue.

[Enter-tainer]

typst.ts https://github.com/Myriad-Dreamin/typst.ts proposes a way to directly render the document to canvas, this might be the fastest approach available.

[xsro]

I am trying to render only current page in png in my fork.
The preview looks faster but still slower than official webapp.

[tmistele]

Fwiw, I've toyed around with rendering directly to a slint UI without a pdf/png roundtrip. Performance seems to be quite good. Since that approach involves a separate window being opened by slint it probably isn't suitable for vscode integration. On the other hand, the advantage is that it works with any editor with lsp support.

slint.webm.mp4

[Myriad-Dreamin]

@tmistele what's the performance of slint on pageless document? I'm working on these case in typst-preview. You can test it by editting following file.

#set page(height: auto)

== Introduction

edit me

#lorem(25000)