Why Yjs server in ts when you can use pycrdt instead ?

[davidbrochart]

I just came across this project, it seems very cool!
Looking at the code base quickly, it seems that the server is written in Django but there is a special server for Yjs written in TypeScript. I don't know the motivations for choosing to have a special server in another language, but you may want to look at pycrdt, a Python library providing bindings for Yrs, the Rust port of Yjs. Pycrdt is used for real-time collaboration in JupyterLab.

[virgile-dev]

That looks really cool ! Curious to have your thoughts on this @YousefED

[lunika]

The main problem I see is managing a websocket with django. We have to switch to an ASGI application, use django-channels. To scale you have to use the redis backend and there are many issues open about memory leaks.
FastAPI is probably a better candidate.

[davidbrochart]

Pycrdt can be used with Django, we had a contribution in django_channels_consumer.py. I've never used it myself, but happy to provide support if something is missing. There is also an ASGI server.
In general, pycrdt-websocket should be server-agnostic. We use it in jupyter_server which is based on Tornado, and in jupyverse which is based on FastAPI.

[SylvainCorlay]

It could be interesting for us to compare notes (between the suitenumerique/docs and JupyterLab teams) on usage of YJS for collaborative editing.

[YousefED]

That looks really cool ! Curious to have your thoughts on this @YousefED

I would check this with Kevin - I feel this is a bit outside of my area of expertise tbh. It could be an interesting option - but it might be quite some work (so imo we'd need a strong reason to switch)

[davidbrochart]

One such reason could be the ability to modify a shared document from Python. In the Jupyter ecosystem this is done e.g. by an AI agent, in which case the backend is just another collaborative peer.
BTW pycrdt was created to replace Ypy, the official Python bindings to Yrs, which is unfortunately unmaintained. Kevin started the implementation of collaborative editing in JupyterLab.

[SylvainCorlay]

BTW pycrdt was created to replace Ypy, the official Python bindings to Yrs, which is unfortunately unmaintained. Kevin started the implementation of collaborative editing in JupyterLab.

This makes me think we should probably archive ypy.

[lunika]

This makes me think we should probably archive ypy.

We installed it in docs recently ^^ (See #570)
I will migrate to pycrdt because we have an error (PanicException) when using it on some documents. I hope we will not have these errors with pycrdt.

[SylvainCorlay]

I will migrate to pycrdt because we have and error (PanicException) when using it on some documents. I hope we will not have these errors with pycrdt.

Pycrdt is being used quite extensively in Jupyter-Collaboration.

[lunika]

I made the migration from Ypy to pycrdt. If you want to take a look and give me your feedback of what I'm probably doing wrong : #809

The inital idea is to extract from an existing Ydoc (created with Yjs) all the images url contained in.

[lunika]

@davidbrochart @SylvainCorlay Would you be interested to come at the Hackathon the DINUM is organizing in june ? You can take a look at this short presentation here https://docs.numerique.gouv.fr/docs/96b46e7d-e2ec-46c5-ae25-641e34c5ebfe/
Probably we can take at a look at we can do with pycrdt or other related subjects ?

[SylvainCorlay]

@davidbrochart @SylvainCorlay Would you be interested to come at the Hackathon the DINUM is organizing in june ? You can take a look at this short presentation here https://docs.numerique.gouv.fr/docs/96b46e7d-e2ec-46c5-ae25-641e34c5ebfe/

We would love to participate. Happy to coordinate through other channels if it is relevant.

[sampaccoud]

@SylvainCorlay my understanding is that it is not trivial to move to Python for Ydoc because the server needs to understand the BlockNotejs syntax which is custom (for blocks etc.)
@YousefED can tell us a bit more and for sure it should not be too hard but we need to make sure we will maintain compatibility throughout languages.

[davidbrochart]

my understanding is that it is not trivial to move to Python for Ydoc because the server needs to understand the BlockNotejs syntax which is custom (for blocks etc.)

I don't know if it's BlockNotejs or the Hocuspocus server that adds a layer on top of Yjs's sync protocol.

[YousefED]

@SylvainCorlay my understanding is that it is not trivial to move to Python for Ydoc because the server needs to understand the BlockNotejs syntax which is custom (for blocks etc.)
@YousefED can tell us a bit more and for sure it should not be too hard but we need to make sure we will maintain compatibility throughout languages.

If you want to do operations on the document (change it's content) through the BlockNote API, or convert it to / from Markdown / HTML - this is easiest in a Node environment indeed.

However, as far as I know, the Hocuspocus server you're using does not do any BlockNote specific processing. There's a separate Node server that covnerts to markdown - but I don't think this is connected directly to the sync server. I think @AntoLC can confirm

[sampaccoud]

Ok so that's the thing : we could replace hocuspocus for collaboration but would still need a nodejs container for document conversion to/from markdown. Might still be interesting for performance, simplicity... especially with the upcoming work on attribution, comments and diffs.
We should definitely plan a workshop between the Docs and the Jupyter teams. Want to join our next open forum?

[SylvainCorlay]

We should definitely plan a workshop between the Docs and the Jupyter teams. Want to join our next open forum?

We would love to. I think there are some really good avenues for collaboration.

[virgile-dev]

@SylvainCorlay @davidbrochart next forum is on the 10th of april. Write to us at [email protected] we'll send you the details to attend :)