[UI] Support repo snapshots #462

senwang86 · 2023-08-18T04:32:48Z

Summary

Creating a new database table called YDocSnapshot that stores repo snapshots at the request of users.
The implement features a strict database consistency, once receiving a request, it reads the yDocBlob in the Repo table and inserts it into YDocSnapshot. It avoids conflicts between Repo.yDocBlob in the event of network interruption.
"Rename/restore/delete" operations will be added after discussion.

Test

Attach the shell of the api docker container
Run npx prisma migrate dev
Restart api container

lihebi · 2023-08-18T04:42:00Z

api/src/typedefs.ts

+    repoId: String
+    createdAt: String
+    message: String
+    yDocBlob: String


This blob doesn’t need to be sent to the front end.

This blob doesn’t need to be sent to the front end.

SG, I was thinking about the "restore" operation, I am not exactly sure if it's better to pass the yDocBlob in a single query.

Restore should probably happen in the backend yjs server, and front end will retrieve from yjs. I’ll think about this.

lihebi · 2023-08-18T04:46:16Z

api/src/resolver_repo.ts

+      id: await nanoid(),
+      yDocBlob: repo.yDocBlob,
+      message: message,
+      repoId: repoId,


Use the connect clause to connect the two tables.

lihebi · 2023-08-18T04:47:13Z

api/src/resolver_repo.ts

+ */
+async function getRepoSnapshots(_, { repoId }) {
+  const snapshots = await prisma.yDocSnapshot.findMany({
+    where: { repoId: repoId },


{repo: {id: repoId}}

lihebi · 2023-08-18T04:49:01Z

api/prisma/schema.prisma

+    createdAt DateTime @default(now())
+    message   String?
+    yDocBlob  Bytes?
+    Repo      Repo?    @relation(fields: [repoId], references: [id])


Lower case “repo” field name.

lihebi · 2023-08-18T18:25:39Z

api/prisma/schema.prisma

+    createdAt DateTime @default(now())
+    message   String?
+    yDocBlob  Bytes?
+    deleted   Boolean  @default(false)


I don't want to add deleted here since it adds complexity. We could reply on DB backup if we really want to ensure data safety.

I am not so familiar with DB backup, any reference for its pros and cons? I am considering a scenario where users might decide to delete a snapshot, but then want to undo the deletion later. Furthermore, if there are 1,000 users performing these operations at different times, can DB backups gracefully address this scenario?

I'd skip this scenario for now. If the user deletes it, it is deleted.

I'd skip this scenario for now. If the user deletes it, it is deleted.

OK, let me revert the change.

lihebi · 2023-08-18T18:51:25Z

api/prisma/schema.prisma

+    message   String?
+    yDocBlob  Bytes?
+    deleted   Boolean  @default(false)
+    repo      Repo?    @relation(fields: [repoId], references: [id])


yDocBlob and repo/repoId sound to be required, not optional. I.e.,

yDocBlob Bytes repo: Repo repoId: String

This reverts commit 880c04c.

lihebi · 2023-08-18T20:51:07Z

api/prisma/schema.prisma

-    yDocBlob  Bytes?
-    repo      Repo?    @relation(fields: [repoId], references: [id])
-    repoId    String?
+    yDocBlob  Bytes
+    repo      Repo    @relation(fields: [repoId], references: [id])
+    repoId    String
 }


You need to generate a new DB migration to reflect this change.

bb35923 addes the prisma migration.

lihebi · 2023-08-19T01:34:53Z

One last comment: merge the two migrations into one? I.e. remove the migrations and re-generate one.

senwang86 · 2023-08-19T03:20:54Z

One last comment: merge the two migrations into one? I.e. remove the migrations and re-generate one.

d291948 address this.

lihebi · 2023-08-19T04:10:26Z

Awesome, thanks!

lihebi · 2023-08-20T17:44:55Z

Since this PR involves DB schema change, I'd like to ensure that the restoring would work under this design before merging.

I was thinking about how to restore a snapshot. Two methods:

Option 1 (not ideal): Directly replacing the ydoc to a previous state. This isn't ideal, as the peers see a different yDoc with different history, impossible to sync.
Option 2 (ideal): Add a new "restoring" changeset on top of the latest ydoc. This should be the ideal way.

In terms of implementation, I found that Yjs has a "Y.Snapshot". So instead of storing a whole YDocBlob into snapshots, we could store a Y.Snapshot binary and use Y.snapshot APIs for restoring. But this snapshot seems to only support method 1, which is not ideal.

To support method 2, we might need to implement a "restoring" changeset, like mentioned here.

I'll need some time to think about it before we merge this PR and work on restoring.

senwang86 · 2023-08-30T23:56:18Z

Realized that the YDocSnapshot needs a field of userId to separate the snapshots among various collaborators. WDYT, @lihebi ?

lihebi · 2023-08-30T23:59:50Z

I don't think so. The snapshot is associated with the repo. Each repo has only one line of snapshots, regardless of which user triggered it.

senwang86 · 2023-08-31T00:07:47Z

I don't think so. The snapshot is associated with the repo. Each repo has only one line of snapshots, regardless of which user triggered it.

Will the snapshot list show snapshots from all other collaborators?

lihebi · 2023-08-31T00:13:39Z

Yes. There's only one snapshot line for the repo; everyone sees and operates on it. Keep it simple.

lihebi · 2023-08-31T00:16:07Z

There are more things to consider.

I've been thinking about how to make the revision system to:

play well with the Git system and
to be able to function on scopes

Also, the revision system code wouldn't need to be included in the public-facing desktop app. It would be exclusive to SaaS app, meaning the code would be in another repo.

Sen Wang added 2 commits August 17, 2023 19:35

Add repo snapshots

1e3c096

refine gql and clean up

1b63040

senwang86 requested a review from lihebi August 18, 2023 04:32

lihebi reviewed Aug 18, 2023

View reviewed changes

Sen Wang added 3 commits August 18, 2023 10:11

Address comments

4ff91b0

Add deleted filed in YDocSnapshot

880c04c

clean up

1dd7c37

lihebi reviewed Aug 18, 2023

View reviewed changes

Sen Wang added 2 commits August 18, 2023 13:16

Revert "Add deleted filed in YDocSnapshot"

2ece924

This reverts commit 880c04c.

Address comments

edb7d0f

lihebi reviewed Aug 18, 2023

View reviewed changes

Update_prisma_schema

bb35923

clean prisma migration sql

d291948

Merge branch 'main' into UI_repo_snapshots

b3c03e0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[UI] Support repo snapshots #462

[UI] Support repo snapshots #462

senwang86 commented Aug 18, 2023

lihebi Aug 18, 2023

senwang86 Aug 18, 2023

lihebi Aug 18, 2023

lihebi Aug 18, 2023

lihebi Aug 18, 2023

lihebi Aug 18, 2023

lihebi Aug 18, 2023

senwang86 Aug 18, 2023

lihebi Aug 18, 2023

senwang86 Aug 18, 2023

lihebi Aug 18, 2023 •

edited

Loading

lihebi Aug 18, 2023

senwang86 Aug 18, 2023

lihebi commented Aug 19, 2023

senwang86 commented Aug 19, 2023

lihebi commented Aug 19, 2023

lihebi commented Aug 20, 2023

senwang86 commented Aug 30, 2023 •

edited

Loading

lihebi commented Aug 30, 2023

senwang86 commented Aug 31, 2023

lihebi commented Aug 31, 2023

lihebi commented Aug 31, 2023

[UI] Support repo snapshots #462

Are you sure you want to change the base?

[UI] Support repo snapshots #462

Conversation

senwang86 commented Aug 18, 2023

Summary

Test

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lihebi Aug 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lihebi commented Aug 19, 2023

senwang86 commented Aug 19, 2023

lihebi commented Aug 19, 2023

lihebi commented Aug 20, 2023

senwang86 commented Aug 30, 2023 • edited Loading

lihebi commented Aug 30, 2023

senwang86 commented Aug 31, 2023

lihebi commented Aug 31, 2023

lihebi commented Aug 31, 2023

lihebi Aug 18, 2023 •

edited

Loading

senwang86 commented Aug 30, 2023 •

edited

Loading