Sync service by GabrielMartinezRodriguez · Pull Request #777 · HappyChainDevs/happychain

GabrielMartinezRodriguez · 2025-05-16T14:36:23Z

Linked Issues

closes https://linear.app/happychain/issue/HAPPY-284/backend-driven-user-information-sync

Description

This PR is the first version of the sync service, which implements a backend to handle CRUD operations and includes modifications in the iframe to synchronize permissions and watched assets

Include all relevant context (but no need to repeat the issue's content).
Draw attention to new, noteworthy & unintuitive elements.

Toggle Checklist

Checklist

Basics

B1. I have applied the proper label & proper branch name (e.g. norswap/build-system-caching).
B2. This PR is not so big that it should be split & addresses only one concern.
B3. The PR targets the lowest branch it can (ideally master).

Reminder: PR review guidelines

Correctness

C1. Builds and passes tests.
C2. The code is properly parameterized & compatible with different environments (e.g. local,
testnet, mainnet, standalone wallet, ...).
C3. I have manually tested my changes & connected features.

< INDICATE BROWSER, DEMO APP & OTHER ENV DETAILS USED FOR TESTING HERE >

< INDICATE TESTED SCENARIOS (USER INTERFACE INTERACTION, CODE FLOWS) HERE >

C4. I have performed a thorough self-review of my code after submitting the PR,
and have updated the code & comments accordingly.

Architecture & Documentation

D1. I made it easy to reason locally about the code, by (1) using proper abstraction boundaries,
(2) commenting these boundaries correctly, (3) adding inline comments for context when needed.
D2. All public-facing APIs are documented in the docs.
Public APIS and meaningful (non-local) internal APIs are properly documented in code comments.
D3. If appropriate, the general architecture of the code is documented in a code comment or
in a Markdown document.
D4. An appropriate Changeset has been generated (and committed) with make changeset for
breaking and meaningful changes in packages (not required for cleanups & refactors).

GabrielMartinezRodriguez · 2025-05-16T14:36:43Z

chore: sync chains #984
Sync service #777 👈 (View in Graphite)
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

cloudflare-workers-and-pages · 2025-05-16T14:43:35Z

Deploying happychain with Cloudflare Pages

Latest commit:	`f7764cc`
Status:	✅ Deploy successful!
Preview URL:	https://4795d85d.happychain.pages.dev
Branch Preview URL:	https://gabriel-sync-config.happychain.pages.dev

View logs

norswap · 2025-05-21T01:16:05Z

demos/react/src/demo-components/SessionKeyDemo.tsx

+      hex += b.toString(16).padStart(2, '0');
+    }
+    return '0x' + hex;
+  }


Seems to be leftover code to be removed, but in any case, viem has a function for this.

norswap · 2025-05-21T01:16:28Z

apps/settings-service/tsconfig.build.json

+    "compilerOptions": {
+        "strict": true
+    },
+    "include": ["src", "./package.json"]


Suggested change

"include": ["src", "./package.json"]

"include": ["src"]

I need it because it’s used in index.ts to extract the package version

norswap · 2025-05-21T01:17:20Z

apps/settings-service/tsconfig.build.json

+    "compilerOptions": {
+        "strict": true
+    },


Suggested change

"compilerOptions": {

"strict": true

},

It's already in the base file.
If you copied this over from somewhere, can you fix that over there too?

norswap · 2025-05-21T01:17:40Z

apps/settings-service/tsconfig.json

+    "compilerOptions": {
+        "strict": true
+    },


Suggested change

"compilerOptions": {

"strict": true

},

norswap · 2025-05-21T01:18:36Z

apps/settings-service/src/utils/logger.ts

+export const logger = Logger.create("SettingsService")
+
+const responseLogger = Logger.create("Response", LogLevel.TRACE)
+export const logJSONResponseMiddleware = createMiddleware(async (c, next) => {


Suggested change

export const logger = Logger.create("SettingsService")

const responseLogger = Logger.create("Response", LogLevel.TRACE)

export const logJSONResponseMiddleware = createMiddleware(async (c, next) => {

export const logger = Logger.create("SettingsService")

const responseLogger = Logger.create("Response", LogLevel.TRACE)

export const logJSONResponseMiddleware = createMiddleware(async (c, next) => {

norswap · 2025-05-21T01:19:37Z

apps/settings-service/src/utils/isUUID.ts

+
+export function isUUID(str: string): str is UUID {
+    return validate(str) && version(str) === 4
+}


Should move to the common package.

norswap · 2025-05-21T01:21:56Z

apps/settings-service/package.json

+    "devDependencies": {
+        "@happy.tech/happybuild": "workspace:0.1.1",
+        "typescript": "^5.6.2",
+        "hono-openapi": "^0.4.4"


Duplicate from dependencies.

norswap · 2025-05-21T01:22:41Z

apps/settings-service/Makefile

+include ../../makefiles/bundling.mk
+include ../../makefiles/help.mk
+
+start: ## Starts the settings service


Suggested change

start: ## Starts the settings service

dev: ## Starts the settings service

For consistency with other packages.

norswap · 2025-05-21T01:24:08Z

apps/settings-service/build.config.ts

+    exports: [".", "./migrate"],
+    bunConfig: {
+        minify: false,
+        target: "node",


Do we want to run this with bun? Currently the submitter uses bun, and I think the other services use node. Did we have a specific reason to avoid bun at the time? I think maybe Mikro-ORM?

Yes, the reason was that Bun isn't compatible with Mikro-ORM.

norswap · 2025-05-21T01:24:51Z

apps/settings-service/.env.example

norswap · 2025-05-21T01:32:35Z

apps/settings-service/src/db/types.ts

+    // The app to which the permission is granted.
+    invoker: AppURL
+    // This is the EIP-1193 request that this permission is mapped to.
+    parentCapability: "eth_accounts" | string // TODO only string or make specific


let's just make this string + at the site this was coming from and remove the TODO

norswap · 2025-05-21T01:33:00Z

apps/settings-service/src/db/types.ts

+ *
+ * This type is copied from Viem (eip1193.ts)  but we add a user field.
+ */
+export type WalletPermissionTable = {


Suggested change

export type WalletPermissionTable = {

export type WalletPermission = {

Or possibly WalletPermissionRow if you want to use WalletPermission separately, but actually I don't think rows from the DB will be loaded with this type? So maybe not WalletPermisisonRow.

norswap · 2025-05-21T01:35:42Z

apps/settings-service/src/dtos.ts

dtos as "data transfer objects"?

norswap · 2025-05-21T01:37:33Z

apps/settings-service/src/env.ts

+const parsedEnv = envSchema.safeParse(process.env)
+
+if (!parsedEnv.success) {
+    console.log(parsedEnv.error.issues)
+    throw new Error("There is an error with the server environment variables")
+}


Suggested change

const parsedEnv = envSchema.safeParse(process.env)

if (!parsedEnv.success) {

console.log(parsedEnv.error.issues)

throw new Error("There is an error with the server environment variables")

}

const parsedEnv = envSchema.parse(process.env)

The default exception it throws has a good information on what went wrong during the parse, the extra code just suppressed this here.

norswap · 2025-05-21T02:05:53Z

apps/settings-service/src/index.ts

+serve({
+    port: env.APP_PORT,
+    fetch: app.fetch,
+})


Super curious why we don't need this in the bun version. cc @not-reed
I don't think it's just a bun thing?

Yes, it's weird. If you read the documentation of the node-server (https://github.com/honojs/node-server), they mention that this is needed for compatibility reasons with Node. However, we're running this service with Bun, so I removed it

norswap · 2025-05-21T02:07:21Z

apps/settings-service/src/repositories/permissionsRepository.ts

+    return {
+        user: permission.user,
+        invoker: permission.invoker,
+        parentCapability: permission.parentCapability,
+        caveats: JSON.stringify(permission.caveats),
+        date: permission.date,
+        id: permission.id,
+        updatedAt: permission.updatedAt,
+        createdAt: permission.createdAt,
+        deleted: permission.deleted,
+    }


Suggested change

return {

user: permission.user,

invoker: permission.invoker,

parentCapability: permission.parentCapability,

caveats: JSON.stringify(permission.caveats),

date: permission.date,

id: permission.id,

updatedAt: permission.updatedAt,

createdAt: permission.createdAt,

deleted: permission.deleted,

}

return {

...permission,

caveats: JSON.stringify(permission.caveats),

}

Isn't this approach better? Ditto for below.

norswap · 2025-05-21T02:10:09Z

apps/settings-service/src/db/migrations/Migration20250515123000.ts

+        .addColumn("parentCapability", "text")
+        .addColumn("caveats", "jsonb")
+        .addColumn("date", "integer")
+        .addColumn("id", "text", (col) => col.notNull())


We should make this a primaryKey.

And shouldn't a lot (most?) of the other fields be non-null?

Regarding the ID, yes, we can do it. But for the other fields, if we make them non-null, the problem is that if we later want to make them nullable, SQLite doesn't allow altering the column. That happened to me in the randomness service, so I think we don't gain much by adding the non-null restriction. I think that it's better to enforce it in the service logic, and if we ever want to change the rule, we can do so more easily

norswap · 2025-05-21T02:33:33Z

apps/iframe/src/state/permissions.ts

+    },
+    persist: {
+        plugin: ObservablePersistLocalStorage,
+        name: 'config-legend',


Is this the name of the local storage key?

norswap · 2025-05-21T02:50:29Z

apps/iframe/src/components/interface/permissions/useAppsWithPermissions.ts

-    const permissionsMap = useAtomValue(permissionsMapAtom)
-    const account = useAccount()
+    const appsWithPermissions = () => {
+        const permissions = permissionsMapLegend.get()
+        return Object.values(permissions).filter((permission) => !isWallet(permission.invoker)).reduce((acc, permission) => {
+            const existing = acc.find(([app]) => app === permission.invoker)
+            if (existing) {
+                existing[1][permission.parentCapability] = permission
+            } else {
+                acc.push([permission.invoker, {
+                    [permission.parentCapability]: permission
+                }])
+            }
+            return acc
+        },  [] as [AppURL, AppPermissions][])
+    }

-    return entries(permissionsMap[account?.address ?? "0x0"] ?? {}) //
-        .filter(([app]) => !isWallet(app))
+    return use$(() => appsWithPermissions())


This confused me a good deal, but I see now it's an unfortunate consequence of the changing shape of permissionsMapLegend vs permissionsMapAtom. See my other comment in state.ts, but I think we might be able to preserve the existing shape?

Actually, we can change it because the atom isn't being used directly in the iframe. It's consumed through some typed methods where I manipulate the list type to convert it to the expected version so nothing breaks. It's working fine now

norswap · 2025-05-21T03:06:59Z

apps/iframe/src/state/permissions.ts

-            return hasPermissions(app, permissionsRequest)
+
+
+export const permissionsMapLegend = observable(syncedCrud({


The Legend State docs says that this returns by default returns a Record with keys ... so are we just free to use any key we want?

I see you return WalletPermission[] from list — but the doc says you need to you use as: ... in the object you pass to syncedCrud.

This absolutely needs a type.

The default value returned is a record where the key is the ID of the object—that’s the type it returns. Returning an array directly isn’t an option when consuming the state in the frontend. As stated in the docs (https://legendapp.com/open-source/state/v3/sync/supabase/#as):

Note that array is not an option because arrays make it hard to efficiently and correctly add, update, and remove elements by ID.

The type is already inferred; the observable has this shape: Observable<Record<string, WalletPermission>>, where the string is the ID

As for your question about why the list method returns an array: it's because the list should provide all the permissions that have changed. Internally, Legend State then stores them as a record

norswap · 2025-05-21T03:11:13Z

apps/settings-service/.env.example

Btw the name of the directory should be sync-service because it doesn't only deal in settings :)

norswap · 2025-05-21T03:11:24Z

apps/iframe/src/state/permissions.ts

+    if (app === getWalletURL()) {
+        // Permissions don't exist, create them.
+        // The iframe is always granted the `eth_accounts` permission.
+        const permissionId = createUUID()


Using the UUID as key is rather problematic, because as indicated in its comment on WalletPermission, it's not something we really use or need and so it's just filled by generated BS.

Which means it's not deterministic, which means it presents a problem for sync, as if two clients fall out of sync and have both granted connection permission (eth_accounts) to an app, then they will appear as though they are different permissions because of the different UUID.

The same case can also happen here (and this doesn't need out of sync scenarios): the permissions to the iframe are auto-granted.

We could use (user address + appURL) as key instead. That does however make the value arrays of app permissions (or even a recrod) instead of plain permissions. We could also just use the user address (first level of the map), given values of type Record<AppURL, AppPermissions> aka Record<AppURL, Record<string, WalletPermission>> — is using these kinds of values supported? Does the sync work on them? If so, we should use that (at least, at first). So we can preserve our existing schema and logic.

The real question is: what do update actually look like? if I change a field down the object graph, does it only send that field? But also can we send similarly partial objects from the server? (Well I guess we control the list function, so technically yes we can send these partial updates and apply them ourselves to the existing object in the list function)

^ Most of the value of the review is in this comment.

I changed it to use the key: const id = ${user.address}-${app}-${permissionName}, and with this change it's working fine

Now the data structure we save is a WalletPermissions, and when we use the list method, it returns a WalletPermissions[], which is then converted to a Record<string, WalletPermissions>, where the string is the ID of the WalletPermissions. When we perform an update, we only send the specific WalletPermissions that changed

norswap · 2025-05-21T03:11:32Z

apps/iframe/src/state/permissions.ts

+    initial: {},
+    fieldCreatedAt: 'created_at',
+    fieldUpdatedAt: 'updatedAt',
+    fieldDeleted: 'deleted',


If you specify this, it doesn't call delete which can be omitted but update instead.

In the end, we're not using it because I changed the approach from only fetching the differences to loading the entire state each time, as the previous method was causing syncing issues. After that change, we no longer need soft deletes and can fully delete instead

not-reed

didn't run it yet, but looks cool so far!

not-reed · 2025-05-22T16:17:48Z

apps/sync-service/src/db/driver.ts

+
+import { env } from "../env"
+
+const dbPath = env.SETTINGS_DB_URL || ":memory:"


probably makes sense for the default to be set globally in .env ?

not-reed · 2025-05-22T16:18:39Z

apps/sync-service/build.config.ts

+        target: "node",
+        external: ["better-sqlite3"],


Suggested change

target: "node",

external: ["better-sqlite3"],

target: "bun",

if your using bun for the runtime in prod (looks like it will be) 😈

not-reed · 2025-05-22T16:19:57Z

apps/sync-service/src/env.ts

+import { z } from "zod"
+
+const envSchema = z.object({
+    NODE_ENV: z.enum(["development", "production"]),


Suggested change

NODE_ENV: z.enum(["development", "production"]),

NODE_ENV: z.enum(["development", "production"]).default('development'),

not-reed · 2025-05-22T16:20:17Z

apps/sync-service/src/env.ts

+const envSchema = z.object({
+    NODE_ENV: z.enum(["development", "production"]),
+    APP_PORT: z.string().transform((s) => Number(s)),
+    SETTINGS_DB_URL: z.string().trim(),


Suggested change

SETTINGS_DB_URL: z.string().trim(),

SETTINGS_DB_URL: z.string().trim().default(':memory:'),

not-reed · 2025-05-22T16:20:28Z

apps/sync-service/src/env.ts

+
+const envSchema = z.object({
+    NODE_ENV: z.enum(["development", "production"]),
+    APP_PORT: z.string().transform((s) => Number(s)),


Suggested change

APP_PORT: z.string().transform((s) => Number(s)),

APP_PORT: z.coerce.number(),

not-reed · 2025-05-22T16:22:22Z

apps/sync-service/src/errors.ts

@@ -0,0 +1,23 @@
+import type { ContentfulStatusCode } from "hono/utils/http-status"
+
+export abstract class HappySettingsError extends Error {


I don't think we have used it anywhere, but hono does have an HttpException class (seems thats basically what this is implementing?) https://hono.dev/docs/api/exception#throw-httpexception

not-reed · 2025-05-22T16:23:29Z

apps/sync-service/src/handlers/deleteConfig/validation.ts

+
+export const deleteConfigSchema = z
+    .object({
+        id: z.string().openapi({ example: "78b7d642-e851-4f0f-9cd6-a47c6c2a572a" }),


i guess the id is no longer a uuid? (i think i saw that on the front end)

linear · 2025-06-25T14:24:45Z

HAPPY-284 Backend-driven user information sync

We will want to synchronize the user's information (see list below) between multiple instances of its wallets (tabs, devices, standalone, …).

We used to do sync localStorage-driven sync on the same device, but that's doesn't actually work when the wallet iframe is embedded under different domains. It never covered the cross-device use-case which we do want anyway.

Reference to the previous conversation on storage-based sync: HAPPY-137

Information to sync (pretty much anything that sits in storage):

transaction history
permissions
added chains
watched assets
added ABIs
session keys
- BUT encrypted to a key derived from controlling EOA, so that setting up in a new storage context brings session keys along

What not to sync:

user details (come from auth)
local keyshare once we have that

Auth

First, user info should be private, so there is a need to auth the user. For oauth, this should probably be JWT driven. But how do we make this work for injected wallet controlled account?

Signing a challenge (~ Sign in with Ethereum or SIWE) works, but it's annoying because that's an extra user interaction each time we need to refresh auth.
Auth session key: derive a private key from a wallet signature, needed once per storage context (storage partition x browser-install). Bit better.

Where to sync

Firebase: but can we make this to work with key verification?
Jazz is a promising library that can host a DB and sync it locally.
- but the docs are pretty WIP
- hard to say how pluggable authentication is for our needs
Other / custom

How to sync

Will depend on the "where", but assuming either a DB or an API endpoint, basically send an atomic "SQL-like" update.

There's a question of whether such updates can cover all of our use-cases. Something that (1) reads the storage, then (2) issues an update based on it, could cause race conditions. I believe if the read part of the storage is written as part of the update, then that might alleviate the issue? Not 100% sure.

If that doesn't work, then we need API endpoint that perform the read + write transaction atomically on the server.

…ssion state

GabrielMartinezRodriguez changed the title ~~feat(setting-service): first version~~ Settings service May 16, 2025

GabrielMartinezRodriguez added the draft Not ready for review label May 16, 2025

GabrielMartinezRodriguez force-pushed the gabriel/sync-config branch from cacfaa9 to b457552 Compare May 20, 2025 13:17

norswap reviewed May 21, 2025

View reviewed changes

apps/settings-service/.env.example Outdated

Copy link

Collaborator

norswap May 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(empty)

GabrielMartinezRodriguez reacted with rocket emoji

norswap reviewed May 21, 2025

View reviewed changes

GabrielMartinezRodriguez force-pushed the gabriel/sync-config branch from cae317b to 484364e Compare May 22, 2025 14:24

not-reed reviewed May 22, 2025

View reviewed changes

GabrielMartinezRodriguez force-pushed the gabriel/sync-config branch from 484364e to 8c7a36c Compare June 3, 2025 14:09

GabrielMartinezRodriguez force-pushed the gabriel/sync-config branch from 8c7a36c to 0ff84bd Compare June 24, 2025 10:18

GabrielMartinezRodriguez self-assigned this Jun 25, 2025

GabrielMartinezRodriguez changed the title ~~Settings service~~ Sync service Jun 25, 2025

GabrielMartinezRodriguez added reviewing-1 Ready for, or undergoing first-line review and removed draft Not ready for review labels Jun 25, 2025

GabrielMartinezRodriguez marked this pull request as ready for review June 25, 2025 14:15

GabrielMartinezRodriguez added 17 commits June 26, 2025 10:45

feat(setting-service): first version

b60f5bb

feat(settins-service): update and delete methods

484790a

chore(settings-service): wallet integration

e60087a

chore(settings-service): replace jotai atom for legend state in permi…

90745c2

…ssion state

chore(settings): when set, clean all the previous elements

7322108

fix(sync-service): fix conditions races

915da75

chore: pr review

d129cf4

chore: sse events sync service

d9ef015

chore: syns service happy version dependency

ea5f15d

chore: sync service sse

36b6fee

chore: watch tokens

80abc18

chore(sync-service): deploy

f62149d

chore: force gh actions

4a093ba

chore: format

a552390

chore: duplicated actions name

dc7b763

chore: format

3b30a9e

chore: fix iframe build

a4907bc

GabrielMartinezRodriguez force-pushed the gabriel/sync-config branch from 95031e8 to a4907bc Compare June 26, 2025 09:00

GabrielMartinezRodriguez added 2 commits June 26, 2025 11:11

chore: change docs servers

5b51513

chore: final changes

f7764cc

GabrielMartinezRodriguez mentioned this pull request Jun 27, 2025

chore: sync chains #984

Draft

11 tasks

	start: ## Starts the settings service
	dev: ## Starts the settings service

	export type WalletPermissionTable = {
	export type WalletPermission = {

		return hasPermissions(app, permissionsRequest)


		export const permissionsMapLegend = observable(syncedCrud({


		import { env } from "../env"

		const dbPath = env.SETTINGS_DB_URL \|\| ":memory:"

	NODE_ENV: z.enum(["development", "production"]),
	NODE_ENV: z.enum(["development", "production"]).default('development'),

	SETTINGS_DB_URL: z.string().trim(),
	SETTINGS_DB_URL: z.string().trim().default(':memory:'),

	APP_PORT: z.string().transform((s) => Number(s)),
	APP_PORT: z.coerce.number(),

		@@ -0,0 +1,23 @@
		import type { ContentfulStatusCode } from "hono/utils/http-status"

		export abstract class HappySettingsError extends Error {

Conversation

GabrielMartinezRodriguez commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Linked Issues

Description

Checklist

Basics

Correctness

Architecture & Documentation

Uh oh!

GabrielMartinezRodriguez commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloudflare-workers-and-pages bot commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying happychain with Cloudflare Pages

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

norswap May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

norswap May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

norswap May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

norswap May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GabrielMartinezRodriguez May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

norswap May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

GabrielMartinezRodriguez commented May 16, 2025 •

edited

Loading

GabrielMartinezRodriguez commented May 16, 2025 •

edited

Loading

cloudflare-workers-and-pages bot commented May 16, 2025 •

edited

Loading

norswap May 21, 2025 •

edited

Loading

norswap May 21, 2025 •

edited

Loading

norswap May 21, 2025 •

edited

Loading

norswap May 21, 2025 •

edited

Loading

GabrielMartinezRodriguez May 22, 2025 •

edited

Loading

norswap May 21, 2025 •

edited

Loading

norswap May 21, 2025 •

edited

Loading