Add guild.xyz integration

[amrro]

Linear Ticket

PRO-692 - Guild Integration Consumer

Description

Main Changes:

1. Integrated NATS Messaging to establish a connection and retrieve messages containing the guild ID from Feature/csv uploader #358 with couple of changes. visit helpers/nats.ts for more documentations.
2. Made API calls to fetch guild's basic info, roles and users associated with these roles using @guildxyz/sdk.
3. Added the required API calls to store and log these fetched data. You can review the permissions file to check the newly added APIs. helpers/db.ts file for used calls.
4. There are no changes to the database schema.

Caveat

In order to a guild import history record to the database, guild id is needed. Consequenlty, If failed to fetch any data from guild's sdk, there will be no record for a failed guild import to show in import history table.

API examples

• https://api.guild.xyz/v1/guild/caribbean-blockchain-alliance
• https://api.guild.xyz/v1/guild/our-guild (for larger set of data: 60k members)
• https://api.guild.xyz/v1/role/25305

Required .env values

CHAIN_TYPE_ID=1
PROTOCOL_URL="http://localhost:4000/graphql"
GUILD_IMPORT_TOKEN=<your token>

Testing & Debugging

To fetch a guild named our-guild from the NATS server stream named guild-import-job, follow these steps:

1. Install nats-server locally via a Package Manager.
2. Install NATS Command Line Interface. This step is optional but highly recommended for a smoother debugging experience.
3. Add env variables mentioned in the previous section.
4. Run protocol API: yarn nx run protocol-api:serve.
5. run nats server: nats-server --jetstream
6. Publish messages to nats server with a stream name: guild-import-job. You can acheive that using one of two ways:
   • nats CLI: nats -s nats://localhost:4222 publish "guild-import-job.row" "our-guild". It is recommended to complete step 7 at least once before running this command.
   • Alternatively, call the writeMessages() utility function from /helpers/nats.ts after you successfully establish a connection.
7. To check if a message is successfully published: nats stream report. To view messages: nats stream view
8. Run guild import job: yarn nx run guild-import-job:serve.

Unfortunately, nats-server doesn't include a built-in logging or debugging mode. However, nats CLI provide a convenient ways to interact with the server's state using nats stream help, accessing various functionalities and insights about nats server.

Here's a Loom screencast to prepare nats-server for testing:
https://www.loom.com/share/18fad924307f4c14a87fdd0b0ef1458c

[alexkeating]

Minor comments but this is looking really good

[amrro]

⚠️ Careful with with graphql queries file in this commit: 9b9dc20. It mixes the diffs between two fragments.

I noticed an unexpected behavior where null values in roles are included within the guild response. This why I filtered it.

[amrro]

still unable to retrieve messages, I have provided additional details about the stream to facilitate further debugging.

Information for Stream guild-import-job created 2023-03-29 20:21:47

             Subjects: guild-import-job.*
             Replicas: 1
              Storage: File

Options:

            Retention: Limits
     Acknowledgements: true
       Discard Policy: Old
     Duplicate Window: 2m0s
    Allows Msg Delete: true
         Allows Purge: true
       Allows Rollups: false

Limits:

     Maximum Messages: unlimited
  Maximum Per Subject: unlimited
        Maximum Bytes: unlimited
          Maximum Age: unlimited
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited


State:
             ↘️
             Messages: 5
                Bytes: 295 B
             FirstSeq: 1 @ 2023-03-29T18:21:47 UTC
              LastSeq: 5 @ 2023-03-29T18:43:59 UTC
     Active Consumers: 0
   Number of Subjects: 1

[amrro]

These changes 556f146 to pullMessage() helped in messages retrieval.

-  const msgs = await js.fetch(stream, durable, {
-    batch: batch,
-    expires: expires,
-  });

+  const subscription = await js.pullSubscribe(`${stream}.row`, {
+    mack: true,
+    config: {
+      durable_name: durable,
+      ack_policy: AckPolicy.Explicit,
+      ack_wait: nanos(4000),
+    },
+  });

  const done = (async () => {
+    await subscription.pull({ no_wait: true, batch, expires });
    for await (const m of subscription) {
      console.log(`Received message ${m.data}`);
      await callback(nc, m);
      m.ack();
    }
  })();

[alexkeating]

Give comment

[alexkeating]

Give more feedback about consumer

[alexkeating]

Look more closely