Skip to content

refactor: migrate to typescript, undici, and improve contributing #42

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 8, 2025

Conversation

semoal
Copy link
Contributor

@semoal semoal commented Dec 9, 2024

Hi guys, hope this pull-request suits you well

I've had a boring weekend and had time to give some love to this library, QuestDb is cool but db clients experience must be a requirement for you as a company, so since it's open-source I'm happy to collaborate with you 🤟

What's new?

  • From npm to pnpm for installing dependencies, much faster and stable, more common in JavaScript ecosystem lately.
  • Eslint 9.x with better defaults, also added Prettier to handle style-guide.
  • Migrate the entire source-code to Typescript, I've removed ALL anys and everything is type-safe.
  • Bunchee for compiling the library, this emits the dual-cjs approach that is complicated to generate, now we're compatible with esm and cjs without any problems, also the typings are auto-generated.
  • Migrated from Jest to Vitest, much more maintained, faster and has a lot of features that are super-useful for testing.
  • Upgraded JSDoc and regenerated the documentation site.
  • Added a CONTRIBUTING.md guide for making it easier to collaborate and introduce new features.

Breaking changes

  • Drop non-LTS version of Node, now we're compatible from Node 20 or greater, this is also tested on the matrix Github workflow testing suite.
  • http.Agent and http.Agents are cool, but latest Node version ship with a library called Undici which is by far better, easier to handle retry, faster, and more maintained and its core contributors are Node maintainers.

https://lh7-rt.googleusercontent.com/docsz/AD_4nXc-Tezi5wDPfpdy_Cmt_6O16iurV2w2Fz63wej7AoBmJTr4K9AvYpzKnpJla-CcDoW_sY53fHb6BH0Ndf5xDitj-9xSXoJGEY9AcnaN-d4BSuPVc_WbHdKzqHcbi-h4LnkNaVMMiQ?key=rd3x2HIhs4vf6dVo5p--Aq8N

Performance improvements

Getting rid of http.agent and using Undici made that when I've installed the library in my project and run some benchmarks creating 2M rows in 64 iterations with a time limit of 5000ms of run we had a nice result:

┌─────────┬───────────┬────────────────────────┬───────────────────────────┬────────────────────────────┬───────────────────────────┬─────────┐
│ (index) │ Task name │ Latency average (ns)   │ Latency median (ns)       │ Throughput average (ops/s) │ Throughput median (ops/s) │ Samples │
├─────────┼───────────┼────────────────────────┼───────────────────────────┼────────────────────────────┼───────────────────────────┼─────────┤
│ 0       │ 'v4.0.0'  │ '54626798.72 ± 27.37%' │ '43224916.50 ± 17958.50'  │ '22 ± 3.08%'               │ '23'                      │ 106     │
│ 1       │ 'v3.0.0'  │ '63162652.05 ± 5.29%'  │ '59678416.50 ± 291166.50' │ '16 ± 2.81%'               │ '17'                      │ 80      │
└─────────┴───────────┴────────────────────────┴───────────────────────────┴────────────────────────────┴───────────────────────────┴─────────┘
  1. Latency (average):

    • v4.0.0 achieves a ~14% reduction in average latency, indicating faster performance.
  2. Latency (median):

    • v4.0.0 shows a significant improvement in median latency, reducing it by ~28%.
  3. Throughput (average):

    • Throughput increased by ~37.5% in v4.0.0.
  4. Throughput (median):

    • v4.0.0 delivers a ~35% improvement in median throughput.

Summary of Improvements

  • Substantial reduction in both average and median latency (~14% and ~28%, respectively).
  • Significant increase in both average and median throughput (~37.5% and ~35%, respectively).
  • Overall, v4.0.0 demonstrates superior performance and efficiency compared to v3.0.0.

These results confirm the improvements introduced in v4.0.0, making it a faster and more capable version.

@CLAassistant
Copy link

CLAassistant commented Dec 9, 2024

CLA assistant check
All committers have signed the CLA.

Copy link

@woss woss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent PR!

I have left a few comments.

@semoal
Copy link
Contributor Author

semoal commented Dec 12, 2024

@puzpuzpuz @glasstiger a cool review here would be super nice:) sorry for pinging just want to move this forward

@puzpuzpuz puzpuzpuz added the enhancement New feature or request label Dec 13, 2024
@nwoolmer
Copy link
Contributor

Hi @semoal ,

Thanks for the contribution!

Do you know if this resolves your socket issue, or is this an ongoing problem?

#40

@semoal
Copy link
Contributor Author

semoal commented Dec 13, 2024

Hi @semoal ,

Thanks for the contribution!

Do you know if this resolves your socket issue, or is this an ongoing problem?

#40

Yes, it does. Undici handles way better the connection under the hood

@puzpuzpuz
Copy link
Collaborator

Wow, that's a biggish PR! We may need some time to review it, so please keep this in mind.

@semoal
Copy link
Contributor Author

semoal commented Dec 13, 2024

Wow, that's a biggish PR! We may need some time to review it, so please keep this in mind.

Yes, no hurry from my side, we forked internally and using it already in production, just wanted to ping to make sure that you're "subscribed" to this repo :)

@nwoolmer nwoolmer linked an issue Dec 13, 2024 that may be closed by this pull request
@semoal
Copy link
Contributor Author

semoal commented Dec 18, 2024

To make you a real demo, I've reviewed some heavy load (it's bandwidth coming through our production QuestDB machine):

  • The initial peak of this graph generated the error socket: hang out without the new version, from 400K jobs easily 15k were failed because socket hang out
  • The latest peak, was handled with the new version (v4.0.0) , 0 errors, 90% cpu and handled without any problems the connection.

This demonstrates how well Undici manages connection pooling under the hood and how fast it is, easily managed a throughput of 2x or 3x, we were limited to run our asynchronous pipelines to 15 machines because this error, and with the new version we put 100 machines at the same time, and went amazingly good.

Screenshot 2024-12-18 at 15 57 35

- Bumped versions of several devDependencies in package.json and pnpm-lock.yaml, including eslint, prettier, and testcontainers.
- Enhanced the Sender class by introducing a flushPromiseChain to manage flush operations sequentially, ensuring data integrity during high load.
- Refactored the HTTP request handling to improve error logging and response management.
- Added new private methods for buffer management and data sending logic.
- Updated tests to reflect changes in error messages and ensure proper functionality under high load conditions.
@glasstiger
Copy link
Collaborator

@semoal, sorry, it took a long long time to get to it, but I am reviewing your PR.

This PR could have been cut into 3-4 separate ones, tbh.
At least migrating to typescript, and the code reformatting could have been done on separate PRs, but it is ok.

Copy link
Collaborator

@glasstiger glasstiger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In spite of the many comments I am approving this PR.
We were planning to port the codebase to typescript, and introducing undici as transport is a very good idea.

I will be raising multiple PRs to address the comments on this PR.

@@ -1,10 +1,14 @@
const { Sender } = require('@questdb/nodejs-client')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could add typescript examples too.


jobs:
test:
name: Build with Node.js ${{ matrix.node-version }}
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [16, latest]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to keep compatibility with node 18 at least, but it is ok if we cannot.

@@ -2,16 +2,17 @@ name: build

on:
push:
schedule:
- cron: '15 2,10,18 * * *'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This build was scheduled to run 3 times a day to catch possible incompatibility issues with the server.
Would be good to keep it that way.
The server moves very fast, and it is good to see early on, if we broke compatibility with the clients.

@glasstiger glasstiger merged commit 9018102 into questdb:main Jul 8, 2025
1 check passed
@JonDum
Copy link

JonDum commented Jul 16, 2025

Just wanted to chime in that this was a great move, @semoal did you guys a massive solid. All the libs he chose are de facto A+ tools in a modern TS/Node ecosystem that will give the questdb node client new life.

@nwoolmer
Copy link
Contributor

Just wanted to chime in that this was a great move, @semoal did you guys a massive solid. All the libs he chose are de facto A+ tools in a modern TS/Node ecosystem that will give the questdb node client new life.

Hey Jon, thanks for your comment - we agree fully! This contribution went far above and beyond, and we are very excited to finally release these improvements, alongside support for the new N-dimensional array type.

Feedback and support from our OSS community is absolutely critical to ensure the software is robust and fit for purpose.

@semoal definitely deserves more kudos and applause for his contribution!

@glasstiger
Copy link
Collaborator

@semoal, @JonDum, I am sorry it did not really come through in my merge comment that this PR provides tremendous value for us.
The move to typescript has been planned for a while, but we have never got to it.
Switching to Undici is a very good idea, and the update to modern tooling is also an invaluable contribution.
Many thanks for this PR!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Error: socket hang up, with cpu at 90%
7 participants