Add `npm` package using napi-rs #182

remorses · 2025-06-19T08:16:27Z

Fix #6

The npm package is currently published as @xmorse/markdown-rs

This npm package will also support WASM. Most of the CI has been ported from the oxc-resolver package.

Still a WIP, the exported functions need to be thought out better, for now I am experimenting.

Things to do before merge:

re-enable windows builds
rename from @xmorse/markdown-rs to markdown-rs (or other available name)
fix benchmark, which now does not work because of a Vitest bench bug, all benchmarks take 0ms

wooorm · 2025-06-20T06:51:55Z

Wow, cool work! It’s a lot, will take some time to review. And, also I am sure you can find stuff if you self-review.
Do you have any questions for me? Anything you want me to look at?

remorses · 2025-06-20T09:10:05Z

Here is the benchmark result on the Zod v3 mdx documentation

clk: ~3.25 GHz
cpu: Apple M2 Pro
runtime: node 24.2.0 (arm64-darwin)

benchmark                           avg (min … max) p75 / p99    (min … top 1%)
--------------------------------------------------- -------------------------------
markdown-rs parse                     16.33 ms/iter  16.57 ms             █▄
                              (15.29 ms … 17.08 ms)  17.04 ms        ▅ ▅ ▅██ ▅  ▅
                            (  1.21 mb …   1.22 mb)   1.21 mb ▅▁▁▁▅█▅█▅█▁█████▅▅█▁█

remark parse                          42.02 ms/iter  43.75 ms ██
                              (39.96 ms … 46.92 ms)  46.26 ms ██
                            ( 10.46 mb …  44.35 mb)  15.31 mb ███▁██▁▁▁▁▁▁█▁▁▁▁█▁▁█

summary
  markdown-rs parse
   2.57x faster than remark parse

--------------------------------------------------- -------------------------------
markdown-rs parse without parsing js  16.14 ms/iter  16.31 ms          █
                              (15.63 ms … 16.81 ms)  16.64 ms      ▅ ▅▅█ ▅  ▅ ▅
                            (  1.21 mb …   1.21 mb)   1.21 mb ▅▅█▅▁█▅████████▁█▅▁▅▅

remark parse                          41.73 ms/iter  41.63 ms    █
                              (39.68 ms … 47.57 ms)  45.96 ms █ ██
                            (841.99 kb …  14.55 mb)  11.21 mb █▁█████▁▁▁▁█▁▁▁▁▁▁▁▁█

summary
  markdown-rs parse without parsing js
   2.59x faster than remark parse

--------------------------------------------------- -------------------------------
markdown-rs html                       7.86 ms/iter   8.03 ms         ▃ █▃
                                (7.34 ms … 8.29 ms)   8.29 ms     ▃█▆▃█▃███ ▃▃ ▃ ▃▃
                            (186.48 kb … 186.64 kb) 186.51 kb ▄▆▄▁█████████████████

remark html                           41.38 ms/iter  42.53 ms     █
                              (39.16 ms … 46.77 ms)  43.76 ms ▅▅▅▅█ ▅ ▅  ▅   ▅ ▅  ▅
                            (  6.65 mb …  14.85 mb)  12.03 mb █████▁█▁█▁▁█▁▁▁█▁█▁▁█

summary
  markdown-rs html
   5.27x faster than remark html

remorses · 2025-06-20T09:40:52Z

Notice that the bigger files like index.js and browser.js are generated by NAPI-RS

The important parts live in the lib.rs file. That is the interface exported to the npm package. You can also take a look at the tests, which mostly consist of inline snapshots.

ChristianMurphy

Thanks @remorses!
I appreciate the time put into this.

I am a bit cautious on the maintainer burden. (See #6 (comment) )
With NAPI and native bindings, now we need to test both across node versions and across a broad set of operating systems and instruction sets.
Many of which I don’t have a machine I could test a platform specifical bug on.

I would strongly lean towards WASM first, as its one binary that can be delivered to many native platforms and to the web, with minimal overhead (less than 1.5x difference from native).

ChristianMurphy · 2025-06-21T16:30:01Z

.github/contribute.md

+
+This is the package that handles NAPI bindings for markdown-rs. It also supports WASM. It uses napi-rs to generate the npm packages published on npm.
+
+Tests use vitest, expect().toMatchInlineSnapshot() mostly. Most tests should be run with `pnpm vitest --run -u` to update snapshots, then read the test file again and make sure they have the expected result.


I'd recommend npm first over requiring one of the many alternative package managers (yarn, pnpm, jsr, etc).

most popular projects use pnpm now, it's become the standard for workspaces and a must as soon as you hit one of the many bugs in npm, which is barely maintained now

most popular projects use pnpm now

That is objectively false.
None of the top 10 packages by download use pnpm.
And most in the top 1000 don't either.
https://github.com/wooorm/npm-high-impact

More important, unified, which this is broadly part of does not https://github.com/unifiedjs/unified

soon as you hit one of the many bugs in npm

I hit far more pnpm bugs than npm

barely maintained now

Last release less than a week ago https://www.npmjs.com/package/npm?activeTab=versions
And they come at a regular cadence.

ChristianMurphy · 2025-06-21T16:31:28Z

.github/workflows/release-napi.yml

+  workflow_dispatch:
+  push:
+    paths:
+      - napi/package.json # make new release on version change of napi package


Start with being able to release locally.
Before adding automation, I dont think we would want every merge to main to become a release.

unfortunately to release a cross platform native binary you need to use the platform with the same architecture, GitHub actions is perfect for this. This code was taken for the most part from the oxc-resolver which is state of the art regarding this kind of publishing from Rust to npm

Again see my main comment.
I don't think we should release platform specific versions yet.
WASM is universal.

ChristianMurphy · 2025-06-21T16:33:34Z

napi/README.md

@@ -0,0 +1,3 @@
+# markdown-rs
+
+CommonMark compliant markdown parser in Rust with ASTs and extensions. Packaged for npm with full WASM support.


The readme would need to be fairly complete, a lot could probably be lifted from https://github.com/remarkjs/remark or https://github.com/micromark/micromark
Which this is based off

Sure, I think this can be done in a follow-up PR after the first version is released on npm

ChristianMurphy · 2025-06-21T16:35:22Z

napi/index.d.ts

@@ -0,0 +1,54 @@
+/* auto-generated by NAPI-RS */
+/* eslint-disable */
+export interface CompileOptions {


Avoid keeping a separate..d.ts that needs to be maintained separate from the source.
Use JSDoc mode for TypeScript https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html

Which can both type check and generate a definitions file.

This file is generated by NAPI, based on the Rust lib.rs source. It needs to be in the source code, not sure exactly why, all projects using NAPI do this.

ChristianMurphy · 2025-06-21T16:36:05Z

napi/index.js

+// @ts-nocheck
+/* auto-generated by NAPI-RS */
+
+const { createRequire } = require('node:module')


Please use ESM https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c

I can try but I am not an expert on this and there could be bugs in NAPI when using the new esm option, because not many projects use it and it's not adopted much. If not strictly necessary I would avoid it.

remorses · 2025-06-21T19:06:24Z

I am not sure releasing a WASM only build of markdown-rs would be worth it becaues performance would be worse that remark js. I don't think there is need to test the markdown-rs build in other platforms, they will almost surely always work given this project doesn't have any dependencies on OS specific syscalls but only serde and an id generation package.

ChristianMurphy · 2025-06-21T19:51:22Z

I am not sure releasing a WASM only build of markdown-rs would be worth it becaues performance would be worse that remark js.

How so?
Do you have numbers to back that up?

I don't think there is need to test the markdown-rs build in other platforms, they will almost surely always work given this project doesn't have any dependencies on OS specific syscalls but only serde and an id generation package.

unfortunately to release a cross platform native binary you need to use the platform with the same architecture, GitHub actions is perfect for this.

I disagree.
If I need to maintain a specific configuration, there needs to be a way to build and test it both in CI and locally.
I don’t have a way to run several of these locally.

I'll defer to @wooorm as BDFL, but this is not something I'd be able to support.

remorses added 30 commits June 18, 2025 12:04

feat(napi): add napi bindings and test

6901039

works on mac

9974eab

adding mdx support

da4c5d2

adding napi release process

250df48

Update package.json

ddbf3ae

Create .node-version

00d9dad

Update release-napi.yml

6f96c73

use setup node from actions

eeb9d07

package.json

b163178

Update release-napi.yml

c67a78a

Update release-napi.yml

0923d3e

Update release-napi.yml

ee92349

package

de77211

Update package.json

5c78406

package

368f47b

Update pnpm-workspace.yaml

d2d1689

disable windows for now

c43301d

fix root package.json

0c24e1f

workaround napi-rs bug with -

cc68c55

lock to older version

b14cf0c

run build once

2acdf12

remove -

db0ff41

add publishConfig

9d48fa7

fix wrong readme.md

1201c14

fix mdx test

d17d1e9

adding more options in the exports

1c4a8fd

new version

92adec7

added more options

0096e70

esm works

2e8ff1e

using napi package.json

6f2e20f

remorses added 9 commits June 18, 2025 16:32

remove freebsd

911945a

adding benchmark, does not work tho

efaa769

add support for fronmatter

2561138

meta strings

48971b8

heading handling

8d11d99

add gfm option

f5977e4

remove positions from tests, simpler split into sections

a5fe72a

smaller npm package

845ac34

package

6beb6b3

remorses added 7 commits June 20, 2025 10:23

shorter readme

6dc1af1

remoe splitIntoSections function

27b14ef

re enable windows

e78532f

rename to tests

d02b6ad

add benchmark, add toHtml function

b30edd9

better benchmark, use the zod markdown long doc

871037f

Update package.json

af708ca

fix version check

758f944

remorses marked this pull request as ready for review June 20, 2025 10:55

remorses added 6 commits June 20, 2025 12:56

Update release-napi.yml

e5a6bb6

Update package.json

b72e973

add missing generated files

53000fe

update generated files

61aeda7

scripts in root are wrong

cad5c32

fix artifacts step in ci

229f7df

ChristianMurphy reviewed Jun 21, 2025

View reviewed changes


		This is the package that handles NAPI bindings for markdown-rs. It also supports WASM. It uses napi-rs to generate the npm packages published on npm.

		Tests use vitest, expect().toMatchInlineSnapshot() mostly. Most tests should be run with `pnpm vitest --run -u` to update snapshots, then read the test file again and make sure they have the expected result.

		@@ -0,0 +1,3 @@
		# markdown-rs

		CommonMark compliant markdown parser in Rust with ASTs and extensions. Packaged for npm with full WASM support.

Uh oh!

Add npm package using napi-rs #182

Are you sure you want to change the base?

Add npm package using napi-rs #182

Uh oh!

Conversation

remorses commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wooorm commented Jun 20, 2025

Uh oh!

remorses commented Jun 20, 2025

Uh oh!

remorses commented Jun 20, 2025

Uh oh!

ChristianMurphy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

remorses Jun 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

remorses Jun 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

remorses commented Jun 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ChristianMurphy commented Jun 21, 2025

Uh oh!

Uh oh!

Add `npm` package using napi-rs #182

Add `npm` package using napi-rs #182

remorses commented Jun 19, 2025 •

edited

Loading

remorses Jun 21, 2025 •

edited

Loading

remorses Jun 21, 2025 •

edited

Loading

remorses commented Jun 21, 2025 •

edited

Loading