Switch to QuickJS for CouchDB JS engine

[garethbowen]

Describe the performance issue

View rebuilding is slow which is a big problem particularly when doing an update which changes a view requiring the whole index to be rebuilt.

Describe the improvement you'd like

CouchDB 3.4.x introduced a new JS engine called QuickJS which promises 4-5x speed improvement and 6x improvement in memory usage. It's off by default. Try turning it on and see if we can get similar gains.

Describe alternatives you've considered

To Reproduce

1. Get a db with a lot of realistic data
2. Back up the db so you can re-run the test easily
3. Change a view (even a whitespace change) to trigger a complete rebuild
4. Measure the time, cpu usage, memory usage, disk usage until completion
5. Restore the backup
6. Change the JS engine to QuickJS (docs)
7. Change the view and measure the time again

Measurements

• Time to complete
• CPU usage average and peak
• Memory usage average and peak
• Disk usage peak

Environment

• Instance: A cloud instance would be ideal, but as long as the specs are the same for both runs it should be ok
• Browser: N/A
• Client platform: N/A
• App: couchdb
• Version: latest

Additional context

Blocked by #9303

[lorerod]

Moved to 4.15.0 so as not to hold up the release.

[latin-panda]

Moving it to 4.16

[dianabarsan]

I used a database with 1 million records.
I used this script to monitor resource usage: https://gist.github.com/dianabarsan/c65ad5d861740da33f0caae5b98f91cd and then made minor changes in views:

• 1 view in medic ddoc
  
• 1 view in medic-client ddoc
  
• 2 views in medic-client ddoc
  

For medic, it seems that spidermonkey and quickjs are equivalent. There is no significant difference in time it took to index, disk, CPU and memory usage.
However, for medic-client the results are quite different: we can see a significantly quicker view indexing time: 15 minutes vs 11.5 minutes (for two views) and a significant increase in memory usage while indexing: max 22GB for spidermonkey and max 32GB for quickjs.
CPU and disk usage are very similar.

Results can be seen here:
https://docs.google.com/spreadsheets/d/19z6N7YabCngsGZgPnHDLIoR-HwY_zYPSyjcFiWFY6ko/edit?gid=0#gid=0

[dianabarsan]

Due to the differences in time and memory when indexing medic-client (case 2 from above), I ran a new test where I launched the docker container with a 20G memory limit, to see if the short indexing time for quickJS is due to better memory usage.

It turns out that even with limited memory, quickjs is quicker to index views by 16% .

[dianabarsan]

However, there is apparently a scanner plugin that will detect view incompatibilities between the two engines, described here: https://docs.couchdb.org/en/stable/config/quickjs.html#quickjs

I enabled this plugin and I'm seeing a bunch of weird stuff in the logs:

[error] 2024-12-01T17:40:15.171255Z  OS Process Error <0.896.0> :: {<<"TypeError">>,{[]}}
[error] 2024-12-01T17:40:15.171522Z  OS Process Error <0.894.0> :: {<<"TypeError">>,{[]}}
[error] 2024-12-01T17:40:15.171755Z  OS Process Error <0.896.0> :: {<<"TypeError">>,{[]}}
[error] 2024-12-01T17:40:15.171978Z  OS Process Error <0.894.0> :: {<<"TypeError">>,{[]}}
[error] 2024-12-01T17:40:15.172106Z  OS Process Error <0.896.0> :: {<<"TypeError">>,{[]}}
[error] 2024-12-01T17:40:15.172245Z  OS Process Error <0.894.0> :: {<<"TypeError">>,{[]}}
[error] 2024-12-01T17:40:15.172347Z  OS Process Error <0.896.0> :: {<<"TypeError">>,{[]}}
[error] 2024-12-01T17:40:15.172550Z  OS Process Error <0.894.0> :: {<<"TypeError">>,{[]}}
[error] 2024-12-01T17:40:15.172643Z  OS Process Error <0.896.0> :: {<<"TypeError">>,{[]}}
[error] 2024-12-01T17:40:15.172774Z  OS Process Error <0.894.0> :: {<<"TypeError">>,{[]}}
[error] 2024-12-01T17:40:15.172885Z  OS Process Error <0.896.0> :: {<<"TypeError">>,{[]}}
[error] 2024-12-01T17:40:15.424935Z  OS Process Error <0.896.0> :: {<<"TypeError">>,{[]}}
[warning] 2024-12-01T17:40:15.424973Z  couch_quickjs_scanner_plugin s:1733074794-4b72e236f644 db:medic/00000000-15555554 ddoc:_design/medic doc:0a6bce12-973c-4e91-97fe-8654c8483493 doc validation failed {vdu_doc,{error,{throw,{<<"TypeError">>,{[]}}}},#{}}
[error] 2024-12-01T17:40:15.425235Z  OS Process Error <0.894.0> :: {<<"TypeError">>,{[]}}
[error] 2024-12-01T17:40:15.425470Z  OS Process Error <0.896.0> :: {<<"TypeError">>,{[]}}
[error] 2024-12-01T17:40:15.425853Z  OS Process Error <0.894.0> :: {<<"TypeError">>,{[]}}
[error] 2024-12-01T17:40:15.426087Z  OS Process Error <0.896.0> :: {<<"TypeError">>,{[]}}
[error] 2024-12-01T17:40:15.426395Z  OS Process Error <0.894.0> :: {<<"TypeError">>,{[]}}
[error] 2024-12-01T17:40:15.426715Z  OS Process Error <0.896.0> :: {<<"TypeError">>,{[]}}
[error] 2024-12-01T17:40:15.427088Z  OS Process Error <0.894.0> :: {<<"TypeError">>,{[]}}
[error] 2024-12-01T17:40:15.427337Z  OS Process Error <0.896.0> :: {<<"TypeError">>,{[]}}
[error] 2024-12-01T17:40:15.427571Z  OS Process Error <0.894.0> :: {<<"TypeError">>,{[]}}

There are lots of these errors.
I'm reluctant to proceed with a green light. We would need to identify which are the incompatibilities and edit the view that have them. This would mean that updating the JS engine will have potentially heavy reindexing as a side-effect when partners upgrade.