Create versioned releases of pandoc-wasm (released as wasm-pandoc on npmjs.com) #10
There are no files selected for viewing
|
There was a problem hiding this comment. I see that you used haskell-wasm/pandoc#1 to create this, makes sense 👍 Now that I see the minimized diff again, I realize that the only remaining thing that is actually patched in the Haskell code here is the addition of the
@johanneswilm In your use case, do you use the There was a problem hiding this comment. @amesgen That sounds promising. Yes, I took your patch in order to make the diff smaller. My use case is this: I have created an open source word processor similar to Google Docs/Microsoft Word 365 Online, etc. for a specific niche market. I have written a number of export filters myself to common formats (like DOCX, ODT, HTML, EPUB, JATS, etc.). These are written in JS and all run in the end users browser. I have even written an import filter for ODT files that works the same way. Now I have a number of users wanting to import and export from other, more exotic formats. So I've created import/export filters in JS to the pandoc internal json format. And I then let the client send the pandoc json to a server where pandoc is run in server mode, converting the json to one of the other formats and sending it back to the client. This is a bit problematic as these conversions take up processing power on the server and it's quite complex to deploy pandoc to a number of different architectures for which there are no pre-compiled binaries. There could even be security issues about sending various files back and forth. So my idea is to instead use this, if the user clicks on export to or import from an "exotic" format for the first time, the browser will download the pandoc.wasm file (to cache for future use) and then to do the conversion in the users own browser instead - taking the users own processing power and not that of the server. I assume you are referring to the terms "reactor" and "command" as they are defined here [1]. Given that I only want to execute the conversion based on one input and then to close down again (only caching the binary so it doesn't have to be downloaded again), I assume this corresponds to pandoc-cli more than pandoc-server. I haven't yet tried whether it is actually possible. Based on the web demo it seems like it should though. I don't know the who-is-who of the haskell-wasm world either. So I don't know which one of all of you can make any decisions here and who would be a good candidate to maintain an npm package of pandoc-wasm. I do maintain several open source packages, but those are written in languages that I use daily (like JS/TS or Python). So if one of you would want to step forward and do this, I'd be in favor. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,229 @@ | ||
| diff --git a/cabal.project b/cabal.project | ||
| index 4ca6da52e630..b3a4ffcbb87b 100644 | ||
| --- a/cabal.project | ||
| +++ b/cabal.project | ||
| @@ -2,10 +2,147 @@ packages: . | ||
| pandoc-lua-engine | ||
| pandoc-server | ||
| pandoc-cli | ||
| -tests: True | ||
| -flags: +embed_data_files | ||
| +tests: False | ||
| + | ||
| constraints: skylighting-format-blaze-html >= 0.1.1.3, | ||
| skylighting-format-context >= 0.1.0.2, | ||
| -- for now (commercialhaskell/stackage#7545): | ||
| data-default-class <= 0.2, data-default <= 0.8 | ||
|
|
||
| +allow-newer: all:zlib | ||
| + | ||
| +package aeson | ||
| + flags: -ordered-keymap | ||
| + | ||
| +package crypton | ||
| + ghc-options: -optc-DARGON2_NO_THREADS | ||
| + | ||
| +package digest | ||
| + flags: -pkg-config | ||
| + | ||
| +package pandoc | ||
| + flags: +embed_data_files | ||
| + | ||
| +package pandoc-cli | ||
| + flags: -lua -server | ||
| + | ||
| +allow-newer: | ||
| + all:Cabal, | ||
| + all:Cabal-syntax, | ||
| + all:array, | ||
| + all:base, | ||
| + all:binary, | ||
| + all:bytestring, | ||
| + all:containers, | ||
| + all:deepseq, | ||
| + all:directory, | ||
| + all:exceptions, | ||
| + all:filepath, | ||
| + all:ghc, | ||
| + all:ghc-bignum, | ||
| + all:ghc-boot, | ||
| + all:ghc-boot-th, | ||
| + all:ghc-compact, | ||
| + all:ghc-experimental, | ||
| + all:ghc-heap, | ||
| + all:ghc-internal, | ||
| + all:ghc-platform, | ||
| + all:ghc-prim, | ||
| + all:ghc-toolchain, | ||
| + all:ghci, | ||
| + all:haskeline, | ||
| + all:hpc, | ||
| + all:integer-gmp, | ||
| + all:mtl, | ||
| + all:os-string, | ||
| + all:parsec, | ||
| + all:pretty, | ||
| + all:process, | ||
| + all:rts, | ||
| + all:semaphore-compat, | ||
| + all:stm, | ||
| + all:system-cxx-std-lib, | ||
| + all:template-haskell, | ||
| + all:text, | ||
| + all:time, | ||
| + all:transformers, | ||
| + all:unix, | ||
| + all:xhtml | ||
| + | ||
| +constraints: | ||
| + Cabal installed, | ||
| + Cabal-syntax installed, | ||
| + array installed, | ||
| + base installed, | ||
| + binary installed, | ||
| + bytestring installed, | ||
| + containers installed, | ||
| + deepseq installed, | ||
| + directory installed, | ||
| + exceptions installed, | ||
| + filepath installed, | ||
| + ghc installed, | ||
| + ghc-bignum installed, | ||
| + ghc-boot installed, | ||
| + ghc-boot-th installed, | ||
| + ghc-compact installed, | ||
| + ghc-experimental installed, | ||
| + ghc-heap installed, | ||
| + ghc-internal installed, | ||
| + ghc-platform installed, | ||
| + ghc-prim installed, | ||
| + ghc-toolchain installed, | ||
| + ghci installed, | ||
| + haskeline installed, | ||
| + hpc installed, | ||
| + integer-gmp installed, | ||
| + mtl installed, | ||
| + os-string installed, | ||
| + parsec installed, | ||
| + pretty installed, | ||
| + process installed, | ||
| + rts installed, | ||
| + semaphore-compat installed, | ||
| + stm installed, | ||
| + system-cxx-std-lib installed, | ||
| + template-haskell installed, | ||
| + text installed, | ||
| + time installed, | ||
| + transformers installed, | ||
| + unix installed, | ||
| + xhtml installed | ||
| + | ||
| +-- https://github.com/haskell/network/pull/598 | ||
| +source-repository-package | ||
| + type: git | ||
| + location: https://github.com/haskell-wasm/network.git | ||
| + tag: ab92e48e9fdf3abe214f85fdbe5301c1280e14e9 | ||
| + | ||
| +source-repository-package | ||
| + type: git | ||
| + location: https://github.com/haskell-wasm/foundation.git | ||
| + tag: 8e6dd48527fb429c1922083a5030ef88e3d58dd3 | ||
| + subdir: basement | ||
| + | ||
| +source-repository-package | ||
| + type: git | ||
| + location: https://github.com/haskell-wasm/hs-memory.git | ||
| + tag: a198a76c584dc2cfdcde6b431968de92a5fed65e | ||
| + | ||
| +source-repository-package | ||
| + type: git | ||
| + location: https://github.com/haskell-wasm/xml.git | ||
| + tag: bc793dc9bc29c92245d3482a54d326abd3ae1403 | ||
| + subdir: xml-conduit | ||
| + | ||
| +-- https://github.com/haskellari/splitmix/pull/73 | ||
| +source-repository-package | ||
| + type: git | ||
| + location: https://github.com/amesgen/splitmix | ||
| + tag: 5f5b766d97dc735ac228215d240a3bb90bc2ff75 | ||
| + | ||
| +source-repository-package | ||
| + type: git | ||
| + location: https://github.com/amesgen/cborg | ||
| + tag: c3b5c696f62d04c0d87f55250bfc0016ab94d800 | ||
| + subdir: cborg | ||
| diff --git a/pandoc-cli/pandoc-cli.cabal b/pandoc-cli/pandoc-cli.cabal | ||
| index 5b904b9906bd..66d92a1875f3 100644 | ||
| --- a/pandoc-cli/pandoc-cli.cabal | ||
| +++ b/pandoc-cli/pandoc-cli.cabal | ||
| @@ -61,7 +61,7 @@ common common-options | ||
|
|
||
| common common-executable | ||
| import: common-options | ||
| - ghc-options: -rtsopts -with-rtsopts=-A8m -threaded | ||
| + ghc-options: -rtsopts -with-rtsopts=-H64m | ||
|
|
||
| executable pandoc | ||
| import: common-executable | ||
| @@ -74,6 +74,10 @@ executable pandoc | ||
| text | ||
| other-modules: PandocCLI.Lua | ||
| , PandocCLI.Server | ||
| + | ||
| + if arch(wasm32) | ||
| + ghc-options: -optl-Wl,--export=__wasm_call_ctors,--export=hs_init_with_rtsopts,--export=malloc,--export=wasm_main | ||
| + | ||
| if flag(nightly) | ||
| cpp-options: -DNIGHTLY | ||
| build-depends: template-haskell, | ||
| diff --git a/pandoc-cli/src/pandoc.hs b/pandoc-cli/src/pandoc.hs | ||
| index 019d0adedb15..520a858c89a2 100644 | ||
| --- a/pandoc-cli/src/pandoc.hs | ||
| +++ b/pandoc-cli/src/pandoc.hs | ||
| @@ -1,5 +1,7 @@ | ||
| {-# LANGUAGE CPP #-} | ||
| +{-# LANGUAGE ScopedTypeVariables #-} | ||
| {-# LANGUAGE TemplateHaskell #-} | ||
| + | ||
| {- | | ||
| Module : Main | ||
| Copyright : Copyright (C) 2006-2024 John MacFarlane | ||
| @@ -34,6 +36,13 @@ import qualified Language.Haskell.TH as TH | ||
| import Data.Time | ||
| #endif | ||
|
|
||
| +#if defined(wasm32_HOST_ARCH) | ||
| +import Control.Exception | ||
| +import Foreign | ||
| +import Foreign.C | ||
| +import System.IO | ||
| +#endif | ||
| + | ||
| #ifdef NIGHTLY | ||
| versionSuffix :: String | ||
| versionSuffix = "-nightly-" ++ | ||
| @@ -44,6 +53,24 @@ versionSuffix :: String | ||
| versionSuffix = "" | ||
| #endif | ||
|
|
||
| +#if defined(wasm32_HOST_ARCH) | ||
| + | ||
| +foreign export ccall "wasm_main" wasm_main :: Ptr CChar -> Int -> IO () | ||
| + | ||
| +wasm_main :: Ptr CChar -> Int -> IO () | ||
| +wasm_main raw_args_ptr raw_args_len = | ||
| + catch act (\(err :: SomeException) -> hPrint stderr err) | ||
| + where | ||
| + act = do | ||
| + args <- words <$> peekCStringLen (raw_args_ptr, raw_args_len) | ||
| + free raw_args_ptr | ||
| + engine <- getEngine | ||
| + res <- parseOptionsFromArgs options defaultOpts "pandoc.wasm" $ args <> ["/in", "-o", "/out"] | ||
| + case res of | ||
| + Left e -> handleOptInfo engine e | ||
| + Right opts -> convertWithOpts engine opts | ||
| +#endif | ||
| + | ||
| main :: IO () | ||
| main = E.handle (handleError . Left) $ do | ||
| prg <- getProgName |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you are going to put a version number on these builds, shouldn't they also be pinned to a specific commit (probably a release tag) for the underlying Pandoc version too?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@alerque Yes, you are right. I am trying to figure out how to best do this. It should probably be easy to get it to build with the most recent pandoc version and we probably need another digit to the number to show the version of the pandoc-wasm package. So
3.6.3.xinstead of just3.6.3. If you have a proposal of how to do the versioning in the most standard complying and simple way - I'm all for it.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just adding a segment will make it hard to parse because Pandoc follows PVP and has a variable number of segments. I think you're going to need a different segment operator that plays nice with distro versioning (I think
+is the most robust option). The question is probably does it make more sense to version this project first, then append the relevant Pandoc version, or the other way around? e.g.pandoc-wasm-3.6.3+0.1orpandoc-wasm-0.1+3.6.3. I think the latter probably makes more sense but it depends on the expected release channels and use workflows I guess. I don't really have a handle on that.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@alerque OK, as someone working mainly with JS/TS in browsers, I would expect it to become available in npm. But I'm open to others needing it other places - maybe? I like both of your versioning proposals, and unless there is reason not to do so, I'd go for the second one then.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@alerque This seems to now be working with tagged versions, etc. . Only thing remaining is put it it on npm. As I assume that @TerrorJack or @tweaf or @alerque will want to have control over the npm repository after merging this PR (or writing something similar), I will not add that part.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@alerque It looks like npm does not allow this versioning scheme. It only allows 3 digit semver. So I split the two numbering apart. It's still really easy to update the pandoc version though.