build-your-own-parser

[dougwilson]

If you look at the middleware, after I had refactored it, all they are is a call to typeis and a call to read. We should add something like bodyParser.generic() to let people roll their owner simple body parsers.

[jonathanong]

I think this lib is pretty easy to understand. IMO they should just fork this lib if they wanna do more stuff. Most of the logic should be in separate libs anyways

[Fishrock123]

I think it would be better to have the interface and people have parsing libs than lots of body-parser forks.

[dougwilson]

I think it would be better to have the interface

This module should be the interface. json and urlencoded should be the separable libs that use this module ;)

[jonathanong]

You guys are just adding more responsibility to your plate hah

[Fishrock123]

Yeah, that's the issue, may be too much to be worthwhile to support.

[dougwilson]

i.am.machine :DDD

[mikermcneil]

This module should be the interface. json and urlencoded should be the separable libs that use this module ;)

I can tackle this if it's helpful

[dougwilson]

This is actually nearly done, as it was intended for the goal of bringing body-parser back into express core.

Out of curiosity, what parser were you planning to build that we don't have already :)?

[mikermcneil]

Out of curiosity, what parser were you planning to build that we don't have already :)?

good question

[mikermcneil]

xml

[mikermcneil]

lol

[dougwilson]

xml

lol. Somehow I knew that was the answer ;) Currently the best you can do is to use bodyParser.text and then feed the text into a XML parser.

[dougwilson]

The reason why I had a feeling is because I use this module to parse XML all the time, but of course using text + parser requires the request body to buffer up instead of feeding it to an incremental parser (CSV is another common one, which I also use!).

This new stuff will actually be out sooner rather than later since I should not be distracted with express core.

[rlidwka]

Out of curiosity, what parser were you planning to build that we don't have already :)?

JSON5? I use that in a few APIs, because it's easier to debug such api with a curl (and b/w compatibility with json is perfect). Even created express-json5 based on bodyParser.

So this use-case is more real than it seems.

[dougwilson]

Even created express-json5 based on bodyParser.

Though it looks like as body-parser is currently, it can be significantly simplified by wrapping bodyParser.text ;)

[dougwilson]

@rlidwka or, looking at your code now, you could technically wrap bodyParser.json and if it errors, check err.body and parse that with the json5 parser :) I'm not actually saying this invalids the issue at had, here, because it's still valid and I'm working on it, haha.

[lopugit]

How do you build may I ask?

[jonchurch]

Reviving this issue as a more central place to discuss all the Generic parser work.

tldr;

Im in favor of creating a generic parser middleware interface folks can extend to create custom middlewares for their advanced usecases. However, Im against exposing an opts.parser function on existing parsers which users can use as an escape hatch

The Current PRs

We have several PRs open right now to land a Generic implementation. I don't want to leave feedback on all of these, or the additional closed ones, so Im writing this comment instead. cc @wesleytodd @UlisesGascon @ctcpip @Phillip9587

• Generic Body Parser #561
• refactoring changes to generic body parser impl #575
• yet another rebase! generic body parser #574

The existing approach here exposes an opts.parser option, which I don't want to accept.

Flexibility vs Simplicity

I'm open to offering a Generic interface for custom parsers, but I don’t believe we should expose an opts.parser option in our existing middleware parsers. I prefer we export well known and well tested parsers which are hard to muck up and simpler to reason about for us and users. (We do that already with urlencoded extended, and I've read enough issues to know that confuses people.)

The goal of body-parser (IMO) is to provide simple, reliable parsers for common HTTP body formats like JSON and urlencoded data, covering common usecases therein. By sticking to common use cases and avoiding overcomplication, the core middleware is easier for users to adopt and maintain long term.

Extending a generic interface to create a more custom parser is a viable way to provide users flexibility while keeping the simplicity of the core middlewares.

Customization needs are rare compared to the more common issues reported, which suggests that the current parsers cover most users' needs.

We’ve seen a small but recurring set of issues where users want to extend or change the behavior of existing parsers. Examples include:

• Parsing BigInts Any plans to support larger than 53-bit integers? #278
• Additional config options passed to qs Allow extended options to be passed to qs library #98 Pass custom parameters to qs #453
• New formats such as XML, ndjson, or even YAML (!?) xml parsing ? #370 support for ndjson #478 Introduce YAML Input Format #369
• Streaming JSON parsers use non blocking json parser #132

Most of these requests represent edge cases or specialized needs, which is why I think a Generic parser interface is the best way to address them without overcomplicating the core parsers.

[Phillip9587]

I agree with @jonchurch that we should focus on maintaining and improving our core parsers while allowing users to create their own custom parsers as needed. Keeping the core parsers simple and reliable should remain our priority.

Additionally, I think we should start over rather than trying to rebase the existing PRs. A fresh approach would give us a cleaner foundation to work from and ensure we implement this in the best possible way.

That said, there is #551, which introduces the normalizeOptions function. This PR standardizes and validates common parser options like inflate, limit, type and verify, ensuring consistent defaults and reducing duplication across the parsers. I think we should merge it as foundational work, which would help keep the scope of this effort smaller.

Looking forward to hearing everyone's thoughts!