Bug: the concept of `jstrencode(1)` appears to be both very wrong and very buggy

[xexyl]

Is there an existing issue for this?

• I have searched for existing issues and did not find anything like this

Describe the bug

Based on JavaScript encoding of JSON, there appear to be multiple problems with our tool.

First of all, it appears that the \ being converted to a \\ is wrong. I will give examples in the what we should expect section.

The next problem is that code points should be converted to unicode symbols just like with jstrdecode(1). This is how it is with JavaScript too: both encode and decode need to do this.

Another problem is that the \ escape chars should not be done the way we have it. See what to expect for examples.

What you expect

In order of the problems above, here are examples of what jstrencode(1) does and what JavaScript does.

If we have the JavaScript:

const json_to_encode = {
          name: "\u0f0ff\bo"
        };

it SHOULD (i.e. this is what JavaScript does) encode to:

{"name":"༏f\bo"}

but our tool converts the string to:

$  jstrencode '"\u0f0ff\bo"'
\"\\u0f0ff\\bo\"

Notice the double \ before the b! Now on the subject of the escaped quotes in the beginning see the anything else section.

Now as far as the code points go, javascript of:

const json_to_encode = {
          name: "\u0f0f"
        };

converts to:

{"name":"༏"}

but our tool does this:

$ jstrencode '{"name": "\u0f0f"}'
{\"name\": \"\\u0f0f\"}

or as just a string:

$ jstrencode '"\u0f0f"'
\"\\u0f0f\"

Now if the \uxxxx was converted to a unicode symbol it might just be that the \" surrounding the output would be the difference but I am not sure of this.

The third problem appears to be even worse. There might be other cases where something like this happens but anyway the JavaScript:

const json_to_encode = {
          name: "\u0f0ff\co"
        };

... turns into:

{"name":"༏fco"}

Notice how the \c has the \ silently removed and the c is by itself (or rather it's after the unicode symbol and before the o).

But our tool does something extremely wrong. First as a string by itself:

$ jstrencode '"\u0f0ff\co"'

0;xexyl@xexyz:~$

.. or as the json with {}:

$ jstrencode {"name":"\u0f0ff\co"}

0;xexyl@xexyz:~$

As can be seen the encoding concept we have appears to be totally wrong.

Environment

• OS: linux for tool tests, macOS for JavaScript but really should be n/a
• Device: n/a
• Compiler: n/a

jparse_bug_report.sh output

n/a

Anything else?

As for the escaped " surrounding the string. I guess it depends on how this tool will be used but even so it would appear to be the wrong default.

As for the doubling of \ it also depends on how we need to use it.

But clearly the \ of non-valid escape chars seems to be wrong.

Of course it could be that it's because I am tired or because I do not really know JavaScript but it might not be. I think it probably isn't either of those. But of course depending on what our tool needs to do that deviates from a normal json encoder ....

[xexyl]

One thing that should be done and which I will do is to rename the utf8encode() function to not suggest encoding or decoding but rather unicode. This way there is no confusion in the decode function.

[xexyl]

Something I do not understand, looking at the so-called json spec and the JavaScript output, is that for the non valid \- characters, it would appear that the \ should just be part of the string. But then in JavaScript it looks like the \ is removed and the character is there.

I presume that this is not understanding the standard correctly (if we can call something about it 'correct' :-) ) or because I am tired and not reading the grammar right. I rather suspect it's a bit of both.

[lcn2]

We are in the process of making some minor edits to jstrdecode(1) (and friends). We will apply these same edits to the mkiocccmetry repo. This in order to fix some minor issues with make quick_www and make www in the "other (temp-test-ioccc) repo".

We do not expect these edits to me major, just a few tweaks needed.

Once the make quick_www and make www work well, we will issue a PR with respect to this repo as well as an commit update to the mkiocccmetry repo and finally a commit to the "other (temp-test-ioccc) repo".

Please, @xexyl, stand by / try to withhold any edits and PRs for short term. We will advise when we are done.

[xexyl]

We are in the process of making some minor edits to jstrdecode(1) (and friends). We will apply these same edits to the mkiocccmetry repo. This in order to fix some minor issues with make quick_www and make www in the "other (temp-test-ioccc) repo".

We do not expect these edits to me major, just a few tweaks needed.

Once the make quick_www and make www work well, we will issue a PR with respect to this repo as well as an commit update to the mkiocccmetry repo and finally a commit to the "other (temp-test-ioccc) repo".

Please, @xexyl, stand by / try to withhold any edits and PRs for short term. We will advise when we are done.

Thank you! I won't make any edits. I am done for the day and given that (as I brought up in the other thread) I don't even know what you have in mind for the functionality I probably can't do much anyway.

[lcn2]

We have a solution that appears to work well for make www. We are doing to dinner and then we will come back a re-test. If all does well we will issue a PR for this repo as well as commits for the other repos.

[lcn2]

We have a solution that appears to work well for make www. We are doing to dinner and then we will come back a re-test. If all does well we will issue a PR for this repo as well as commits for the other repos.

UPDATE 0

When we return after dinner, we plan to update the documentation as per our minor edits.

[lcn2]

NOTE: In all of these example, we do NOT have a trailing newline.

Regarding a string with a trailing backslash, and here our examples, consider this string:

foo\

When encoded by jstrencode(1), would produce, by default, the following JSON string:

"foo\\"

All good. But let's now mess with things. 🤭

What should happen if we were to give this to jstrdecode(1):

"foo\"

In the above, we have 4 characters foo\ enclosed with double quotes.

What should happen? An ERROR should be thrown.

Depending on how you want to slice the problem, one could consider the string to end with \" and thus lacks the necessary trailing end double quote, OR the string does have enclosing double quotes but has a dangling backslash.

Either way, the JSON encoded string is invalid and the resulting JSON is not valid.

It's your choice as to what error you'd prefer to throw.

We recommend such a test case be added to the "invalid JSON" test suite.

[lcn2]

We have a solution that appears to work well for make www. We are doing to dinner and then we will come back a re-test. If all does well we will issue a PR for this repo as well as commits for the other repos.

UPDATE 0

When we return after dinner, we plan to update the documentation as per our minor edits.

See PR #29

UPDATE 0

With pr #29 and release 1.6.13 2024-11-16 of the mkiocccentry repo and commit c47f3abbe47cc63c33a5f4b99ac6ca35cb773dfe of the temp-test-ioccc repo we have now entered a soft code freeze for the mkiocccentry repo.

UPDATE 1a

Please see, @xexyl, GH-issuecomment-2480929883.

Please also see GH-issuecomment-2481266359.

[xexyl]

Good things above. I'll reply now. Unfortunately did not have time to address anything before the mkiocccentry sync but these things are not really important for there I guess. I don't think you addressed everything I brought up in this issue but we can discuss those later.

[xexyl]

NOTE: In all of these example, we do NOT have a trailing newline.

Regarding a string with a trailing backslash, and here our examples, consider this string:

foo\

When encoded by jstrencode(1), would produce, by default, the following JSON string:

"foo\\"

All good. But let's now mess with things. 🤭

What should happen if we were to give this to jstrdecode(1):

"foo\"

In the above, we have 4 characters foo\ enclosed with double quotes.

What should happen? An ERROR should be thrown.

Depending on how you want to slice the problem, one could consider the string to end with \" and thus lacks the necessary trailing end double quote, OR the string does have enclosing double quotes but has a dangling backslash.

Either way, the JSON encoded string is invalid and the resulting JSON is not valid.

It's your choice as to what error you'd prefer to throw.

We recommend such a test case be added to the "invalid JSON" test suite.

Other than this issue there are others as well.

It appears to me that we would have to keep track of each character in the decoding code. That would complicate matters greatly. We deliberately designed the regexp in the parser/lexer to not worry about certain details since we could take care of them in the code outside the parser/scanner. But the ending quotes would be easily solved by passing the strings through the parser for validation.

Some might say this would be overly complicated but it might be the cleaner way since otherwise we would have to add even more code to the decode functions (and maybe encode too?). It would be far easier to use the parser to do this and after all, isn't that the point of it - to verify it's valid and to provide a way to traverse it? We do not have to traverse it but it using the parse function would sure simplify these tools when it comes to errors.

What do you think?