Smart Quotation Marks in Formattable Strings.

[Drodt]

Close #70.

I implemented a couple of changes to enable hayagriva to better render quotation marks in titles or other formattable strings.

The changes are:
When pushing a string to the buffer, we modify the string with the same heuristics as typst uses for smart quotes (taking into account any surrounding quotes created by hayagriva). To generate the quotation marks we add, we use the given style and language.

Changes in passing tests

Adds 6 tests to passing but removes one: With the new changing of quotation marks, we change flipflop_OrphanQuote to a slightly different local test.

Todo:

• Better parsing of quotation marks (following typst's heuristics)
• Additional Rust tests for quote funcions

Todo in biblatex:

• Handling quotation commands (\enquote, \mkbibquote)

Todo in typst:

• Move SmartQuoter to its own crate, or a util crate usable by typst and hayagriva

[reknih]

Should this also work for single quotes? The ChunkKind could track the "doubleness" with a boolean.

[DerDrodt]

Should this also work for single quotes? The ChunkKind could track the "doubleness" with a boolean.

There is no need for that, and it might even produce problems.

1. When rendering, hayagriva automatically checks whether it should use inner or outer quotation marks.
2. Single quotes are more ambiguous in titles; think of Dijkstra's algorithm and the like.

So long as double quotes are part of the spec, I think it should be fine.

[reknih]

If I'm reading your code correctly, BibTeX' normal chunks will continue to yield regular quotes. I think that we should parse BibTeX normal chunks to also emit the quote chunk so that both formats behave more similarly.

[Drodt]

If I'm reading your code correctly, BibTeX' normal chunks will continue to yield regular quotes. I think that we should parse BibTeX normal chunks to also emit the quote chunk so that both formats behave more similarly.

That would be good but how do we want this to work? In Bib(la)TeX, you don't use " to make quotation marks but \enquote. So we would need to handle this command in biblatex. But there we cannot just add some unicode symbols like for other commands.

The best way I see to do this is to make another ChunkKind in biblatex that handles the quotation commands. Is that fine by you?

[Drodt]

Another thing: I do think we will run into issues at some point by just parsing everything between " as a quote. (At some point someone will open an issue about weird behavior because they have two measurements in inches in their title.)

I think we should add support for simple typst commands like quote, smallcaps, etc. Thoughts?

[reknih]

Ultimately, smart quotes are a heuristic. Typst implements it here: https://github.com/typst/typst/blob/c20b6ec6e157b753e90aa60ccd5b50f6f31cb7f6/crates/typst/src/text/smartquote.rs#L91. This handles something like quotes between numbers (e.g. for measurements in inches). It could be worthwhile to extract this into a crate and use the same code here and there. Additionally, I would propose to tokenize only the quote inside a chunk instead of the complete citation it wraps and approach this as purely string replacement instead of semantic.

Adding a non-standard chunk to biblatex is not very appealing to me. Maybe the Normal chunks could have a method that parses the appropriate commands and returns a vector with an enum?

[Drodt]

Once you find the time, this PR is ready to merge @reknih @PgBiel

[PgBiel]

Thanks! Overall, mostly fine. I can apply most of my suggestions myself, but I still have a few concerns which will need a bit of additional discussion.

[PgBiel]

Interesting that the original test, which expected " at both positions, failed. I wonder if Hayagriva is just doing something that citeproc deliberately isn't? Can we find a reason for this to see if it is reasonable to override the test like that?

[Drodt]

On main the test passes. Hayagriva handles this correctly (aside from this branch using fancy quotation marks)

[PgBiel]

I do wonder if Typst should somehow wrap this in a smartquotes elements, which would render your changes potentially problematic... I'll try to ask the rest of the team on this.

[PgBiel]

After some initial discussion with part of the team, it's possible that we could want the smartquoter to emit an ElemChild::SmartQuote so that Typst can later emit a smartquote element, which would allow customization via show and set rules. The idea is then that the default plain text renderer would just convert it into text with the surrounding quotes. In addition, the chosen quotes stored in that element would be None if the CSL locale doesn't override the quote terms to allow for set rules to work on the Typst side.

However, we're still not sure regarding the algorithm. We would like to know how BibLaTeX approaches smart quoting so we can make sure existing .bib files behave at least somewhat as expected, converting one approach to another if necessary. It could be nice to have some further investigation in this regard.

Still, we won't make a decision before feedback from @reknih (who is currently unavailable), so don't worry about making too many changes right now, although feel free to investigate about BibLaTeX as that would be helpful.

[Drodt]

Sounds good.

I already looked into how biblatex does this; I‘ll write it up some time next week

[Drodt]

Biblatex does not handle " in any special way. For quotation marks, they provide the \mkbibquote command, which either produces American quotation marks (and, typically, moves punctuation like . or , inside the quote) or uses \enquote by the csquotes packages. The latter uses language-dependent quotation marks.

For conversion, we could translate \mkbibquote and \enquote to use normal quotation marks, which are then handled by hayagriva.