File name sanitation

[geoo89]

Gitsync generates filenames based on question/category titles.
I believe this is done by using a character blacklist to be replaced, but from my experience this still sometimes gives troubles.

For example, degree symbols appear to be problematic (e.g. Finding the Trigonometry Ratios in a 30°, 60° and 90° Triangle): We had a category with degree symbols, and while create_repo did create the appropriate files, they do not appear in the manifest. I've also had issues with degree symbols (and sometimes other symbols) when trying to open files e.g. in python scripts or referencing them in an XML attribute. (In this case e.g. UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc2' in position 122: surrogates not allowed)

What about using a whitelist instead of a blacklist for characters (though that may strip letters with accents altogether rather than removing the accent), or some existing slugify library or function (which will nicely handle international characters)?

[EJMFarrow]

I've had a look into this and it's a minefield. The reason for the blacklist is to ensure absolute consistency across UNIX and Windows while still allowing international characters e.g. Arabic and Chinese (not just characters with accents). I'd be happy to learn about a better way of achieving that, that isn't going to cause its own problems!

I've tested Gitsync with degree symbols in question titles and categories and it works fine for me on both UNIX and Windows and even copying files between them. Do you have the latest version? Maybe there is something odd with your setup.

The fact that files containing the symbol internally are failing to open is apparently Python looking for ASCII rather than UTF-8:
"Python Surrogate Escaping: When Python encounters bytes it cannot decode using a specific codec (like trying to read a UTF-8 file as ASCII), it may use "surrogateescape" to represent the unknown bytes. The byte 0xC2 becomes the surrogate character \udcc2"

[geoo89]

Hi Edmund,

thanks for looking into this. I upgraded to the latest version of gitsync and I cannot reproduce the issue anymore (i.e. the missing entry in the manifest). And looking at it, the entries with those symbols are 0x00B0 unicode characters now. The corresponding Python script is not throwing errors anymore either.

While I like my filenames without weird characters, I see how non-Latin scripts make the whole situation more complicated and I'm not aware of robust slugify solutions that handle that.