use a visually more appealing encoding

[christian-monch]

Fixes #232

This PR uses Unidecode to translate unicode characters into the ASCII-range before employing any dataverse-specific character quotations.

If unidecode() returns an empty string, the name "__not_representable_<X>" is used, where <X> is the length of the original string.

[christian-monch]

It should be noted that the results of unidecode() might still contain a lot of characters that are not allowed in dataverse directory names or dataverse file names. Those are still encoded in the format -<HEXDIGIT><HEXDIGIT> in the result of mangle_path().

[christian-monch]

@mih : I can rebase this PR once PR #257 is merged

[mih]

Why is

If unidecode() returns an empty string, the name "_not_representable" is used

done, rather than only the fallback on the hexcodes?

I cannot see from the test diff alone how it would look. Need to handcraft a test dataset and try.

[christian-monch]

Why is

If unidecode() returns an empty string, the name "_not_representable" is used

done, rather than only the fallback on the hexcodes?

I cannot see from the test diff alone how it would look. Need to handcraft a test dataset and try.

Good question. The answer is that we aimed at a human readable representation, and using the hexcodes would probably be confusing. I think it is a good idea though. If we would do that, we have to decide whether we want to distinguish hex-code-file names that are generated because unidecode() did not return anything and hex-code-file names that just exist in the dataset. If we want to distinguish that reliably, we need an escape mechanism to signal whether a file name belongs to the first class or the latter. That would also influence "normal" file names that contain the escape character.

We could also leave to interpretation to the user, who might know, which file names are "genuine" dataset file names and which file names are just a hex-code representation of names that are mapped on empty-strings by unidecode().

All in all, the simplest approach might be to use hex-codes if the unidecode()-result is empty, but without any escaping mechanism. I think the collision probability is low, comparable to the collision probability of using unidecode().

I will change the code.