Remove dependency on LEVELS()

[lionel-]

Closes #1726

[lionel-]

I had to remove the tests for "unknown" encodings, i.e. for strings that are unmarked. The new API charIsUTF8() returns true for these strings and I couldn't find an API conformant way of detecting them.

Davis, you added these tests in #1187. Do you have a good sense of how important this case is? I'm not sure how this arises in practice. Also I would expect edge cases to be less common in general now that Windows supports UTF-8.

[DavisVaughan]

We determined the one place this is actually used right now is in rlang's chr_detect_dups() function here

rlang/src/internal/utils.c

Line 228 in 23f0f30

x = KEEP(r_obj_encode_utf8(x));

Which is used by names_as_unique()

rlang/src/internal/names.c

Line 49 in 23f0f30

r_obj* dups = KEEP(chr_detect_dups(new_names));

Which is a common name repair strategy in the tidyverse.

Since no tests failed we determined this is probably going to work fine.

---

It is also worth noting that this is not currently used in vctrs right now. We did do the work to expose this in the rlang C API as an rlang callable, but we never went back and updated vctrs to use it (instead it uses its own local copy of a similar function).

---

Lastly, here is the main change:

https://github.com/wch/r-source/blob/7f80ed7ae28959026e071a7c58cd38fa3a3a8f9f/src/main/sysutils.c#L962

The one case Rf_charIsUTF8 seems to be different for us is when:

• String is not "marked" as UTF8
• But utf8locale is set to true

In this case we previously assumed it was not UTF8 because it was "unknown" to us and we can't access utf8locale

But R can access utf8locale so I guess it assumes that if the string is not marked as anything and not NA and utf8locale is set, then it must be UTF8

This case is probably quite rare, and the only difference in behavior is that we used to call Rf_translateCharUTF8() on these "unknown" strings, but R seems to think they are UTF8 anyways, so that was probably already a no-op in terms of actual behavior (possibly it would add the UTF8 marking, but it probably didn't change the underlying bytes).