Output of emojis seems to be somewhat broken

[thomasfowler]

Expected Behavior

I am seeing some odd character output with certain emojis. Not all emojis, but certainly some, are not being output correctly.

Current Behavior

When asking Llama2 model to respond using mostly emojis, some are output as follows:

🏋️\u200d♀️💪🏼👟🥊🚀🎉🤩👍

The \u200d plus the female symbol seems to be some kind of encoding issue

Environment and Context

llama-cpp-python Version: 0.1.77 (latest at time of writing)
Python Version: 3.11.4
Platform: Apple M1 Macbook 16GB
Llama2 Model: llama-2-7b-chat.ggmlv3.q4_1.bin via https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML

Steps to Reproduce

Here is a basic example, using the model file noted above, that seems to reliably re-create the issue

from llama_cpp import Llama

prompt = """
[INST] <<SYS>>
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.
<</SYS>>

Show me a list of fitness related emojis.[/INST]
"""

llm = Llama(
    model_path="models/llama-2-7b-chat.ggmlv3.q4_1.bin",
    n_gpu_layers=1,
    n_ctx=2048,
    n_batch=512,
    f16_kv=True,
    verbose=True
)

llm(prompt=prompt)

Produces the following output:

{'id': 'cmpl-61a22458-6094-4ebc-9498-eac52c29570d',
'object': 'text_completion',
'created': 1691949830,
'model': '/Users/thomas/Hobbies/fff-pledge/llama-2-7b-chat.ggmlv3.q4_1.bin',
'choices': [{'text': '🏋️\u200d♀️💪🏼👟🥊🚀🎉🤩👍',
'index': 0,
'logprobs': None,
'finish_reason': 'stop'}],
'usage': {'prompt_tokens': 58, 'completion_tokens': 40, 'total_tokens': 98}}

I have tried multiple versions of the llama2 model, from TheBloke, and despite the overall performance differences, the specific issue I am seeing remains.

[thomasfowler]

I've been doing some further digging and the issue may be related to the underlying way the models are generated using llama.cpp

I have run a similar test just using the llama.cpp console interactive mode application, thus taking llama-cpp-python out of the equation, and have had similar results:

I'm pasting a screenshot below because pasting the chars here, they seem to be getting lost:

Specifically, look at the "person meditating" emoji, and it has the same female symbol adjacent to the emoji as well as the placeholder chars where my terminal has no idea what the character should be.

However, this comment in the llama.cpp repo would suggest that there may be an implementation fix that could resolve the problem here:

ggml-org/llama.cpp#2231 (comment)

I have zero idea where to even start in this codebase to see if thats an applicable fix, but if any of the maintainers here would point me to the right place to get started, I'd be willing to give it a bash.

[David-Boucher-Dotmodus]

It would seem like this might be the cause:

llama-cpp-python/llama_cpp/llama.py

Line 1053 in 66fb034

"text": self.detokenize([token]).decode(

It attempts to decode each individual token, in this case the byte is part of a multibyte sequence. It errors (which is ignored) and you end up with a blank text representation of the token. We could put in a patch as suggested above but I don't know if that goes against the design intent for this function.

[tarpeyd12]

Related: #372