Cannot use Guidance with AZURE - ConstraintException

[doomgrave]

I'm working on a project that was made with llama2 and Guidance.
Now I wanted to transfer it to Azure using Gpt-Turbo, the problem it's that i cannot use the most basic Guidance function cause i get ConstraintException error.

I Share here the code and the error that i get really similar for both of these basic guidance query:

--- CODE ---

azure_api_key = 'MY_KEY'
azure_full_endpoint = 'https://my_portal.openai.azure.com/openai/deployments/AITEST/chat/completions?api-version=2023-07-01-preview'
azure_tokenizer_model="gpt-35-turbo"

azureai_model = guidance.models.AzureOpenAIChat(azure_endpoint=azure_full_endpoint,api_key=azure_api_key,model=azure_tokenizer_model)

with system():
    lm = azureai_model + "You are a helpful assistant."
with user():
    lm += "What is the meaning of life?"

with assistant():
    lm += "Answer: \"" + gen("meaning",max_tokens=50,stop="\"",suffix="\"",temperature=0.5)

print(lm)


with system():
    lm = azureai_model + "You are a helpful assistant. Only answer YES, NO"
with user():
    lm += """Life its Hard? 
    Only answer: ['YES', 'NO']"""
with assistant():
    lm += "Answer = " + select(['YES','NO'],name='answer')

print(lm)

--- ERROR---
Exception has occurred: ConstraintException
The model attempted to generate b's an AI language' after the prompt b'...f life?<|im_end|><|im_start|>assistant\nA', but that does
not match the given grammar constraints! Since your model is a remote API that does not support full guidance
integration we cannot force the model to follow the grammar, only flag an error when it fails to match.
You can try to address this by improving the prompt, making your grammar more flexible, rerunning with
a non-zero temperature, or using a model that supports full guidance grammar constraints.
File "C:\ProgrammiX\PYTHON_SCRIPTS\EXAMPLES\Guidance_Examples\Guidance_Llama_llamacpp_Chat_Azure.py", line 43, in
lm += "Answer: "" + gen("meaning",max_tokens=50,stop=""",suffix=""",temperature=0.5)
guidance.models._model.ConstraintException: The model attempted to generate b's an AI language' after the prompt b'...f life?<|im_end|><|im_start|>assistant\nA', but that does
not match the given grammar constraints! Since your model is a remote API that does not support full guidance
integration we cannot force the model to follow the grammar, only flag an error when it fails to match.
You can try to address this by improving the prompt, making your grammar more flexible, rerunning with
a non-zero temperature, or using a model that supports full guidance grammar constraints.

[hrusli]

I am having the same problem? any luck?

[SaarthShah]

Struggling with the same thing but with OpenAI

[hrusli]

OpenAI endpoint don't have direct support for guidance grammars, but through optimistic running we can still control them in ways that match the model type.

Based on that, it seems that we cant use partial generation or prompt with the "Assistant role".

with system():
    lm = chat_lm + "You are a helpful assistant."
with user():
    lm += "What is the meaning of life?\n"
    lm += "Answer: \""

with assistant():
    lm += gen("meaning",max_tokens=50,stop="\"",suffix="\"",temperature=0.5)

This should work even though its not exactly the same.

[hrusli]

This seems to work as well:

with system():
    lm = chat_lm + "You are a helpful assistant."
with user():
    lm += "What is the meaning of life?\n"

with assistant():
    lm += "Answer: \"" 

with assistant():
    lm += gen("meaning",max_tokens=50,stop="\"",suffix="\"",temperature=0.5)

[doomgrave]

With this we solve half of this problem!!!
Unfortunately still cannot use gen(regex="(YES|NO)") or select(['YES','NO'])

with system():
    lm = azureai_model + "You are a helpful assistant. Only answer YES, NO"
with user():
    lm += """Life its Hard? 
    Only answer: ['YES', 'NO']"""
with assistant():
    lm += "Answer = "
 with assistant():
    lm += select(['YES','NO'],name='answer')

When using select or regex it seems to reply always "b''" indeed if i use

lm += gen(regex="b''", name='answer')

It run without errors.

[hrusli]

great, hm it should work with the select with no problem, however I tried with gpt-3.5-turbo not azureai.
But yea the regex seems to have problem.
another workaround which is a bit redundant is to use python re::

with assistant():
    lm += gen("answer")
    # Use regex to find numbers
    numbers = re.findall(r'\d+', str(lm['answer']))

    # numbers will be a list of string numbers. To get the first number:
    if numbers:
        first_number = numbers[0]
    print(first_number) # this will print 7, and you can use this output and append it to lm (but use a temp gen to hold the prev value of lm)

not a perfect solution

[doomgrave]

We can always use solution like a regex after a full reply, but it almost kill the point of using guidance to force a response!

I made it work both with select and regex, both improving the system prompt and increasing the temperature.

with system():
    lm = azureai_model + "Answer to the questions only with YES or NO."
with user():
    lm += """Life its Hard? """
with assistant():
    lm += "Answer = "

##USE SELECT OR GEN
# with assistant():
#     lm += select(['YES','NO'],'answer')
    
# with assistant():
#     lm += gen(regex="(YES|NO)",'answer',temperature=0.1)

Cause sometimes it can fail a try/catch increasing the temperature at each iteration may work. Not a perfect solution either but better than nothing!

[jammiguelyfish]

any update to this? having the same issue with select