Description

• Fixes Support cross-region inference for Amazon Bedrock #1114. See this issue for context.
• Supersedes Add Llama 3.2 models to Amazon Bedrock Chat #1014.
• Adds support for Llama 3.2 models on Amazon Bedrock.

This PR automatically uses cross-region inference (CRI) on all models supporting CRI on Amazon Bedrock. To do so, the BedrockChat provider maintains a list of all models which support CRI in the us region area, and automatically prepends us. to the model ID internally.

Testing instructions

• Clone this branch and comment out this section in aws.py:

      if model_id in self.cri_models:
          model_id = "us." + model_id

• Send a message Llama 3.2 on Bedrock, and verify the request fails with the exception:

  ValueError: Error raised by bedrock service: An error occurred (ValidationException) when calling the InvokeModelWithResponseStream operation: Invocation of model ID meta.llama3-2-3b-instruct-v1:0 with on-demand throughput isn’t supported. Retry your request with the ID or ARN of an inference profile that contains this model.
  

• Uncomment the aforementioned section and restart Jupyter AI.
• Send another message to Llama 3.2 on Bedrock, and verify the request succeeds.

Demo