Conversation
* feat: add LiteLLM as AI gateway provider * chore: tighten litellm pin to >=1.80.0,<1.87.0 * fix: address review feedback on LiteLLM provider - Add embedding client initialization in create_client() to match BaseOpenAI pattern, ensuring RAG evaluators work with BaseLiteLLM - Add defensive checks for empty response.choices and None content in send_messages() to prevent IndexError and literal 'None' strings - Skip litellm tests when litellm is not installed (fixes CI test job) - Fix flake8 E302 blank line violation in test file - Add tests for empty choices and None content edge cases --------- Co-authored-by: Aarish Irani <rheagalfire@gmail.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces BaseLiteLLM, a base class for LLM evaluators routing through LiteLLM, along with corresponding unit tests and setup requirements. Feedback highlights a critical security concern regarding the specified litellm version range in setup.py, which includes compromised and vulnerable versions; updating the constraint to >=1.84.0 is recommended. Additionally, removing the OpenAI-specific configuration validation in BaseLiteLLM is suggested to ensure compatibility and flexibility across different providers.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
|
|
||
| agent_requirements = _read_requirements("./requirements/agent.txt") | ||
| hhem_requirements = _read_requirements("./requirements/hhem_integration.txt") | ||
| litellm_requirements = ["litellm>=1.80.0,<1.87.0"] |
There was a problem hiding this comment.
The specified LiteLLM version range includes versions with known critical security vulnerabilities and compromises. Specifically, CVE-2026-42208 is a critical SQL injection vulnerability affecting versions >=1.81.16 to <1.83.7, and versions 1.82.7 and 1.82.8 were compromised on PyPI with a malicious backdoor. To ensure security, please update the version constraint to exclude these vulnerable and compromised versions. It is highly recommended to require at least version 1.84.0.
| litellm_requirements = ["litellm>=1.80.0,<1.87.0"] | |
| litellm_requirements = ["litellm>=1.84.0,<1.90.0"] |
| extra_params = cls.dynamic_config.model_extra or {} | ||
| cls.validate_config(extra_params) |
There was a problem hiding this comment.
Calling cls.validate_config(extra_params) enforces OpenAI-specific parameter constraints (such as restricting temperature strictly between 0 and 2). Since BaseLiteLLM is designed to support 100+ different providers via LiteLLM, these strict OpenAI-specific limits may block valid configurations for other models/providers that use different parameter ranges. Since LiteLLM and the underlying provider APIs already validate parameters and return descriptive errors, consider removing the local cls.validate_config(extra_params) call to allow full provider flexibility.
| extra_params = cls.dynamic_config.model_extra or {} | |
| cls.validate_config(extra_params) | |
| extra_params = cls.dynamic_config.model_extra or {} |
2 participants
No description provided.