add a new unified route for count_tokens endpoint #1293
Conversation
|
DescriptionSummary By MatterAI
🔄 What ChangedA new unified 🔍 Impact of the ChangeThis change adds a dedicated and standardized endpoint for token counting, significantly enhancing the gateway's capabilities by providing a consistent method to estimate token usage across various AI models. It centralizes the token counting logic, which improves maintainability and simplifies future integrations with additional providers. 📁 Total Files Changed
🧪 Test AddedN/A 🔒Security VulnerabilitiesPotential for unvalidated input in MotivationType of Change
How Has This Been Tested?
Screenshots (if applicable)Checklist
Related Issues. Tip Quality Recommendations
Tanka Poem ♫
Sequence DiagramsequenceDiagram
participant U as User
participant HonoApp as Hono App
participant MCH as MessagesCountTokensHandler
participant HU as HandlerUtils
participant EAP as External AI Provider
U->>HonoApp: POST /v1/messages/count_tokens (requestBody, headers)
Note over HonoApp: Route matched
HonoApp->>MCH: handle(c: Context)
MCH->>MCH: Parse request body (await c.req.json())
MCH->>MCH: Extract raw headers (Object.fromEntries(c.req.raw.headers))
MCH->>HU: constructConfigFromRequestHeaders(requestHeaders)
HU-->>MCH: camelCaseConfig
MCH->>HU: tryTargetsRecursively(c, camelCaseConfig, request, requestHeaders, 'messagesCountTokens', 'POST', 'config')
Note over HU: Selects provider based on config
HU->>EAP: POST /messages/count_tokens (or provider-specific endpoint)
EAP-->>HU: tokenCountResponse
HU-->>MCH: tryTargetsResponse
MCH-->>HonoApp: tryTargetsResponse
HonoApp-->>U: tokenCountResponse
alt Error Handling
MCH->>MCH: catch (err: any)
Note over MCH: Log error: console.log('messagesCountTokens error', err.message)
MCH-->>HonoApp: Error Response (status: 400/500, message)
HonoApp-->>U: Error Response
end
|
![matter-code-review[bot]](https://avatars.githubusercontent.com/in/474549?s=60&v=4){kind=small}
| /** | ||
| * Handles the '/messages' API request by selecting the appropriate provider(s) and making the request to them. | ||
| * | ||
| * @param {Context} c - The Cloudflare Worker context. | ||
| * @returns {Promise<Response>} - The response from the provider. | ||
| * @throws Will throw an error if no provider options can be determined or if the request to the provider(s) fails. | ||
| * @throws Will throw an 500 error if the handler fails due to some reasons | ||
| */ |
There was a problem hiding this comment.
📝 Documentation Gap
Issue: JSDoc comment describes '/messages' API but function handles '/messages/count_tokens' endpoint
Fix: Update documentation to match actual functionality
Impact: Prevents developer confusion about handler purpose
| /** | |
| * Handles the '/messages' API request by selecting the appropriate provider(s) and making the request to them. | |
| * | |
| * @param {Context} c - The Cloudflare Worker context. | |
| * @returns {Promise<Response>} - The response from the provider. | |
| * @throws Will throw an error if no provider options can be determined or if the request to the provider(s) fails. | |
| * @throws Will throw an 500 error if the handler fails due to some reasons | |
| */ | |
| /** | |
| * Handles the '/messages/count_tokens' API request by selecting the appropriate provider(s) and making the request to them. | |
| * | |
| * @param {Context} c - The Cloudflare Worker context. | |
| * @returns {Promise<Response>} - The response from the provider. | |
| * @throws Will throw an error if no provider options can be determined or if the request to the provider(s) fails. | |
| * @throws Will throw an 500 error if the handler fails due to some reasons | |
| */ |
| ): Promise<Response> { | ||
| try { | ||
| let request = await c.req.json(); | ||
| let requestHeaders = Object.fromEntries(c.req.raw.headers); |
There was a problem hiding this comment.
⚡️ Performance Improvement
Issue: Object.fromEntries creates unnecessary object copy when headers are already iterable
Fix: Pass headers directly to constructConfigFromRequestHeaders if it accepts Headers object
Impact: Reduces object creation overhead and improves memory efficiency
| let requestHeaders = Object.fromEntries(c.req.raw.headers); | |
| let requestHeaders = c.req.raw.headers; |
Tested by making request to count tokens (attached curls for reference)
**NOTE : ** Bedrock does not support counting tokens
vertex
anthropic