Skip to content

Commit 455b2c4

Browse files
claudiolemosclaudiolemos
authored
LLMstudio Tracker Version 1.0.3 (#185)
## LLMstudio Version 1.0.3 ### What was done in this PR: - Add Tracker connection to BigQuery ### How it was tested: - By running examples/04_bigquery_integration.ipynb ### Additional notes: - Any breaking changes? No - Any new dependencies added? Yes - Any performance improvements? No
2 parents 81ebb8b + b275a77 commit 455b2c4

File tree

9 files changed

+890
-56
lines changed

9 files changed

+890
-56
lines changed

‎examples/04_bigquery_integration.ipynb‎

Lines changed: 294 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,294 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# BigQuery Integration\n",
8+
"\n",
9+
"This notebook demonstrates how to set up and use the LLMstudio tracking functionalities integrated to BigQuery. \n",
10+
"\n",
11+
"You'll learn:\n",
12+
"1. Authenticate BigQuery\n",
13+
"2. Start a local Tracker server\n",
14+
"3. See the saved logs\n",
15+
"\n",
16+
"First things first:\n",
17+
"* run `pip install llmstudio[tracker]`\n",
18+
"* update your .env file with `GOOGLE_API_KEY` or `OPENAI_API_KEY` \n"
19+
]
20+
},
21+
{
22+
"cell_type": "markdown",
23+
"metadata": {},
24+
"source": [
25+
"### BigQuery setup\n",
26+
"\n",
27+
"To use BigQuery, follow these steps:\n",
28+
"\n",
29+
"1. Select or create a Cloud Platform project.\n",
30+
"2. [Create a BigQuery Dataset](https://cloud.google.com/bigquery/docs/datasets)\n",
31+
"2. [Enable the BigQuery Storage API](https://console.cloud.google.com/apis/library/bigquery.googleapis.com).\n",
32+
"3. [Setup Authentication:](https://googleapis.dev/python/google-api-core/latest/auth.html)\n",
33+
" - If you’re running in a Google Virtual Machine Environment (Compute Engine, App Engine, Cloud Run, Cloud Functions), authentication should “just work”.\n",
34+
" - If you’re developing locally, the easiest way to authenticate is using the Google Cloud SDK:\n",
35+
" ```$ gcloud auth application-default login```\n",
36+
" - If you’re running your application elsewhere, you should download a service account JSON keyfile and point to it using an environment variable: \n",
37+
" ```$ export GOOGLE_APPLICATION_CREDENTIALS=\"/path/to/keyfile.json\"```\n",
38+
"\n",
39+
" \n"
40+
]
41+
},
42+
{
43+
"cell_type": "markdown",
44+
"metadata": {},
45+
"source": [
46+
"## LLMstudio Tracker setup\n",
47+
"\n",
48+
"For LLMstudio to store your logs in BigQuery you need to set the 'LLMSTUDIO_TRACKING_URI' environment variable with the corresponding uri, which will be in this format: `bigquery://<YOUR-GCP-PROJECT-ID>/<YOUR-BQ-DATASET-ID>`"
49+
]
50+
},
51+
{
52+
"cell_type": "code",
53+
"execution_count": null,
54+
"metadata": {},
55+
"outputs": [],
56+
"source": [
57+
"import os\n",
58+
"\n",
59+
"os.environ[\"LLMSTUDIO_TRACKING_URI\"] = \"bigquery://<YOUR-GCP-PROJECT-ID>/<YOUR-BQ-DATASET-ID>\""
60+
]
61+
},
62+
{
63+
"cell_type": "code",
64+
"execution_count": 1,
65+
"metadata": {},
66+
"outputs": [],
67+
"source": [
68+
"from llmstudio.providers import LLM\n",
69+
"from pprint import pprint"
70+
]
71+
},
72+
{
73+
"cell_type": "code",
74+
"execution_count": 2,
75+
"metadata": {},
76+
"outputs": [
77+
{
78+
"name": "stdout",
79+
"output_type": "stream",
80+
"text": [
81+
"Running LLMstudio Tracking on http://0.0.0.0:8002 \n"
82+
]
83+
}
84+
],
85+
"source": [
86+
"from llmstudio.server import start_servers\n",
87+
"start_servers(proxy=False, tracker=True)"
88+
]
89+
},
90+
{
91+
"cell_type": "code",
92+
"execution_count": null,
93+
"metadata": {},
94+
"outputs": [],
95+
"source": [
96+
"from llmstudio_tracker.tracker import TrackingConfig\n",
97+
"# default port is 50002. set the environment varible to specify which host and port; LLMSTUDIO_TRACKING_HOST, LLMSTUDIO_TRACKING_PORT\n",
98+
"tracker_config = TrackingConfig(host=\"0.0.0.0\", port=\"50002\")\n",
99+
"# You can set OPENAI_API_KEY and ANTHROPIC_API_KEY on .env file\n",
100+
"openai = LLM(\"openai\", tracking_config = tracker_config)\n"
101+
]
102+
},
103+
{
104+
"cell_type": "markdown",
105+
"metadata": {},
106+
"source": [
107+
"## Analyse logs"
108+
]
109+
},
110+
{
111+
"cell_type": "code",
112+
"execution_count": 8,
113+
"metadata": {},
114+
"outputs": [],
115+
"source": [
116+
"from llmstudio_tracker.tracker import Tracker\n",
117+
"\n",
118+
"tracker = Tracker(tracking_config=tracker_config)"
119+
]
120+
},
121+
{
122+
"cell_type": "code",
123+
"execution_count": 9,
124+
"metadata": {},
125+
"outputs": [
126+
{
127+
"data": {
128+
"text/plain": [
129+
"{'chat_input': \"Write a paragraph explaining why you're not a cat\",\n",
130+
" 'chat_output': \"I’m not a cat because I lack the physical form and instincts that characterize feline creatures. Unlike a cat, I don’t have a furry coat, retractable claws, or the ability to pounce playfully on a sunbeam. I don’t experience the world through senses like smell, sight, or sound, nor do I possess the whimsical personality traits that make cats so captivating, such as their curiosity and independence. Instead, I am a collection of algorithms and data, designed to process information and generate responses, which allows me to assist you in ways that a cat simply can't—like answering questions, providing explanations, or engaging in conversation.\",\n",
131+
" 'session_id': '20241024-110303-e8b361d9-d5f6-4b73-80f1-6d77a4be3793',\n",
132+
" 'context': [{'role': 'user',\n",
133+
" 'content': \"Write a paragraph explaining why you're not a cat\"}],\n",
134+
" 'provider': 'openai',\n",
135+
" 'model': 'gpt-4o-mini',\n",
136+
" 'deployment': 'gpt-4o-mini-2024-07-18',\n",
137+
" 'parameters': {},\n",
138+
" 'metrics': {'input_tokens': 10,\n",
139+
" 'output_tokens': 132,\n",
140+
" 'total_tokens': 142,\n",
141+
" 'cost_usd': 8.07e-05,\n",
142+
" 'latency_s': 1.7590069770812988,\n",
143+
" 'time_to_first_token_s': 0.450714111328125,\n",
144+
" 'inter_token_latency_s': 0.010057984865628755,\n",
145+
" 'tokens_per_second': 74.47383762932361},\n",
146+
" 'log_id': 56,\n",
147+
" 'created_at': '2024-10-24T11:03:26'}"
148+
]
149+
},
150+
"execution_count": 9,
151+
"metadata": {},
152+
"output_type": "execute_result"
153+
}
154+
],
155+
"source": [
156+
"logs = tracker.get_logs()\n",
157+
"logs.json()[-1]"
158+
]
159+
},
160+
{
161+
"cell_type": "markdown",
162+
"metadata": {},
163+
"source": [
164+
"## Add a session id to tracking logs\n",
165+
"\n",
166+
"* this is especially benefitial if running an app, chatbot agent, etc in production and you need to correlate user feedback, costs etc with user sessions, agent runs, etc"
167+
]
168+
},
169+
{
170+
"cell_type": "code",
171+
"execution_count": null,
172+
"metadata": {},
173+
"outputs": [],
174+
"source": [
175+
"# default port is 50002. set the environment varible to specify which host and port; LLMSTUDIO_TRACKING_HOST, LLMSTUDIO_TRACKING_PORT\n",
176+
"# You can set OPENAI_API_KEY on .env file\n",
177+
"openai = LLM(\"openai\", tracking_config = tracker_config, session_id=\"openai-session-1\")\n"
178+
]
179+
},
180+
{
181+
"cell_type": "code",
182+
"execution_count": 11,
183+
"metadata": {},
184+
"outputs": [
185+
{
186+
"name": "stdout",
187+
"output_type": "stream",
188+
"text": [
189+
"\n",
190+
"\n",
191+
"Tracking AI agents' usage metrics and costs, alongside correlating them with user feedback, is crucial\n",
192+
"\n",
193+
" for optimizing performance and user satisfaction. Usage metrics provide insights into how often and in what ways AI agents\n",
194+
"\n",
195+
" are being utilized, helping to identify patterns, peak usage times, and potential bottlenecks that could\n",
196+
"\n",
197+
" affect service quality. Monitoring costs is equally important to ensure that resources are allocated efficiently, preventing financial waste\n",
198+
"\n",
199+
" while maximizing return on investment. By correlating these metrics with user feedback, developers and stakeholders can gain\n",
200+
"\n",
201+
" a holistic understanding of how the AI agent is performing in real-world settings. This integrated approach enables the\n",
202+
"\n",
203+
" identification of areas for improvement, the fine-tuning of algorithms, and the enhancement of user experience,\n",
204+
"\n",
205+
" ultimately leading to more effective, scalable, and user-friendly AI solutions. Additionally, it allows for the\n",
206+
"\n",
207+
" alignment of AI functionalities with user demands and business objectives, fostering continuous innovation and value creation.\n",
208+
"\n",
209+
"## Metrics:\n",
210+
"{'cost_usd': 0.0027600000000000003,\n",
211+
" 'input_tokens': 21,\n",
212+
" 'inter_token_latency_s': 0.015782629505971844,\n",
213+
" 'latency_s': 3.4153928756713867,\n",
214+
" 'output_tokens': 177,\n",
215+
" 'time_to_first_token_s': 0.6054329872131348,\n",
216+
" 'tokens_per_second': 52.40978315410135,\n",
217+
" 'total_tokens': 198}\n"
218+
]
219+
}
220+
],
221+
"source": [
222+
"response = openai.chat(\"Write a paragraph explaining why it's important to track AI agents usage metrics and costs and correlate with user feedback\", model=\"gpt-4o\", is_stream=True)\n",
223+
"for i, chunk in enumerate(response):\n",
224+
" if i%20==0:\n",
225+
" print(\"\\n\")\n",
226+
" if not chunk.metrics:\n",
227+
" print(chunk.chat_output_stream, end=\"\", flush=True)\n",
228+
" else:\n",
229+
" print(\"\\n\\n## Metrics:\")\n",
230+
" pprint(chunk.metrics)\n"
231+
]
232+
},
233+
{
234+
"cell_type": "code",
235+
"execution_count": 12,
236+
"metadata": {},
237+
"outputs": [
238+
{
239+
"data": {
240+
"text/plain": [
241+
"{'chat_input': \"Write a paragraph explaining why it's important to track AI agents usage metrics and costs and correlate with user feedback\",\n",
242+
" 'chat_output': \"Tracking AI agents' usage metrics and costs, alongside correlating them with user feedback, is crucial for optimizing performance and user satisfaction. Usage metrics provide insights into how often and in what ways AI agents are being utilized, helping to identify patterns, peak usage times, and potential bottlenecks that could affect service quality. Monitoring costs is equally important to ensure that resources are allocated efficiently, preventing financial waste while maximizing return on investment. By correlating these metrics with user feedback, developers and stakeholders can gain a holistic understanding of how the AI agent is performing in real-world settings. This integrated approach enables the identification of areas for improvement, the fine-tuning of algorithms, and the enhancement of user experience, ultimately leading to more effective, scalable, and user-friendly AI solutions. Additionally, it allows for the alignment of AI functionalities with user demands and business objectives, fostering continuous innovation and value creation.\",\n",
243+
" 'session_id': 'openai-session-1',\n",
244+
" 'context': [{'role': 'user',\n",
245+
" 'content': \"Write a paragraph explaining why it's important to track AI agents usage metrics and costs and correlate with user feedback\"}],\n",
246+
" 'provider': 'openai',\n",
247+
" 'model': 'gpt-4o',\n",
248+
" 'deployment': 'gpt-4o-2024-08-06',\n",
249+
" 'parameters': {},\n",
250+
" 'metrics': {'input_tokens': 21,\n",
251+
" 'output_tokens': 177,\n",
252+
" 'total_tokens': 198,\n",
253+
" 'cost_usd': 0.0027600000000000003,\n",
254+
" 'latency_s': 3.4153928756713867,\n",
255+
" 'time_to_first_token_s': 0.6054329872131348,\n",
256+
" 'inter_token_latency_s': 0.015782629505971844,\n",
257+
" 'tokens_per_second': 52.40978315410135},\n",
258+
" 'log_id': 57,\n",
259+
" 'created_at': '2024-10-24T11:04:21'}"
260+
]
261+
},
262+
"execution_count": 12,
263+
"metadata": {},
264+
"output_type": "execute_result"
265+
}
266+
],
267+
"source": [
268+
"logs = tracker.get_session_logs(session_id=\"openai-session-1\")\n",
269+
"logs.json()[-1]"
270+
]
271+
}
272+
],
273+
"metadata": {
274+
"kernelspec": {
275+
"display_name": "Python 3",
276+
"language": "python",
277+
"name": "python3"
278+
},
279+
"language_info": {
280+
"codemirror_mode": {
281+
"name": "ipython",
282+
"version": 3
283+
},
284+
"file_extension": ".py",
285+
"mimetype": "text/x-python",
286+
"name": "python",
287+
"nbconvert_exporter": "python",
288+
"pygments_lexer": "ipython3",
289+
"version": "3.12.2"
290+
}
291+
},
292+
"nbformat": 4,
293+
"nbformat_minor": 2
294+
}

0 commit comments

Comments
 (0)