Skip to content

Commit 7588562

Browse files
committed
Trim log docs to one window read and one tail example
The tail recipe with its seen dict, overlap paragraphs and the 0.5.x and 0.6.0 migration tables are gone; the example shows a window read and a tail loop over a single generator. Signed-off-by: Honglin Cao <hocao@nvidia.com>
1 parent 9ccc81a commit 7588562

2 files changed

Lines changed: 49 additions & 164 deletions

File tree

‎README.md‎

Lines changed: 19 additions & 115 deletions
Original file line numberDiff line numberDiff line change
@@ -55,135 +55,39 @@ delete the deployment automatically.
5555

5656
### Deployment logs SDK example
5757

58-
`fetch_logs()` is the one way to read deployment logs: it fetches a revision's
59-
stored log lines within a time window (`start_time`/`end_time`, epoch ms,
60-
inclusive; omit either bound to leave that side unbounded) and yields them lazily
61-
as chunks of up to `chunk_size` `DeploymentLogEvent` — each line at most once,
62-
each event carrying its pod name, with bounded memory however large the window.
63-
`newest_first` selects only the order chunks arrive: `False` (the default) walks
64-
the window oldest chunk first, `True` newest chunk first; lines inside every chunk
65-
are always in ascending `(timestamp, id)` order. So the bare call reads the full
66-
retained history chronologically, `newest_first=True` starts from the newest
67-
stored line and walks backward, and `start_time` alone catches up from a known
68-
point to the present. By default every pod of the revision is merged into one
69-
stream; pass `pod=` to read a single pod — discover names with
70-
`get_deployment_pods()` (terminated pods still within log retention are included):
58+
`fetch_logs()` is the one way to read deployment logs: it fetches one pod's stored
59+
log lines within a time window (`start_time`/`end_time`, epoch ms, inclusive) and
60+
yields them lazily, oldest first, as chunks of up to `chunk_size`
61+
`DeploymentLogEvent` — each line at most once, with bounded memory however long
62+
the stream. Discover pod names with `get_deployment_pods()` (terminated pods
63+
still within log retention are included). `start_time` defaults to the moment of
64+
the call; with `end_time` set the iterator terminates once the window is
65+
delivered:
7166

7267
```python
73-
for chunk in cclient.fetch_logs(DEPLOYMENT_ID, REVISION, start_time=t1_ms, end_time=t2_ms):
68+
for chunk in cclient.fetch_logs(DEPLOYMENT_ID, REVISION, pod, start_time=t1_ms, end_time=t2_ms):
7469
for event in chunk:
7570
print(event.pod, event.message)
7671
```
7772

78-
The iterator always terminates, and its exhaustion is the only termination signal:
79-
one that yields nothing means the window holds no stored lines (aged out of
80-
retention, before the deployment existed, an unknown pod, or genuinely empty).
81-
Reading backward, note that a line the log store receives late for a time region
82-
the walk has already passed is absent from that call: everything older than
83-
roughly the read's start minus the ingest lag (~15s) is complete, and when
84-
completeness of the newest lines matters, read forward. There is no follow mode:
85-
tailing is a caller loop that re-calls a forward `fetch_logs` with a later
86-
`start_time` and deduplicates by `event.id`:
73+
Without `end_time` the same generator tails: it never terminates, and once caught
74+
up it yields an empty chunk each time nothing new is stored yet — the caller
75+
decides when to sleep or break:
8776

8877
```python
8978
import time
9079

91-
OVERLAP_MS = 30_000 # covers the server's ~15s late-arrival re-delivery span
92-
POLL_SECONDS = 2.0
93-
94-
seen = {} # event id -> timestamp, trimmed to the overlap window each round
95-
boundary = int(time.time() * 1000) # newest timestamp seen so far
96-
while True:
97-
for chunk in cclient.fetch_logs(
98-
DEPLOYMENT_ID, REVISION, start_time=max(boundary - OVERLAP_MS, 0), newest_first=False
99-
):
100-
for event in chunk:
101-
if event.id in seen:
102-
continue
103-
seen[event.id] = event.timestamp
104-
boundary = max(boundary, event.timestamp)
105-
print(event.pod, event.message)
106-
cutoff = boundary - OVERLAP_MS
107-
seen = {event_id: ts for event_id, ts in seen.items() if ts >= cutoff}
108-
time.sleep(POLL_SECONDS)
109-
```
110-
111-
Two properties of this loop matter. Consecutive windows overlap on purpose: the log
112-
store may deliver a line up to ~15 seconds after its timestamp, so starting each call
113-
`OVERLAP_MS` below the newest seen line is what keeps late arrivals from being
114-
skipped — do not advance `start_time` past that span to avoid the duplicates. And the
115-
dedup state is bounded: only ids inside the overlap window can come back again, so
116-
`seen` is trimmed to that window each round and never grows with the stream.
117-
118-
`python examples/sdk/get_deployment_logs.py` runs a newest-first peek, a full
119-
chronological read and this tail loop. `get_deployment_logs()`, `get_deployment_logs_range()` and
120-
`deployment_log_session()` still work but are deprecated in favor of `fetch_logs()`
121-
and raise a `DeprecationWarning` on use.
122-
123-
### Migrating deployment log reads from 0.5.x
124-
125-
`get_deployment_logs()` kept its name but not its signature, and is now deprecated:
126-
`start_time`, `end_time`, `line_count`, `start_from_head` and `stream` are gone, and
127-
`fetch_logs()` is the replacement for every read. A 0.5.x call raises `TypeError`
128-
(or a validation error, if its arguments were positional) rather than returning
129-
something wrong, so no call site fails silently.
130-
131-
| To | 0.5.x | now |
132-
|---|---|---|
133-
| Read a time window | `get_deployment_logs(id, rev, start_time=, end_time=)` | `fetch_logs(id, rev, start_time=, end_time=)` |
134-
| Stream a window lazily | the same call with `stream=True` | `fetch_logs(...)` — chunks are yielded as they are fetched |
135-
| Take the newest lines first | `start_from_head=False` | `fetch_logs(id, rev, newest_first=True)` |
136-
| Read from the beginning | `start_from_head=True` | `fetch_logs(id, rev)` — oldest first is the default |
137-
| Cap what one iteration hands you | `line_count=n` | `chunk_size=n` |
138-
| Tell which pod a line came from | parse `kubernetes.pod_name` out of `message` | `event.pod` |
139-
| Keep tailing past the window | not supported | re-call `fetch_logs` with overlapping windows (the tail loop above) |
140-
141-
A whole-window read loses its envelope parsing, because `message` is now the log line
142-
itself rather than a JSON record wrapping it:
143-
144-
```python
145-
# 0.5.x
146-
events = cclient.get_deployment_logs(DEPLOYMENT_ID, REVISION, start_time=t1, end_time=t2)
147-
for event in events:
148-
record = json.loads(event["message"])
149-
print(record["kubernetes"]["pod_name"], record["log"])
150-
151-
# now
152-
for chunk in cclient.fetch_logs(DEPLOYMENT_ID, REVISION, start_time=t1, end_time=t2):
80+
for chunk in cclient.fetch_logs(DEPLOYMENT_ID, REVISION, pod):
81+
if not chunk:
82+
time.sleep(2)
83+
continue
15384
for event in chunk:
15485
print(event.pod, event.message)
15586
```
15687

157-
A `stream=True` loop becomes a `fetch_logs()` loop, which yields each chunk as it
158-
arrives just as the old generator yielded pages:
159-
160-
```python
161-
# 0.5.x
162-
for event in cclient.get_deployment_logs(
163-
DEPLOYMENT_ID, REVISION, start_time=t1, end_time=t2, stream=True
164-
):
165-
print(json.loads(event["message"])["log"])
166-
167-
# now
168-
for chunk in cclient.fetch_logs(DEPLOYMENT_ID, REVISION, start_time=t1, end_time=t2):
169-
for event in chunk:
170-
print(event.message)
171-
```
172-
173-
One contract change to check error handling against: a revision that does not exist
174-
now answers 404 where the old endpoint answered 400.
175-
176-
The 0.6.0 readers — `get_deployment_logs()` page anchoring, `get_deployment_logs_range()`
177-
and `deployment_log_session()` — still work but are deprecated and warn on use; the
178-
look-behind anchoring they exposed is handled inside `fetch_logs()`:
179-
180-
| 0.6.0 | now |
181-
|---|---|
182-
| `get_deployment_logs(id, rev, pod)` — newest page of one pod | `fetch_logs(id, rev, pod=pod, newest_first=True)` and take the first chunk |
183-
| `get_deployment_logs(id, rev, pod, after=events)` — page newer than held events | `fetch_logs(id, rev, pod=pod, start_time=boundary_ms)` (the tail loop above for repeated polling) |
184-
| `get_deployment_logs_range(id, rev, start_time=, end_time=)` | `fetch_logs(id, rev, start_time=, end_time=)` — chunked and lazy instead of one list |
185-
| `deployment_log_session(...).fetch_older()` loop | `fetch_logs(id, rev, pod=pod, newest_first=True)` — one iterator walks back to the start |
186-
| `session.fetch_newer()` polling | the tail loop above |
88+
`python examples/sdk/get_deployment_logs.py` runs both. `get_deployment_logs()`,
89+
`get_deployment_logs_range()` and `deployment_log_session()` still work but are
90+
deprecated in favor of `fetch_logs()` and raise a `DeprecationWarning` on use.
18791

18892
### Un-installation
18993

‎examples/sdk/get_deployment_logs.py‎

Lines changed: 30 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,8 @@
66
# --- Configuration ---
77
DEPLOYMENT_ID = 1234 # Replace with your deployment ID
88
REVISION_NUMBER = 10
9-
RECENT_LINES = 20 # How many of the newest stored lines to peek at
9+
WINDOW_MINUTES = 10 # How far back the window read looks
1010
TAIL_LINES = 20 # How many tailed lines to print before stopping the tail loop
11-
OVERLAP_MS = 30_000 # Covers the server's ~15s late-arrival re-delivery span
1211
POLL_SECONDS = 2.0
1312

1413

@@ -19,58 +18,40 @@ def format_event(event) -> str:
1918

2019
def main():
2120
with get_centml_client() as cclient:
22-
# The newest stored lines, without reading the whole history: newest_first
23-
# walks the window backward, one chunk at a time. Direction changes only
24-
# the order chunks arrive — lines inside each chunk are always ascending.
25-
print(f"Newest {RECENT_LINES} lines of deployment {DEPLOYMENT_ID} revision {REVISION_NUMBER}:\n")
26-
printed = 0
27-
for chunk in cclient.fetch_logs(DEPLOYMENT_ID, REVISION_NUMBER, newest_first=True, chunk_size=RECENT_LINES):
21+
# Discover pod names; terminated pods still within log retention are included.
22+
pods = cclient.get_deployment_pods(DEPLOYMENT_ID, REVISION_NUMBER)
23+
pod = pods[0]
24+
25+
# A window read: start_time/end_time are epoch ms, inclusive. With end_time
26+
# set the iterator terminates once the window is delivered. fetch_logs is
27+
# lazy — chunks are yielded as they are fetched, with bounded memory
28+
# however large the window.
29+
now_ms = int(time.time() * 1000)
30+
print(f"Last {WINDOW_MINUTES} minutes of pod {pod}:\n")
31+
count = 0
32+
for chunk in cclient.fetch_logs(
33+
DEPLOYMENT_ID, REVISION_NUMBER, pod, start_time=now_ms - WINDOW_MINUTES * 60_000, end_time=now_ms
34+
):
2835
for event in chunk:
2936
print(format_event(event))
30-
printed += len(chunk)
31-
if printed >= RECENT_LINES:
32-
break
33-
34-
# A full chronological read, all pods merged. fetch_logs is lazy — chunks
35-
# are yielded as they are fetched, with bounded memory however large the
36-
# window — and always terminates once caught up. Bound the window with
37-
# start_time/end_time (epoch ms, inclusive) when the history is long.
38-
count = 0
39-
for chunk in cclient.fetch_logs(DEPLOYMENT_ID, REVISION_NUMBER):
4037
count += len(chunk)
41-
print(f"\nFull retained history holds {count} lines.")
38+
print(f"\nThe window holds {count} lines.")
4239

43-
# Tailing is a caller loop: re-call a forward fetch_logs with the next
44-
# window starting OVERLAP_MS below the newest seen line, so lines the log
45-
# store delivers late (up to ~15s after their timestamp) are not skipped,
46-
# and deduplicate by event.id. Only ids inside the overlap window can come
47-
# back again, so trimming `seen` to it keeps the loop's memory bounded.
48-
print(f"\nTailing; stopping after {TAIL_LINES} new lines...")
49-
seen = {} # event id -> timestamp, trimmed to the overlap window each round
50-
boundary = int(time.time() * 1000) # newest timestamp seen so far
40+
# A tail: without end_time the same generator never terminates. start_time
41+
# defaults to the moment of the call, and once caught up the generator
42+
# yields an empty chunk each time nothing new is stored yet — the caller
43+
# decides when to sleep or break. No line is ever delivered twice.
44+
print(f"\nTailing pod {pod}; stopping after {TAIL_LINES} new lines...")
5145
printed = 0
52-
while printed < TAIL_LINES:
53-
for chunk in cclient.fetch_logs(
54-
DEPLOYMENT_ID, REVISION_NUMBER, start_time=max(boundary - OVERLAP_MS, 0), newest_first=False
55-
):
56-
for event in chunk:
57-
if event.id in seen:
58-
continue
59-
seen[event.id] = event.timestamp
60-
boundary = max(boundary, event.timestamp)
61-
print(format_event(event))
62-
printed += 1
63-
cutoff = boundary - OVERLAP_MS
64-
seen = {event_id: ts for event_id, ts in seen.items() if ts >= cutoff}
65-
time.sleep(POLL_SECONDS)
66-
67-
# A single pod (discover names with get_deployment_pods; terminated pods
68-
# still within log retention are included), with caller-sized chunks:
69-
# pods = cclient.get_deployment_pods(DEPLOYMENT_ID, REVISION_NUMBER)
70-
# for chunk in cclient.fetch_logs(
71-
# DEPLOYMENT_ID, REVISION_NUMBER, pod=pods[0], start_time=t1_ms, end_time=t2_ms, chunk_size=500
72-
# ):
73-
# ...
46+
for chunk in cclient.fetch_logs(DEPLOYMENT_ID, REVISION_NUMBER, pod):
47+
if not chunk:
48+
time.sleep(POLL_SECONDS)
49+
continue
50+
for event in chunk:
51+
print(format_event(event))
52+
printed += len(chunk)
53+
if printed >= TAIL_LINES:
54+
break
7455

7556

7657
if __name__ == "__main__":

0 commit comments

Comments
 (0)