Skip to content

Commit adf6a2a

Browse files
lindongwlindongw
committed
add trace tool to support libva trace process
Signed-off-by: Lindong Wu <[email protected]>
1 parent 4af6a66 commit adf6a2a

27 files changed

+6388
-0
lines changed

‎tracetool/README.md

+157
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
# Linux trace tool
2+
3+
4+
## Introduction
5+
6+
This python tool helps to analysis media stack trace logs combining ftrace events from libva, media driver and Linux kernel mode driver (e.g. i915).
7+
8+
9+
## Linux trace capture
10+
11+
1. Install trace-cmd:
12+
13+
sudo apt-get install trace-cmd
14+
15+
2. Grant write access to trace node for application:
16+
17+
sudo chmod 777 /sys/kernel/debug/
18+
sudo chmod 777 /sys/kernel/debug/tracing/
19+
sudo chmod 777 /sys/kernel/debug/tracing/trace_marker_raw
20+
21+
3. Enable libva trace:
22+
23+
export LIBVA_TRACE = FTRACE
24+
25+
to enable libva buffer data capture
26+
27+
export LIBVA_TRACE_BUFDATA = 1
28+
29+
4. Run application under trace-cmd in a proxy mode:
30+
31+
trace-cmd record -e i915 <workflow-cmd-line>
32+
33+
5. Output is "trace.dat"
34+
35+
Alternatively you can collect trace data in separate terminal.
36+
It is useful if you want to profile daemon or a service:
37+
38+
1. Start trace capture:
39+
40+
sudo trace-cmd record -e i915
41+
42+
2. Run test app in another terminal
43+
3. Stop capturing in the first terminal
44+
4. Output is "trace.dat"
45+
46+
47+
## Trace post-processing and analysis
48+
49+
python3 main.py [-raw] file.dat|file.etl [file.dat|file.etl ...]
50+
51+
Options:
52+
53+
* `-raw` - Parse trace events and dump into <trace-file>.csv file.
54+
55+
Output:
56+
57+
* `<trace-file>.json.gz` - visualized trace activity, open in `<chrome://tracing/>` or `<edge://tracing/>`
58+
* `<trace-file>_stat.csv` - statistic of trace activity, open in Excel
59+
* `<trace-file>_surface.csv` - surface attributes and runtime usage in GPU, open in Excel
60+
* `<trace-file>_rtlog.txt` - iHD driver runtime log
61+
62+
63+
## Trace tool manifests
64+
65+
Trace tool uses manifests to decode trace data. Each trace module available in trace tool
66+
needs to have a manifest file in manifests folder. The manifest file could be either in
67+
MSFT ETW XML manifest format or in json format. Current supported traces:
68+
69+
* [libva_trace.man](./manifests/libva_trace.man) - libva trace manifest in MSFT ETW XML
70+
* [Intel-Media-Open.json](./manifests/Intel-Media-Open.json) - iHD media driver trace manifest in json
71+
72+
## Trace Modules
73+
74+
Trace tool loads trace modules from the [modules](./modules) folder. Two types of modules
75+
are supported:
76+
77+
* Trace readers
78+
* Trace handlers
79+
80+
Readers support reading data from the trace without performing any action on the read data.
81+
Handlers perform actions over read data. Effectively trace readers provide input to trace
82+
handlers.
83+
84+
Trace tool loads modules by class names.
85+
86+
### Trace core
87+
88+
Trace core loads trace modules making them available for trace tool. Key interfaces:
89+
90+
| Interface | Description |
91+
| --------- | ----------- |
92+
| `core.regParser(id, parser) -> int` | Registers trace header `parser` to the core. `id` is 4bytes trace identifier. |
93+
| `core.regHandler(sys, name, handler) -> None` | Registers event handler. Set name to None for common trace handler. |
94+
| `core.getContext() -> dict` | Get share context from the core. |
95+
96+
### Trace readers
97+
98+
Trace reader module is responsible for parsing trace file into trace events and call
99+
trace handlers one by one in event timestamp order.
100+
101+
Trace reader is driven by trace core. The following interfaces are required to be
102+
provide by the trace reader module implentation:
103+
104+
| Interface | Description |
105+
| --------- | ----------- |
106+
| `open(self, file, options) -> int` | Open trace file, returns 0 for sucess, < 0 for failures. User command line options are provided in args. |
107+
| `setParser(self, parsers) -> None` | Set trace header parsers. Since all Linux user space traces share single trace_marker_raw entry, each user trace(libva and iHD) need register its own header parser to identify itself. |
108+
| `syncSource(self, src) -> int` | For sync timestamp across multi trace readers. Return -1 for no support. |
109+
| `process(self, filter, handler) -> None` | Starts trace event process with filter and handler callbacks. Filter callback could speed up event process. |
110+
111+
Currently supported trace reader modules:
112+
113+
| Module | Description |
114+
| ---------- | ----------- |
115+
| ftrace.py | Linux ftace file reader, trace file from trace-cmd |
116+
117+
### Trace handlers
118+
119+
Trace event handler module customizes events handling. Since all handler modules are
120+
seperate instances, trace core provides shared context to share data between modules.
121+
122+
By default, shared context provides the following:
123+
124+
| Name in context | Description |
125+
| --------------- | ----------- |
126+
| `UI` | Instance of class `writeUI` (see [writeUI.py](./writeUI.py). Class writes events for chrome://tracing. |
127+
| `Stack` | Instance of class `callStack` (see [callStack.py](./callStack.py). Class provides call stack of trace event. Call stack is built from event sequence from the same process id and thread id. |
128+
| `Stat` | Instance of class `statistic` (see [statistic.py](./statistic.py). Class provides statistics for specific events. |
129+
| `Output` | Output path string in case module needs to create its own output file. |
130+
131+
Handler module only interact with core, should not export interface to external directly. Module register its own event handlers to core through `core.regHandler(sys, name, handler)`.
132+
133+
It is possible that multi modules register their own handlers for the same event. Core will call these callbacks one by one when target event occurs.
134+
135+
Handler module could write output into `UI` or `Stat` in share context, also could create and write its own output file and format. Output file path is available in share context `Output`.
136+
137+
In case a handler module is targeted to provide a common service, it could export its name and instance in share context. Its name in share context should be unique, other module use this name to get service instance. The service interface is defined by module itself. see example [surface.py](./modules/surface.py).
138+
139+
To add new trace support, handler module for this new trace MUST register event header parser to core, through `core.regParser(id, parser)`. This event header parser is to detect and parse trace header, otherwise trace reader could not recognize this new trace. The id for this new trace should be unique. refer example [libva.py](./modules/libva.py).
140+
141+
Currently supported handler modules:
142+
| Module | Description |
143+
| ---------- | ----------- |
144+
| i915.py | i915 trace handler to extract GPU workload submit & execution timing |
145+
| libva.py | libva trace handler |
146+
| iHD.py | Intel iHD open source media driver trace handler |
147+
| surface.py | Handler tracks surface object & attributes across iHD and i915 traces |
148+
149+
## Making changes in the tool
150+
151+
Make sure to run unit tests before creating PR:
152+
153+
cd tracetool
154+
python3 -m unittest
155+
156+
Make sure trace event and event data are backward compatible.
157+

‎tracetool/callStack.py

+70
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
#
2+
# Copyright (C) Intel Corporation. All rights reserved.
3+
# Licensed under the MIT License.
4+
#
5+
6+
# build call stack from events with the same process and thread id
7+
class callStack:
8+
9+
def __init__(self):
10+
self.context = {} # maintain call stack
11+
12+
# get latest pushed call event in stack
13+
def current(self, pid, tid):
14+
if pid not in self.context or tid not in self.context[pid]:
15+
return None
16+
if self.context[pid][tid]:
17+
return self.context[pid][tid][0]
18+
return None
19+
20+
# get full call stack record
21+
def get(self, pid, tid):
22+
if pid not in self.context:
23+
self.context[pid] = {}
24+
if tid not in self.context[pid]:
25+
self.context[pid][tid] = []
26+
return self.context[pid][tid]
27+
28+
# push event into stack
29+
def push(self, evt):
30+
if evt['pid'] not in self.context:
31+
self.context[evt['pid']] = {}
32+
if evt['tid'] not in self.context[evt['pid']]:
33+
self.context[evt['pid']][evt['tid']] = []
34+
self.context[evt['pid']][evt['tid']].insert(0, evt)
35+
36+
# pop event from stack
37+
def pop(self, evt):
38+
if evt['pid'] not in self.context:
39+
return None
40+
if evt['tid'] not in self.context[evt['pid']] or not self.context[evt['pid']][evt['tid']]:
41+
thrds = self.context[evt['pid']]
42+
for t in thrds.values():
43+
if t and t[0]['name'] == evt['name']:
44+
return t.pop(0)
45+
return None
46+
ctx = self.context[evt['pid']][evt['tid']]
47+
name = evt['name']
48+
idx = 0
49+
ret = None
50+
# find target in the stack
51+
for i in range(len(ctx)):
52+
if ctx[i]['name'] == name:
53+
idx = i+1
54+
ret = ctx[i]
55+
break
56+
# remove target from stack
57+
del ctx[0:idx]
58+
return ret
59+
60+
# find top event with the same sys id + pid + tid
61+
def find(self, evt):
62+
if evt['pid'] not in self.context or evt['tid'] not in self.context[evt['pid']]:
63+
return None
64+
for e in self.context[evt['pid']][evt['tid']]:
65+
if e['sys'] == evt['sys']:
66+
return e
67+
return None
68+
69+
def __del__(self):
70+
del self.context

‎tracetool/core.py

+160
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
#
2+
# Copyright (C) Intel Corporation. All rights reserved.
3+
# Licensed under the MIT License.
4+
#
5+
6+
import os
7+
import sys
8+
import importlib
9+
from writeUI import writeUI
10+
from statistic import statistic
11+
from callStack import callStack
12+
from util import *
13+
14+
class core:
15+
16+
def __init__(self):
17+
self.source = None
18+
self.sharedCtx = {} # shared context for all handlers, dict for flexible usage
19+
self.handlers = {}
20+
self.instances = []
21+
self.readers = []
22+
self.parsers = {}
23+
self.dumpRaw = False
24+
25+
cur = os.path.abspath(os.path.dirname(__file__))
26+
sys.path.append(cur+os.sep+'modules')
27+
for py in os.listdir('modules'):
28+
name = os.path.splitext(py)[0]
29+
m = importlib.import_module(name)
30+
if hasattr(m, 'traceHandler'):
31+
cls = getattr(m, 'traceHandler')
32+
# create handler instace, the class init should call register of this instance
33+
instance = cls(self)
34+
# just for keep instance ref
35+
self.instances.append(instance)
36+
elif hasattr(m, 'traceReader'):
37+
cls = getattr(m, 'traceReader')
38+
self.readers.append(cls)
39+
40+
# open trace file
41+
def open(self, input, options) -> int:
42+
ret = -1
43+
if isinstance(input, list) and len(input) == 1:
44+
input = input[0]
45+
if isinstance(input, list):
46+
# enumerate and open trace files
47+
names = []
48+
readers = []
49+
for i in input:
50+
for cls in self.readers:
51+
reader = cls()
52+
sts = reader.open(i, options)
53+
if sts == 0:
54+
names.append(i)
55+
readers.append(reader)
56+
break
57+
if len(input) == len(readers):
58+
# sync time stamp across multi trace files, need find single source reader
59+
print('Multi trace input files, sync time line ...')
60+
for i in readers:
61+
for j in readers:
62+
if i != j and i.syncSource(j) == 0:
63+
self.source = i
64+
self.sharedCtx['sourceFile'] = names[readers.index(i)]
65+
break
66+
if self.source != None:
67+
break
68+
if self.source != None:
69+
print('done')
70+
ret = 0
71+
else:
72+
print('Error! could not syn time line')
73+
else:
74+
for cls in self.readers:
75+
reader = cls()
76+
sts = reader.open(input, options)
77+
if sts == 0:
78+
self.source = reader
79+
self.sharedCtx['sourceFile'] = input
80+
ret = 0
81+
break
82+
# setup handlers and output if success
83+
if ret == 0:
84+
self.source.setParser(self.parsers)
85+
86+
baseName = self.sharedCtx['sourceFile']
87+
baseName = os.path.splitext(baseName)[0]
88+
self.sharedCtx['Output'] = baseName
89+
self.sharedCtx['UI'] = writeUI(baseName)
90+
self.sharedCtx['Stat'] = statistic(baseName)
91+
self.sharedCtx['Stack'] = callStack()
92+
self.sharedCtx['Opt'] = options
93+
return ret
94+
95+
# start process event from trace file
96+
def process(self) -> None:
97+
self.source.process(self.filter, self.callback)
98+
99+
# close
100+
def __del__(self):
101+
del self.source
102+
del self.readers
103+
del self.instances
104+
del self.handlers
105+
del self.sharedCtx
106+
107+
# test if event handler is set for this event
108+
def filter(self, evt) -> bool:
109+
if 'raw' in self.sharedCtx['Opt']:
110+
return True
111+
if 'sys' not in evt or 'name' not in evt:
112+
return False
113+
if evt['sys'] not in self.handlers:
114+
return False
115+
handler = self.handlers[evt['sys']]
116+
if evt['name'] not in handler and 'allEvent' not in handler:
117+
return False
118+
return True
119+
120+
# call back function to process event with handler
121+
def callback(self, evt) -> None:
122+
if evt['sys'] not in self.handlers:
123+
return
124+
# get handler, could be a list, multi handler for single event
125+
hnd = self.handlers[evt['sys']]
126+
flag = 0
127+
if evt['name'] in hnd:
128+
for h in hnd[evt['name']]:
129+
sts = h(evt)
130+
if sts != None and sts < 0:
131+
flag = 1
132+
# call all event handler at last step, skip if any handler has returned -1
133+
if 'allEvent' in hnd and flag == 0:
134+
for h in hnd['allEvent']:
135+
h(evt)
136+
137+
# register event handler
138+
def regHandler(self, sys, name, handler) -> None:
139+
if name == None:
140+
name = 'allEvent' # name = None means handler for all events of this trace system
141+
if sys not in self.handlers:
142+
self.handlers[sys] = {}
143+
# add handler to list
144+
hnd = self.handlers[sys]
145+
if name in hnd:
146+
hnd[name].append(handler)
147+
else:
148+
hnd[name] = [handler]
149+
150+
# register event head parser from raw message
151+
def regParser(self, id, parser) -> int:
152+
if id in self.parsers:
153+
print('Warning! duplicated event header id')
154+
return -1
155+
self.parsers[id] = parser
156+
return 0
157+
158+
# get shared context
159+
def getContext(self) -> dict:
160+
return self.sharedCtx

0 commit comments

Comments
 (0)