DeepQL

[Umaaz]

DeepQL is the primary interaction with Deep. We need to be able to create, delete, modify and search deep using this
protocol.

DeepQL - Create

Creating needs to be straight forward within Deep. Below are some ideas:

Common controls

In all cases there are shared controls that can be used. These include:

• Fire Count - the number of times to trigger this action
• Rate Limit - the number minimum time between subsequent triggers
• Window - The time window in which this trigger is active
• Targeting - The target resource to trigger on

Fire Count

This control allows you to set the number of times this action should be triggered be fore being disabled. Currently,
this is limited to act only on the clients. If fire count is set to 10, then each client can trigger this action a
maximum of 10 times. This control is reset each time the config of a client changes. Resulting is possibly more triggers
than expected.

see #81

Rate limit

This controls how frequently an action can be triggered. This prevents a trigger from running again until this time
elapses. If set to 1000ms then the trigger will only fire once per second.

Window

This control allows for a time window to be set for the trigger. A window can be a combination of window_start
and window_end, each are optional and need not be set. window_start will default to now, while window_end will
default to different values depending on the action type. The values can be absolute times, or relative times (e.g. 7d);
relative times are resolved on creation.

Targeting

This controls the targets that are provided the triggers. This allows you to install a trigger only on specific
services, or applications. The client resource definition is used to determine these targets.

If a client has a resource { 'service.name': 'my_app', 'app_version': '12' } and another client
has { 'service.name': 'my_app', 'app_version': '14' } you can install a trigger on just the client app_version: 14
by using the targeting: trigger{ app_version="14" }

Generic Trigger

A generic trigger can be created which allows you full control on the actions. This can be useful when creating more
complex actions.

# create snapshot and log on a line
trigger{ file="some_file.py" line=12 snapshot=true watch="self.name" log="inject a log" }

Full options

Full options

trigger{
 file="some_test.py"                           # the file name/path of the source file
 line=12                                       # the source line number
 method="name"                                 # the method name to look for in the file (cannot define line and method)
 
 # Optional Values
 
 # control options
 fire_count=-1                                 # the number of times to fire (-1 represents forever)
 rate_limit=1000                               # the minimum time in ms between fires
 window_end=48h                                # the relative or absolute time to stop collecting (0 means never end)
 window_start=now                              # the relative or absolute time to start collecting
 condition="{expression}"                      # conditional statement to control fire
 
 # snapshot options
 snapshot=false                                # should this log also capture a snapshot (default false)
 watch="expression"                            # define watch expressions to attach to the snapshot add multiple expressions as required
 watch="expression"
 snapshot_label_my_label="{expression}"        # snapshot labels for collected snapshot follow the same pattern as log labels
 snapshot_label_other_label="static"

 # metric options
 metric=false                                  # should this log also capture a metric (default false)
 metric_name="log_some_test_py_12"             # metric name for collected metric
 metric_label_my_label="{expression}"          # metric labels for collected metric follow the same pattern as log labels
 metric_label_other_label="static"
 
 # span options
 span=None                                     # should this log also create a span (default None)
 span_name="log_some_test_py_12"               # the name of the span
 span_label_my_label="{expression}"            # span labels for create span follw the same pattern as log labels
 span_label_other_label="static"
 
 # optional log values
 log="some injected log with {expression}"     # the injected log message with expressions
 log_label_my_label="{expression}"             # labels to attach to the log, `label_` is remove from the key to 
 log_label_other_label="static value"          # form the real key, value can be expression of static value
}

Logs

When creating logs it is generally the case you want this log to log always unless specified. However, we do not want
logs to continue for all time. As they can easily be forgotten, so the defaults for logs should be:

• Fire Count - Forever
• Rate Limit - 1000ms
• Window - now - +48hrs

So the below log created on Monday 12:00 will continue until Wednesday 12:00 where it will stop.

log{file="some_test.py" line=12 msg="some injected log with {expression}"}

Full options

Full options

log{
 file="some_test.py"                           # the file name/path of the source file
 line=12                                       # the source line number
 msg="some injected log with {expression}"     # the injected log message with expressions
 
 # optional log values
 label_my_label="{expression}"                 # labels to attach to the log, `label_` is remove from the key to 
 label_other_label="static value"              # form the real key, value can be expression of static value

 # Optional Values
 
 # control options
 fire_count=-1                                 # the number of times to fire (-1 represents forever)
 rate_limit=1000                               # the minimum time in ms between fires
 window_end=48h                                # the relative or absolute time to stop collecting (0 means never end)
 window_start=now                              # the relative or absolute time to start collecting
 condition="expression"                        # conditional statement to control fire
 
 # snapshot options
 snapshot=false                                # should this log also capture a snapshot (default false)
 watch="expression"                            # define watch expressions to attach to the snapshot add multiple expressions as required
 watch="expression"
 
 # metric options
 metric=false                                  # should this log also capture a metric (default false)
 metric_name="log_some_test_py_12"             # metric name for collected metric
 metric_label_my_label="{expression}"          # metric labels for collected metric follow the same pattern as log labels
 metric_label_other_label="static"
 
 # span options
 span=None                                     # should this log also create a span (default None)
 span_name="log_some_test_py_12"               # the name of the span
 span_label_my_label="{expression}"            # span labels for create span follw the same pattern as log labels
 span_label_other_label="static"
}

Snapshot

When collecting a snapshot you generally are looking for a small collection of data so you can understand the data that
is in the area. To assist this the defaults are set to:

• Fire Count - 100
• Rate Limit - 10000ms
• Window - now - +7d

# line snapshot - capture data at start of line
snapshot{file="some_file.py" line=12}

# method entry snapshot - capture data at start of method
snapshot{file="some_file.py" method="my_method"}

# line capture snapshot - capture line exception and runtime
snapshot{file="some_file.py" line=12 target="capture"}

# line end snapshot - capture data at end of line and exception and runtime
snapshot{file="some_file.py" line=12 target="end"}

# method exit snapshot - capture data at end of method with return/exception and runtime
snapshot{file="some_file.py" method="my_method" target="end"}

# method entry snapshot - capture data at start of method with return/exception and runtime
snapshot{file="some_file.py" method="my_method" target="capture"}

Full options

Full options

snapshot{
 file="some_test.py"                           # the file name/path of the source file
 line=12                                       # the source line number (cannot define line and method)
 method="some_method"                          # the method name to look for in the file (cannot define line and method)
 
 # Optional snapshot values
 watch="expression"                            # define watch expressions to attach to the snapshot add multiple expressions as required
 watch="expression"
 
 # Optional Values
 
 # control options
 fire_count=100                                # the number of times to fire (-1 represents forever)
 rate_limit=10000                              # the minimum time in ms between fires
 window_end=7d                                 # the relative or absolute time to stop collecting (0 means never end)
 window_start=now                              # the relative or absolute time to start collecting
 condition="expression"                        # conditional statement to control fire
 
 # log options
 log="some injected log with {expression}"     # the injected log message with expressions (default null)
 log_label_my_label="{expression}"             # labels to attach to the log, `label_` is remove from the key to 
 log_label_other_label="static value"          # form the real key, value can be expression of static value
 
 # metric options
 metric=false                                  # should this log also capture a metric (default false)
 metric_name="log_some_test_py_12"             # metric name for collected metric
 metric_label_my_label="{expression}"          # metric labels for collected metric follow the same pattern as log labels
 metric_label_other_label="static"
 
 # span options
 span=None                                     # should this log also create a span (default None)
 span_name="log_some_test_py_12"               # the name of the span
 span_label_my_label="{expression}"            # span labels for create span follw the same pattern as log labels
 span_label_other_label="static"
}

Metric

When collecting a metric these are generally for a longer period as we want to track a value over a period of time. To
assist this the defaults are set to:

• Fire Count - Forever
• Rate Limit - 500ms
• Window - now - +28d

# Create guage of LRU caceh size
metric{file="some_file.py" line=12 value="len(self.cache)" name="lru_cache_size" type="gauge"}

# Track calls to endpoints by users
metric{file="some_file.py line=12 name="calls_to_endpoint" label_user_id="{user.id}" label_endpoint="login" type="counter"}
metric{file="some_other_file.py line=22 name="calls_to_endpoint" label_user_id="{user.id}" label_endpoint="logout" type="counter"}

# Track function performance by user
metric{file="some_file.py" method="some_method" value="deep.target.runtime" type="histogram", label_user="{user.id}"}

Full options

Full options

metric{
 file="some_test.py"                           # the file name/path of the source file
 line=12                                       # the source line number
 method="some_method"                          # the method name to look for in the file (cannot define line and method)
 
 # Optional metric values
 type="counter"                                # the type of metric to create (default coutner)
 value="expression"                            # the value to use for the metric, can be expression or fixed value (default 1)
 label_my_label="{expression}"                 # metric labels for collected metric follow the same pattern as log labels
 label_other_label="static"
 
 # Optional Values
 
 # control options
 fire_count=100                                # the number of times to fire (-1 represents forever)
 rate_limit=10000                              # the minimum time in ms between fires
 window_end=28d                                # the relative or absolute time to stop collecting (0 means never end)
 window_start=now                              # the relative or absolute time to start collecting
 condition="expression"                        # conditional statement to control fire
 
 # log options
 log="some injected log with {expression}"     # the injected log message with expressions (default null)
 log_label_my_label="{expression}"             # labels to attach to the log, `label_` is remove from the key to 
 log_label_other_label="static value"          # form the real key, value can be expression of static value

 # snapshot options
 snapshot=false                                # should this log also capture a snapshot (default false)
 watch="expression"                            # define watch expressions to attach to the snapshot add multiple expressions as required
 watch="expression"
 
 # span options
 span=None                                     # should this log also create a span (default None)
 span_name="log_some_test_py_12"               # the name of the span
 span_label_my_label="{expression}"            # span labels for create span follw the same pattern as log labels
 span_label_other_label="static"
}

Span

When collecting a span, it is likely you want to augment your observability until a permanent change can be made. So the
defaults are:

• Fire Count - Forever
• Rate Limit - 100ms
• Window - Always

# Create span around function
span{file="some_file.py" method="method_name"}

# Create span around line
span{file="some_file.py" line=12}

# Create span around method with attributes
span{file="some_file.py" method="method_name" label_user="{user.id}"}

Full options

Full options

span{
 file="some_test.py"                           # the file name/path of the source file
 line=12                                       # the source line number
 method="some_method"                          # the method name to look for in the file (cannot define line and method)

# optional span options

name="some_test_py:12"                        # the name of the span (default to location e.g. class_name:method_name or file_name:line_no)
label_my_label="{expression}"                 # span labels for create span follw the same pattern as log labels
label_other_label="static"

# Optional Values

# control options

fire_count=-1                                 # the number of times to fire (-1 represents forever)
rate_limit=100                                # the minimum time in ms between fires
window_end=0                                  # the relative or absolute time to stop collecting (0 means never end)
window_start=now                              # the relative or absolute time to start collecting
condition="expression"                        # conditional statement to control fire

# log options

log="some injected log with {expression}"     # the injected log message with expressions (default null)
log_label_my_label="{expression}"             # labels to attach to the log, `label_` is remove from the key to
log_label_other_label="static value"          # form the real key, value can be expression of static value

# snapshot options

snapshot=false                                # should this log also capture a snapshot (default false)
watch="expression"                            # define watch expressions to attach to the snapshot add multiple expressions as required
watch="expression"

# metric options

metric=false                                  # should this log also capture a metric (default false)
metric_name="log_some_test_py_12"             # metric name for collected metric
metric_label_my_label="{expression}"          # metric labels for collected metric follow the same pattern as log labels
metric_label_other_label="static"
}

DeepQL - Search

Now we need to query the data that is collected by deep. This can be data in many areas (logs, traces, metrics,
snapshots), primarily the query will deal with snapshots. query for logs should be done in the log provider.

Basic query statements

# find snapshots by trigger (tracepoint)
{ tracepoint="atracepointid" }

# find by service name
{ service.name="myApp" }

# find by resource values
{ rs.app_version="14" }

# find by attribute values
{ file="some_file.py" }

# find with wildcard
{ file=~"*.py" }

Advanced query statements

# find snapshots where variable has value
{ var.user_id="auserid" }

# find snapshots where watch expression has value
{ watch.value="some value" }

# basic analytics
# create metric from count of snapshots where user_id is null
count({ var.user_id == null })[5m]

# plot user email into appropriate chart
plot(user.email)

DeepQL - Interactions

As we can create triggers via DeepQL it should also be possible to list and delete them.

Basic interactions

# list triggers
list{}

# list triggers by file
list{ file="some_file.py" }

# delete trigger by id
delete{ id="atracepointid" }

# delete triggers on file
delete{ file="some_file.py" }

[Umaaz]

The nomenclature of trigger vs tracepoint is under review and will be addressed in time.

[Umaaz]

When dealing with values that can be either an expression or a static value, e.g. labels. It will be necessary to specify if it is a static value or expression by wrapping expressions in {}. In the same say expressions within log messages are evaluated. In this way it will be possible to mix static and dynamic values. e.g.

metric{ label_name="name is {user.name}" }