Release Testing

This page intends to list the tests that we shall/must do before we do a release for KUKSA Databroker. It is based on the old kuksa.val release testing page.

The intention of the release test is to do a reasonable regression test. It is assumed that new features of individual components have been thoroughly tested, the focus of the release test is to verify that there are no regressions. For capacity reasons the tests have no intention to cover all possible use cases.

Prerequisites

No outstanding Pull Requests that may affect component behavior.
CI Docker builds on latest master has succeeded
Pre-release of kuksa-client exists on PyPI (If dependencies exist)

Test regressions in Kuksa Python SDK

Kuksa Python SDK use Kuksa Databroker as a submodule to get access to Proto files. Update the submodule reference and create a (draft) pull request and check CI results.

If it succeeds hopefully everything is good
If it fails check if the failure is expected
- For example intended backward incompatible changes in Databroker

Basic testing with Databroker (Insecure, No TLS)

erik@debian3:~/kuksa.val/kuksa_databroker$ cargo run --bin databroker -- --metadata data/vss-core/vss_release_4.0.json --insecure

Python Client test

(kuksa-client) erik@debian3:~/kuksa.val/kuksa-client$ kuksa-client grpc://127.0.0.1:55555


Test Client> setValue Vehicle.Speed 33
OK

Test Client> getValue Vehicle.Speed

Databroker CLI

NOTE: Verify that correct version number is shown in the "hello message!*

erik@debian3:~/kuksa.val/kuksa_databroker$ cargo run --bin databroker-cli
...
kuksa.val.v1 > publish Vehicle.Speed 44
[publish]  OK  
kuksa.val.v1 > get Vehicle.Speed
[get]  OK  
Vehicle.Speed: 44.00 km/h

Provider smoke tests

We want to know if providers work - if not we need to know why

CAN Provider

They rely on the client, so some testing is necessary. Two options exist, either build and install kuksa-client locally, or upload an alpha package to pypi. Testing with an uploaded package is preferable as that also tests that Pypi packaging works as expected.

Start DBC Feeder using default settings

erik@debian3:~/kuksa.val.feeders/dbc2val$ ./dbcfeeder.py

Let it run for a while, verify with Python client (or databroker CLI) that Vehicle.Speed change now and then (it takes some time to see the first change).

Test Client> getValue Vehicle.Speed                          
{
    "path": "Vehicle.Speed",
    "value": {
        "value": 77.0,
        "timestamp": "2023-07-20T13:17:26.946733+00:00"
    }
}

Test Client> getValue Vehicle.Speed
{
    "path": "Vehicle.Speed",
    "value": {
        "value": 54.0,
        "timestamp": "2023-07-20T13:17:33.924064+00:00"
    }
}

Basic testing with Databroker (Tokens and TLS)

erik@debian3:~/kuksa.val/kuksa_databroker$ cargo run --bin databroker -- --metadata data/vss-core/vss_release_4.0.json --tls-cert certificates/Server.pem --tls-private-key certificates/Server.key --jwt-public-key certificates/jwt/jwt.key.pub

Kuksa-client

Verify with Kuksa-client that connection works

erik@debian4:~/kuksa-python-sdk/kuksa-client$ kuksa-client grpcs://127.0.0.1:55555 --cacertificate /home/erik/kuksa-common/tls/CA.pem --tls-server-name Server

     ⢀⣤⣶⣾⣿⢸⣿⣿⣷⣶⣤⡀
    ⣴⣿⡿⠋⣿⣿   ⠈⠙⢿⣿⣦
   ⣾⣿⠋  ⣿⣿  ⣶⣿  ⠙⣿⣷
  ⣸⣿⠇   ⣿⣿⠠⣾⡿⠃   ⠸⣿⣇  ⣶ ⣠⡶⠂ ⣶  ⢰⡆ ⢰⡆⢀⣴⠖ ⢠⡶⠶⠶⡦   ⣰⣶⡀
  ⣿⣿    ⠿⢿⣷⣦⡀     ⣿⣿  ⣿⢾⣏   ⣿  ⢸⡇ ⢸⡷⣿⡁  ⠘⠷⠶⠶⣦  ⢠⡟⠘⣷
  ⢹⣿⡆   ⣿⣶⠈⢻⣿⡆   ⢰⣿⡏  ⠿ ⠙⠷⠄ ⠙⠷⠶⠟⠁ ⠸⠇⠈⠻⠦ ⠐⠷⠶⠶⠟ ⠠⠿⠁ ⠹⠧
   ⢿⣿⣄  ⣿⣿  ⠿⣿  ⣠⣿⡿
    ⠻⣿⣷⡄⣿⣿   ⢀⣠⣾⣿⠟    kuksa-client CLI
     ⠈⠛⠇⢿⣿⣿⣿⣿⡿⠿⠛⠁     0.0.0

Default tokens directory: /home/erik/kuksa-python-sdk/submodules/kuksa.val/kuksa_certificates/jwt

Connecting to VSS server at 127.0.0.1 port 55555 using KUKSA GRPC protocol.
TLS will be used.
INFO 2023-11-16 10:36:56,850 kuksa_client.grpc Using TLS with Root CA from /home/erik/kuksa-common/tls/CA.pem
INFO 2023-11-16 10:36:56,855 kuksa_client.grpc Using client private key and certificates, mutual TLS supported if supported by server
INFO 2023-11-16 10:36:56,855 kuksa_client.grpc.aio Establishing secure channel
INFO 2023-11-16 10:36:56,855 kuksa_client.grpc.aio Using TLS server name Server
INFO 2023-11-16 10:36:56,909 kuksa_client.grpc.aio Unauthenticated channel started
Secure gRPC channel connected.


Test Client> authorize /home/erik/kuksa-common/jwt/actuate-provide-all.token 
"Authenticated"

Test Client> setValue Vehicle.Speed 33
OK

Test Client> getValue Vehicle.Speed
{
    "path": "Vehicle.Speed",
    "value": {
        "value": 33.0,
        "timestamp": "2023-11-16T09:37:49.769857+00:00"
    }
}

Databroker-cli

erik@debian3:~/kuksa.val/kuksa_databroker$ cargo run --bin databroker-cli -- --token-file jwt/provide-all.token --ca-cert certificates/CA.pem --server https://127.0.0.1:55555
    Finished dev [unoptimized + debuginfo] target(s) in 0.20s
     Running `/home/erik/kuksa.val/target/debug/databroker-cli --token-file ../jwt/provide-all.token --ca-cert ../kuksa_certificates/CA.pem`

...

kuksa.val.v1 > publish Vehicle.Speed 44
[publish]  OK  
kuksa.val.v1 > get Vehicle.Speed
[get]  OK  
Vehicle.Speed: 44.00 km/h
kuksa.val.v1 >

Databroker Docker

Verify first that Databroker:main is recently built and there is no pending/queued builds.

Note: You need to adapt examples to your own environment. Examples based on that kuksa-common is cloned!

erik@debian3:~/kuksa.val/kuksa_databroker$ docker pull ghcr.io/eclipse-kuksa/kuksa-databroker:main
erik@debian3:~/kuksa.val/kuksa_databroker$ docker run --net=host --rm -it  -p 55555:55555/tcp -v /home/erik/kuksa-databroker/certificates:/tls -v /home/erik/kuksa-common/jwt:/jwt   ghcr.io/eclipse-kuksa/kuksa-databroker:main --tls-cert /tls/Server.pem --tls-private-key /tls/Server.key --jwt-public-key /jwt/jwt.key.pub

Then repeat test for local build

databroker-cli docker

erik@debian4:~/kuksa.val$ docker run --rm -it --net=host -v /home/erik/kuksa-common/tls:/tls -v /home/erik/kuksa-common/jwt:/jwt ghcr.io/eclipse-kuksa/kuksa-databroker-cli:main --token-file /jwt/provide-all.token --ca-cert /tls/CA.pem --server https://127.0.0.1:55555

KUKSA VISS Interface

Run

cargo run --bin databroker --features viss -- --enable-viss --metadata data/vss-core/vss_release_4.0.json --insecure

Then use two instances of kuksa-client (as VISS interface ONLY supports setting of target value and reading of actual value)

Step	VISS client	gRPC client
Start client	`kuksa-client ws://127.0.0.1:8090`	`kuksa-client`
Set Speed		`Test Client> setValue Vehicle.Speed 332`
Read Speed	`Test Client> getValue Vehicle.Speed`
Set target tripmeter	`Test Client> setTargetValue Vehicle.TripMeterReading 32`
Read target tripmeter		`Test Client> getTargetValue Vehicle.TripMeterReading`

Testing Databroker kuksa.val.v2 API methods with gRPCurl

There are multiple ways to install gRPCurl. If you have a working Docker environment using Docker might be the most simple solution. If using Docker the examples in this section needs to be adapted similar to the example below.

Smoke Test

docker run --network=host fullstorydev/grpcurl  -d '{ "root": "Vehicle.Speed" }' -plaintext localhost:55555 kuksa.val.v2.VAL.ListMetadata

Expect something like

{
  "metadata": [
    {
      "id": 882,
      "dataType": "DATA_TYPE_FLOAT",
      "entryType": "ENTRY_TYPE_SENSOR",
      "description": "Vehicle speed.",
      "unit": "km/h"
    }
  ]
}

GetValue

rpc GetValue(GetValueRequest) returns (GetValueResponse);
Returned GRPC error codes:

NOT_FOUND if the requested signal doesn't exist

./grpcurl -d '{ "signal_id": { "path": "Path_doesnt_exist" } }' -plaintext localhost:55555 kuksa.val.v2.VAL.GetValue

UNAUTHENTICATED if no credentials provided or credentials has expired (start databroker with --jwt-public-key certificates/jwt/jwt.key.pub)

./grpcurl -d '{ "signal_id": { "path": "Vehicle.Speed" } }' -plaintext localhost:55555 kuksa.val.v2.VAL.GetValue

PERMISSION_DENIED if access is denied
INVALID_ARGUMENT if the request is empty or provided path is too long
- MAX_REQUEST_PATH_LENGTH: usize = 1000; Command: -> provide a path longer than 1000 characters:

./grpcurl -d '{ "signal_id": { "path": "provide_path_too_long" } }' -plaintext localhost:55555 kuksa.val.v2.VAL.GetValue

GetValues

rpc GetValues(GetValuesRequest) returns (GetValuesResponse);
Returned GRPC error codes:

NOT_FOUND if any of the requested signals doesn't exist.

./grpcurl -d '{ "signal_ids": { "path": "Path_doesnt_exist" } }' -plaintext localhost:55555 kuksa.val.v2.VAL.GetValues

UNAUTHENTICATED if no credentials provided or credentials has expired (start databroker with --jwt-public-key certificates/jwt/jwt.key.pub)

./grpcurl -d '{ "signal_ids": { "path": "Vehicle.Speed" } }' -plaintext localhost:55555 kuksa.val.v2.VAL.GetValues

PERMISSION_DENIED if access is denied for any of the requested signals.
INVALID_ARGUMENT if the request is empty or provided path is too long
- MAX_REQUEST_PATH_LENGTH: usize = 1000; Command: -> provide a path longer than 1000 characters:

./grpcurl -d '{ "signal_ids": { "path": "provide_path_too_long" } }' -plaintext localhost:55555 kuksa.val.v2.VAL.GetValues

Subscribe

rpc Subscribe(SubscribeRequest) returns (stream SubscribeResponse);
Returned GRPC error codes:

NOT_FOUND if any of the signals are non-existant.

./grpcurl -d '{ "signal_paths": ["Path_doesnt_exist"]}' -plaintext localhost:55555 kuksa.val.v2.VAL.Subscribe

UNAUTHENTICATED if no credentials provided or credentials has expired (start databroker with --jwt-public-key certificates/jwt/jwt.key.pub)

./grpcurl -d '{ "signal_paths": ["Vehicle.Speed"]}' -plaintext localhost:55555 kuksa.val.v2.VAL.Subscribe

PERMISSION_DENIED if access is denied for any of the signals
INVALID_ARGUMENT if the request is empty or provided path is too long or buffer_size too big.
- MAX_REQUEST_PATH_LENGTH: usize = 1000;
- If subscription buffer_size exceeds 1000;

./grpcurl -d '{ }' -plaintext localhost:55555 kuksa.val.v2.VAL.Subscribe

./grpcurl -d '{ "signal_paths": ["provide_path_too_long"]}' -plaintext localhost:55555 kuksa.val.v2.VAL.Subscribe

./grpcurl -d '{ "signal_paths": ["Vehicle.Speed"], "buffer_size": 10000}' -plaintext localhost:55555 kuksa.val.v2.VAL.Subscribe

SubscribeById

rpc SubscribeById(SubscribeByIdRequest) returns (stream SubscribeByIdResponse);
Returned GRPC error codes:

NOT_FOUND if any of the signals are non-existant.

./grpcurl -d '{ "signal_ids": [55555]}' -plaintext localhost:55555 kuksa.val.v2.VAL.SubscribeById

UNAUTHENTICATED if no credentials provided or credentials has expired (start databroker with --jwt-public-key certificates/jwt/jwt.key.pub)

./grpcurl -d '{ "signal_ids": [882]}' -plaintext localhost:55555 kuksa.val.v2.VAL.SubscribeById

PERMISSION_DENIED if access is denied for any of the signals
INVALID_ARGUMENT if the request is empty or provided path is too long or buffer_size too big.
- MAX_REQUEST_PATH_LENGTH: usize = 1000;
- If subscription buffer_size exceeds 1000;

./grpcurl -d '{ }' -plaintext localhost:55555 kuksa.val.v2.VAL.SubscribeById

./grpcurl -d '{ "signal_ids": ["provide_id_too_long"]}' -plaintext localhost:55555 kuksa.val.v2.VAL.SubscribeById

./grpcurl -d '{ "signal_ids": [54], "buffer_size": 10000}' -plaintext localhost:55555 kuksa.val.v2.VAL.SubscribeById

Actuate

rpc Actuate(ActuateRequest) returns (ActuateResponse);
Returned GRPC error codes:

NOT_FOUND if the actuator does not exist.

./grpcurl -d '{ "signal_id": { "path": "Path_doesnt_exist" },  "value": { "float": 75.5 }}' -plaintext localhost:55555 kuksa.val.v2.VAL.Actuate

UNAUTHENTICATED if no credentials provided or credentials has expired (start databroker with --jwt-public-key certificates/jwt/jwt.key.pub)

./grpcurl -d '{ "signal_id": { "path": "Vehicle.Speed" },  "value": { "float": 75.5 }}' -plaintext localhost:55555 kuksa.val.v2.VAL.Actuate

PERMISSION_DENIED if access is denied for the actuator
UNAVAILABLE if there is no provider currently providing the actuator

./grpcurl -d '{ "signal_id": { "path": "Vehicle.ADAS.PowerOptimizeLevel" },  "value": { "uint32": "5" }}' -plaintext localhost:55555 kuksa.val.v2.VAL.Actuate

INVALID_ARGUMENT

if the provided path is not an actuator

./grpcurl -d '{ "signal_id": { "path": "Vehicle.Speed" },  "value": { "float": "75.5" }}' -plaintext localhost:55555 kuksa.val.v2.VAL.Actuate

if the data type used in the request does not match the data type of the addressed signal

./grpcurl -d '{ "signal_id": { "path": "Vehicle.Speed" },  "value": { "float": "75.5" }}' -plaintext localhost:55555 kuksa.val.v2.VAL.Actuate

if the requested value is not accepted, e.g. if sending an unsupported enum value

./grpcurl -d '{ "signal_id": { "path": "Vehicle.Body.Lights.LightSwitch" },  "value": { "string": "NOT_ALLOWED_VALUE" }}' -plaintext localhost:55555 kuksa.val.v2.VAL.Actuate

if the provided value is out of the min/max range specified

./grpcurl -d '{ "signal_id": { "path": "Vehicle.ADAS.PowerOptimizeLevel" },  "value": { "uint32": "50" }}' -plaintext localhost:55555 kuksa.val.v2.VAL.Actuate

BatchActuate

rpc BatchActuate(BatchActuateRequest) returns (BatchActuateResponse);
Returned GRPC error codes:

NOT_FOUND if any of the actuators are non-existant.

./grpcurl -d '{ "actuate_requests": [{ "signal_id": { "path": "Path_doesnt_exist" }, "value": { "float": 75.5 }}] }' -plaintext localhost:55555 kuksa.val.v2.VAL.BatchActuate

UNAUTHENTICATED if no credentials provided or credentials has expired (start databroker with --jwt-public-key certificates/jwt/jwt.key.pub)

./grpcurl -d '{ "actuate_requests": [{ "signal_id": { "path": "Vehicle.Speed" }, "value": { "float": 75.5 }}] }' -plaintext localhost:55555 kuksa.val.v2.VAL.BatchActuate

PERMISSION_DENIED if access is denied for any of the actuators
DATA_LOSS is there is a internal TransmissionFailure
UNAVAILABLE if there is no provider currently providing an actuator

./grpcurl -d '{ "actuate_requests": [{ "signal_id": { "path": "Vehicle.ADAS.PowerOptimizeLevel" }, "value": { "uint32": "5" }}] }' -plaintext localhost:55555 kuksa.val.v2.VAL.BatchActuate

INVALID_ARGUMENT

if any of the provided path is not an actuator

./grpcurl -d '{ "actuate_requests": [{ "signal_id": { "path": "Vehicle.Speed" }, "value": { "float": 75.5 }}] }' -plaintext localhost:55555 kuksa.val.v2.VAL.BatchActuate

if the data type used in the request does not match the data type of the addressed signal

./grpcurl -d '{ "actuate_requests": [{ "signal_id": { "path": "Vehicle.TripMeterReading" },  "value": { "string": "hello" }}] }' -plaintext localhost:55555 kuksa.val.v2.VAL.BatchActuate

if the requested value is not accepted, e.g. if sending an unsupported enum value

./grpcurl -d '{ "actuate_requests": [{ "signal_id": { "path": "Vehicle.Body.Lights.LightSwitch" }, "value": { "string": "NOT_ALLOWED_VALUE" }}] }' -plaintext localhost:55555 kuksa.val.v2.VAL.BatchActuate

if any of the provided actuators values are out of the min/max range specified

./grpcurl -d '{ "actuate_requests": [{ "signal_id": { "path": "Vehicle.ADAS.PowerOptimizeLevel" }, "value": { "float": 75.5 }}] }' -plaintext localhost:55555 kuksa.val.v2.VAL.BatchActuate

ListMetadata

rpc ListMetadata(ListMetadataRequest) returns (ListMetadataResponse);
Returned GRPC error codes:

NOT_FOUND if the specified root branch does not exist

./grpcurl -d '{ "root": "Path_doesnt_exist" }' -plaintext localhost:55555 kuksa.val.v2.VAL.ListMetadata

UNAUTHENTICATED if no credentials provided or credentials has expired (start databroker with --jwt-public-key certificates/jwt/jwt.key.pub)

./grpcurl -d '{ "root": "Vehicle.Speed" }' -plaintext localhost:55555 kuksa.val.v2.VAL.ListMetadata

PERMISSION_DENIED if access is denied
INVALID_ARGUMENT if the provided path or wildcard is wrong

./grpcurl -d '{ "root": "** **d. *Da" }' -plaintext localhost:55555 kuksa.val.v2.VAL.ListMetadata

PublishValue

rpc PublishValue(PublishValueRequest) returns (PublishValueResponse);
Returned GRPC error codes:

NOT_FOUND if any of the signals are non-existant.

./grpcurl -d '{  "signal_id": { "path": "Path_doesnt_exist" },  "data_point": { "timestamp": "2024-10-29T12:04:31.463835877Z", "value": { "float": 75.5 }  }}' -plaintext localhost:55555 kuksa.val.v2.VAL.PublishValue

UNAUTHENTICATED if no credentials provided or credentials has expired (start databroker with --jwt-public-key certificates/jwt/jwt.key.pub)

./grpcurl -d '{  "signal_id": { "path": "Vehicle.Speed" },  "data_point": { "timestamp": "2024-10-29T12:04:31.463835877Z", "value": { "float": 75.5 }  }}' -plaintext localhost:55555 kuksa.val.v2.VAL.PublishValue

PERMISSION_DENIED if access is denied for any of the signals
INVALID_ARGUMENT

if the data type used in the request does not match the data type of the addressed signal

./grpcurl -d '{  "signal_id": { "path": "Vehicle.TripMeterReading" },  "data_point": { "timestamp": "2024-10-29T12:04:31.463835877Z", "value": { "string": "hello" }  }}' -plaintext localhost:55555 kuksa.val.v2.VAL.PublishValue

if the published value is not accepted, e.g. if sending an unsupported enum value

./grpcurl -d '{  "signal_id": { "path": "Vehicle.Body.Lights.LightSwitch" },  "data_point": { "timestamp": "2024-10-29T12:04:31.463835877Z", "value": { "string": "NOT_ALLOWED_VALUE" }  }}' -plaintext localhost:55555 kuksa.val.v2.VAL.PublishValue

if the published value is out of the min/max range specified

./grpcurl -d '{  "signal_id": { "path": "Vehicle.ADAS.PowerOptimizeLevel" },  "data_point": { "timestamp": "2024-10-29T12:04:31.463835877Z", "value": { "float": 75.5 }  }}' -plaintext localhost:55555 kuksa.val.v2.VAL.PublishValue

GetServerInfo

rpc GetServerInfo(GetServerInfoRequest) returns (GetServerInfoResponse);

./grpcurl -plaintext localhost:55555 kuksa.val.v2.VAL.GetServerInfo

Expect something like below

{
  "name": "databroker",
  "version": "0.4.7-dev.1",
  "commitHash": "8d9a3b25471dde2779322f5d2358fa225582ef3b"
}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release Testing

Prerequisites

Test regressions in Kuksa Python SDK

Basic testing with Databroker (Insecure, No TLS)

Python Client test

Databroker CLI

Provider smoke tests

CAN Provider

Basic testing with Databroker (Tokens and TLS)

Kuksa-client

Databroker-cli

Databroker Docker

databroker-cli docker

KUKSA VISS Interface

Testing Databroker kuksa.val.v2 API methods with gRPCurl

Smoke Test

GetValue

GetValues

Subscribe

SubscribeById

Actuate

BatchActuate

ListMetadata

PublishValue

GetServerInfo

Clone this wiki locally