From 1e403fb949bde2e022d42ddd81247c1086c912ee Mon Sep 17 00:00:00 2001 From: Fabian Hartung Date: Wed, 5 Mar 2025 15:30:04 +0100 Subject: [PATCH 1/4] ccc: Add logging and debugging section Signed-off-by: Fabian Hartung --- docs/source/development.rst | 2 + docs/source/getting_started.rst | 1 + .../development_debugging_and_logging.inc | 114 ++++++++++++++++++ 3 files changed, 117 insertions(+) create mode 100644 includes/development_debugging_and_logging.inc diff --git a/docs/source/development.rst b/docs/source/development.rst index db317fb..634d3bc 100644 --- a/docs/source/development.rst +++ b/docs/source/development.rst @@ -109,3 +109,5 @@ how to checkout a dedicated EVerest release. #. Follow the steps under the section :ref:`firmware_customization` to install your PKI certificate, pack the modified root filesystem image again into the firmware update image, and test the new firmware image. + +.. include:: ../../includes/development_debugging_and_logging.inc diff --git a/docs/source/getting_started.rst b/docs/source/getting_started.rst index 971be3a..6e05283 100644 --- a/docs/source/getting_started.rst +++ b/docs/source/getting_started.rst @@ -229,6 +229,7 @@ to restart the EVerest charging stack to apply the changes: start. Therefore, it is recommended to back up the original configuration file before making changes. +.. _start_charging_and_monitoring: Starting and Monitoring the Charging Process -------------------------------------------- diff --git a/includes/development_debugging_and_logging.inc b/includes/development_debugging_and_logging.inc new file mode 100644 index 0000000..46a79f8 --- /dev/null +++ b/includes/development_debugging_and_logging.inc @@ -0,0 +1,114 @@ +.. _logging_and_debugging: + +Logging and Debugging +===================== + +There are different ways to analyze the EVerest charging software. In the following, different +ways are described. In preparation for starting this chapter, it is necessary to be connected to the board +via Ethernet. This is also necessary to copy logs and traces from the board to the local PC via an FTP +tool like WinSCP `_ or FileZilla `_. + +journalctl logs +--------------- +As already described in the :ref:`start_charging_and_monitoring` section, the EVerest framework uses +journalctl for logging. journalctl is a system service that collects and stores logging data. +To view the logs, connect to the charge controller via SSH and run the following command: + +.. code-block:: console + + journalctl -f -u everest + +The -f flag is used to follow the logs live when they are written. The -u flag is used to filter the logs +for the everest service. To stop following the logs, press :code:`Ctrl + C`. + +The logs are stored by the systemd journal service in a binary format in +"/var/log/journal". To open the binary in a human-readable format, use the following command: + +.. code-block:: console + + journalctl -u everest --file /path/to/your/file.journal + + +As shown in the :ref:`start_charging_and_monitoring` section, a log line consists, among other things, +of a timestamp, a log level, a module name, and a message. The log level can be one of the following: + + - "VERB" (Verbose) + - "DEBG" (Debug) + - "INFO" (Information) + - "WARN" (Warning) + - "ERRO" (Error) + - "CRIT" (Critical) + +The log level can be configured for each module separately in the logging configuration file. This file +is located in the "/etc/everest/default_logging.cfg" directory. +Here is an example to configure the message filter to change the log level of the EvseManager module +to DEBUG: + +.. code-block:: console + + # The filter expression consists of two parts: + # 1. Logs with severity level INFO or higher (affects all EVerest modules). + # 2. Logs from the EvseManager process with severity level DEBUG or higher. + Filter="(%Severity% >= INFO) or (%Process% contains EvseManager and %Severity% >= DEBG)" + +pcap traces +----------- +pcap traces can be used to debug and analyze the high-level communication (HLC) between SECC and EVCC. +In general, there are two ways to capture pcap traces: + +#. Capture via tcpdump command + + .. code-block:: console + + tcpdump -i eth1 -w /srv/charge_com_01.pcap + + This command captures all packets on the eth1 (QCA interface) interface and writes them to + a pcap file. Stop the capture by pressing :code:`Ctrl + C`. + + .. note:: + Please stop the trace before copying the file to a local machine. The file may be incomplete if + copied while the tcpdump is still running. + +#. Using the EVerest "PacketSniffer" module + + The `PacketSniffer `_ module + is part of EVerest-core and can be used to capture pcap traces automatically. Please look into + the module manifest to see how to configure the PacketSniffer module. + +pcap traces with V2G communication can be analyzed using Wireshark with a V2G plugin like from `dSPACE `_. + +.. note:: + TLS encrypted communication cannot be analyzed with Wireshark without the TLS session key. Please + look in the module manifest which is responsible for the TLS handshake and check if there is an option + to export the TLS session key. E.g. the `IsoMux module `_ + is able to export the TLS session key. + +Session logging of the EVSEManager +---------------------------------- +The EVerest "EVSEManager" is able to generate session log files with information about e.g. current session state, +configured, CP state, duty cycle and is also able to decode and log the content of the V2G messages. +Please look into the module manifest to see how to configure the EVSEManager module to activate session logging. + +.. note:: + The session logging is pretty performance intensive and should only be used for debugging purposes. + +MQTT logs +--------- + +It is also possible to observe the internal MQTT communication between the EVerest modules. +When developing new modules, this can be helpful to better understand the internal processes regarding +the use of the interfaces. +We recommend using the `MQTT Explorer `_ tool to monitor and analyze the MQTT communication. +It is also possible to use the following command to monitor the MQTT communication, but this is not so easy to read: + +.. code-block:: console + + mosquitto_sub -F "[@H:@M:@S.@N] %t %p" -t 'everest/#' -t 'everest_api/#' > /srv/charge_com_mqtt_01.log + +Summary +------- +There are several ways to debug and analyze the EVerest framework. Which way to choose depends on the +specific use case and the information needed. The most common way is using journalctl logs. If it is +necessary to analyze the high-level communication between SECC and EVCC, pcap traces are the best choice. +If it is not clear how the internal communication between the EVerest modules works, the MQTT logs can +be helpful. From e32aa823fbbf2b486aa24244a3cda00d8cab8ccb Mon Sep 17 00:00:00 2001 From: Fabian Hartung Date: Mon, 10 Mar 2025 10:43:04 +0100 Subject: [PATCH 2/4] rework after review Signed-off-by: Fabian Hartung --- .../development_debugging_and_logging.inc | 29 +++++++++---------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/includes/development_debugging_and_logging.inc b/includes/development_debugging_and_logging.inc index 46a79f8..89058b9 100644 --- a/includes/development_debugging_and_logging.inc +++ b/includes/development_debugging_and_logging.inc @@ -16,10 +16,10 @@ To view the logs, connect to the charge controller via SSH and run the following .. code-block:: console - journalctl -f -u everest + journalctl -f -n 100 -u everest -The -f flag is used to follow the logs live when they are written. The -u flag is used to filter the logs -for the everest service. To stop following the logs, press :code:`Ctrl + C`. +The -f flag is used to follow the logs live, and the -n flag is used to view the last 100 lines. +The -u flag is used to filter the logs for the everest service. To stop following the logs, press :code:`Ctrl + C`. The logs are stored by the systemd journal service in a binary format in "/var/log/journal". To open the binary in a human-readable format, use the following command: @@ -28,9 +28,8 @@ The logs are stored by the systemd journal service in a binary format in journalctl -u everest --file /path/to/your/file.journal - As shown in the :ref:`start_charging_and_monitoring` section, a log line consists, among other things, -of a timestamp, a log level, a module name, and a message. The log level can be one of the following: +of a timestamp, a log level, a module id, a module name, and a message. The log level can be one of the following: - "VERB" (Verbose) - "DEBG" (Debug) @@ -39,8 +38,9 @@ of a timestamp, a log level, a module name, and a message. The log level can be - "ERRO" (Error) - "CRIT" (Critical) -The log level can be configured for each module separately in the logging configuration file. This file -is located in the "/etc/everest/default_logging.cfg" directory. +The log level can be configured for each EVerest module separately in the logging configuration file. +The logging configuration file is located at "/etc/everest/default_logging.cfg". The filter can be configured +using the log level and the module identifier as defined in the EVerest configuration. Here is an example to configure the message filter to change the log level of the EvseManager module to DEBUG: @@ -48,8 +48,8 @@ to DEBUG: # The filter expression consists of two parts: # 1. Logs with severity level INFO or higher (affects all EVerest modules). - # 2. Logs from the EvseManager process with severity level DEBUG or higher. - Filter="(%Severity% >= INFO) or (%Process% contains EvseManager and %Severity% >= DEBG)" + # 2. Logs from the EvseManager with module id "connector" with severity level DEBUG or higher. + Filter="(%Severity% >= INFO) or (%Process% contains connector and %Severity% >= DEBG)" pcap traces ----------- @@ -62,7 +62,7 @@ In general, there are two ways to capture pcap traces: tcpdump -i eth1 -w /srv/charge_com_01.pcap - This command captures all packets on the eth1 (QCA interface) interface and writes them to + This command captures all packets on the eth1 (QCA) interface and writes them to a pcap file. Stop the capture by pressing :code:`Ctrl + C`. .. note:: @@ -72,10 +72,10 @@ In general, there are two ways to capture pcap traces: #. Using the EVerest "PacketSniffer" module The `PacketSniffer `_ module - is part of EVerest-core and can be used to capture pcap traces automatically. Please look into + is part of everest-core and can be used to capture pcap traces automatically. Please look into the module manifest to see how to configure the PacketSniffer module. -pcap traces with V2G communication can be analyzed using Wireshark with a V2G plugin like from `dSPACE `_. +pcap traces with V2G communication can be analyzed using Wireshark with a V2G plugin like from `dSPACE `_. .. note:: TLS encrypted communication cannot be analyzed with Wireshark without the TLS session key. Please @@ -85,8 +85,8 @@ pcap traces with V2G communication can be analyzed using Wireshark with a V2G pl Session logging of the EVSEManager ---------------------------------- -The EVerest "EVSEManager" is able to generate session log files with information about e.g. current session state, -configured, CP state, duty cycle and is also able to decode and log the content of the V2G messages. +The EVerest "EVSEManager" can generate session log files with information about the current session state, +configured CP state and duty cycle. It can also decode and log the content of the V2G messages. Please look into the module manifest to see how to configure the EVSEManager module to activate session logging. .. note:: @@ -94,7 +94,6 @@ Please look into the module manifest to see how to configure the EVSEManager mod MQTT logs --------- - It is also possible to observe the internal MQTT communication between the EVerest modules. When developing new modules, this can be helpful to better understand the internal processes regarding the use of the interfaces. From 8cdde1bf33f39676d9b34f23add12f052c46e673 Mon Sep 17 00:00:00 2001 From: Fabian Hartung Date: Mon, 10 Mar 2025 11:25:28 +0100 Subject: [PATCH 3/4] Rework after review Signed-off-by: Fabian Hartung --- includes/development_debugging_and_logging.inc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/includes/development_debugging_and_logging.inc b/includes/development_debugging_and_logging.inc index 89058b9..edf6143 100644 --- a/includes/development_debugging_and_logging.inc +++ b/includes/development_debugging_and_logging.inc @@ -5,8 +5,8 @@ Logging and Debugging There are different ways to analyze the EVerest charging software. In the following, different ways are described. In preparation for starting this chapter, it is necessary to be connected to the board -via Ethernet. This is also necessary to copy logs and traces from the board to the local PC via an FTP -tool like WinSCP `_ or FileZilla `_. +via Ethernet. This is also necessary to copy logs and traces from the board to the local PC via a SFTP +tool like `WinSCP `_ or `FileZilla `_. journalctl logs --------------- @@ -62,7 +62,7 @@ In general, there are two ways to capture pcap traces: tcpdump -i eth1 -w /srv/charge_com_01.pcap - This command captures all packets on the eth1 (QCA) interface and writes them to + This command captures all packets on the eth1 (PLC) interface and writes them to a pcap file. Stop the capture by pressing :code:`Ctrl + C`. .. note:: From 6de256a482ca28b8b601e61762440c5e051c1d30 Mon Sep 17 00:00:00 2001 From: Fabian Hartung Date: Mon, 10 Mar 2025 11:26:27 +0100 Subject: [PATCH 4/4] Add links to the new logging_and_debugging chapter Signed-off-by: Fabian Hartung --- docs/source/getting_started.rst | 3 ++- includes/troubleshooting_contact.inc | 5 +++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/source/getting_started.rst b/docs/source/getting_started.rst index 6e05283..1e29681 100644 --- a/docs/source/getting_started.rst +++ b/docs/source/getting_started.rst @@ -238,7 +238,8 @@ Before we start the first charging session, we shall open the EVerest log to mon process. The EVerest log is stored in the systemd journal and can be accessed via the journalctl command. The journalctl command provides a lot of options to filter the log messages. Now just type "journalctl -f -u everest -n 50" to see the last 50 log messages of the EVerest -charging stack and to follow the charging process in real time. +charging stack and to follow the charging process in real time. For more information about the +EVerest log, see the :ref:`logging_and_debugging` chapter. The EVerest log should look like this: diff --git a/includes/troubleshooting_contact.inc b/includes/troubleshooting_contact.inc index 4fa4ea8..6ef3911 100644 --- a/includes/troubleshooting_contact.inc +++ b/includes/troubleshooting_contact.inc @@ -14,6 +14,11 @@ Do you have general questions about EVerest, please use the EVerest community's Do you have questions about the chargebyte BSP (incl. Yocto), please use `our support desk `_. +.. note:: + Before submitting a support request, make sure that all relevant log files have been captured. + The chapter :ref:`logging_and_debugging` describes the debugging options EVerest offers and shows + which logs are relevant for the respective case. + Address ^^^^^^^