I am experiencing a problem with agent logs disappearing, which occurs when I use a job that has a very short duration but generates a large number of logs. (This is an example, the same will happen if iterations generate a lot of logs on agent for any reason, not only by using TAPI job).
This results in the absence of all logs from a certain agent beyond a certain point. After some analysis, I identified the reasons for this behavior:
- Sending logs to the Test Engine (TE) and test execution work in parallel.
- In cases where there is a "quick" job that generates many logs, log sending is slow. For example, when the 7th iteration is running, it is only sending logs for the 3rd iteration.
- There is a ring buffer (
log_buffer in lib/loggerta/logger_ta.c) which contains all unsent logs on the Test Agent. When the log_buffer is full but the TA wants to add a new log, it deletes the oldest log from the ring buffer and adds the new one (this occurs in the lgt_rb_allocate_head() function, which is called from ta_log_dynamic_user_ts()). This leads to the disappearance of some logs, and I haven't found any way to wait here until the logger processes more logs and the ring buffer is no longer full.
- In the end, when the TE runs the last (100th) iteration, the logger is only processing logs from the 50th iteration. After that, the TE flushes the logs but doesn't wait long enough, and in the final log, I see that the agent logs end at the 66th iteration. Here again, I haven't found any wait function that can wait for the complete flush of the TA ring buffer.
I am experiencing a problem with agent logs disappearing, which occurs when I use a job that has a very short duration but generates a large number of logs. (This is an example, the same will happen if iterations generate a lot of logs on agent for any reason, not only by using TAPI job).
This results in the absence of all logs from a certain agent beyond a certain point. After some analysis, I identified the reasons for this behavior:
log_bufferinlib/loggerta/logger_ta.c) which contains all unsent logs on the Test Agent. When thelog_bufferis full but the TA wants to add a new log, it deletes the oldest log from the ring buffer and adds the new one (this occurs in thelgt_rb_allocate_head()function, which is called fromta_log_dynamic_user_ts()). This leads to the disappearance of some logs, and I haven't found any way to wait here until the logger processes more logs and the ring buffer is no longer full.