[BUG] "Pausing CURL read for upload handle x" without new upload handle ready #452
Labels
Comments
|
Hi @bachtran02, can you please share verbose logs demonstrating this issue? What credential provider are you using? What is the exact frequency of this issue? |
|
I ran a long running test using the kvssink sample application with IoT credentials that has been running since yesterday. It was intermittently disconnected from the internet multiple times until this morning and it is still running. I'm able to playback the media on the viewer as well. Successful IoT credentials rotation: There are no problems disconnecting and reconnecting to the internet while using Producer C with IoT credentials. We have not been able to reproduce the reported issue with our sample application. |
|
That is one tricky thing about the bug. I wasn't able to consistently reproduce it either, but it tends to happen several hours after running. Here is a different example of debug log of the problem. I wasn't able to get verbose log because the camera running the KVS app doesn't have sufficient storage but I can really try to get that if more info is needed. One thing I noticed from the log is
As you can see the app ran fine for 11 hours since 9:20AM before finally stopped working at 8:04PM as reflected in the log. |
|
Thanks for sharing the details. Is the network on your device stable? The error is a curl failure which could have been a result of no connectivity too as it was successful the first time you were able to connect |
|
I would say the camera's network is quite stable. However, in the event that the network is indeed unstable how should I handle the error and keep retrying to connect? |
|
Hi @niyatim23 ! I would like to follow up on this discussion. Upon further inspection of the logs, I have understood the error a bit more. I think the issue I really encountered was with As upload handle switching happens every time the credentials are updated using AWS IoT, I initially thought the errors were with AWS IoT. Here are the logs (sorry I still can't find a way to get verbose logs because of camera storage) of upload handle switching working normally and not working properly. I'm aware that the core issue could still be with bad network but I hope I could get some help on how to handle this error or if it is possible for the producer library to handle this error. Upload handle switched normally from handle 3 to 4 Upload handle failed to switch and I don't know if the next part of the log is necessary but in my current code if As you can see after the stream was manually restarted it started working again. Please let me know if my current understanding of the issue is correct or if there is any additional information I can provide! |
bachtran02
changed the title
|
Hi Peter, can you please tell me are you able to reproduce this issue using our sample applications? Also wondering, is there a gap between the |
|
Noted. I will get back to you on this ASAP. |
|
Hi @stefankiesz My apologies for the belated response!
The two log lines come directly after each other. I don't think there is any gap in between!
I have been testing this KVS sample app on the camera (aarch64) for the past couple days and were able to reproduce the error. While the error messages are not completely the same, the patterns I noticed are similar: the error usually comes during or shortly after the IoT credentials rotating process, which causes the app to stop sending stream and logging. The app also didn't exit after encountering the error. Additionally, I was able to reproduce the error much faster after reducing the AWS IoT credential duration. Here are the logs of 3 error instances. Due to the storage capacity of the camera, I wasn't able to get Here is the Google Drive URL to the three full log files if you want to look into them more. 0911 0910 0909 Please let me know if you want to know anything more on this! |
|
Hi @bachtran02, are you able to reproduce this while using the curl bundled with the SDK? I see that you mentioned that you're using |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Media pipeline:
getFrame()from camera SDK which continuously returns h264 frames are then passed toputKinesisVideoFrame(), I-type frames marked asFRAME_FLAG_KEY_FRAMELogging
Describe the bug
The bug occurred roughly 12 hours after the app started. Everything was working fine before. There was a few curl issues (
status=0x16000001) here and there but were not breaking errors. This one breaks the application.SDK version number
v1.5.2Screenshot
To Reproduce
N/A
Expected behavior
The bug to not occur and the stream continues to work.
Desktop (please complete the following information):
aarch64Additional context
The app runs on Axis camera and streams to AWS Kinesis
Code Summary:
vdostreamwhich is responsible for accessing h264 frames from camera livestreamvdostream, construct KinesisFramewith the data and send toputKinesisVideoFrame()OpenSSL and Curl are both cross-compiled for aarch64.
The text was updated successfully, but these errors were encountered: