feat: collects all stats like cpu and mem in background for up to 5 minutes

[amir20]

• chore: cleans up goroutines to bubble with errors
• feat: streams all stats in background routine

[amir20]

@EDIflyer @akash-ramaswamy I wonder if you two can take a moment to test this PR with amir20/dozzle:pr-2740. I have moved all the stats collection to the API. So now when you visit Dozzle you should instantly see a history of stats. This is personally something that will help a lot with debugging. You can even see stats of containers that were shutdown.

[EDIflyer]

Hi @amir20 - have tested the pr-2740 image - seems to load fine although I'm not quite clear what is meant to show up differently vs previous versions? I tried stopping a container but it then disappeared altogether even although I had 'show stopped containers' set to true. Interesting of the two other stopped containers only one of them is greyed out! I've checked back though and this behaviour seems the same as on older versions so I don't think it's a new thing (I just hadn't noticed it before!)

glances was the one I stopped - you can see it in my Portainer list but not on the Dozzle page

Sorry not sure if that's helpful or not!!

I then tried restarting glances and stopped it again and it now shows in the list at the bottom but still not greyed out for some reason:

[akash-ramaswamy]

Hi @amir20 ! As @EDIflyer mentioned, Im not able to see any noticeable changes in this version. Also the stopped containers are not showing if refreshed it seems. Also its not greyed out like previous version.

[amir20]

I tried stopping a container but it then disappeared altogether even although I had 'show stopped containers' set to true.

This is actually a bug. I just pushed a fix. Try pulling again

So the change is some what subtle but big if you realize what is happening. Before, Dozzle was collecting the stats in the browser. Meaning, if you closed the tab and open Dozzle again, it would have lost the history of CPU and mem. Now it should continue while the tabs are closed.

[amir20]

Thanks @akash-ramaswamy try again. I think I had a bug.

Look for the stats on the top right. It should have last 300 seconds of data even when the browser has been closed. I guess it's good thing that not much is noticeable because a lot is happening behind the scenes. Which by the way this might be a good start to doing a notifications 🤓

[akash-ramaswamy]

yeah @amir20 you are right 🤓. I noticed that the stopped containers do not have stats in older version. But this version maintains stats even if the tab is closed and reopened.

It should have last 300 seconds of data even when the browser has been closed

Regarding this point, I couldn't exactly feel this.

But still, the stopped container's greyed out color is missing :)

[akash-ramaswamy]

But still, the stopped container's greyed out color is missing :)

Still have this issue @amir20 :)

[amir20]

Ah wait a minute, I think you are right. If the container stops, I don't update the status. Can you confirm the steps to reproduce?

1. Create a container
2. Go to Dozzle see the container and it should be active
3. Stop the container, I am guessing it updates in Dozzle
4. But after refreshing, it is still active.

Does that sound about right? I knew doing a lot of logic would be complicated. :)

[akash-ramaswamy]

Yes! Exactly these steps reproduce the bug I am facing :)

[akash-ramaswamy]

Stop the container, I am guessing it updates in Dozzle

@amir20 this also doesn't update for me in dozzle it seems :) I meant the color of the listed container doesn't grey out.

[amir20]

@akash-ramaswamy ok try again. I realized this is a little more complicated. Will do more testing on my side.

[amir20]

I think something broke and I can't figure out what I did 🤔 So pause testing until I figure it out

[amir20]

OK try again @akash-ramaswamy

[EDIflyer]

Haha was stuck in a meeting - looks like you've been hard at it! Still need testing or all looking good now?

[amir20]

Test master now if you can.

[EDIflyer]

OK pulled master - the glances container doesn't disappear when stopped anymore but still doesn't go grey immediately. If I refresh the page it does. The mkdocs-checkforupdates shows at the bottom with other stopped containers but also doesn't go grey - when I click on it then it does say container stopped. In Portainer it just shows as 'created' and the logs are:

So I'm not sure if that's the reason why it isn't grey?

I can see the graphs on the new version:

Have more detail than on the old version:

I'm assuming that's due to more data due to it logging it anyway? Sorry I must confess I hadn't really noticed 🫢😀

[amir20]

@EDIflyer I wasn't able to reproduce that.

I did

1. docker run ....
2. Stopped the container with ctrl+c

Screen.Recording.2024-02-01.at.1.36.20.PM.mov

Here is a video. The container shows up and then grays out.

How are you able to do that?

[EDIflyer]

Haha - I'm stopping it from within Dozzle using the new functionality from a month or so ago 😉

For the mkdocs-checkforupdates container I'm not stopping it - it's already stopped (it just exists for Watchtower to check) but never goes grey despite it not running (although as mentioned it has a status of Created rather than Exited)

[amir20]

@akash-ramaswamy actually worked on that 🎉

@EDIflyer I tried doing exactly the same thing and it worked. Do you have possibly an older version? Can you pull master again and try again?

[EDIflyer]

Hmm I had literally just re pulled it and it seemed to fix the other issues. Will try again tomorrow.

[akash-ramaswamy]

Hi @amir20 ! I've spent some time reviewing the code changes, and I noticed the shift from streaming stats at the time of browser requests to including them with the containers-changed event during streaming.

I understand the technical adjustments, but could you kindly provide a brief explanation of why this change was necessary? I appreciate your time and clarification! 😊

FYI: (just an immediate fact I noticed)
Also I noticed that the dozzle CPU usage spiked up upto 4 % in the top right stats for each refresh I made to the browser tab ( which was earlier maximum of 1 - 2 %, below that almost every time...)

[amir20]

Excellent question @akash-ramaswamy! Let me try to explain it and hopefully the need for this makes sense.

I use Docker a lot and when a container dies, I have to investigate the cause. Sometimes the errors are shown in the log but other times it might be an "out of memory" error. So I would need to go back in time and find the container to see how much memory it was using. Unfortunately, this is not possible right now with Docker as the stats are streamed but not stored. Something like Datadog can be used but that costs money. Dozzle also doesn't help here because as you are aware it requires the browser tab to be open. You can see this happening right now if you open Dozzle, wait a couple a minutes and then refresh Dozzle. All the stats, memory, and CPU are reset because the tab is restarted.

So to do fix this, it's obvious that Dozzle needs to have some kind of temporary data for stats and logs.

On logs, I think you are already very familiar with this. Currently, notifications can't be sent. Logs can't be analyzed, etc...

But to me, the stats was even more important. Now I have created ContainerStore in the API which at all times has active containers and their stats. I think in the future, it can also compute avg CPU and start alerting when over 90%. None of these features would have been possible if tab was required to be open.

On performance, it should actually be better. Before, for each user I would start listening on all containers. So that was something like N X M where N is users and M is the number of containers. Now, I only listen to the containers once but stream all this data to all users. So if more users connect, CPU shouldn't go up.

Does this help?

I am finding some little bugs and will try to release when I think it's stable.

[akash-ramaswamy]

Thanks a lot for your detailed explanation. I can feel your intense efforts and engineering from this single PR.

Now I have created ContainerStore in the API which at all times has active containers and their stats.

Yeah @amir20 ! I read through the ContainerStore API and the RingBuffer DS to store atmost 5 minutes of latest stats at anytime :) It was indeed an excellent opportunity for me to learn from your code! This also gives me a new direction to think for the notification! Im really excited to try and tinker with your ContainerStore and explore more ideas from it.

Before, for each user I would start listening on all containers. So that was something like N X M where N is users and M is the number of containers.

This sounds great! I wasn't thinking this direction. This really reduces the CPU usage a lot.

Once again, thanks a lot for your clear explanation :)

[EDIflyer]

@EDIflyer I tried doing exactly the same thing and it worked. Do you have possibly an older version? Can you pull master again and try again?

Just done that and identical to yesterday evening. Go into glances container, two dots menu > stop container. Container drops to the bottom and shows as stopped in the main log page but still white text in the list on the left. Hit refresh and text goes grey.

Interestingly when starting the container as well as jumping back up in the list it also goes back to white text.

[amir20]

You are testing master, right? Can you share a video.

[EDIflyer]

Yep I re-pulled master just now. Video is the same as in the post above - #2740 (comment)

{"level":"info","msg":"Dozzle version v6.1.1","time":"2024-02-01T21:04:39Z"}

[amir20]

{"level":"info","msg":"Dozzle version v6.1.1","time":"2024-02-01T21:04:39Z"} Doesn't look right. It should say Dozzle version master.

Are you pulling amir20/dozzle:master?

[EDIflyer]

I'm a total numpty @amir20 !! - I was so busy switching it from a pr-xx version in the docker compose without thinking I put latest rather than master and then didn't spot the difference 🤦‍♂️ 🤦‍♂️ It now does indeed work fine!! - glad I posted that log entry!

[amir20]

Ha great. So it looks this is as an actual bug in latest version. Glad to know I fixed something I didn't know about.

[EDIflyer]

Eek - possibly spoke to soon - have found a bug the other way round now I'm afraid!!

Screen recording below (and I promise this is definitely on master!):

1. Select running container
2. 2-dot menu and select Stop
3. Container stops, name goes grey and goes to bottom of list
4. 2-dot menu correctly switches Stop to Start
5. Refresh page
6. Container now jumps back up to main running list and name switches back to white, despite container still being stopped. 2-dot menu now shows Stop rather than Start (I'm guessing tied in to the container name being in white rather than grey?). In the video I then restarted the container from the menu just to show it did come back up.

[amir20]

Ahhh good catch. I did not cover start or restart. :)