PVS loosing WiFi connectivity over time

[relyef]

I have integration running for a couple of months with update interval set to 300 seconds. It looks like with my setup PVS memory issue is more pronounces; PVS lost WiFi connectivity completely. From the description it looks like you've experienced the issue first hand:

I went through a heck of a time with a PVS that began to fail pushing to Sunpower's cloud

Were you able to resolve the issue? Is there a way to clean-up the memory you could share?
With recent SunStrong acquisition of SunPower monitoring system I'm not concerned with PVS cloud connectivity, but the mere fact that the system is in corrupted state is worrisome nonetheless

[relyef]

It looks like every request to PVS module via LAN results in some logs written to system's flash memory - the graph shown "Flash Available" metric over a few days. During nighttime most of data is zero, so the amount of logs is smaller than during the daytime.

I've read from multiple sources that once flash memory is depleted, the monitor goes into "rainbow mode" where it can no longer boot. I wonder if there's a command to stop logging, otherwise it appears that monitoring via LAN/maintenance API is killing the monitor

[jwelter1971]

I think the Sunpower cloud push stops sending (but still stores it on flash in a queue to push to cloud) when the service port is in use and it only resumes after a period of time. If your sunpower integration is setup to fetch too often the cloud push will never resumes and you fill up the flash. Running with a lower interval allows time for the cloud-push to keep caught up.

[koleson]

I've read from multiple sources that once flash memory is depleted, the monitor goes into "rainbow mode" where it can no longer boot. I wonder if there's a command to stop logging, otherwise it appears that monitoring via LAN/maintenance API is killing the monitor

"Rainbow mode" is from flash wear, not running out of space on the partition.

When the eMMC has used 80% or more of its estimated lifetime writes, the system automatically changes from writing logs to flash to writing logs to a RAMdisk. It also stops automatically updating the firmware.

[jwelter1971]

I've read from multiple sources that once flash memory is depleted, the monitor goes into "rainbow mode" where it can no longer boot. I wonder if there's a command to stop logging, otherwise it appears that monitoring via LAN/maintenance API is killing the monitor

"Rainbow mode" is from flash wear, not running out of space on the partition.

When the eMMC has used 80% or more of its estimated lifetime writes, the system automatically changes from writing logs to flash to writing logs to a RAMdisk. It also stops automatically updating the firmware.

Any way to force rainbow mode?

[koleson]

"Rainbow Mode" is a permanent condition where your PVS6 won't boot. The LEDs only go wacky like that when the eMMC refuses reads because it cannot guarantee the integrity of reads - usually due to write exhaustion.

There's no user-facing way to force the system to use a RAMdisk/refuse firmware updates. You would have to edit some scripts in the firmware.

[relyef]

I think the Sunpower cloud push stops sending (but still stores it on flash in a queue to push to cloud) when the service port is in use and it only resumes after a period of time. If your sunpower integration is setup to fetch too often the cloud push will never resumes and you fill up the flash. Running with a lower interval allows time for the cloud-push to keep caught up.

My integration is configured with update interval of 300 secs, much less frequently than suggested but the memory is still shrinking. I don't think the assumption that updates are queued in flash memory is correct, as the flash memory has very limited amount of writes. If anything is queued, it'll be in volatile memory

[relyef]

"Rainbow mode" is from flash wear, not running out of space on the partition.

When the eMMC has used 80% or more of its estimated lifetime writes, the system automatically changes from writing logs to flash to writing logs to a RAMdisk. It also stops automatically updating the firmware.

If I continue monitoring the system via the integration, what happens when "Flash Available" is reduced to zero?

[koleson]

If I continue monitoring the system via the integration, what happens when "Flash Available" is reduced to zero?

I think this is what is described in the README - the system will still boot and dl_cgi will still provide data to the integration, but the system will no longer send data to SunPower/SunStrong servers and there's not really any way to get it "unstuck" from that without ssh access besides replacing the PVS6.

That said, I don't think you'll continue seeing a monotonic decrease in available flash if your PVS is allowed to access the internet and the cloud servers. It seems to either flush or delete data before free space gets too scarce. (Maybe this is a bug fix in the firmware over the past two years?)

[motopower2]

If I continue monitoring the system via the integration, what happens when "Flash Available" is reduced to zero?

I think this is what is described in the README - the system will still boot and dl_cgi will still provide data to the integration, but the system will no longer send data to SunPower/SunStrong servers and there's not really any way to get it "unstuck" from that without ssh access besides replacing the PVS6.

That said, I don't think you'll continue seeing a monotonic decrease in available flash if your PVS is allowed to access the internet and the cloud servers. It seems to either flush or delete data before free space gets too scarce. (Maybe this is a bug fix in the firmware over the past two years?)

What if you block the PVS from accessing the internet, will it cause an issue with not being able to clear the flash?