[TrimUI Brick] Wifi does not come back up after failed deep sleep

[DrFlarp]

Submitting this to further document #578 and #581. Although both have been resolved in 6.7.0RC1 by throwing away Allwinner's wifi daemon and libs, it seems the root causes remain even in the tg5050 branch. Deep sleep still has a high probability of failing with an error code.

In the suspend script:

• if echo mem > /sys/power/state fails the script prematurely exits. This means that - if applicable - wifi and bt services have not been restarted after being torn down in preparation for sleep. The best way to resolve this is not a given - either 1) we handle the error in-script with set +e and keep track of the error code, or 2) we split the script into pre- and post-steps and force the calling exe to keep track of the state of wifi and bt, which is more "correct" but puts the burden of doing this correctly downstream.

Is it intended for other paks call the suspend script directly? if so, I think we stick with option 1).

The reason for the sleep failure itself is not fully clear. rmmod for allwinner's wifi drivers seems to reduce the chances of failure but not 100%, so this is going to get its own issue.

[frysee]

I think we want the suspend to be as self-contained as possible, since community paks might call suspend to implement sleep support on their own. Unloading the wifi/bt modules could be an interesting thing to try and test, but especially on tg5050 it takes a bit to load them again - I assume it would be similar on other platforms.

[DrFlarp]

Alright. It also seems when sleep fails, we should move the retry logic into the suspend script so 1) we don't have to burden every pak with having to handle this themselves and 2) we don't tear down and restart networking every 5 seconds.

[DrFlarp]

So, the tradeoff with handling retries in the suspend script is that the power button cannot be used to wake the system while the script is running. In practice I've found that takes around 4-6 seconds of retries (every 2 seconds) before the wifi driver finally lets us sleep, but if the user is unlucky enough to try waking their system in that window they might think it's a bug and force reset the device.

However, I'm electing to not touch LEDs during this process. If the user has their info_brightness up then they will see the breathing animation pause during the retries - then the LEDs will power off when the suspend is successful. Maybe this is good enough?

Anyways I'm submitting #595 for this, would you be able to test this on the SPS?

[DrFlarp]

(also, would it just be better to just set +e the whole script? I think we always want to run everything regardless of errors)

[DrFlarp]

Got really frustrated with root-cause debugging the kernel. If the wifi module is removed I sometimes (5-10% chance) get a suspend failure. Checking /sys/kernel/debug says it's related to suspend_noirq(). There doesn't appear to be a driver attached to this failure.

A little more testing is still required to see if rmmod xradio_wlan would be better - how much longer would it delay or interfere with bringing wifi back up?

[frysee]

Got really frustrated with root-cause debugging the kernel. If the wifi module is removed I sometimes (5-10% chance) get a suspend failure. Checking /sys/kernel/debug says it's related to suspend_noirq(). There doesn't appear to be a driver attached to this failure.

A little more testing is still required to see if rmmod xradio_wlan would be better - how much longer would it delay or interfere with bringing wifi back up?

Just checked on TSPS, its less impactful than I remembered, actually:

# cat /tmp/nextui_boottime 
before mount 7.16 34.75
after mount 7.17 34.83
after splash 7.27 35.47
after cpugov 7.71 46.95
before pkg install 7.72 46.98
after pkg install 7.73 47.00
after update install 7.74 47.01
before leds 8.17 47.78
after bluetooth 8.53 47.95
after wifi 8.63 48.01

A bit less than half a second for both wifi and bt to load on startup, combined. Wifi especially seems pretty inconsequential, BT takes a bit longer it seems.

[DrFlarp]

Unfortunately removing the module puts the device in a state where sometimes the device properly suspends but kernel still returns a non-zero error code. This puts us in a state where the user wakes the machine, the script sees the error code, considers this as a failure and re-attempts the sleep. The user would have to wake the device multiple times before it actually stays awake.

I'm going to drop this rmmod idea and submit this as-is, though still pending testing on TSP and TSPS.

[foxflyy]

I tried to rmmod wifi modules there was bigger issue: 209.217656] sunxi-mmc sdc1: smc 2 p1 err, cmd 5, RTO !! [ 209.220895] sunxi-mmc sdc1: smc 2 p1 err, cmd 5, RTO !! [ 209.224136] sunxi-mmc sdc1: smc 2 p1 err, cmd 5, RTO !! [ 209.227376] sunxi-mmc sdc1: smc 2 p1 err, cmd 5, RTO !!
Like the action of removing and re insert sdio devices with wrong processing.
The benefit could be wifi i2c has been set to low which be little help to reduce power consumption during deep sleep.
I'm curios this issue seems not duplicated on stock firmware, will look into it later.

[DrFlarp]

Yeah, it appears I've corrupted my dev SD card while working on how not have this happen. Had to nuke the partition table on the card and start over. I think what we have is good enough, if only it could be made a little more responsive in that 6-second window.