[Question] Why SAETrainer do not resample dead neurons?

[jasonrichdarmawan]

I suspect there is no logic for resampling dead neurons?

It seems the sparsity/dead_features metric never decrease and I can't find code which resample dead neurons

There is logic for sampling dead neurons. However, it's only for metrics and not to resample the dead neurons

If you are interested, you can see the WandB runs below

Note: gated_detach-16384 run variants use via_gate_reconstruction = pi_gate_act @ self.W_dec.detach() + self.b_dec.detach() to compute the auxiliary_reconstruction_loss. Meanwhile gated-16384 run variants use the unmodified version of SAELens discussed in #539

https://wandb.ai/jasonrichdarma/sonar_sae?nw=nwuserjasonrichdarmawan

[chanind]

Some SAE architectures include an aux loss to revive dead neurons, some don't. GatedSAEs do not include a resampling loss.

[chanind]

One thing to try if you're not already is to add a l1 warmup to the SAE training - this tends to help with dead latents