Fix critical architecture bugs and refactor ChaosGrad and Trainer

[theomgdev]

This pull request introduces several enhancements and refinements to the ChaosGrad optimizer and the RealNetTrainer class to improve training stability, adaptability, and configurability. The most notable changes include the addition of new hyperparameters for adaptive learning rate smoothing, plateau noise control, and input gradient health detection, as well as improvements to parameter classification and gradient clipping logic. These updates make the optimizer more robust and flexible for diverse network architectures and training scenarios.

ChaosGrad Optimizer Enhancements:

• Added new hyperparameters for improved adaptive learning rate and plateau escape: adaptive_ema (smoothing factor for adaptive LR variance), plateau_noise_intensity (multiplier for plateau noise), loss_history_min (minimum steps for loss history), and sentinel_threshold (threshold for input health detection). These are now configurable through the optimizer and preset methods. [1] [2] [3] [4] [5] [6] [7] [8]
• Improved adaptive learning rate smoothing by replacing the fixed EMA factor with a configurable value (adaptive_ema).
• Enhanced plateau noise injection logic for core parameters, making noise intensity configurable and more robust.
• Refined input gradient health calculation by introducing a configurable threshold for vanishing input detection.
• Updated parameter classification logic to better handle naming conventions and projections, improving optimizer targeting.

Training Logic Improvements:

• Added max_grad_norm as a configurable argument to RealNetTrainer, allowing flexible gradient clipping during training. [1] [2] [3]
• Modified gradient persistence logic to synchronize persistent gradients with the active AMP scaling factor, improving stability in mixed precision training.

Stability and Robustness:

• Updated spectral clipping logic to maintain momentum buffer alignment, ensuring optimizer stability after parameter scaling.
• Improved loss history tracking by making its minimum length configurable, supporting more reliable plateau detection.

These changes collectively make the optimizer and trainer more configurable, adaptive, and robust for a wide range of use cases.

[Copilot]

Pull request overview

This PR updates RealNet’s training stack by expanding ChaosGrad configurability (adaptive LR smoothing, plateau behavior, and input-gradient health detection) and making RealNetTrainer gradient clipping configurable, with additional tweaks to stability-related optimizer/training logic.

Changes:

• Added new ChaosGrad hyperparameters (adaptive_ema, plateau_noise_intensity, loss_history_min, sentinel_threshold) and wired them through presets/defaults.
• Refined ChaosGrad behavior for parameter grouping, plateau noise injection, spectral clipping buffer handling, and input gradient health computation.
• Added max_grad_norm to RealNetTrainer and adjusted gradient persistence to account for AMP scaling.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
realnet/training/trainer.py	Adds configurable grad clipping (max_grad_norm) and adjusts AMP-related gradient persistence behavior.
realnet/training/chaos_optimizer.py	Introduces new optimizer hyperparameters and modifies plateau/adaptive/spectral/sentinel logic for stability and configurability.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[theomgdev]

@copilot any more?

[Copilot]

@theomgdev I've opened a new pull request, #9, to work on those changes. Once the pull request is ready, I'll request review from you.