Harden SwiftBuddy model loading and align local server settings #308

Job	Run time
build_and_unit_test	13m 0s
dflash-speculative-decoding	15m 34s
speculative-decoding-eval	10m 54s
ssd-draft-memory-guard	1m 23s
integration_matrix (opencode)	3m 43s
integration_matrix (vision)	1m 25s
integration_matrix (graph)	1m 8s
integration_matrix (audio)	2m 37s
integration_matrix (omni)	2m 36s
speculative-decoding	12m 34s
integration_matrix (server)	1m 37s
	1h 6m 31s

Provide feedback