fix clustering problems with singleton processes

[LKlemens]

Proposition of fixing clustering in teiserver 🤞

Fix

• Cluster before starting singletons. Added libcluster; Cluster.Supervisor starts first in the supervision tree (no-op when the node isn't distributed). Nodes now connect at boot
• Teiserver.SingletonSupervisor replaces the plain Coordinator.DynamicSupervisor as the home of the four singletons. Children of a dead or stopping node are automatically restarted on a surviving node
• Cooperative conflict resolution for the residual race (unclustered boot, netsplit heal): singletons register via :via tuples in ServerRegistry at start_link and handle the trapped {:EXIT, _, {:name_conflict, ...}} with {:stop, :normal}. A merge loser now steps down cleanly instead of zombie-surviving (catch-all handle_info would otherwise swallow the kill) or respawning into a kill loop. Registry keys are unchanged, so all lookup sites work as before.
• Matchmaking.QueueSupervisor.start_queue!/1 treats {:error, {:already_started, pid}} as success - previously the second node's boot aborted entirely because the default queues already ran elsewhere in the cluster.
• Removed dead QueueWaitRegistry/QueueMatchRegistry

Why env reads moved to runtime.exs

config.exs/test.exs are evaluated at build time: for a release, a System.get_env there bakes in the build machine's environment and is ignored at server start (it only appears to work in dev because Mix re-evaluates config each run). runtime.exs is evaluated at boot in every environment and loads last, overwriting earlier config per key.

Testing

you can run different nodes using just command that automatically bumps required ports to avoid conflicts

just node
just node 2
just node 3

https://github.com/casey/just

[LKlemens]

@Teifion @geekingfrog Please let me know what do u think 🙏

[Teifion]

Thank you for taking the time to make this PR, I've not had a chance to go through it properly at all but the first thought is there's a little bit of context missing here. The main part being many of these sections are going to be removed as we swap from the Spring protocol to the Tachyon protocol and this is probably not the right time for this PR (though some of the config stuff may well be very appropriate).

If you are on the discord (there is no obligation to be there) please feel free to ping me if you see me online and I'll happily jump into a voice chat with you to answer questions in real time if that is of interest to you.

[LKlemens]

Yeah, I'm not really into teiserver :D
Thank u for invitation to the discord channel but unfortunately I'm really busy rn to dig dipper into project 🙏
I just saw deletion of horde in this PR - and drived by curiosity decided to try to fix the clustering problem
If it is not a right time to do it - that's totally fine - feel free to close the PR.
Thanks for review!

[Teifion]

I don't intend to close the PR without a proper examination and evaluation of what parts should be cherry picked out :)

[geekingfrog]

There is an AI policy that you almost certainly ignored, the description screams llm.

The author of the PR must write the description in part to demonstrate their understanding of the chnage. [sic]

Then, this PR does too many things.

• automatic cluster formation with libcluster
• "fix" singleton conflicts
• just file to start several nodes locally

That should be broken down.

And finally, we are not going to do clustering for the lobby protocol, fixing the singleton conflict for these is a complete waste of time.
For tachyon (and matchmaking), maybe that makes sense, but it still doesn't fix the problem of what happen when the node running the queue goes down. What happen to the state then?
Simply fixing the error at startup isn't enough.

[LKlemens]

Thanks for honest review!
Like I said, it was just proposition - if u consider it as AI slop - that's fine (sorry, I didn't want to offend anyone 🙏)