[Feature Proposal] - Total Allocations Tracking

[nrwiersma]

Is your feature request related to a problem? Please describe.
At no point does Agones track or expose the number of allocations (GameServers moving into Allocated state). This however is an extremely useful metric in general, but also specifically for autoscaling.

Describe the solution you'd like
Add TotalAllocations to the FleetStatus, which is a uint64 counter of the total number of allocations observed over time. This should not update the Fleet for every allocation observed, but rather rate limit changes to the Fleet for this counter to 1s. A prom metric should also be exposed with this data.

The TotalAllocations field should be provided in the FleetAutoscaleRequest to webhooks and WASM.

This allows users to reason about the number of allocations on a fleet and react to this data in a custom autoscaler, which should allow for better buffer size optimisation.

An example of the new FleetStatus:

type FleetStatus struct {
	Replicas int32 `json:"replicas"`
	ReadyReplicas int32 `json:"readyReplicas"`
	ReservedReplicas int32 `json:"reservedReplicas"`
	AllocatedReplicas int32 `json:"allocatedReplicas"`
	TotalAllocations uint64 `json:"totalAllocations"`
	Players *AggregatedPlayerStatus `json:"players,omitempty"`
	Counters map[string]AggregatedCounterStatus `json:"counters,omitempty"`
	Lists map[string]AggregatedListStatus `json:"lists,omitempty"`
}

Describe alternatives you've considered
It is possible to do this outside of Agones in a custom controller or webhook, and then inject the data where needed.

Additional context
Add any other context or screenshots about the feature request here.

Link to the Agones Feature Proposal (if any)
The link to the AFP PR.

Discussion Link (if any)

[markmandel]

but rather rate limit changes to the Fleet for this counter to 1s.

Only other potential thought I had was that since a GameServer (or more) moves to an allocated state, it may not need a 1s time, you may be able to pick up the change with the GameServerSet update, since it should happen relatively quickly after an allocation.

https://github.com/googleforgames/agones/blob/b9bbac2db88365a60df0aae9f6b86a9bdfa7d545/pkg/fleets/controller.go#L117-L126

Just a theory though - may just need investigating.

Otherwise sounds good!

[nrwiersma]

The issue I see there is when you have 1 GameServer move into Allocated and 1 out in roughly the same time, I am not sure the GameServerSet would then pick it up.

I had 2 possible angles of implementation here:

1. Get the fleet controller to implement in interface used by the gameserver controller to mark when it sees an allocation (this trades some efficiency for some light coupling)
2. Have an additional GS informer watch in the fleet controller looking only for this condition

In both cases they could update a an atomic value in a syncmap (which gets deleted when the fleet is deleted) and either queue the fleet or delay queue the fleet (I think i have some rate limiting implementation kicking around somewhere to do this). Then basically every time the fleet updates, it checks the map and adds to the counter somewhat opportunistically.

[markmandel]

The issue I see there is when you have 1 GameServer move into Allocated and 1 out in roughly the same time, I am not sure the GameServerSet would then pick it up.

Yeah 100% agree - that's my concern as well. Was just an idea - wasn't sure if it would stick or not.

In both cases they could update a an atomic value in a syncmap (which gets deleted when the fleet is deleted)

I'd probably not go with a syncMap just because it tends to not be great for general usage - but the idea tracks. Just have some mutexes instead. Could even be encapsulated into it's own struct.

Most code should use a plain Go map instead, with separate locking or coordination, for better type safety and to make it easier to maintain other invariants along with the map content.

I was going to suggest sharing the map between the fleet controller and the allocation service -- but they are in different whole binaries! So that doesn't work.

I think you are right - keep the GS Informer watching into the fleet controller -- that seems like the best bet.

[nrwiersma]

Awesome, I will try to secure some time from the benevolent overlords soon to tackle this.