You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We're looking at an lb target cluster. It is often customary to put it in the same availability zone (AZ) as the source. I.e. the local targets are preferred.
The problem scenario arises when all the local targets go down but the NT itself stays up. In this case, we'd like to continue sending the data.
Potential solution
Make a new kind of cluster called, say, lb_priority. It will work the same way as the lb cluster but will have 2 kinds of targets: primary and secondary. Secondary targets will be marked in the config, all others will be primary which will also be the default. When at least 1 primary target is available, all traffic will go to the set of primary targets. If all primaries are down, NT will start using the set of secondary targets.
Does this solution bring value? Trade-offs.
This will solve the problem mentioned in the beginning. However, there's a number of concerns:
The scenario described is rather rare. It also does not fit into the abstraction of AZ. If AZ goes down as a whole it is not expected for the NT continue working. If there's a network partition that cuts-off an AZ, NT will work well and this fix is not necessary.
This solution is intended to make NT more reliable. However, the scenario it covers is exotic, so it's benefits will be rare. On the other hand, it will result in a more complex system that by itself reduces the reliability. It is not possible to give a numeric estimate at the moment, but there's a good chance that net value of this solution will be zero of even negative. Taking into account the costs described below, the value of this solution becomes even smaller.
The targets in each AZ will have to be provisioned with a much higher capacity that will be sufficient to accommodate traffic from other AZs. This is necessary for the solution to work. Otherwise, data will be lost because of overload.
There will be a number of additional failure scenarios this feature will introduce: e.g. AZ flapping, addition of feedback loops in code, less configuration abstraction, etc. These can be elaborated in detail later.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
What problem are we trying to solve?
We're looking at an
lbtarget cluster. It is often customary to put it in the same availability zone (AZ) as the source. I.e. the local targets are preferred.The problem scenario arises when all the local targets go down but the NT itself stays up. In this case, we'd like to continue sending the data.
Potential solution
Make a new kind of cluster called, say,
lb_priority. It will work the same way as thelbcluster but will have 2 kinds of targets: primary and secondary. Secondary targets will be marked in the config, all others will be primary which will also be the default. When at least 1 primary target is available, all traffic will go to the set of primary targets. If all primaries are down, NT will start using the set of secondary targets.Does this solution bring value? Trade-offs.
This will solve the problem mentioned in the beginning. However, there's a number of concerns:
All reactions