Reworks balloon for sending balloons in freeze #3305
Conversation
|
Mhh come to think of it, we don't really have the full old behavior anymore. Because now problems that are only solved in the freeze will never receive a balloon regardless of the value. What do you think, should the old config var be reinstated? |
tuupke
force-pushed
the
Feature/balloon-rework
branch
2 times, most recently
from
60343b5 to
682de1f
Compare
|
Possibly related to #2778 |
Yes thank you! I was looking for this exact issue but couldn't find it for some reason. These indeed serve the same purpose. |
|
I'm quite sure the remarks phpstan makes are incorrect. Since otherwise it would not work at all while it does 🙃 |
Probably the docblocks for the methods being called are wrong. Use cd webapp
vendor/bin/phpstan analyze --configuration phpstan.dist.neonto check locally btw. |
Kevinjil
left a comment
Kevinjil
left a comment
There was a problem hiding this comment.
Just a general question: has something (e.g. at SEERC) influenced why we want this feature compared to the last time it was discussed IRL at the hackathon?
tuupke
force-pushed
the
Feature/balloon-rework
branch
from
682de1f to
52d58f0
Compare
Partly. Some were just because we we're already looking at this code because of a bug (hence Nicky's PR) Some discussion and talks 'inspired' me on how to implement and that this is a nice feature. It was also the case that there were some teams that only solved some of the easy problems in the freeze. Because this is somewhat common -especially on the early rungs of the tournament ladder- it might be a nice motivational feature to add. The replacing of the old config flag came after (incorrectly) thinking that this change offered at least the same functionality as before. Tbh I cannot recall this discussion, but if we're opposed we can close the PR; I don't think it hurts anything while it does add a new functionality. When declining the pr we should cherry-pick 706147e since it DRYs up the retrieving of balloon updates. |
tuupke
force-pushed
the
Feature/balloon-rework
branch
from
52d58f0 to
37f27e8
Compare
vmcj
left a comment
vmcj
left a comment
There was a problem hiding this comment.
I think we shouldn't merge this without tests. I do agree it happens relatively often but some in the team had strong opinions on such a feature, because of the same opinion we don't test this ourselves at competitions so I think if we ship this we should test it in a different way (with Unit tests for example..)
|
I'm not in favor of adding this because it adds extra complexity and weird exceptions. For example, what if a teams solves their first problem in the freeze, but it is an as yet unsolved problem? Highly unlikely maybe, but then that exposes crucial information during the freeze. There are likely similar edge-case issues, which I think is the reason we should not add this. |
|
I think a better way to solve this, is when |
So, this proposal basically shifts the balloon selection from a technical problem to an organizational problem. One could e.g. decide that such balloons are not handed out, unless the contest jury decides to grant it. |
I think the balloon runners are not always the most experienced with all the rules and edge cases on when something would disclose information. So in that case I would say we should also include that information (handing this out provides additional crucial information), and if you already have that information you also have the edge cases. So leaving the choice up to the balloon runners feels wrong to me. |
Sure, I'll go ahead and start adding some tests. Good to have for balloons anyways. Will probably/hopefully also help address some of @eldering's concerns. |
This is exactly why these are only shown for problems that have been solved pre-freeze by teams in the same sortorder. So in your example the team would not get a balloon. The entire goal is to not leak any information that was not available pre-freeze while still being able and hand out balloons when desired. I've tried making this abundantly clear in the code and the doc -more in the first than in the second- any suggestions for how to improve? Adding the testcases as suggested by @vmcj will help discover and prevent regressions on these possible edge-cases. I've given this quite some thought and could not come up with relevant edge-cases within the scope of the feature. I'm open to any and all edge-cases you can think of @eldering. Would be great to address those beforehand. |
I completely agree! Solving this at an organizational level is not a good idea. The expectation that (balloon) volunteers are aware of enough intricacies and act accordingly does not hold. We've all organized and been to plenty of contests where this does not even hold for the pre-freeze situation let alone for things that happen post-freeze. |
|
To add to the edge-case discussion. The following scenario would potentially leak information:
The major issue with this scenario is that it assumes the value of balloons to hand out post freeze is set to a number so high that this well performing team still gets a balloon. Any team with a sufficient number of accepted submissions will not receive any more balloons regardless of which problem they solve. I'd like to stress that the behavior is disabled by default. Only after setting the configuration variable to a strictly positive value will balloon notifications appear for correct post-freeze submissions. There is thus less potential for information leaking than with the old implementation of handing out all balloons post-freeze. For completeness sake, I will be also be adding back the old behavior of handing out balloons to all accepted post-freeze submissions somehow. Not yet sure whether this is with the old or a new config flag. |
Ok, I overlooked that you handled that case. However, I'm still not a fan as this opens up a can of worms of inconsistencies. Next, someone comes and complains that the balloons API endpoint is not in sync with submissions/judgings/scoreboard, and I really don't want to go there.
I can't think of any right now (didn't think too carefully), but this feels the same as the first to solve balloon in contest where we uncovered various bugs, and finally gave up on fixing it. And tests are not a silver bullet. Anyways, if no-one else objects, then I'd say we must at least have a big disclaimer that we don't guarantee consistency between balloon and other data, so we can keep this completely self-contained inside balloon-only code. |
I kind of agree with @eldering that this feels like skating on thin ice. There are numerous situations, for now leaving out the likelyhood, that will still leak information. Consider for example:
This example is exaggerated to illustrate the potential leakage. One could define some How we should weigh this risk against the potential joy perceived by teams still being handed a balloon, is where I'm kind of undecided. Right now, I lean towards having a robust automated system and having the jury (NOT the balloon runners) explicitly decide on encouragement balloons. |
|
This whole feature is still behind a config option. If you don't want to leak this information, disable the option? I do agree with @eldering that we should add a big fat warning when enabling the feature btw. |
Fair point, a big warning (also on the config checker page) seems right here. |
tuupke
force-pushed
the
Feature/balloon-rework
branch
from
37f27e8 to
73a38f3
Compare
|
I totally agree that there is a potential for a data leak but since this behavior is (1) both disabled by default and (2) there is now a potential for (at least) less data leakage and inconsistencies compared to the old setting. I always found it surprising we did not have this disclaimer before so definitely a good idea and will add it for sure. Just an observation but what I find kind of funny is that there are suggestions that this can be solved on an organizational level with the old (non-disclaimer, 'show every balloon during the freeze') option because people can reason about all these cases, while at the same time arguing that there are so many edge-cases that we need a disclaimer to warn them. (Which I again agree with) Wrt to the (internal) inconsistencies. This is why it is should all be DRYed up by using the -- -- |
tuupke
force-pushed
the
Feature/balloon-rework
branch
from
ccd8f28 to
46c8b18
Compare
Kevinjil
left a comment
Kevinjil
left a comment
There was a problem hiding this comment.
I think this is a fine balance. This change might also deserve an entry in the changelog as a (breaking) change in config.
Balloons can be quite a motivator for a team and it can be frustrating to spend >4 hours solving a problem to only get an AC during the freeze. We used to have the show_balloons_postfreeze configflag to send balloons during the freeze but this would always send balloons during the freeze. show_balloons_postfreeze has now been replaced with 'minimum_number_of_balloons'. This setting now expresses the minimum number of balloons to send to a team even during the freeze. To prevent an information leak only balloons for problems that have been solved before the freeze can be sent out. Leaving this value to 0 keeps the 'old' behavior of not sending any balloons while setting it to a value >#contestproblems results in the old behaviour of sending all balloons during the freeze.
The computation for which balloons need to be sent out is a bit more complex than before. Reusing the logic of the BalloonService keeps it nice and DRY.
The old behavior allowed handing out all balloon notifications during the freeze regardless of whether the problem was solved before the freeze. Instead of reusing the old configuration variable a new variable is chosen to better reflect the interaction with `minimum_number_of_balloons`.
tuupke
force-pushed
the
Feature/balloon-rework
branch
from
46c8b18 to
fe178e5
Compare
| type: int | ||
| default_value: 0 | ||
| public: true | ||
| description: How many balloons to hand out ignoring freeze time. Only hands out balloons for problems solved pre-freeze. |
There was a problem hiding this comment.
| description: How many balloons to hand out ignoring freeze time. Only hands out balloons for problems solved pre-freeze. | |
| description: How many balloons to hand out ignoring freeze time. Only hands out balloons for problems solved pre-freeze by other teams. |
| description: How many balloons to hand out ignoring freeze time. Only hands out balloons for problems solved pre-freeze. | |
| description: How many balloons to hand out ignoring freeze time. Limited on problems solved pre-freeze. |
| default_value: 0 | ||
| public: true | ||
| description: How many balloons to hand out ignoring freeze time. Only hands out balloons for problems solved pre-freeze. | ||
| - name: any_balloon_postfreeze |
There was a problem hiding this comment.
How is this option different than #Balloons>#Problems?
If the intent is to disclose all the information it's currently unclear.
6 participants
Replaces
show_balloons_postfreezewithminimum_number_of_balloonswhich allows for sending out balloons to teams even during the freeze while still supporting the old behavior if desired.This is useful as a motivational tool for teams that have not yet solved 'enough' problems before entering the freeze. e.g. 2 problems before the freeze while the expected number of problems to solve before the freeze is 3. By setting this new value to 3 the team will receive a balloon as long as the problem they've solved has been solved before the freeze.
Kept on draft until I fix the documentation and do some minor cleanup.