Handling beta kata retirement more gracefully and better informing new kata authors of beta process quality assurance

[Voileexperiments]

Coming from #1645.

Currently beta katas are retired after receiving a very low satisfaction rate, which can happen quickly when a new beta kata has significant issues. The problem here is:

• 1. This process happens too fast. Usually kata authors aren't fast enough to catch on the issues before everyone voted according to kata quality factors and the beta kata gets retired.
  (Maybe it can be solved by making retirement temporary for 1 day where existing votes can still be changed to stop the retirement, then permanent?)
• 2. There are no documentations of retirement feature anywhere. We can't even tell the exact criteria for a kata to be retired, or that katas are even retired in the first place. For most users who aren't around long enough, they wouldn't know anything except that their kata got retired for no apparent reasons, which doesn't make any sense and results in a salty experience.
• 3. Kata best practice is still hidden to practically everyone. This and this are buried so deep unless someone knows they exist or are sent a link of them, they wouldn't even know they existed, and so they wouldn't know what constitutes the quality in the first place.
• 4. There are no mentions that beta katas are supposed to be CW content either, which have to be curated and assured of their quality. Basically, once a kata is published from draft, it stops being the user's baby anymore and they cannot just do whatever to the kata as they please.
• 5. The wording around the entire beta process is... not very ideal and perhaps a bit too hostile. e.g:
  

Incidentally (and unsurprisingly), stackoverflow meta has encountered such kind of issues for a long time, and they have some very good suggestions.

---

• 2: Added Auto-Retire section

┆Issue is synchronized with this Clickup by Unito

[CliffStamp]

This has to be made a priority to address, you can see new authors getting kata's retired immediately in the same day, often with no issues raised. The kata is published, it is gang voted down and retired.

I just had a kata retired in the middle of me adding random tests to it when I noted the solution lacked them, and there were no comments, no issues raised. It comes up and goes down on the whim of just a handful of people. And I guarantee if you look at the host of auto-retired kata you will see the same people involved.

The way it is now, you might as well give a select few people the ability to just auto retire katas based on what they feel alone is worthwhile with no mandate to adhere to any community/site standard because that is exactly what has been is achieved.

And if this isn't going to be changed, put a big red flag on the screen when new katas are getting made to the effect of

"NOTE - if just a couple of people don't like this kata, it can be retired for no other reason than they don't like it. You are not guaranteed to get any comments, nor any change to fix it, it will just be retired."

If you are going to be extremely unfair and hostile to new users, at a minimum, let them know what to expect.

[Voileexperiments]

@CliffStamp Like I have said before, if you insist on accusing everyone for being strict, hostile elitists who shuts down every kata for arcane reasons, please provide a proof for this, because among all PUs you're literally the only one who's making such accusations. If you cannot provide such proof and yet keep saying so sprinkling salt all over us, you're not just not helping the situation, you're making it worse.

Also, your point is off-topic to this issue because strictly speaking, "some users are too strict about beta process" is not the topic on this issue. This issue is about "how to make beta process more transparent to everyone". If you want to complain about particular users, go make another issue about it and explicitly state it as such (e.g "that user whose name's md5 hash is b4f14574a9f95bfa174bbd255752121f does not obey beta standards and is definitely an alien").

[Blind4Basics]

I'm not sure there is an ideal way to handle all of this, but, about the retirement process being to quick, what about one of those:

• automatic unpublishing with notification to the author for a beta having an issue opened (only for recent published kata, otherwise old betas will definitely disappear)
• maybe the same with a new "not satisfied" vote? (tho there might be the problem of the lacking feeback about what to do)

This, way, the kata isn't retired yet, isn't accessible "anymore/for now" so that should let time to the author to look into this. and being put to draft, the kata isn't completable anymore (no submit final) so for PU who scrutinize their dashboard and access the kata seeing the issue raised, they still can raise other issues, but cannot downvote (for now), so that'd stop the "hemorrhage".

Just ideas...

[CliffStamp]

• automatic unpublishing with notification to the author for a beta having an issue opened (only for recent published kata, otherwise old betas will definitely disappear)

There are lots of issues spammed which are not actual issues (i.e. the kata isn't actually broken), they are just personal preferences. There are PU's who don't like any kind of story/description for example and just want a problem stated like it was from a math textbook. Because someone has such an opinion doesn't mean the kata should get unpublished. I could keep everyone one of your katas unpublished by just raising as an issue that it should be translated into French as well as English, it seems absurd that anyone's personal and arbitrary whims should have that kind of influence. The real problem is there is nothing stopping abuse, and there is rampant abuse of the current system.

https://www.codewars.com/kata/reorder-the-text/solutions/javascript

Second kata retired by the same guy. First one had no comments at to the problem, just auto-retired. In the second he fixed every issue raised promptly to ensure he wasn't auto-retired, still retired because there are people who will vote to retire a kata for nothing aside from a personal whim without saying why they are voting to do it. Yes, that is exactly how you make Codewars a place to encourage learning and to foster new kata content.

And again, it is the same people, lets not pretend this is a community approach to curation.

Here is the reality

If you have something which is supposed to be a community driven perspective, BUT, you tolerate extreme actions of a vocal minority - then all you will have is that extreme vocal minority. Because of the lack of any kind of community moderation, that is what has been created, a lot of power has been given to a select few without moderation and that will always lead to abuse, this isn't a well known fact, do any reading at all about what happens if you power divide a community without moderation.

You want a solution, you only have two choices

• you take away the ability of those people who are constantly spamming down votes to auto-retire katas based on their perspective only

• you change the counting mechanism so that their extreme view isn't the only one heard

The second one is very easy, you simply give beta's a longer grace period, say a month before that retirement is evaluated. You have to realize not everyone is on CW every day, some people have other jobs/lives and some issues are not at all easy to resolve, reference solutions which miss edge cases, generators which miss edge cases, etc. . That gives enough time for the community to evaluate it, and for authors to respond.

But as notes, while this isn't fixed, and you have such a hostile environment, the least you can do is warn any new author what they are facing. You know the problem exists, you know it isn't fixed, so warn people is it there so you don't have people waste their time who don't want to deal with that.

[kazk]

The calculation of satisfaction percent had a bug when the number of "Not Satisfied" was more than "Very Satisfied" (#1292). This made some kata to auto-retire much quicker.

I'll be deploying the fix for this either tomorrow or early next week.

@CliffStamp only one user down voted that kata.

[Blind4Basics]

@CliffStamp

you're talking about abuses, personal preferences and those who are screaming the loudest... But do you even realize that you fit extremely well into each one of theses categories too? Meaning that you don't help the community either, currently.

You actually level up the average stress by a lot, mostly because of your general tone and your behavior. And that's sad because you may have some good points/ideas in the middle of all your rants...

[Voileexperiments]

I think I should perhaps add a number 6 to the list in the OP:

Since beta katas award honor for ranking and satisfaction votes, and you lose the ability to make ranking votes after it's approved (and both after it's unpublished/retired), it encourages users to leave a vote no matter what. Which was good in the past because the approval criteria was high and you need up to 12 votes with at least 90% satisfaction rating to get past beta, but not needed anymore with beta approval requirement being as low as it is right now, and in fact is now being somewhat harmful in two ways:

• user voted "very satisfied" out of common courtesy, made problematic katas past 80% satisfaction criteria as a result
• user voted "somewhat satisfied" or "not satisfied" because there are significant issues with the kata, but the votes are coming too quick! There is no time for the kata author to react

I don't really think giving missable honor for voting is a good idea at now when most of the time votes are made just to not miss out that additional tiny bit of honor. At least there should be good reasons for us to hold out our votes for something, which currently is nothing.

[Blind4Basics]

@kazk : what are the current exact rules that trigger the auto-retirement? (That would be a good idea to document that somewhere in the wiki, btw. Maybe in the kata creation section? )

[kazk]

@Blind4Basics the rule hasn't changed.

Auto-retire conditions:

1. If 4 or more votes and satisfaction percent is 0 (4+ consecutive "Not Satisfied")
2. If n or more votes and satisfaction percent is <= 20 where n is the same as the number of votes needed to be approved

It should be the same as before. I only fixed a bug in the satisfaction percent calculation which was breaking the first one.

I didn't realize the second rule was using the number of votes needed to be approved. It makes no sense to me. This means a kata written by a new user requires more votes to be auto-retired :/ I guess this assumes a terrible kata by a new user is handled by the first rule.

---

• Added Auto-Retire section

[FArekkusu]

Personally, I liked the system before the fix more :(

It's unfair that some katas get retired quickly, but nobody seems to care about the fact that the majority of them are objectively very bad, and it may be a much better idea to kill those.

For example, this kata is plain worthless: alias an existing method with input validation thrown in; it has no novelty (and most probably a duplicate), and dealing with the input validation brings nothing game-changing to the task (and usually it's plain annoying). Yet with 0 very, 1 somewhat and 4 not satisfied votes it's still up...