|
| 1 | +--- |
| 2 | +layout: post |
| 3 | +title: My Adventure with Late Answers |
| 4 | +comments: yes |
| 5 | +tags: community |
| 6 | +--- |
| 7 | + |
| 8 | + |
| 9 | + |
| 10 | +Last week, I finally got around to analyzing |
| 11 | +[a change](http://meta.stackexchange.com/q/266696/1438) to the Late |
| 12 | +Answers review queue. For those who aren't familiar with Stack Exchange, |
| 13 | +[review queues](http://stackoverflow.com/help/privileges/access-review-queues) |
| 14 | +give experienced users a chance to look over posts from new users for |
| 15 | +problems that can be corrected or hidden gems. The later is |
| 16 | +particularly important for answers that come over a month after a |
| 17 | +question was asked, since those answers often get ignored. Increasing |
| 18 | +the pool of new users eligible for consideration seemed like a good |
| 19 | +way to combat the commonly-held perception that Stack Overflow is |
| 20 | +rude. |
| 21 | + |
| 22 | + |
| 23 | +[{: .wrap}](https://xkcd.com/1428/) |
| 24 | + |
| 25 | +My main concern was that by adding more reviews, we might get into a |
| 26 | +situation where more answers were being added to the queue than the |
| 27 | +reviewers could handle. That can be a problem since ever-growing |
| 28 | +queues |
| 29 | +[demoralize reviewers](http://meta.stackoverflow.com/q/255270/1438). So |
| 30 | +most of my analysis was geared toward predicting how many extra tasks |
| 31 | +my change would create. By my estimate, it would add roughly 300 tasks |
| 32 | +a day on Stack Overflow and maybe a dozen on other large sites. Given |
| 33 | +Stack Overflow users review tens of thousands of posts a day, these |
| 34 | +numbers seemed manageable. |
| 35 | + |
| 36 | + |
| 37 | +Site Name | New Tasks |
| 38 | +:------------- | --------: |
| 39 | +Stack Overflow | 314 |
| 40 | +Super User | 13 |
| 41 | +Server Fault | 9 |
| 42 | +Apple | 7 |
| 43 | +Gaming | 5 |
| 44 | +Ubuntu | 4 |
| 45 | +All others | 3 or fewer |
| 46 | + |
| 47 | +After this analysis, I consulted a colleague and decided to just |
| 48 | +change the site setting and see what happens: |
| 49 | + |
| 50 | +[](http://meta.askubuntu.com/q/14469/173605 "4.4k Late Answer reviews") |
| 51 | + |
| 52 | +So my estimate was about 4 _orders of magnitude_ wrong. With a single |
| 53 | +change of a variable, I'd increased this site's outstanding reviews 10 |
| 54 | +fold. And I'd made the change on every site on the network. Not only |
| 55 | +that, but there is no easy way to remove review tasks except by, well, |
| 56 | +reviewing them. I'd managed to inadvertently create a bunch of work |
| 57 | +for thousands of people. My immediate reaction was panic. |
| 58 | + |
| 59 | +Then I got down to business figuring out what had happened. It quickly |
| 60 | +was clear that the change picked up late answers that were older than |
| 61 | +a day. Initially, I thought the code was only picking up a month's |
| 62 | +worth of posts (more on this later), but comments on |
| 63 | +[my answer](http://meta.stackexchange.com/a/267080/1438) fixed my |
| 64 | +misunderstanding. In fact, the change put _all answers ever_ that fit |
| 65 | +the new criteria up for review. |
| 66 | + |
| 67 | +Meanwhile, I'd notified the network's moderators of the change and, |
| 68 | +thanks to their encouragement, featured the meta post describing the |
| 69 | +change on the sidebar across the network. Many sites had already |
| 70 | +started |
| 71 | +[noticing the change](http://stackexchange.com/filters/205229/late-answers) |
| 72 | +and answering the meta questions became an addition task for |
| 73 | +moderators and involved users. I couldn't keep up with those posts, |
| 74 | +but |
| 75 | +[looking back a week later](https://www.google.com/search?q=site:meta.*.stackexchange.com+"late+answers") |
| 76 | +it's gratifying to see so many informed (and considered) answers from |
| 77 | +other users. |
| 78 | + |
| 79 | +Also meanwhile, people were working hard to remove tasks from the queue: |
| 80 | + |
| 81 | + |
| 82 | + |
| 83 | +This activity is on [Ask Ubuntu](http://askubuntu.com/), but I could |
| 84 | +show similar charts from all sorts of sites. Traditionally, Late |
| 85 | +Answers tend to be a backwater of the review queues, so the sudden |
| 86 | +spike was a rare opportunity to earn badges for that queue. The week |
| 87 | +before my change, there were 5 Custodian badges—the bronze level |
| 88 | +for doing just one review. In the following week: |
| 89 | + |
| 90 | +Badge |Queue |Awarded |
| 91 | +:--------|:----------|------: |
| 92 | +Custodian|Late Answer| 6 |
| 93 | +Reviewer |Late Answer| 7 |
| 94 | +Steward |Late Answer| 1 |
| 95 | + |
| 96 | +The [Steward badge](http://askubuntu.com/help/badges/126/steward) |
| 97 | +requires 1,000 Reviews, which is pretty hard to accomplish when there |
| 98 | +are only 300 or so a week divided among hundreds of interested users. |
| 99 | + |
| 100 | +As the initial crisis began to die down, I started to wonder why Stack |
| 101 | +Overflow hadn't exploded. The answer, which I'd noticed in the code |
| 102 | +before starting research: |
| 103 | + |
| 104 | + (Current.Site.IsStackOverflow ? "and p.CreationDate > GETUTCDATE() - 7" : "") |
| 105 | + |
| 106 | +So I was saved by a special case that was, no doubt, built in for just |
| 107 | +this purpose. On Stack Overflow, the code does not consider answers |
| 108 | +older than a week, so it only spiked to 1500 or so. Why did I think |
| 109 | +the code looked back 30 days? I don't really know, but I think I got |
| 110 | +that from the definition of a "late answer": 30 days after the |
| 111 | +question was asked. |
| 112 | + |
| 113 | +## Lessons learned |
| 114 | + |
| 115 | +1. **Reading the code is no substitute for testing.** Well, I've known |
| 116 | + this for _at least_ 20 years. But thanks to my high self-esteem, I |
| 117 | + need to relearn this yearly, it seems. We have a development |
| 118 | + environment which would have been a perfect place to discover that |
| 119 | + the queue was going to spike, if only I'd thought to try there |
| 120 | + first. Alas, I did not so think. |
| 121 | + |
| 122 | +2. **If you are about to make a change that might effect thousands of |
| 123 | + other people, you might want to test it first.** The bit I didn't |
| 124 | + consider was that people don't just see the number of outstanding |
| 125 | + reviews when they go to the review page. The number _also_ gets |
| 126 | + displayed for people with 500+ reputation on the top bar visible on |
| 127 | + every page on the site. So plenty of people noticed even if they |
| 128 | + don't make a habit of reviewing. |
| 129 | + |
| 130 | +3. **Don't make a dramatic change if you can make several small |
| 131 | + changes instead.** Many of the comments/forceful suggestions I |
| 132 | + received in the aftermath of my fateful change hinted that a more |
| 133 | + controlled increases in reviews would have eased the concern people |
| 134 | + had. There were any number of ways I could have |
| 135 | + [boiled the frog](http://conservationmagazine.org/2011/03/frog-fable-brought-to-boil/), |
| 136 | + but the easiest would have been to increase the reputation limit in |
| 137 | + increments of 5 or 10 followed by quiescence until we reach the |
| 138 | + desired limit. The thought never crossed my mind. |
| 139 | + |
| 140 | +4. **Even if you plan to make a change no matter what, it helps to |
| 141 | + notify people in advance.** Obviously, _knowing_ the actual impact |
| 142 | + of the change is a prerequisite to announcing a coming change. But |
| 143 | + if I'd known, I still would have wanted to increase the Late Answer |
| 144 | + pool. Obviously, I couldn't ask _everyone_ who has a |
| 145 | + stake. However, it would be good to let the moderators know what to |
| 146 | + expect at least a few hours in advance. |
| 147 | + |
| 148 | +## Future work |
| 149 | + |
| 150 | +I'd really like to see about increasing the reputation level |
| 151 | +to 100. But before I do that, I really should take advantage of the |
| 152 | +natural experiment created by the change to 50. How did people handle |
| 153 | +the flood of late answers? Overall, did we find hidden gems or just |
| 154 | +superfluous answers? In the weeks to come, will people who benefited |
| 155 | +from upvoted answers in this queue tend to contribute more and better |
| 156 | +answers? Or has the feedback come too late? |
| 157 | + |
| 158 | +<!-- LocalWords: LocalWords http png https xkcd chatroom ubuntu |
| 159 | + --> |
| 160 | + |
| 161 | +<!-- LocalWords: GETUTCDATE CreationDate IsStackOverflow |
| 162 | + --> |
0 commit comments