Usability study 1

This is a discussion page for the results of a usability study that Colorado ran. (The text is taken from his e-mail to our mailing list, [email protected].). User study completed on 6 September 2013.

Usability study

Using the think aloud protocol, I've been asking various AI-oriented students at Berkeley to use metacademy for 5 or 10 minutes while I observe their interactions. I only provided a brief verbal overview of what metacademy is trying to accomplish, and then set them loose on metacademy.org's landing page. So far I've surveyed three users, and somewhat surprisingly, my notes are nearly identical for all three. Here's the summary:

1. The search functionality is a bit confusing, all three users said some form of "what should I search for" then typed in: machine learning (14 responses, but is a seemingly random hodgepodge of topics that use the phrase "machine learning") deep learning (0 responses) bayesian regression (3 good responses)

   • [Roger] I guess it makes sense that people start by searching for broad subject areas, since that's how we're used to learning about things. Once we have roadmap creation set up, we can create roadmaps for general topics like "practical machine learning" or "Bayesian machine learning," and include these in the search results.
   • [Colorado] We might also want to provide links to "courses," which would be unordered lists of concepts. This might be more natural for searches like "machine learning"

2. None of the users seemed to notice the new "change visualization" arrow pointing to the explore/learn transition during the first second of viewing the learning list

3. None of the users found the explore view on their own (I pointed it out to them after they scrolled around the learning view for awhile and thought they had exhausted all of the implemented functionality: often providing feedback like "it would be great if we could visualize the unrolled dependency structure explicitly, in addition to this list")

4. None of the users found the clear/show learned buttons (again, I pointed it out to them after they clicked around for awhile and thought it would be a good idea if they could remove concepts they already knew)

5. None of the users clicked the green checkmarks when exploring the concepts in the learning view, though they did click the checkmarks in the explore view

6. one user thought clicking on nodes indicated that she had learned the concept, since the checkmark stayed visible; this user was annoyed by the way the node summaries kept hiding other nodes)

7. The learning list is confusing, it's not obvious that it provides a topological sort of the dependency structure in the explore view: 2/3 users weren't sure if the learning and explore view provided the same content. The one user that recognized that they contained the same concept was exploring logistic regression (a simple concept).

   • [Roger] An alternative interface would be a sidebar on the left with the list of concepts. People would be more used to thinking of sidebars as ordered lists, and it would also solve the problem of people not noticing that the dependencies were there.
   • [Colorado] Yes! I think this would be much more natural. I was personally a bit dissatisfied watching the users slowly scroll all the way to the the top of the learning list (it took a long time for some of the more advanced topics), especially once they started expanding topics. Should we keep all concepts loaded in the list view still and provide a sidebar as a quick-navigation, or should we use the sidebar as the central navigation tool where clicking on a title loads that concept into the main viewing port, i.e. only one concept is viewable at a time? I think the latter option would be more natural.
     • [Roger] I agree -- use the sidebar as the main navigation, and only show one concept at a time. The current version could still work well for the mobile site, though.

8. 2/3 users encountered shortcut nodes in the explore view and did not understand what the dashed edges meant

9. None of the users recognized the explore-to-learn-view transition arrows provided in the hover-summary

10. One user thought it would be nice if the resources indicated whether the user had previously visited that resource (i.e. the specific location or the general resource)

    • [Roger] Should we do this as hovertext, or should it be displayed on the page itself? If it's the latter, what should the format be? I've been thinking of showing not only which concepts are already learned, but also how many steps it takes to get there given what the user already knows. (E.g. "You have already learned this concept" or "6 steps to learn this concept".)
    • [Colorado] I think placing a small icon before/after the visited resources would look reasonable, maybe a little checkmark... Informing the user of how many steps they are from learning the concept is a good idea. But, for the time being (and probably the foreseeable future), the majority of our users aren't registered users. So this will typically be the total number of prereqs the concept has. So I would opt for making this information very subtle, e.g. a small number in the corner of the list-view that has explanatory onhover text.

11. It appeared that nobody read " Read/watch one starred resource, and go to any of the others for additional clarification." At the end of session, no one knew the difference between starred/unstarred resources.

    • [Roger] Would they notice the directions if they're actually using the page to learn about something?
    • [Colorado] I don't know. We can leave it as-is for the moment, and readdress this issue if we find other users are confused by the star/bullet difference.

12. one user initially thought it was confusing that clicking on the title of the resource did not take him directly to the resource, but once he found the "location" link, he seemed satisfied

    • [Roger] Yeah, it probably is a bit confusing to have the main resource links be so prominent. They should probably still be there, but maybe there's a less conspicuous way to display them.
    • [Colorado] Perhaps a little CSS-fu can soften the effect.

13. One user said he would like to use this for a class, especially if it had a "review mode."

    • [Roger] One idea I've been thinking about is including a spaced repetition feature. It would automatically add cards corresponding to everything you've marked as learned. This would mostly be for simple things like definitions and statements of theorems. (At one point, I attempted to use spaced repetition to learn more complex things like proofs, but never found a good way to do it.)
    • [Colorado] I totally agree. I've been using Anki for various subjects since January. While not perfect, my rentention of various programming languages, math tricks, and machine learning concepts has certainly improved. This will take some finesse to address properly, so I vote for holding off on this idea for now. (PS) we discussed this idea previously, see the first entry under Goals and Ideas#ideas; it's probably a good sign that it keeps reemerging

14. One user mentioned that the clear/show learned buttons are in a confusing place, since traditionally, the header has site-wide navigation/operations.

15. One user didn't like that the back button didn't take them back to the previous view

16. Two users mentioned that they would like to use the arrow keys to navigate the explore view

17. One user, who had an awkward scroll wheel, wanted zoom-in/zoom-out functionality embedded into the explore view

18. there's a bug that 2 users encountered that causes the main application to "scroll down" and hide the header; I think this has to do with the application being bigger than the page itself -- I'll look into this

    • [Colorado] fixed

19. one user accidentally selected all titles in the explore view and was unable to unselect them by clicking on the background of the application (this has happened to me before) -- I'll see if I can fix this

20. each user immediently zoomed out once they encountered the explore view (perhaps we should start a bit more zoomed out)

    • [Roger] I second the part about starting more zoomed out.
    • [Colorado] I agree, setting a static zoom-level is trivial, but trying to e.g. include the entire graph will be a bit trickier