Refactor scraper to capture section-specific details

[ramonechen]

What?

This is a really big pull request; probably bigger than it should be.

Closes #17, closes #45, closes #34, closes #29.

This pull request introduces:

• Reorganization of the output JSON to display all class/section details instead of aggregating into courses.
  • CRNs
  • Section numbers
  • Section-specific titles, descriptions, credits, restrictions, faculty, etc.
• Addition of meeting block information to the output JSON.
  • Start and end hours
  • Meeting credit hours
  • Campus codes and descriptions
  • Building codes and descriptions
  • Building room
  • Meeting category (lecture, test, lab/recitation, etc.)
  • Start and end dates
  • Days of the week
• Adjustment of faculty data to display meetings that they are a part of as well as meetings that they are the primary instructor for.
  • Denoted by allMeetings and primaryMeetings lists filled with integer IDs that correspond to meeting blocks.
  • Each meeting block is assigned a local id that counts up from 1.
    • This was my solution to avoiding data duplication in the case that every meeting block has its own list of faculty.
• Addition of waitlist data to help complement enrollment data.
• New functionality to scrape courses that are normally hidden from SIS class search via discovery by crosslists.
• Improved logging and verbosity.
• Improved scraping/network performance.
  • The biggest change was using a single, shared TCPConnector for all client sessions instead of creating a new one for every session created.
    • This means that only one connection with the server is established, which is shared by individual sessions that have their own set of cookies.
    • This greatly cut down on the volume of TCP handshakes and connection teardowns, which previously wasted a considerable amount of time.
  • This change also imposes a global maximum number of simultaneous requests instead of a local maximum per client session.
    • This effectively makes the flow of requests to the SIS server much more constant, as the old architecture created a "thundering herd" problem in which one session could blast the server with many requests at once and cause it to throttle responses.
    • Client sessions for each term now fire requests in a more round-robin fashion, so all processing terms gradually approach 100% in sync.
    • As a side effect, scraper logs are now very quiet during the scraping process and all writing happens in a burst at the end of the process.

Why?

In a nutshell, more granular data good, aggregating section data into courses bad because it omits lots of important information about each section.

See linked issues for more details.

How?

New functions such as get_class_details(), get_class_enrollment(), and get_class_faculty_meetings() were added to sis_api.py to facilitate scraping of hidden classes.

A major refactor to the code in sis_scraper.py was needed to support the new section-level scraping and especially the hidden class scraping.

Testing?

The code has been tested incrementally and has shown to produce correct results several times in a row on all available terms at once. Other than that, trust me bro.

Anything else?

Have a nice day.