v6.5.0

6.5.0 - 2021-04-22

I can sum up this release of Quepid as either the Hey friend, come join me in improving search release or the so much technical debt has been paid down release.

For those of you upgrading your Docker based install, there are two new ENV variables you need to add to your docker-compose.yml file:

- RAILS_LOG_TO_STDOUT=true
- RAILS_SERVE_STATIC_FILES=true

Hey friend, come join me in improving search

We have added features that make it easier for you to invite your colleagues to come join your team and start rating documents. We
have reworked the initial case creation wizard to be smarter about popping up only the first time you decide to create your own case, instead of having it jump into the user flow when you join an existing team with existing cases. This should make it easier to bring folks interested only in rating docs into Quepid. As part of this, we've also added support for deleting individual Cases and User accounts. So if you invite someone, and change your mind, you can clean up after yourself.

so much technical debt has been paid down

We are finally off Rails 4.2 and are now on the latest Rails 5 release, and we have a shiny new favicon to go with it! We've updated all of our frontend dependencies to the latest versions possible, congruent with the core app still being based on Angular1. The lift from Rails 4 to Rails 5 was a massive effort, and a huge round of thanks goes out to everyone who reported bugs. We've shrunk our list of dependencies by 10% over the v6.4.1 release, and 29% compared to v6.0.1 release, and we're set up now to look at new front end technologies!

We also did some housecleaning by ripping out some features that hadn't seen adoption by users, including "soft delete" of queries and the "unit test" style of custom scorers.

Features

• Invite your friends to join your team on Quepid! You can now send an email with an invite code to join Quepid and your specific team. #259 by @epugh. #305 by @epugh introduced better support for tracking of if someone had gone through the create case wizard or not, extending this feature.

• Add support for sending emails via SMTP, or use Postmark, or don't send emails. #276 by @gabauer fixes #275.

• Let a user (or an Administrator) delete their account from Quepid, handling their cases, scorers, and team memberships. #315 by @epugh fixes #311.

• You can now Delete a case altogether! Historically we had an Archive function, so that you could restore an old case. However, if you are like me, you create lots and lots of throwaway cases, so this allows you to clean up your Quepid setup. This PR also fixed some data modeling issues, and the problem of sometimes have a Try of zero, instead of the default first try of One! We also always include the Try number even if you have named the Try, since that is a big deal. #288 by @epugh fixes #250. Thanks @DmitryKey for help QA'ing this code.

Improvements

• We now have a favicon for Quepid! #317 by @DmitryKey.

• Upgrade to Rails 5 and Ruby 2.7.2! We have been stuck on Rails 4.2 for years, and this unlocks a lot of new capabilities. #256 by @epugh with inspiration from @worleydl.

• Renamed "Export Case" and "Import Ratings" to just "Export" and "Import". #270 by @abhishekChoudhary93 fixes #195.

• Pay down some tech debt by removing our use of Coffee Script in Quepid. #283 by @epugh.

• Build time on CircleCI dropped from ~135 seconds down to ~105 seconds by removing sass gem in favour of sassc. #285 by @epugh.

• We have simplified our queries that involve cases and users that are part of teams by making the simplifying assumption that all owners of teams are also members, which turns out to be true in practice. We also have audited our use of .includes to load child objects (avoiding the N+1 problem), and used more .preload to try and deal with running out of temp space in our database. #286 by @epugh.

• Turns out we had a ERD diagram all along, but it was hidden. Now you can see it on our Data Mapping page, plus we have how to recreate it documented and integrated. #287 by @epugh.

• Remove obscure quepidIf.js file that doesn't seem to do anything. #293 by @worleydl.

• The Export All Cases feature only supported the old "Detail" format, and none of the other export formats, like LTR, Basic, or RRE. Plus the filtering options of "All, Owned, Shared" was pretty simplistic. It wasn't being used, and it added some complexity and performance issues to List All Cases page. So we removed it. #295 by @epugh fixes #294.

• Unit Test style custom scorers let you run a unit test that asserted specific things about specific docs at specific ranks in your search results. This logic however was always convoluted, and only 15 people since 2019 have used it, and I suspect by accident ;-) We want better ways of handling this type of function, so removing this to pay down some tech debt, simplify the database architecture, and open the door to new approach. #296 by @epugh fixes #290. #313 also dealt with this.

• We have removed the Soft Delete for Queries to simplify how Quepid works. If you delete a query in Quepid it is now fully deleted from the database! This is a nice bit of paying down tech debt. Huge thanks to @DmitryKey for testing this PR. #299 by @epugh fixes #298 by @DmitryKey.

• The /case url is one we frequently end up with, and it generates a 404. Thanks @DmitryKey for suggesting we have it route to the case listing page /cases instead. #304 by @epugh.

• The logic around when we popped open the "Create a Case" wizard for a brand new user was somewhat split between the backend home_controller.rb and the front end. It also made anyone who was invited to a team just for rating purposes go through the Create a Case Wizard on their first login, which was awkward. So, converted the concept of a "first_login" for the wizard to just a boolean "completed_case_wizard", and now it it checked if you have NO cases, and popped up, or if you click "Add a Case". #305 by @epugh fixes #281.

• Upgraded to the latest version of Angular 1, 1.8.2, and many of the other front end dependencies. #308 and #324 by @epugh and #320 by @worleydl deals with this.

• You can now override the from email address from the default [email protected] to your own email address. #322 by @slawmac. Thanks @slawmac for this improvement!

Bugs

• You can export a rating that has no actual rating value chosen! #266 by @epugh fixes #265.

• Fixing typo in defining aria-hidden html tag. #269 by @abhishekChoudhary93 fixes #268.

• Rails 5 Upgrade Hangover: fixed strong parameters validation issues, fixed wonky left outer joins on cases showing duplicate cases (and killed performance), eliminated the user_case_finder.rb class in favour of simpler scopes. #292 by @epugh w/ assist from @worleydl. #314 by @worleydl is related. #321 by @janhoy fixed by #323 by @epugh.

• Thanks to @DmitryKey for spotting that we were not tracking scorers for cases (bug introduced when we added the Show Only Ratings feature). #303 by @epugh and @worleydl fixes both #306 AND #289. A twofer!

• Thanks to @DmitryKey for spotting that we were not properly storing the Elasticsearch choice. #310 by @epugh fixes #309.

• There is a known issue where expanding/collapsing queries on a larger case, the UI can become completely locked in Chrome, forcing the user to close out the tab. Thanks to some next level debugging by @LiuCao0614 we have a workaround that disables the JQuery based drag and drop sorting feature that appears to avoid this bug. This is not a fix! This is a workaround! Set QUERY_LIST_SORTABLE=false in your environment to disable the sorting of queries feature. #272 tracks this ongoing issue.

6.4.1 - 2021-01-14

In the 6.4.0 release, the correct splainer-search NPM package was missed in the production Docker image.

This release resolves that oversight. Thanks @LiuCao0614 for testing the upgrade process and reporting the issue.

Improvements

• Small housekeeping update for the hosted app version...

v6.4.1

6.4.1 - 2021-01-14

In the 6.4.0 release, the correct splainer-search NPM package was missed in the production Docker image.

This release resolves that oversite. Thanks @LiuCao0614 for testing the upgrade process and reporting the issue.

Improvements

• Small housekeeping update for the hosted app version running on Heroku. #267 by @dworley.

• Clean up last scoring run details on the Cases Listing page. Now better iconography to call out cases that have queries that need rating. #261 by @epugh fixes #219.

Bugs

• Don't export ratings that don't have a rating set for RRE and Basic export formats. #266 by @epugh fixes #265.

v6.4.0

6.4.0 - 2020-12-18

Only Rated toggle is a powerful new feature added to Quepid, our first relevancy centric feature in a long while.

In certain relevancy cases, you may only care about the ordering of a subset of documents within the entire result set. One particular use case may be in e-commerce where you want certain products to be boosted and others to be demoted in the ranking algorithm. While this was doable in previous versions of Quepid, it could get difficult to manage the list of rated documents, especially if the list went many pages deep into the results.

To help solve this problem we worked with Kobler to introduce several new features to Quepid. We strived to make it easier to work with your corpus of rated documents in the main case view, inside of Explain Missing Documents and within the scorers themselves.

This feature was developed by @worleydl in #229, with an initial POC by @janhoy in #220, and resolves issue #163. Much thanks to @janhoy and the folks at Kobler for this feature.

Features

• Added "Show Only Rated" toggle to the main searchResult listing
• Explain Missing Documents modal now defaults to showing all of your rated documents when you first open it up
• Added eachRatedDoc function to ScorerFactory to iterate over rated documents in scoring
• Provided refreshRatedDocs(k) in ScorerFactory for loading a custom amount of rated documents up to k

Improvements

• Migrated scoring to an asynchronous pipeline
• Fixed stuck "Updating queries" message when creating a new case

v6.3.2

6.3.2 - 2020-12-08

Features

• Round trip export and import LTR ranking files! #204 by @epugh fixes #162 by @DmitryKey.

• Disable web signups by setting SIGNUP_ENABLED=false. #238 by @tonomonic.

Improvements

• Easy in-place editing of case name and the try name to encourage using those features! Microinteration FTW! #242 by @epugh.

• Demonstrate richness of queries in Quepid when you use the TMDB dataset. #236 by @epugh fixes #224.

• Update Javascript references to application/javascript. Pay down some tech debt! #223 by @epugh

• Simplify handling doc id's that have periods or slashes in then, and avoid base64 issues by passing that in the JSON payload. #233 by @epugh fixes #228 and #221.

• Some Solr collections need manual setup of the wt=json mimetype. Add better text message for users. #235 by @epugh fixes #178.

• The CSV export format has a CSV injection vulnerability that is now fixed. #245 by @nicholaskwan fixes #231.

• The Javascript based scorers could be used maliciously, so we've added an environment variable COMMUNAL_SCORERS_ONLY that controls if only Admins can create communal scorers for use by users. #246 by @jacobgraves fixes #232.

• Don't show the Sharing icon for communal scorers, since they are implicitly shared globally! #251 by @epugh fixes #247.

• Allow the export and import cycle of ratings using both CSV and RRE formats to include queries with no rated documents. Don't create a partial rating on import where we only have a doc id but no rating. #252 by @epugh fixes #244 by @LiuCao0614

• Make it easier for scrapers and automated test tools to log into Quepid by adding some id and class tags to the login forms. #257 by @epugh.

Bugs

• Discovered that the migrations from communal scorers being @5 to @10 didn't always run cleanly. Commits 94dd239, 8317b54, and 19b0464 to deal with this.

• Discovered that DELETE of ratings didn't work, and had to work around that. Commit 153047c.

• Missing authorization check for Team Owner. #230 by @jacobgraves fixes #230 by @testerTester0123456789.

• Can't rename a case on the Teams page. #240 by @epugh fixes #213

• Fixed scoring of AP@10 and NDCG@10 when you have fewer then 10 results. #253 by @nathancday fixes #225 by @epugh.

v6.3.1.2

6.3.1.2 - 2020-09-16

• Silly cut'n'paste error that should have been caught with more testing before the 6.3.1.1 release, not the day after. Fixed in commit 2e322b3 by @epugh.

v6.3.1

6.3.1 - 2020-09-14

Features

Improvements

• When we rolled out classic graded scorers (CG, DCT, NDCG) we used a 5 point scale, that allows you to pick a "no choice" middle option. However industry standard (for example RRE) is 4 point scale, so lets use 0,1,2,3 as our choices. New Best Practices document https://github.com/o19s/quepid/wiki/Judgement-Rating-Best-Practices by @binarymax is availble to help educate you. #206 by @epugh fixes #205.

• Closely related to the scale change, all the classic scorers used a @5 depth of scoring, however the other industry standard in rating is to score ten deep documents, @10, so #209 by @epugh makes this change as well.

• Encourage use of profile pics by adding help text in Profile page that they come from Gravatar.com. #202 by @epugh fixes #201.

• Use the same header navigation bar in both the main application and the Admin screens. #203 by @epugh refactors this logic.

• Added bin/docker c option to jump into the Rails Console during development, and cleaned up the other options. #200 by @epugh.

Bugs

• Running bin/setup_docker more than once always had hiccups on dropping MySQL, and needed a docker-compose -v, so just make that part of the script. #208 by @epugh fixes this.

• Making HTTP links clickable wasn't working in some cases. #211 by @e-budur fixes #183.

v6.3.0

6.3.0 - 2020-09-01

Features

• We want to export unrated query/doc pairs, which is only supported via exporting a previously created Snapshot. Add support for exporting a Snapshot to the Basic format on the Export screen. #191 by @epugh fixes #185.

Improvements

• When importing queries, you can have it clear all existing queries. This feature only worked for CSV
  files, not other formats like RRE. #193 by @epugh solves this.

• You can now import queries using the Import Ratings screen. Most of this was already supported, however now the modal dialog has better instructions to the user, and nicer validation of CSV formatting. #192 by @epugh solves this.

• When sharing Cases or Scorers, the lookup for what Teams you are part of created a really large JSON response (@epugh would get back a 1.4 MB payload!). We put this API response on a diet! #197 by @epugh solves this.

6.2.2

6.2.2 - 2020-07-09

Improvements

• When exporting for RRE, we need the ES or Solr index name. Extract this from the url for the most recent try and save a step! #167 by @epugh fixes #159.
• If you link to either a case that doesn't exist, or you don't have permission for, or a try that doesn't exist for a case, then provide messaging back in the UI! Let's share some Quepid Cases! 933ed25, 99ac27c, and 37b95b8 by @epugh to master branch fixes #158.
• getCaseByNo only used in tests. #173 by @epugh removes code.
• There was a partially working attempt at a result grid view instead of list view. It wasn't rendering in the UI, and we want to have a more general solution in the future, so removing the code to simplify Quepid. #174 by @epugh fixes #171.

Bugs

• When using the case wizard, you couldn't cut'n'paste in a long list of fields like overview_en, overview_idioms as they became a single tag. #166 by @epugh fixes #165.
• A : in the case name was converted to a when exporting a case. #169 by @epugh fixes #168.
• The implementation of autosaving your notes per query didn't work well. Going back to an explicit save button. #170 by @epugh fixes #164. Thanks @DmitryKey for the issue!
• Remove warning in server log on parameters. #182 by @epugh fixes #180.
• Community member spotted that doc id's with blah-http-blah triggered base64 encoding. Simplified handling docs with an id with a . or / in them. #179 by @worleydl and @epugh fixes #175.

6.2.1

6.2.1 - 2020-06-18

Thanks to some feedback from the community, we figured out that the SQL script for
migrating data from the DefaultScorer table to the Scorers table (and being tagged
as communal) didn't run reliably. We've pulled it out as db/release_6_2_0_merge_default_scorer_into_scorer.sql for folks to look at.

• #157 identified the issue and fixed by @worleydl.
• Commits a1fc942 and 029dd0c and 1a3c997 cover the migration fixes.

6.2.0

6.2.0 - 2020-06-11

This release of Quepid finally introduces the classical search geek scorers like NDCG, DCG, Average Precision, and friends by default. We've also fixed a large number of bugs introduced by changes in Firefox and Chrome browser security models that happened in the first part of 2020, as well as recent updates to Elasticsearch and Solr.

The burgeoning suite of open source tools for relevancy all require judgement lists. We revamped Quepid to handle exporting and importing from your favorite tools like Quaerite and RRE, as well as a Learning to Rank formatted export. You can see this in action in Chorus, an ecommerce search focused stack that includes Quepid.

Ease of development has long been an issue with Quepid. In this release we've worked on developer happiness by auditing all of the dependencies we include. We're removed quite a few Ruby gems and Node packages that weren't used, and upgraded the rest (replacing PhantomJS with Puppeteer for headless testing). yarn.lock dropped from 6706 lines to 1525 and Gemfile.lock from 448 to 330! This allowed us to finally move to ECMAScript 6 and Node version 10. Lastly we have cut the production Quepid Docker image from almost 3 GB down to 2.3 GB.

Upgrade Notes

Follow the steps outlined at https://github.com/o19s/quepid/wiki/Installation-Guide#updating-quepid.

You will need to run some database migrations to update the database and insert the new classical scorers (NDCG, AP, etc).

There is a sql migration script in ./db/release_6_2_0_merge_default_scorer_into_scorer.sql that you might want to look if you have been using Quepid locally for a while. This script is primarily meant to support updating http://app.quepid.com database for the 6.2.0 schema.

QUEPID_DEFAULT_SCORER is a new environment variable specifying the name of the default scorer
for users when they create a new case. QUEPID_DEFAULT_SCORER=AP@5 is what app.quepid.com uses.

Features

• Quepid Default Scorer Admin screen is buried in UI. Add it to the dropdown menu. #96 by @epugh.
• Support Basic Auth in Quepid with Elasticsearch. #114 and o19s/splainer-search#79 by @CGamesPlay fixes #109.
• When installing Quepid, use thor user:create to create an administrator! #112 by @epugh fixes #107.
• Enhanced export ratings function that follows the standard query,docid,rating supports full export/import lifecycle and easier integration with other tools. #128 by @epugh fixes #126.
• Export ratings in RRE and LTR file formats. Import ratings from RRE Judgement JSON file. #137 and #139 and #152 by @epugh fixes #133
• Classical scorers like AP, DCG, and NDCG are now shipping by default. All new users who sign up on Quepid will start with AP@5. This effort is still in progress, for example right now the scorers are named AP@5, DCG@5, NDCG@5 because they are hard coded to only look at the first five ranks. We also need to work on the Q Score Graph's vertical axis handling. Huge effort in #100 by @nathancday and @epugh to get this done. This change also removes the individual query level scorer being assignable, which was a bad idea (#132) and restores the ability to write a custom unit test for a query in Javascript (#120) which is an interesting idea! Thanks to @janhoy for being the impetuous to get this work done!

Improvements

• Community member reported the default demo TMDB Elasticsearch server on port 9200 had no data. Having seem random "drive by vandalism" of our demo Solr and ES servers that run on default ports, we want to move away. Yes, security by obscurity. #103 by @epugh changes default used in Wizard to port 8985 and 9206. #125 and #104 migrate existing URLs to the new ones.
• Increase Developer Happiness by speeding up the bin/setup_docker script by caching RubyGems. 50% speed up! #105 by @nathancday.
• Remove the concept of "communal" i.e shared with everyone scorers. This feature has been in Quepid for years, yet even @softwaredoug didn't remember it! #99 by @epugh fixes #98.
• In the beginning of Quepid, before there was a Search Relevancy community, there was just one, slightly janky scorer, that wasn't like any of the standard search geek scorers (looking at you NDCG) that we use today. We are building towards supporting many scorers, so it's time to remove the DEFAULTS definition of a scorer, and the related scope in ActiveRecord. #97 by @epugh.
• Solr 8.2 tightened up the security profile for accessing it that we depend on via JSONP to have Quepid work. Now the wizard provides you the command to run on your Solr if it can't connect. #95 by @epugh fixes #92.
• Using the default tmdb dataset? Demonstrate the thumb:poster_path feature. #94 by @epugh fixed #72.
• Rename user.username in database to user.email since that is what we use. Clean up API. #113 by @epugh fixes #111. You will need to run database migration for this release!
• Audited code base to prune dependencies and remove unused code. #121, #119, #118, #116 all worked towards this goal.
• Turns out Quepid has a Curate interface. It isn't supported, so in the interest of reducing our codebase, removing it. Points the way for the future however! #123.
• Deal with sameSite cookie warnings from Firefox. #131 by @epugh fixes #130.
• curator variables (i.e knobs and dials) that aren't used in query cause weird UI. #135 by @epugh fixes #64.
• Saving two annotations in a row doesn't work, you need to rescore per annotation. Fixed in #136 by @epugh.
• Users can opt out of community marketing emails. Add a /admin/users.csv export to make keeping track of that easier. Thanks @flaxsearch for the suggestion.
• Inconsistent use of X icon in modal popups is frustrating. #148 and #149 by @worleydl fixes #146 and #145.
• Swap the sorting of tries in the history tab to the newest first, going back in time. #151 by @epugh fixes #143 by @renekrie.
• Be clearer that MySQL is the only supported database platform. #156 by @epugh fixes #155 by @janhoy.

Bugs

• Wizard Autocomplete Didn't work well with Keyboard. Autocomplete suggestion had to be clicked with a Mouse. #94 by @epugh fixes this by upgrading package.
• Multivalued and nest JSON fields didn't display well, you would get [object Object] instead. Now we display arrays and Json properly. #117 by @CGamesPlay fixes #52.
• fixed highlighting throwing an error on Solr date fields by using hl.method=unified in Splainer-Search v2.5.9. #84 created by @janhoy.
• fixed fields with a . like foo.bar failing to be rendered in UI in Splainer-Search v2.5.9. #106 created by @rjurney.
• When a try number is not specified for a case, automatically redirect to the most recent try. #122 by @epugh fixes #110 by @binarymax.
• Case shared via Team with me has NaN for the Try. #134 by @epugh fixes #83.
• Not able to export cases from a Team page. Commit quepid@021bda8627463986f1ff04ffb1c290195c428c5b by @epugh fixed this.
• Race condition in laying out the main Quepid Screen and the Tune Relevance screen. #147 by @worleydl fixes #144 by @epugh.
• Using the "peek" function to page to deeper search results would lose your escapeQueries choice. o19s/splainer-search#87 by @jorgelbg fies this. Bump to splainer search 2.6.0 to include the fix.