Generalize Google Searcher

[maxachis]

While #74 is focusing on automating google searches in the narrow domain of finding homepage urls, I think we can generalize this for something that is more broadly useful for our purposes, and allows us to make the most of automated google searches.

How This Works

Instead of having to create a specially-designed searcher for each subject we want to search, we maintain a queue of google searches which we can populate with all required queries and which are automatically run, 100 per day. As we identify more unique data needs, we can create all relevant variations of these queries and add them to the queue, whereupon they will be deployed and their results stored for later retrieval and organization.

This way, we can perform a wider array of data searches without having to expend considerable time modifying the repository and performing pull requests. All we would need to do is generate the queries and then place them in the queue.

This will work well for discrete searches that we intend to run on a one-time or few-time basis. For regular searches, an alternative design is required.

What the design involves

1. Create a search_queue table in the database. This contains a list of all search queries which are to be performed, and would include columns to enable cross-referencing with other tables in the database if desired. This would also provide a way to quickly identify if a search has been performed, helping to preserve state. This would obviate the agency_url_search_cache table. It would also allow us to review historical searches, whereas currently we don't have a clear record of what query produces what results.
2. Create a search_results table. All results related to a given search would be uploaded here and linked via foreign key to the search queue table, with columns for url and snippet (i.e. short descriptive test supplied by google as to the contents of the web page`. This would be done as an alternative to uploading them for hugging face for a few reasons:

• A: It enables more atomic uploads. The nature of a huggingface database is such that we're incentivized to batch results together into csv files, as having a csv file for each result quickly becomes difficult to organize. However, batching results poses a risk in that we may perform a large amount of searches, then something goes wrong and all those searches will have to be restarted. If each search result is uploaded one at a time, we only risk losing one search if an error occurs
• B: These results can be easily converted into a huggingface dataset at a later time.

3. Modify the Google Searcher such that, instead of specifically focusing on filling agency homepage urls, as it currently does, it pulls searches to perform from the search queue, runs them, and uploads the top 10 results to the search_results table.
4. Instead of hard-coding the search queries, as is currently done with the agency homepage searcher, queries are generated through a one-time script which are then uploaded to the search queue. This allows us to more easily inspect pending queries and to create the queries en-masse before they are sent.

[josh-chamberlain]

Thank you for putting thought into this! Really, we have one use case: searching for a specific kind of data about an agency. You could even consider homepage to just be a type of data. In both cases, the goal is to periodically search for either a homepage or a type of record

My suggestion is to create one table called something like automated_searches. It's basically every combination of agency and snippet (record type options + "homepage").

agency	snippet	last_searched	last_results
agency a	homepage	datetime	<machine readable>
agency a	traffic stops	datetime	<machine readable>
agency a	use of force reports	null	null
agency b	homepage	datetime	<machine readable>

The last row there would be one in the queue. this serves as our to-do list and our log. We could iterate over ones we'd already tried, eventually. We could also add a script to periodically create new queries based on updates to the agencies or record_types, and a status column for archiving searches which we want to stop re-trying.

This script would hit an API endpoint to fetch a paginated list of queries to make, then hit an endpoint to POST updates. Using the API rather than updating or reading the database directly is one of our generally established principles, and we should try to stick to that.

[maxachis]

In terms of the "snippet", we may want to ensure they're discrete categories (possibly a PostgreSQL enum) so as to avoid the error of minor deviations in grammar or capitalization leading to issues.

Additionally, since the different kinds of records we could receive might be quite variable (for example, per #69 , complaints and liability insurance info might end up becoming categories), how would we handle the fact that an agency may have a wide number of possible record types? As new record types are added, would we need to add/modify tables within the database when new categories of record types appear?

[maxachis]

Additionally, where would responses to the queries go? We’d previously discussed having 10 searches at a time for each google search; would we do that again? How do we handle the uncertainty of not knowing if a result or results for an agency-snippet combination are relevant?

[maxachis]

This has been done with the creation of #124 !