Filter function not working properly for Subpoena/Legal Order category #1893
Assignees
Labels
Comments
|
@stephaniesugars To flesh out this issue can you provide some examples of the ones that aren't showing up in search? |
|
Thanks! |
|
hi team! Circling back on this |
|
Priority: Critical |
|
I've done some looking into the situation and it looks like this search error is being caused by My guess is that: pages that were created after we migrated the subpoena/legal order metadata to use the new style did not automatically have the sort order applied. Pages that have been created using the Wagtail admin after the migration will have the correct values for the Fixing this is slightly tricky, I think, since the sort order field is (a) not visible in the admin, (b) not directly editable, and (c) only applied to existing legal order updates when something about that update is changed. I've had some success locally by changing the order and/or the date on the updates, saving a draft, then changing it back and publishing. This might work as a manual process for the affected pages, but I admit it is fiddly. It would be possible to write a command to do this, or manually do it with direct access to the database or management shell in the production environment, but this would take longer. |
|
Let's go forward with writing a script to do this. I leave it to @chigby's discretion whether we write it as a management command or a series of shell commands to coordinate with infra to run as an ad-hoc task. |
|
I've decided we ought to at least do some information gathering before I start writing a script that assumes anything. @enpaul Would it be possible to see the output of running this script on production (note: it will be a fair bit of plain text) from django.db.models import F, Q, OuterRef, Exists
from incident.models import LegalOrder, LegalOrderUpdate
def script():
unordered_updates = LegalOrderUpdate.objects.filter(
legal_order=OuterRef("pk"),
sort_order=None,
)
los = LegalOrder.objects.annotate(
has_unordered_updates=Exists(unordered_updates),
)
for lo in los:
if not lo.has_unordered_updates:
continue
print(f'Incident id: {lo.incident_page.pk}; {lo.incident_page.title}')
print(f' Order id: {lo.pk}; type: {lo.get_order_type_display()}; info requested: {lo.get_information_requested_display()}, status: {lo.get_status_display()}')
for update in lo.updates.order_by(F("date").asc(nulls_first=True)):
sort_order = '<null>' if update.sort_order is None else str(update.sort_order)
print(f' {sort_order:>6} {update.date} {update.get_status_display()}')
script() |
|
Which is to say, the output of this script when run inside a |
|
@chigby Is the output sensitive? Should I post it here or send it via slack/signal/some other mechanism? |
|
It's not sensitive, contains only database IDs and and public information. But you might as well send it to me on Slack rather than here. |
|
Also I hope it's ok, I just edited your script there to make it a bit more copy+paste friendly for the k8s exec terminal |
|
I've identified two incidents that I think need some manual review prior to making a bulk change to the rest of them. Spokesman-Review subpoenaed in defamation lawsuit, second subpoena: Current order on page:
Oberlin College subpoenas legal blogger for communications with source Current order on page:
The reason I'm singling these out is because of the "unknown date," the presence of which makes it harder to automatically determine the correct ordering. If these orders are not correct, they should be updated to be correct. If they are correct, then here's my workaround for saving that order to the database correctly. This
I've tested this procedure locally and it seems to work. I'm sorry this is fiddly, but I'm not sure wagtail allows editing this ordering data in a more direct way. Once these two incidents have been corrected, I can apply a script to the rest of them that will use the date field to infer the order and make the change automatically. |
|
hi @chigby, thanks for this - Oberlin has been updated (the metadata was incorrect), so no longer has an unknown date entry. I followed the steps above for Spokesman-Review. Let me know if anything else is needed! |
|
Great, thanks @sophieongithub -- I'm also now seeing it correctly coming up when searching for incidents whose legal orders have status "Carried out" (link to search), which is promising. I've written up a new script that should update the remaining incidents from django.db.models import F, Q, OuterRef, Exists
from incident.models import LegalOrder, LegalOrderUpdate
def update_orders():
unordered_updates = LegalOrderUpdate.objects.filter(
legal_order=OuterRef("pk"),
sort_order=None,
)
los = LegalOrder.objects.annotate(
has_unordered_updates=Exists(unordered_updates),
)
objects_to_update = []
for lo in los:
current_set = []
for i, update in enumerate(lo.updates.order_by(F("date").asc())):
if not update.date:
current_set = []
print(f'Skipping legal order {lo}')
break
print(f'{update}, sort_order={update.sort_order}, will be changed to {i}')
update.sort_order = i
current_set.append(update)
objects_to_update.extend(current_set)
LegalOrderUpdate.objects.bulk_update(objects_to_update, ['sort_order'])
update_orders()@enpaul -- this is the next script to run. I've tried to write it in your copy/paste friendly style, so let's hope that step is easier. I've tested this on staging and it seems to work, both to remove the null values and to preserve the ordering by date on the existing incidents. |
|
Sounds good @chigby ! To clarify, this script is making modifications to the database, yes? If so I'll just make sure I take a backup before running it. |
|
@enpaul Yes, it does make changes. It will also output the changes that it is making, which I think is an additional layer of help, on top of the backup. |
|
@chigby This is all set now. The script has been run and I have a pre-script backup if necessary. Let me know if I can help with anything else! Script Output |
|
I think this is looking really good! Thank you |
|
@stephaniesugars @sophieongithub We think this is completed—can one or both of you review the accuracy of search now and let us know if you agree? If so, we can close this. |
|
hi @harrislapiroff, I just tested filter searches for unknown, partially upheld, dropped, upheld, ignored, and carried out and they all seem to be working correctly - thank you! |
|
Closing as completed. Thanks all! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I noticed that there are inconsistencies with the filter search pulling accurately pulling cases where the latest status is "upheld," and it may be the case for other metadata options as well. When running the search, it says there are seven cases, but a manual review has turned up at least seven others. There doesn't appear to be any obvious factor as to why some are being caught and others are not (cross listing, multiple legal orders, numerous status updates, etc. appear in both those captured in the filter and those not).
The text was updated successfully, but these errors were encountered: