Release SecureDrop Client 0.7.0

[sssoleileraaa]

Description

This release includes using the latest version of the SDK: freedomofpress/securedrop-builder#299 and is part of a SDW release set.

This issue tracks the SecureDrop Client release 0.7.0. It will be organized by:

• Release Manager: @creviera
• Deputy Release Manager: @rocodes

This release includes the following changes:

• [high level summary of changes]

SecureDrop maintainers and testers: As you QA this release, please report back your testing results as comments on this ticket. File GitHub issues for any problems found, tag them "QA: Release", and associate them with the release milestone for tracking (or ask a maintainer to do so).

Updated Test plan

[Make sure to include the version(s) of the server that need to be tested against. If the release is being coordinated with a server release, specify rc versions of the server that need to be tested and release order. Once completed, insert or link to a test plan here. It can be left out until then.]

[SDK Test Plan] Ensure support for common date string formats (#172)

• You are able to log into the client

Add journalist read receipts (#1417)

• Confirm that when logged in as a user (i.e journalist), the tool-tip of the messages and replies on hover displays the authenticated user's username as the default value.
• Confirm that when logged in as a user (i.e journalist), the tool-tip of the messages and replies on hover displays the authenticated username last if the messages and replies have been read by users other than them.
• Confirm that the username list the tool-tip is initialized to gets updated in a timely manner, in cases where the conversation is read by a new user.
• Confirm that the check-mark is hidden in case the journalist reply fails to deliver.
• Delete a user while logged into the client as a different user and ensure checkmarks are updated as expected

Speed up deletion of local files and database records (#1432, #1458)

• Setup: ensure your instance has a number (> 5) of sources with files, messages, and replies.
• Ensure you can log into your staging server for the first two tests

Migration test

• Database update works and local deletion works

Stress test local source deletion

• Choose a source in the client with file submissions and download at least one file.
• Inspect Sources table and note the source's id (not UUID).
• In your staging server, run the loaddata.py script and add > 10 sources.
• Wait a moment for the client to start syncing
• Delete your target source
• Source disappears from the conversation view once deletion animation completes
• Inspect the ~/.securedrop_client/data/ directory; source's subdirectory is removed and all files are deleted
• Query the sources, messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present

Stress test local conversation deletion

• Choose a source with file submissions in the client and download at least one file submission.
• In your staging server, run the loaddata.py script and add > 10 sources.
• Wait a moment for the client to start syncing
• "Safe-delete" (delete files and messages) for the source
• Source's files and messages disappear in the GUI
• Inspect the ~/.securedrop_client/data/ directory; source's subdirectories should be removed and all relevant files deleted.
• Inspect the sources table and note the source's id (not UUID). Query the messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present.

(Added) No ghost replies

• Safe delete a conversation (delete files and messages), then send reply A, then safe delete again.
• Send reply B
• Send reply C
• Ensure reply A is not temporarily visible in the GUI
• Ensure replies B and C are in correct order, and the deleted 'tear' line and "previous conversation deleted" message appears above them

Regression testing: remote deletion

• Open the client and observe 2 sources in client conversation view. Download files and messages for both.
• Via the Journalist Interface/Tor Browser, delete files and messages for one source, delete the other source entirely
• Wait for sync
• Deleted source is not present in the client conversation view and source files are removed from data directory.
• Files and messages are removed for the other source, including from the ~/.securedrop_client/data/ directory.

Test local deletion with network failures

• Log into the client and select a source with replies, messages, and files. Download some files and observe them in a subdirectory of ~/.securedrop_client/data/
• Open the Qubes Settings -> network settings for sd-whonix, or open a terminal in sd-whonix.
• Safe-delete ("delete files and messages") for the source, then as soon as the UI confirms deletion, quickly kill the network connection by setting the nevtm to none or issuing sudo service tor stop in the sd-whonix terminal.
• Inspect the ~/.securedrop_client/data/ directory; the source's subdirectory should be removed and all relevant files deleted.
• Inspect the sources table and note the source's id and UUID. Query the messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present.
• Inspect the deletedconversation table and ensure that a row with the source uuid is present.
• Re-connect the network and wait for sync.
• Again, query the messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present.
• Ensure the entry with source UUID has been removed from the deletedconversation table.

Add feature to download all files from a single source (#1388)

• Start the server, start the client, select a source.
• Confirm that the conversation and its files are shown as usual.
• Confirm that the "conversation menu" (a.k.a "source menu", a.k.a "kebab menu") contains a sub-section "Download" and an action "All Files".
• Confirm that the "All Files" action is enabled for conversations that have at least one file that hasn't been downloaded before.
• Confirm that the files can be downloaded individually (by activating their "Download" buttons) as usual.
• Confirm that downloading all the files of a conversation one by one (by activating their "Download" buttons) causes the "All Files" action to be disabled.
• Confirm that activating the "All Files" menu action causes all the not-already-downloaded files of the conversation to start downloading.
• Confirm that when downloading all files, the oldest are downloaded first.
• Confirm that once all files are downloaded, the "All Files" action is disabled.
• Stop the client.
• Start the client.
• Confirm that the "All Files" action is disabled for conversations that have no files to download, and enabled for all the other. (Independently of the fact that the files were or not downloaded individually or via "All Files").
• Confirm that everything in the steps above is still true.
• Stop the server (or somehow disconnect the client, this could be done emulating a network issue)
• Wait until the client displays the banner that indicates that the connection to the server was lost.
• Upload new files for existing sources. (Ideally upload file to some sources that already have files to download and other that don't have any yet, so that you can confirm later that the action was enabled as expected.)
• Create new sources, upload files from some sources, and leave other conversations without files
• Start the server (or somehow fix the network issue)
• Confirm that the banner is cleared as usual when the client is automatically sync'd
• Confirm that the new sources are loaded
• repeat download-conversation checks above
• Start downloading "All Files"
• Disconnect the server while the files are downloading (ideally, some were downloaded, others not yet)
• Confirm that the client behaves as usual with respect of failed download tasks.
• Stop the client
• Start the client, do not select any source.
• Confirm that pressing Ctlr+D ("the shorcut") does not trigger any download (even when all conversation have files that could be downloaded)
• Select a source
• Confirm that pressing "the shortcut" causes all files to be downloaded for the conversation that's visible.
• Confirm that the "All Files" action in the conversation menu is disabled once all files were downloaded using the shortcut.
• Select a different source
• Confirm that the shortcut only causes files to be downloaded for the currently selected conversation

Release tasks

• Update changelog
• Create test plan
• Begin formal QA using release candidate(s) (have folks apt install directly into sd-app's template)
• Build production package in standard build environment
• Sign production package
• Perform final pre-flight testing using apt-qa.freedom.press
  • Localization: In a dispVM, change your locale (e.g.: export LANG=es_ES.utf-8; dpkg-reconfigure locales), run the Client, and confirm that the application is translated.
• Publish production package
• Publicize release via support channels

[eloquence]

[SDK Test Plan] Ensure support for common date string formats (#172)

• You are able to log into the client

Add journalist read receipts (#1417)

• Confirm that when logged in as a user (i.e journalist), the tool-tip of the messages and replies on hover displays the authenticated user's username as the default value.
• Confirm that when logged in as a user (i.e journalist), the tool-tip of the messages and replies on hover displays the authenticated username last if the messages and replies have been read by users other than them.
• Confirm that the username list the tool-tip is initialized to gets updated in a timely manner, in cases where the conversation is read by a new user.
• Confirm that the check-mark is hidden in case the journalist reply fails to deliver.
• Delete a user while logged into the client as a different user and ensure checkmarks are updated as expected

[sssoleileraaa]

#1468 might be relevant to QA (and could be pulled into rc2 if repro'd but ro and I are still trying to repro ourselves).

[rocodes]

Here's a proposed update for the Test Plan for deletion changes, to make it a little more streamlined for QAers and also to add some stress testing. if @creviera you like this I'll update the issue. [edit: original issue updated]

New deletion test plan below

Speed up deletion of local files and database records (#1432, #1458)

• Setup: ensure your instance has a number (> 5) of sources with files, messages, and replies.
• Ensure you can log into your staging server for the first two tests

Stress test local source deletion

• Choose a source in the client with file submissions and download at least one file.
• Inspect Sources table and note the source's id (not UUID).
• In your staging server, run the loaddata.py script and add > 10 sources.
• Wait a moment for the client to start syncing
• Delete your target source
• Source disappears from the conversation view once deletion animation completes
• Inspect the ~/.securedrop_client/data/ directory; source's subdirectory is removed and all files are deleted
• Query the sources, messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present

Stress test local conversation deletion

• Choose a source with file submissions in the client and download at least one file submission.
• In your staging server, run the loaddata.py script and add > 10 sources.
• Wait a moment for the client to start syncing
• "Safe-delete" (delete files and messages) for the source
• Source's files and messages disappear in the GUI
• Inspect the ~/.securedrop_client/data/ directory; source's subdirectories should be removed and all relevant files deleted.
• Inspect the sources table and note the source's id (not UUID). Query the messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present.

(Added) No ghost replies

• Safe delete a conversation (delete files and messages), then send reply A, then safe delete again.
• Send reply B
• Send reply C
• Ensure reply A is not temporarily visible in the GUI
• Ensure replies B and C are in correct order, and the deleted 'tear' line and "previous conversation deleted" message appears above them

Regression testing: remote deletion

• Open the client and observe 2 sources in client conversation view. Download files and messages for both.
• Via the Journalist Interface/Tor Browser, delete files and messages for one source, delete the other source entirely
• Wait for sync
• Deleted source is not present in the client conversation view and source files are removed from data directory.
• Files and messages are removed for the other source, including from the ~/.securedrop_client/data/ directory.

Test local deletion with network failures

• Log into the client and select a source with replies, messages, and files. Download some files and observe them in a subdirectory of ~/.securedrop_client/data/
• Open the Qubes Settings -> network settings for sd-whonix, or open a terminal in sd-whonix.
• Safe-delete ("delete files and messages") for the source, then as soon as the UI confirms deletion, quickly kill the network connection by setting the nevtm to none or issuing sudo service tor stop in the sd-whonix terminal.
• Inspect the ~/.securedrop_client/data/ directory; the source's subdirectory should be removed and all relevant files deleted.
• Inspect the sources table and note the source's id and UUID. Query the messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present.
• Inspect the deletedconversation table and ensure that a row with the source uuid is present.
• Re-connect the network and wait for sync.
• Again, query the messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present.
• Ensure the entry with source UUID has been removed from the deletedconversation table.

[sssoleileraaa]

@rocodes your changes look good to me, thanks for the update. One question though: shouldn't the following check be the first thing in the plan?

Test upgrade of existing client

• Ensure database upgrade works and local deletion works

[eloquence]

• Workstation: Qubes 4.0.4 staging
• Server: SecureDrop 2.3.1 prod on hardware

Note: I'm following @rocodes' test plan below, but I am not using the loaddata.py script, as I am testing against a long-running prod server and prefer to avoid running development tooling against it, to preserve prod-like state. So if someone else wants to focus on the stress testing portion, that would be helpful. Prior to deletions, my server had about 20 sources.

Speed up deletion of local files and database records (#1432, #1458)

• Setup: ensure your instance has a number (> 5) of sources with files, messages, and replies.
• Ensure you can log into your staging server for the first two tests

Stress test local source deletion

• Choose a source in the client with file submissions and download at least one file.
• Inspect Sources table and note the source's id (not UUID).
• In your staging server, run the loaddata.py script and add > 10 sources.
• Wait a moment for the client to start syncing
• Delete your target source
• Source disappears from the conversation view once deletion animation completes
• Inspect the ~/.securedrop_client/data/ directory; source's subdirectory is removed and all files are deleted
• Query the sources, messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present

Stress test local conversation deletion

• Choose a source with file submissions in the client and download at least one file submission.
• In your staging server, run the loaddata.py script and add > 10 sources.
• Wait a moment for the client to start syncing
• "Safe-delete" (delete files and messages) for the source
• Source's files and messages disappear in the GUI
• Inspect the ~/.securedrop_client/data/ directory; source's subdirectories should be removed and all relevant files deleted.
• Inspect the sources table and note the source's id (not UUID). Query the messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present.

Ghost issue with deleted sources

❌ However, during entire source deletion, I'm observing the "ghost" problem with deleted sources re-appearing briefly on the next sync, then being cleaned up the following sync. I tried monitoring the deletedsource table while a source deletion operation is underway and I did not observe any UUIDs getting added (I did observe them getting added to the deleteconversation table for the "conversation deletion" path). In a quick skim of #1458 I'm not seeing any code to insert rows to the deletedsource table, but it's possible I'm missing something.

[cfm]

Workstation: Qubes 4.0.4
Client: 0.7.0-rc1-dev-20220413-060636+buster
Server: 2.3.1 staging

I believe the test plan passes as written, but I've flagged some questions below. Per @eloquence in #1465 (comment), I'll plan to run through the stress-testing tomorrow.

[SDK Test Plan] Ensure support for common date string formats (#172)

• You are able to log into the client

Add journalist read receipts (#1417)

• Confirm that when logged in as a user (i.e journalist), the tool-tip of the messages and replies on hover displays the authenticated user's username as the default value.
• Confirm that when logged in as a user (i.e journalist), the tool-tip of the messages and replies on hover displays the authenticated username last if the messages and replies have been read by users other than them.
• Confirm that the username list the tool-tip is initialized to gets updated in a timely manner, in cases where the conversation is read by a new user.
• Confirm that the check-mark is hidden in case the journalist reply fails to deliver.

👉 Yes, but: checkmarks are displayed (with a tooltip listing the sending user) while the reply is still pending. Is this the expected behavior? It surprises me to be able to have "seen" my own reply before it's sent.

• Delete a user while logged into the client as a different user and ensure checkmarks are updated as expected

Speed up deletion of local files and database records (#1432, #1458)

• Setup: ensure your instance has a number (> 5) of sources with files, messages, and replies. For at least one source, save their codename.

Migration

• test migration path with existing db

Test local deletion: happy path

• Choose a source in the client and download files/attachments for them.
• Inspect Sources table and note their id
• Select "Delete Source"
• Source is deleted from database and source documents are deleted from relevant data directory
• Source cannot log in from Tor browser

Test local deletion right after conversation deletion

• Choose a source in the client and download files/attachments for them.
• Inspect Sources table and note their id
• Select "Delete files and messages"
• Quickly select "Delete source" (inside same sync period)
• Observe source is deleted from database and source documents are deleted

Test local deletion: network failure

• Choose a source in the client and download files/attachments for them.
• Inspect Sources table and note their id
• Select "Delete Source", then once GUI animation completes, quickly kill network connection (sudo service tor stop in sd-whonix, or set netvm to None)
• Observe source record is deleted from database (query via ID) and source documents are deleted

Regression testing: remote deletion

• Open the client and observe a source in client conversation view. Download files and messages for the source.
• Delete the source via Journalist Interface/Tor Browser
• Wait for sync
• Source is not present in the client conversation view and source files are removed from data directory.

Test local deletion: happy path

• Log into the client and select a source with replies, messages, and files. Download some files and observe them in a subdirectory of ~/.securedrop_client/data/
• Safe-delete ("delete files and messages") for the source.
• Inspect the ~/.securedrop_client/data/ directory; the source's subdirectory should be removed and all relevant files deleted.
• Inspect the sources table and note the source's id (not UUID). Query the messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present.

Test local deletion with network failures

• Log into the client and select a source with replies, messages, and files. Download some files and observe them in a subdirectory of ~/.securedrop_client/data/
• Open the Qubes Settings -> network settings for sd-whonix, or open a terminal in sd-whonix.
• Safe-delete ("delete files and messages") for the source, then as soon as the UI confirms deletion, quickly kill the network connection by setting the nevtm to none or issuing sudo service tor stop in the sd-whonix terminal.
• Inspect the ~/.securedrop_client/data/ directory; the source's subdirectory should be removed and all relevant files deleted.
• Inspect the sources table and note the source's id and UUID. Query the messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present.
• Inspect the deletedconversation table and ensure that a row with the source uuid is present.
• Re-connect the network and wait for sync.
• Again, query the messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present.
• Ensure the entry with source UUID has been removed from the deletedconversation table.

Ensure no regression when deletion is done on the server

• Download files for a source with files and messages.
• Delete the source's files and messages via the Journalist Interface.
• Wait for sync.
• Ensure files and messages have been deleted from the client as in the steps above.

👉 Yes, but compare:

Contents of ~/.securedrop_client/data after upload of application.ini as source designated mundane_probability:

user@sd-app:~/.securedrop_client/data$ ls -alR
.:
total 12
drwx------ 3 user user 4096 Apr 13 17:30 .
drwx------ 5 user user 4096 Apr 13 17:32 ..
drwx------ 2 user user 4096 Apr 13 17:32 mundane_probability

./mundane_probability:
total 8
drwx------ 2 user user 4096 Apr 13 17:32 .
drwx------ 3 user user 4096 Apr 13 17:30 ..
user@sd-app:~/.securedrop_client/data$ ls -alR
.:
total 12
drwx------ 3 user user 4096 Apr 13 17:30 .
drwx------ 5 user user 4096 Apr 13 17:34 ..
drwx------ 3 user user 4096 Apr 13 17:34 mundane_probability

./mundane_probability:
total 12
drwx------ 3 user user 4096 Apr 13 17:34 .
drwx------ 3 user user 4096 Apr 13 17:30 ..
drwx------ 2 user user 4096 Apr 13 17:34 4-mundane_probability-doc

./mundane_probability/4-mundane_probability-doc:
total 12
drwx------ 2 user user 4096 Apr 13 17:34 .
drwx------ 3 user user 4096 Apr 13 17:34 ..
-rw------- 1 user user  549 Apr 13 17:34 application.ini

Contents of ~/.securedrop_client/data after Delete...Files and Messages from Journalist Interface:

user@sd-app:~/.securedrop_client/data$ ls -alR
.:
total 12
drwx------ 3 user user 4096 Apr 13 17:30 .
drwx------ 5 user user 4096 Apr 13 17:34 ..
drwx------ 2 user user 4096 Apr 13 17:34 mundane_probability

./mundane_probability:
total 8
drwx------ 2 user user 4096 Apr 13 17:34 .
drwx------ 3 user user 4096 Apr 13 17:30 ..

Is this the expected behavior?

Test upgrade of existing client

• Ensure database upgrade works and local deletion works as above

(Added) No ghost replies

• Safe delete a conversation, then send reply A, then safe delete again.
• Send reply B
• Ensure reply A is not temporarily visible in the GUI (note: this is a GUI bug; it's not added to the database)

Add feature to download all files from a single source (#1388)

• Start the server, start the client, select a source.
• Confirm that the conversation and its files are shown as usual.
• Confirm that the "conversation menu" (a.k.a "source menu", a.k.a "kebab menu") contains a sub-section "Download" and an action "All Files".
• Confirm that the "All Files" action is enabled for conversations that have at least one file that hasn't been downloaded before.
• Confirm that the files can be downloaded individually (by activating their "Download" buttons) as usual.
• Confirm that downloading all the files of a conversation one by one (by activating their "Download" buttons) causes the "All Files" action to be disabled.
• Confirm that activating the "All Files" menu action causes all the not-already-downloaded files of the conversation to start downloading.
• Confirm that when downloading all files, the oldest are downloaded first.
• Confirm that once all files are downloaded, the "All Files" action is disabled.
• Stop the client.
• Start the client.
• Confirm that the "All Files" action is disabled for conversations that have no files to download, and enabled for all the other. (Independently of the fact that the files were or not downloaded individually or via "All Files").
• Confirm that everything in the steps above is still true.
• Stop the server (or somehow disconnect the client, this could be done emulating a network issue)

[cfm@dom0 ~]$ qvm-prefs sd-proxy netvm ''

• Wait until the client displays the banner that indicates that the connection to the server was lost.
• Upload new files for existing sources. (Ideally upload file to some sources that already have files to download and other that don't have any yet, so that you can confirm later that the action was enabled as expected.)
• Create new sources, upload files from some sources, and leave other conversations without files
• Start the server (or somehow fix the network issue)

[cfm@dom0 ~]$ qvm-prefs sd-proxy netvm sd-whonix

• Confirm that the banner is cleared as usual when the client is automatically sync'd
• Confirm that the new sources are loaded
• repeat download-conversation checks above
• Start downloading "All Files"
• Disconnect the server while the files are downloading (ideally, some were downloaded, others not yet)
• Confirm that the client behaves as usual with respect of failed download tasks.
• Stop the client
• Start the client, do not select any source.
• Confirm that pressing Ctlr+D ("the shorcut") does not trigger any download (even when all conversation have files that could be downloaded)
• Select a source
• Confirm that pressing "the shortcut" causes all files to be downloaded for the conversation that's visible.
• Confirm that the "All Files" action in the conversation menu is disabled once all files were downloaded using the shortcut.
• Select a different source
• Confirm that the shortcut only causes files to be downloaded for the currently selected conversation

[rocodes]

@creviera Yes - I kind of figured that all the QAers were updating from an existing setup and that the above tests would encompass the basic 'happy path' test, but yes we can add and make that explicit

[rocodes]

QAers: thank you for your detailed reports!

@eloquence: Yikes, you're right about the DeletedSource; it looks like that change was clobbered and we didn't notice. Quick fix incoming.

@cfm: You're right, the local deletion of files and messages removes the parent directory, whereas the server deletion does not (because it's simply updating the current messages, whether that means adding or removing). While neither case is 'wrong' (the directory will be re-created as soon as more files are submitted/downloaded, and since the source still remains in the source list, leaving the parent directory is not divulging any information in a dangerous way), it is an inconsistency, and I think we should fix it.

Regarding seen-by behaviour, I agree that it's a little confusing to see the double checkmarks before a message is sent, and I think that's a gap in the way we specified the feature. I will investigate a little further.

[rocodes]

@cfm I might tap you for review of a couple very small PRs today since Allie is out, if that's ok with you.

[eloquence]

• Workstation: Qubes 4.0.4 / 0.6.0 RPM staging
• Client 0.7.0-rc1-dev-20220414-224612
• Server: 2.3.1 (prod/hardware)

Stress test local source deletion

• Choose a source in the client with file submissions and download at least one file.
• Inspect Sources table and note the source's id (not UUID).
• In your staging server, run the loaddata.py script and add > 10 sources.
• Wait a moment for the client to start syncing
• Delete your target source
• Source disappears from the conversation view once deletion animation completes
• Inspect the ~/.securedrop_client/data/ directory; source's subdirectory is removed and all files are deleted
• Query the sources, messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present

(Added) No ghost replies

• Safe delete a conversation (delete files and messages), then send reply A, then safe delete again.
• Send reply B
• Send reply C
• Ensure reply A is not temporarily visible in the GUI
• Ensure replies B and C are in correct order, and the deleted 'tear' line and "previous conversation deleted" message appears above them

Regression testing: remote deletion

• Open the client and observe 2 sources in client conversation view. Download files and messages for both.
• Via the Journalist Interface/Tor Browser, delete files and messages for one source, delete the other source entirely
• Wait for sync
• Deleted source is not present in the client conversation view and source files are removed from data directory.
• Files and messages are removed for the other source, including from the ~/.securedrop_client/data/ directory.

No more ghosts on repeated deletions with the changes in #1473 present! 🎉

[cfm]

Workstation: Qubes 4.0.4
Client: 0.7.0-rc1-dev-20220414-224612+buster
Server: 2.3.1 staging

Retesting updated section of test plan:

Speed up deletion of local files and database records (#1432, #1458)

• Setup: ensure your instance has a number (> 5) of sources with files, messages, and replies.
• Ensure you can log into your staging server for the first two tests

Migration test

• Database update works and local deletion works

Stress test local source deletion

• Choose a source in the client with file submissions and download at least one file.
• Inspect Sources table and note the source's id (not UUID).
• In your staging server, run the loaddata.py script and add > 10 sources.
• Wait a moment for the client to start syncing
• Delete your target source
• Source disappears from the conversation view once deletion animation completes
• Inspect the ~/.securedrop_client/data/ directory; source's subdirectory is removed and all files are deleted
• Query the sources, messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present

Stress test local conversation deletion

• Choose a source with file submissions in the client and download at least one file submission.
• In your staging server, run the loaddata.py script and add > 10 sources.
• Wait a moment for the client to start syncing
• "Safe-delete" (delete files and messages) for the source
• Source's files and messages disappear in the GUI
• Inspect the ~/.securedrop_client/data/ directory; source's subdirectories should be removed and all relevant files deleted.
• Inspect the sources table and note the source's id (not UUID). Query the messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present.

(Added) No ghost replies

• Safe delete a conversation (delete files and messages), then send reply A, then safe delete again.
• Send reply B
• Send reply C
• Ensure reply A is not temporarily visible in the GUI
• Ensure replies B and C are in correct order, and the deleted 'tear' line and "previous conversation deleted" message appears above them

Regression testing: remote deletion

• Open the client and observe 2 sources in client conversation view. Download files and messages for both.
• Via the Journalist Interface/Tor Browser, delete files and messages for one source, delete the other source entirely
• Wait for sync
• Deleted source is not present in the client conversation view and source files are removed from data directory.
• Files and messages are removed for the other source, including from the ~/.securedrop_client/data/ directory.

Test local deletion with network failures

• Log into the client and select a source with replies, messages, and files. Download some files and observe them in a subdirectory of ~/.securedrop_client/data/
• Open the Qubes Settings -> network settings for sd-whonix, or open a terminal in sd-whonix.
• Safe-delete ("delete files and messages") for the source, then as soon as the UI confirms deletion, quickly kill the network connection by setting the nevtm to none or issuing sudo service tor stop in the sd-whonix terminal.
• Inspect the ~/.securedrop_client/data/ directory; the source's subdirectory should be removed and all relevant files deleted.
• Inspect the sources table and note the source's id and UUID. Query the messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present.
• Inspect the deletedconversation table and ensure that a row with the source uuid is present.
• Re-connect the network and wait for sync.
• Again, query the messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present.
• Ensure the entry with source UUID has been removed from the deletedconversation table.

[eloquence]

• Workstation: Qubes 4.0.4 / 0.6.0 RPM staging
• Client 0.7.0-rc1-dev-20220414-224612
• Server: 2.3.1 (prod/hardware)

Test local deletion with network failures

• Log into the client and select a source with replies, messages, and files. Download some files and observe them in a subdirectory of ~/.securedrop_client/data/
• Open the Qubes Settings -> network settings for sd-whonix, or open a terminal in sd-whonix.
• Safe-delete ("delete files and messages") for the source, then as soon as the UI confirms deletion, quickly kill the network connection by setting the nevtm to none or issuing sudo service tor stop in the sd-whonix terminal.
• Inspect the ~/.securedrop_client/data/ directory; the source's subdirectory should be removed and all relevant files deleted.
• [ x Inspect the sources table and note the source's id and UUID. Query the messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present.
• Inspect the deletedconversation table and ensure that a row with the source uuid is present.
• Re-connect the network and wait for sync.
• Again, query the messages, replies, draftreplies, and files tables for rows with the source id and ensure that no matching rows are present.
• Ensure the entry with source UUID has been removed from the deletedconversation table.

Add feature to download all files from a single source (#1388)

• Start the server, start the client, select a source.
• Confirm that the conversation and its files are shown as usual.
• Confirm that the "conversation menu" (a.k.a "source menu", a.k.a "kebab menu") contains a sub-section "Download" and an action "All Files".
• Confirm that the "All Files" action is enabled for conversations that have at least one file that hasn't been downloaded before.
• Confirm that the files can be downloaded individually (by activating their "Download" buttons) as usual.
• ❌ Confirm that downloading all the files of a conversation one by one (by activating their "Download" buttons) causes the "All Files" action to be disabled. - Some slightly broken behavior, see below.
• Confirm that activating the "All Files" menu action causes all the not-already-downloaded files of the conversation to start downloading.
• Confirm that when downloading all files, the oldest are downloaded first.
• Confirm that once all files are downloaded, the "All Files" action is disabled.
• Stop the client.
• Start the client.
• Confirm that the "All Files" action is disabled for conversations that have no files to download, and enabled for all the other. (Independently of the fact that the files were or not downloaded individually or via "All Files").
• Confirm that everything in the steps above is still true.

Error with repeated "Download all" triggers while files are downloading.

❗ The "Download all" feature was pretty resilient to my repeated attempts to break it. I observed that the feature stayed enabled while any file was still in "Downloading" state, so I tried clicking and hotkeying it repeatedly while files were already downloading, in the manner of a somewhat impatient user. :) In that scenario, I was once able to produce the following issue:

• The "Download all" feature never transitioned to the "disabled" state, even after all files were downloaded
• An error "There was a problem downloading the file" (not the generic network error) appeared after a while, even though all files were downloaded.

[rocodes]

@eloquence Thank you for your thorough testing! I don't have much familiarity with the Download All code, but I wonder if this is related to our lack of a 'pause/stop download' feature.

We should revisit this on Monday and decide if it's worth trying to pull in any changes to this feature. (I'm going to post a broader QA recap in a moment)

[eloquence]

Thanks Ro! I tried reproducing with a freshly initialized client and many downloads, and was not able to. With the debug log enabled, I can see that the duplicate jobs are detected and rejected. So this may have been a more sporadic edge case with some other timing issues in play.

[rocodes]

QA status and todos for Monday
Thanks to awesome QAers, things are looking promising and all of the findings so far have been pretty small fixes.

• Merged: Add locally-deleted Source to DeletedSource table #1473

Approved, ready for merging and inclusion in rc2:

• always show selected source widget as read #1464
• Hide seen-by checkmark for pending messages #1471 (to be included if Allie agrees)

[update] Ready for review:

• guard against a reply being sent to a source that was just deleted #1468: would like to include
• When all submissions are deleted for a source, delete the enclosing directory #1475 (Lowest priority, discuss Monday whether or not to include)

[update] Bug, but not a regression, suggest deferring fix til after the release:

• GUI freezes after sync of many (~200) sources #1476

[sssoleileraaa]

RC2 Test Plan

Make sure to include the version(s) of the server that you are testing against.

No double-check mark for a pending reply (#1470)

• Confirm that the check-mark is hidden while the reply is pending (visually, in a semi-transparent state).
• Confirm that the check-mark is hidden in case the journalist reply fails to deliver.

Selected source remains read in the source list after new message or reply comes in (#1464)

• Confirm no repro of Existing source with unread messages still shows up as unread after the source is selected #1463 (comment)

Selected source remains read in the source list after deleting conversation (#1464)

• Confirm no repro of Existing source with unread messages still shows up as unread after the source is selected #1463 (comment)

Client stress test during source deletion with fresh replies (#1468), No 'ghost sources' (#1473)

• In your staging server, run the loaddata.py script and add about 20 sources.
• Continuously reply to sources and then delete them for 10 or more sources (before, during, and after syncing)
• Confirm the client does not crash
• Confirm sources never reappear in the source list

Global mouse selection cleared after login (#1477)

• Select some text in your terminal
• Now click the middle button on your touchpad or mouse and confirm that it pastes the selected text (so you know it's on the global clipboard)
• Start the client and log in (we don't clear the clipboard until after the login screen to support copy-pasting diceware passwords)
• Select a source and middle click into the replybox
• Confirm no content is copied into the replybox

[sssoleileraaa]

@rocodes - We have some time until rc2 is released, so the new test plan above just needs a review at some point plz

[rocodes]

Test plan LGTM (I added one small item), presuming that we will remove #1475 from the plan if we don't merge it. [Edit: we've decided in today's developer meeting that we'll defer #1475 til after the release so I'm removing it from the test plan]

[sssoleileraaa]

QA Testers, @eloquence and @cfm: the new test plan is here: #1465 (comment) and rc2 can be downloaded and tested directly from https://github.com/freedomofpress/securedrop-dev-packages-lfs/blob/main/workstation/buster/securedrop-client_0.7.0-rc2%2Bbuster_all.deb (we're still testing some new changes that make it so that nightlies don't clobber our release candidate packages, so for now, just dnf install it directly in sd-app, and will make this better next time.

[sssoleileraaa]

It looks like the new rc2 deb is not getting picked up and added to apt-test (right now there is no rc2 on https://apt-test.freedom.press/pool/main/s/securedrop-client/). @conorsch, if you have any tips on how to debug this sort of issue in the future, I think it would help us out quite a bit since it's not that uncommon for things to get stuck here.

[eloquence]

Thanks @creviera. For reference, the process I followed to install the package:

1. Download above deb
2. Copy to sd-small-buster-template by way of dom0
3. dpkg -i install it in sd-small-buster-template
4. Shut down the template and verify that new package is installed in sd-app

• Workstation: Qubes 4.0.4
• Client: 0.7.0-RC2
• Server: 2.3.1 prod on hardware

No double-check mark for a pending reply (#1470)

• Confirm that the check-mark is hidden while the reply is pending (visually, in a semi-transparent state).
• Confirm that the check-mark is hidden in case the journalist reply fails to deliver.

Selected source remains read in the source list after new message or reply comes in (#1464)

• ❓ Confirm no repro of Existing source with unread messages still shows up as unread after the source is selected #1463 (comment)

I temporarily see "Message not yet available" preview text in bold, once it is decrypted, it is marked as read. That behavior seems fine to me, and even helpful to highlight new stuff arriving, but just noting it in case it's not what's desired.

Selected source remains read in the source list after deleting conversation (#1464)

• Confirm no repro of Existing source with unread messages still shows up as unread after the source is selected #1463 (comment)

❗ Other testers, please observe the behavior when new sources are added on the server. Does the client register them as unread? I believe I observed one instance where newly created sources were marked as read, but I was not able to reproduce it - it may have been an observation error.

Client stress test during source deletion with fresh replies (#1468), No 'ghost sources' (#1473)

• In your staging server, run the loaddata.py script and add about 20 sources. (not run on a staging server)
• Continuously reply to sources and then delete them for 10 or more sources (before, during, and after syncing)
• Confirm the client does not crash
• Confirm sources never reappear in the source list

(Tested with about 10 sources)

Global mouse selection cleared after login (#1477)

• Select some text in your terminal
• Now click the middle button on your touchpad or mouse and confirm that it pastes the selected text (so you know it's on the global clipboard)
• Start the client and log in (we don't clear the clipboard until after the login screen to support copy-pasting diceware passwords)
• Select a source and middle click into the replybox
• Confirm no content is copied into the replybox

[sssoleileraaa]

❓ Confirm no repro of #1463 (comment)

I temporarily see "Message not yet available" preview text in bold, once it is decrypted, it is marked as read. That behavior seems fine to me, and even helpful to highlight new stuff arriving, but just noting it in case it's not what's desired.

This is indeed the desired behavior. The issue I was seeing earlier was around the selected source remaining bold until the next sync, so I'll update the STR linked to in the test plan to be clearer. I think we should also get @tina-ux's feedback on this behavior post-release.

[rocodes]

RC2 Test Plan

Make sure to include the version(s) of the server that you are testing against.
Environment: Qubes 4.0.4, staging servers @ 2.3.1, Installation method: debian package in sd-app

No double-check mark for a pending reply (#1470)

• Confirm that the check-mark is hidden while the reply is pending (visually, in a semi-transparent state).
• Confirm that the check-mark is hidden in case the journalist reply fails to deliver.

Selected source remains read in the source list after new message or reply comes in (#1464)

• Confirm no repro of Existing source with unread messages still shows up as unread after the source is selected #1463 (comment)
• ("New message" text is bold but becomes unbold after decryption, expected)

Selected source remains read in the source list after deleting conversation (#1464)

• Confirm no repro of Existing source with unread messages still shows up as unread after the source is selected #1463 (comment)

❌ Note: the behaviour differs between this case and the previous and I think there's a small bug here-- if I safe-delete messages and then reply to a source while it is still selected, the "Message not yet available" text is not bold. STR (cc @eloquence -- it's as you described!):

• Open Tor browser, submit (a) message(s) as a source. Note "message not yet available" text is bold. Select that source. Send another message(s), "message not yet available" text is still bold.
• Keep that source selected and safe-delete conversation. Now send a new reply to the source. "Message not yet available" text is not bold.
• As long as that source remains selected (i.e. you don't click anywhere else), the "Message not yet available" text will remain regular face font for all new messages from that source. As soon as you select a different source and then re-select to the original source, the "Message not yet available" text will return to displaying bold.

Client stress test during source deletion with fresh replies (#1468), No 'ghost sources' (#1473)

• In your staging server, run the loaddata.py script and add about 20 sources.
• Continuously reply to sources and then delete them for 10 or more sources (before, during, and after syncing)
• Confirm the client does not crash
• Confirm sources never reappear in the source list
• Note: Observing logs related to WrongUUIDError in logs after rapidly deleting sources during sync  #1478, but no user-facing behaviour. These logs show the result of deleting sources before their replies have been downloaded. As per my comments in that issue, we can defer this fix to post release.

2022-04-19 11:07:08,639 - securedrop_client.queue:182(process) ERROR: NoResultFound: No row was found for one() # this is the result of using `session.query(Source).filter_by(uuid=uuid).one()`instead of `one_or_none()` 
2022-04-19 11:07:08,733 - securedrop_client.logic:1052(on_delete_source_success) INFO: Source 99969ca4-6ecb-4d7b-a307-4bfdb2c78651 successfully scheduled for deletion at server
2022-04-19 11:07:08,735 - root:923(delete_source_collection) INFO: No source documents for maroon_coffeepot to delete
2022-04-19 11:07:09,092 - securedrop_client.logic:859(on_message_download_failure) ERROR: Could not emit message_download_failed: 'NoResultFound' object has no attribute 'uuid' # this is the result of assuming that we can use a Source object in error-handling, i.e. logger.error("Failed to download items for {}".format(source.uuid)) when the Source may be null

Global mouse selection cleared after login (#1477)

• Select some text in your terminal
• Now click the middle button on your touchpad or mouse and confirm that it pastes the selected text (so you know it's on the global clipboard)
• Start the client and log in (we don't clear the clipboard until after the login screen to support copy-pasting diceware passwords)
• Select a source and middle click into the replybox
• Confirm no content is copied into the replybox

[sssoleileraaa]

❌ Note: the behaviour differs between this case and the previous and I think there's a small bug here-- if I safe-delete messages and then reply to a source while it is still selected, the "Message not yet available" text is not bold. STR (cc @eloquence -- it's as you described!):

That does sound like a bug, @rocodes. The behavior should at least be consistent. I'll take a look to see why the style might not be getting applied as we expect.

[cfm]

Workstation: Qubes 4.0.4
Client: 0.7.0-rc2+buster
Server: 2.3.1 staging

No double-check mark for a pending reply (#1470)

• Confirm that the check-mark is hidden while the reply is pending (visually, in a semi-transparent state).

👉 Clarification: While the reply is pending:

• the reply is semi-transparent (opacity < 100%); and
• the check mark is hidden (opacity = 0%).

• Confirm that the check-mark is hidden in case the journalist reply fails to deliver.

Selected source remains read in the source list after new message or reply comes in (#1464)

• Confirm no repro of Existing source with unread messages still shows up as unread after the source is selected #1463 (comment)

Selected source remains read in the source list after deleting conversation (#1464)

• Confirm no repro of Existing source with unread messages still shows up as unread after the source is selected #1463 (comment)

❌ Yes, but regression:

Steps to reproduce

1. In the Client, follow Existing source with unread messages still shows up as unread after the source is selected #1463 (comment) for a source A.
2. Select another source B.
3. In the Source Interface as source A, submit another message.
4. In the Client, wait for sync.

Expected behavior

After the sync, source A is marked unread (bold) while the new message is downloaded.

Actual behavior

After the sync, source A remains marked read (not bolded).

Client stress test during source deletion with fresh replies (#1468), No 'ghost sources' (#1473)

• In your staging server, run the loaddata.py script and add about 20 sources.
• Continuously reply to sources and then delete them for 10 or more sources (before, during, and after syncing)
• Confirm the client does not crash
• Confirm sources never reappear in the source list

Global mouse selection cleared after login (#1477)

• Select some text in your terminal
• Now click the middle button on your touchpad or mouse and confirm that it pastes the selected text (so you know it's on the global clipboard)
• Start the client and log in (we don't clear the clipboard until after the login screen to support copy-pasting diceware passwords)
• Select a source and middle click into the replybox
• Confirm no content is copied into the replybox

[sssoleileraaa]

^ Now that rc3 is merged, it can be QA'd with the following test plan:

RC3 Test Plan

Follow regression outlined here: #1465 (comment)

• Confirm no more regression

Also look for the ❌ in this QA report and try to reproduce the finding: #1465 (comment)

• Confirm you're unable to repro

While you're at it, feel free to make notes of any unexpected behavior of read/unread (in addition to normal regression testing) since we'll be working on it in the near future to fix bugs like: #1463

[eloquence]

Unfortunately, I still see the issue with new messages being marked as read, which I've filed as #1482.

I don't see the behavior Cory described in #1463 (comment) - the "All files and messages deleted for this source" text is displayed unbolded.

(I've verified that the revert was correctly applied in the version of the code included in the .deb.)

[eloquence]

From my perspective, since we've established that #1482 has been present at least since 0.6.0 and probably longer (possibly since the seen/unseen feature was introduced), and the test plan looks good otherwise, we're good to proceed with releasing SecureDrop Client 0.7.0; then we can debug some of the edge cases we're hitting here at a more relaxed pace.

[sssoleileraaa]

Also, the rc2 change, that we reverted in rc3, fixes a different issue than described in #1482, but since it's so closely related I think it's worth keeping out of the final release. No need to revert the revert.

[rocodes]

RC3 Test Plan

Qubes 4.0.4, staging @ 2.3.1, sd-app @ 0.7.0-rc3

Follow regression outlined here: #1465 (comment)

• Confirm no more regression
• Observed: "All Files and Messages Deleted" did not show up as bold :)
• New message to unselected source does show up as bold

Also look for the ❌ in this QA report and try to reproduce the finding: #1465 (comment)

• Confirm you're unable to repro
• Can still repro intermittently (New messages from existing source sometimes show up as read #1482), but see this comment for our plan of record