Add "Import new releases" emails (#1872)

[daveoconnor]

This is related to #1872.

• Added user emails for "import new releases" and improved logging on "do it all".
• Refactors the release tasks command for reuse.

Testing: confirm that the "do it all" and "import new releases" button in the admin versions ui send an email with a list of completed tasks, and list the function for each task, e.g. 2025-10-01 22:02:21.244245+00:00: Task start - function: import_versions, desc: Importing versions...

[karimarie67]

I deleted version 1.89.0 in http://localhost:8000/admin/versions/version/. When I click import new release I get no email but it does import 1.89.0.

For the Do It All I'm getting:
Task Failed: release_tasks
From:
[email protected]
To:
[email protected]
Your task release_tasks failed.

Am I missing test steps @daveoconnor ?

[sdarwin]

I believe "Do It All" is somewhat deprecated. The purpose of that was to also include the Release Report. However there is some other method to handle that now. The publish scripts have been set back to use the original Import Releases button.

Email would be expected to fail. if you're not set up with official api keys to mailgun, etc. Perhaps not easily testable then.

[karimarie67]

Tested with @daveoconnor

Received emails when the tasks start and when they finish.

Your task release_tasks was started at: 2025-10-07 22:10:25.824684+00:00

The task release_tasks was completed. Task took: 0:35:48.238762

Progress Messages
2025-10-07 22:10:26.730527+00:00: Task start - function: import_versions, desc: importing versions...
2025-10-07 22:10:26.760224+00:00: Task done - function: import_versions, desc: importing versions
2025-10-07 22:10:26.760952+00:00: Task start - command: import_beta_release, desc: importing most recent beta version...
2025-10-07 22:11:36.057563+00:00: Task done - command: import_beta_release, desc: importing most recent beta version
2025-10-07 22:11:36.057714+00:00: Task start - command: update_libraries, desc: importing libraries...
2025-10-07 22:13:18.666395+00:00: Task done - command: update_libraries, desc: importing libraries
2025-10-07 22:13:18.666502+00:00: Task start - function: import_library_versions, desc: saving library-version relationships...
2025-10-07 22:13:18.677919+00:00: Task done - function: import_library_versions, desc: saving library-version relationships
2025-10-07 22:13:18.677977+00:00: Task start - command: update_maintainers, desc: adding library maintainers...
2025-10-07 22:14:30.757791+00:00: Task done - command: update_maintainers, desc: adding library maintainers
2025-10-07 22:14:30.757875+00:00: Task start - command: update_authors, desc: adding library authors...
2025-10-07 22:14:34.069477+00:00: Task done - command: update_authors, desc: adding library authors
2025-10-07 22:14:34.069663+00:00: Task start - command: update_library_version_authors, desc: adding library version authors...
2025-10-07 22:16:02.182470+00:00: Task done - command: update_library_version_authors, desc: adding library version authors
2025-10-07 22:16:02.182555+00:00: Task start - function: import_commits, desc: importing git commits...
2025-10-07 22:20:59.170945+00:00: Task done - function: import_commits, desc: importing git commits
2025-10-07 22:20:59.171149+00:00: Task start - command: sync_mailinglist_stats, desc: syncing mailinglist statistics...
2025-10-07 22:20:59.190025+00:00: Task done - command: sync_mailinglist_stats, desc: syncing mailinglist statistics
2025-10-07 22:20:59.190101+00:00: Task start - command: update_issues, desc: updating github issues...
2025-10-07 22:30:22.860824+00:00: Task done - command: update_issues, desc: updating github issues
2025-10-07 22:30:22.860933+00:00: Task start - command: fetch_slack_activity, desc: updating slack activity buckets...
2025-10-07 22:46:11.315062+00:00: Task done - command: fetch_slack_activity, desc: updating slack activity buckets
2025-10-07 22:46:11.315160+00:00: Task start - function: update_website_statistics, desc: updating website statistics...
2025-10-07 22:46:11.687481+00:00: Task done - function: update_website_statistics, desc: updating website statistics
2025-10-07 22:46:11.687586+00:00: Task start - function: import_ml_counts, desc: importing mailing list counts...
2025-10-07 22:46:14.056792+00:00: Task done - function: import_ml_counts, desc: importing mailing list counts
2025-10-07 22:46:14.056897+00:00: Task start - function: generate_report, desc: generating report...
2025-10-07 22:46:14.062866+00:00: Task done - function: generate_report, desc: generating report

The import_commits task did not find commits for these libraries.
The task may need to re-run.
scope_exit
fusion
proto
icl
math/statistical_distributions
functional/hash
utility/ostream_string
signals
utility/string_ref
math/common_factor
math/octonion
math/quaternion
math/special_functions
compatibility
algorithm/minmax
functional
algorithm
functional/factory
functional/forward
functional/overloaded_function
units
xpressive
io
serialization
date_time
graph_parallel
numeric/ublas
statechart
circular_buffer
polygon
coroutine2
gil
coroutine
numeric/odeint
phoenix
local_function
align
callable_traits
convert
vmd
compute
hof
safe_numerics
parameter_python
static_string
property_map_parallel
multi_array
tti

Your task import_new_versions was started at: 2025-10-08 11:57:09.101095+00:00
The task import_new_versions was completed. Task took: 0:00:52.566134

Progress Messages
2025-10-07 22:10:26.730527+00:00: Task start - function: import_versions, desc: importing versions...
2025-10-07 22:10:26.760224+00:00: Task done - function: import_versions, desc: importing versions
2025-10-07 22:10:26.760952+00:00: Task start - command: import_beta_release, desc: importing most recent beta version...
2025-10-07 22:11:36.057563+00:00: Task done - command: import_beta_release, desc: importing most recent beta version
2025-10-07 22:11:36.057714+00:00: Task start - command: update_libraries, desc: importing libraries...
2025-10-07 22:13:18.666395+00:00: Task done - command: update_libraries, desc: importing libraries
2025-10-07 22:13:18.666502+00:00: Task start - function: import_library_versions, desc: saving library-version relationships...
2025-10-07 22:13:18.677919+00:00: Task done - function: import_library_versions, desc: saving library-version relationships
2025-10-07 22:13:18.677977+00:00: Task start - command: update_maintainers, desc: adding library maintainers...
2025-10-07 22:14:30.757791+00:00: Task done - command: update_maintainers, desc: adding library maintainers
2025-10-07 22:14:30.757875+00:00: Task start - command: update_authors, desc: adding library authors...
2025-10-07 22:14:34.069477+00:00: Task done - command: update_authors, desc: adding library authors
2025-10-07 22:14:34.069663+00:00: Task start - command: update_library_version_authors, desc: adding library version authors...
2025-10-07 22:16:02.182470+00:00: Task done - command: update_library_version_authors, desc: adding library version authors
2025-10-07 22:16:02.182555+00:00: Task start - function: import_commits, desc: importing git commits...
2025-10-07 22:20:59.170945+00:00: Task done - function: import_commits, desc: importing git commits
2025-10-07 22:20:59.171149+00:00: Task start - command: sync_mailinglist_stats, desc: syncing mailinglist statistics...
2025-10-07 22:20:59.190025+00:00: Task done - command: sync_mailinglist_stats, desc: syncing mailinglist statistics
2025-10-07 22:20:59.190101+00:00: Task start - command: update_issues, desc: updating github issues...
2025-10-07 22:30:22.860824+00:00: Task done - command: update_issues, desc: updating github issues
2025-10-07 22:30:22.860933+00:00: Task start - command: fetch_slack_activity, desc: updating slack activity buckets...
2025-10-07 22:46:11.315062+00:00: Task done - command: fetch_slack_activity, desc: updating slack activity buckets
2025-10-07 22:46:11.315160+00:00: Task start - function: update_website_statistics, desc: updating website statistics...
2025-10-07 22:46:11.687481+00:00: Task done - function: update_website_statistics, desc: updating website statistics
2025-10-07 22:46:11.687586+00:00: Task start - function: import_ml_counts, desc: importing mailing list counts...
2025-10-07 22:46:14.056792+00:00: Task done - function: import_ml_counts, desc: importing mailing list counts
2025-10-07 22:46:14.056897+00:00: Task start - function: generate_report, desc: generating report...
2025-10-07 22:46:14.062866+00:00: Task done - function: generate_report, desc: generating report
2025-10-08 11:57:10.018067+00:00: Task start - command: import_versions, desc: importing versions...
2025-10-08 11:57:10.127104+00:00: Task done - command: import_versions, desc: importing versions
2025-10-08 11:57:10.127186+00:00: Task start - command: import_beta_release, desc: importing most recent beta...
2025-10-08 11:58:00.789343+00:00: Task done - command: import_beta_release, desc: importing most recent beta
2025-10-08 11:58:00.789393+00:00: Task start - command: import_development_versions, desc: importing development versions...
2025-10-08 11:58:01.667198+00:00: Task done - command: import_development_versions, desc: importing development versions

@GregKaleka @rbbeeston This needs a review from you. @daveoconnor once approved we can merge. Thank you!

[sdarwin]

@karimarie67 , in the most recent message "Received emails when the tasks start..."

• What did you run to trigger the task? Was it one step, or two, or more?

• If there are two separate tests ("Import New Releases" and "Do It all") could you clearly distinguish between those and what the results were? You wrote "tasks", and then post results. But which is which, between those two?

• Notice this in the output:

...
2025-10-07 22:46:14.062866+00:00: Task done - function: generate_report, desc: generating report
2025-10-08 11:57:10.018067+00:00: Task start - command: import_versions, desc: importing versions...
...

There's apparently a delay of 12 hrs. Was there a long delay?

[karimarie67]

@sdarwin

in the most recent message "Received emails when the tasks start..."

• What did you run to trigger the task? Was it one step, or two, or more? < - one step for each in the Admin console.

• If there are two separate tests ("Import New Releases" and "Do It all") could you clearly distinguish between those and what the results were? <- Import New Releases is import_new_versions and Do It All is release_tasks

You wrote "tasks", and then post results. But which is which, between those two? <- release_tasks is Do It All and import_versions is Import New Releases

• Notice this in the output:

...
2025-10-07 22:46:14.062866+00:00: Task done - function: generate_report, desc: generating report
2025-10-08 11:57:10.018067+00:00: Task start - command: import_versions, desc: importing versions...
...

There's apparently a delay of 12 hrs. Was there a long delay? <- there was not, I believe that was on me. I just ran Do It All again and this is the time: The task release_tasks was completed. Task took: 0:28:03.558970 and versions was The task import_new_versions was completed. Task took: 0:00:47.601381

@gregjkal and/or @rbbeeston - this needs review as well. Thank you!

[gregjkal]

LGTM - nice refactor.