Add historic package lists by andrewnaessens · Pull Request #133 · fedora-eln/content-resolver

andrewnaessens · 2026-08-19T15:11:46Z

Add daily package list snapshots with browse and compare UI features

Add daily snapshots of each view's package lists and counts. Package lists are viewable from chart data points and with browse and compare calendar date pickers.

Resolves #25

Summary

Add daily snapshots with per view SRPM lists and counts
Split daily snapshots into two separate snapshots:
- package lists (browse/compare)
- counts (chart)
Add calendar date pickers for browsing and comparing package lists
Add messaging for package list load failures
Use DataTables table for compare results
Plot weekly and daily counts on a single chart
Scale chart x-axis by date for relative spacing
Add click chart data points to reveal package lists
Move generate_historic_data() before generate_pages() so date list is available to templates
Remove weekly snapshot save (keeps read for historical weekly snapshot data)

coderabbitai · 2026-08-19T15:11:57Z

Important

🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults
Review profile: CHILL
Plan: Advanced
Run ID: 4f18a8a2-a44d-4c99-a440-b073855d4770

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

yselkowitz

There seems to be some boilerplate-like code here, is there an opportunity to create some functions which can be reused in multiple places?

yselkowitz · 2026-08-19T16:51:50Z

+    history_data = {}
+    history_data["date"] = str(now.strftime("%Y-%m-%d"))
+    history_data["workloads"] = {}
+    history_data["envs"] = {}
+    history_data["repos"] = {}
+    history_data["views"] = {}


IIUC we've settled on an initialization syntax for these.

Updated to dict literal syntax in save_current_historic_data_daily.

yselkowitz · 2026-08-19T16:53:06Z

+    # Generate daily Chart.js JSON — same chart types as _generate_chartjs_data
+    # but reading from daily snapshots instead of weekly


Do we need both? Or can we just use daily throughout?

Removed _generate_chartjs_data_daily and chart toggle. Chart is now generated from weekly snapshots for previous historic data and daily snapshots going forward.

yselkowitz · 2026-08-19T16:55:35Z

+  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/flatpickr/dist/flatpickr.min.css">
+  <script src="https://cdn.jsdelivr.net/npm/flatpickr"></script>


This should be like the others, with a version-specific src and integrity, and also make sure it's covered by the update checking workflow.

Pinned to v4.6.13 with SRI and added to check_cdn_updates.py.

yselkowitz · 2026-08-20T17:48:35Z

Also, could the exisitng weekly data be of any use to "backfill" the daily data?

andrewnaessens · 2026-10-05T09:15:45Z

There seems to be some boilerplate-like code here, is there an opportunity to create some functions which can be reused in multiple places?

Extracted listLinks(), fetchViewData(), datePicker(), and showMessage() helpers to reduce duplication.

andrewnaessens · 2026-10-05T09:19:44Z

Also, could the exisitng weekly data be of any use to "backfill" the daily data?

Existing weekly counts are displayed on the chart alongside new daily counts. Package lists are not captured in existing weekly snapshots so browse/compare feature will only contain package lists from the point it is introduced.

bhoy-troy

Looks good, just a couple of minor things to improve it.

bhoy-troy · 2026-10-16T11:12:18Z

+
+        # Sorted for consistent output
+        view_pkgs = {
+            "all_source_names": sorted(list(all_source_names)),


No need to use list(), sorted already returns a list.

There are a number of instances of this, and others that have been there long before we arrived 😃

Removed list().

bhoy-troy · 2026-10-16T11:24:19Z

+    filename = f"historic_data_pkgs-{year}-day_{day_of_year}.json"
+    output_dir = os.path.join(query.settings["output"], "history")
+    os.makedirs(output_dir, exist_ok=True)
+    file_path = os.path.join(output_dir, filename)


This block is duplicated in _save_current_historic_data_daily and _save_current_historic_data_daily_pkgs. There are also other modules that use similar style, that have been there a long time.

Can this be put into a function that can be reused to reduce duplication?

We can update the other modules at a later date when the need arises.

Extracted helper function _daily_snapshot_file_path to reduce duplication.

bhoy-troy · 2026-10-16T11:39:03Z

+
+    merged = {}
+
+    for entry in historic_data_weekly.values():


could be replaced with a dict comprehension

Replaced with dict comprehension.

bhoy-troy · 2026-10-16T11:51:31Z

+    # Filter for daily files only (not weekly)
+    all_filenames = os.listdir(directory)
+    valid_filenames = []
+    for filename in all_filenames:


Could use a more Pythonic approach with a comprehension.

valid_filenames = sorted( f for f in os.listdir(directory) if re.match() )

Used suggested pythonic approach.

bhoy-troy · 2026-10-16T12:08:48Z

+    all_filenames = os.listdir(directory)
+    valid_filenames = []
+    for filename in all_filenames:
+        if bool(re.match(r"historic_data-....-day_....json", filename)):


the . in .json isn't escaped; it works as it's just interpreted by the regex as any char

Also consider using something like r"historic_data-\d{4}-day_\d{3}\.json" if you expect the characters to be digits only.

Improved regex:
_read_historic_data - r"historic_data-\d{4}-week_\d{2}\.json"
_read_historic_data_daily - r"historic_data-\d{4}-day_\d{3}\.json"

andrewnaessens added 8 commits August 13, 2026 13:49

Add daily snapshot: browse, compare, chart toggle

5a6cdb5

Merge upstream master

31aa396

Update onClick handler for Chart.js v4

078acb5

Update form classes for Bootstrap 5

f23661b

Update comments

f9e77a7

Replace date dropdowns with calendar date pickers

908e5d9

Remove redundant browse-date event listener

2473227

Remove unused mode parameter from plot_graph

4fc71af

yselkowitz requested changes Aug 19, 2026

View reviewed changes

andrewnaessens added 2 commits August 20, 2026 15:14

Add versioned flatpickr SRI and update checking

df89e76

Use dict literal in daily snapshot

80c0c45

andrewnaessens added 16 commits September 2, 2026 16:00

Combine chartjs functions for weekly and daily

567d3c1

Rename historic_daily to historic_data_daily

3943a68

Add messages for missing, failed, and empty package lists

d82f44a

Plot weekly and daily data as a single chart view

f97b325

Fix function spacing

cf7c638

Split daily snapshot into counts and package lists

22cd27d

Scale chart x-axis by date for relative spacing

7942a16

Add browse flatpickr fallback and compare messages

42447d0

Add message for view missing from snapshot

f08b0eb

Fix downloadTxt memory leak

d84aedf

Add DataTables and conditional buildroot for compare

3216d4a

Extract helpers to reduce duplication

281eda2

Remove saving of weekly snapshots

019af95

Merge upstream master

54480c5

Use f-strings for consistency

a3f406f

Remove redundant canvas height

616c843

andrewnaessens marked this pull request as ready for review October 9, 2026 13:22

yselkowitz requested a review from bhoy-troy October 16, 2026 10:50

bhoy-troy reviewed Oct 16, 2026

View reviewed changes

PR feedback: extract helper, use pythonic expressions, improve regex

63c5186

bhoy-troy self-requested a review October 20, 2026 14:38

bhoy-troy approved these changes Oct 20, 2026

View reviewed changes

		# Generate daily Chart.js JSON — same chart types as _generate_chartjs_data
		# but reading from daily snapshots instead of weekly

		<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/flatpickr/dist/flatpickr.min.css">
		<script src="https://cdn.jsdelivr.net/npm/flatpickr"></script>

Conversation

andrewnaessens commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add daily package list snapshots with browse and compare UI features

Summary

Uh oh!

coderabbitai Bot commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yselkowitz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewnaessens Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yselkowitz commented Aug 20, 2026

Uh oh!

andrewnaessens commented Oct 5, 2026

Uh oh!

andrewnaessens commented Oct 5, 2026

Uh oh!

bhoy-troy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

andrewnaessens commented Aug 19, 2026 •

edited

Loading

coderabbitai Bot commented Aug 19, 2026 •

edited

Loading

andrewnaessens Sep 3, 2026 •

edited

Loading