You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Handle a case where's a lot of links which only differ with fragment part, such as nix package recipe urls for haskell modules which all come from a single file (a lot of urls differing with a line number only):
It doesn't make sense to check them all as they are a single url in fact.
Possible solutions:
Simple: when checking, group links by fragmentless url. This allows to process a huge batch of new urls quickly, however is not effective solution long-term, as link recheck times are random and not many urls will get into a single batch for recheck.
Instead of checking, take status from a fragmentless url. However, we may not have one in the database (to have one, it should be mentioned in some package explicitly; it's no the case for example above, for instance). We can add such an url, but to work cleanly it would require more complicated refcounting mechanism, as links would reference other links (however that's useful for redirects too).
Strip fragments on the package level, e.g. keep fragmentless urls in the links database, and keep fragment part in the package.
The text was updated successfully, but these errors were encountered:
Handle a case where's a lot of links which only differ with fragment
part, such as nix package recipe urls for haskell modules which all
come from a single file:
https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/development/haskell-modules/hackage-packages.nix#L23075
This is not really a complete solution as its efficiency drops over
time when link rechecks are distributed evenly over time, but it allows
to process new batches of urls much quicker.
…)"
This reverts commit 08bbb27.
After commit 83b6a17 in repology-updater, we no longer have fragments in
links table, so we no longer need this optimization.
Handle a case where's a lot of links which only differ with fragment part, such as nix package recipe urls for haskell modules which all come from a single file (a lot of urls differing with a line number only):
https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/development/haskell-modules/hackage-packages.nix#L23075
It doesn't make sense to check them all as they are a single url in fact.
Possible solutions:
The text was updated successfully, but these errors were encountered: