Deduplicate readmes in registry

[jaydeluca]

Component / area

collector-watcher and java-instrumentation-watcher for code changes, will result in changes to ecosystem-registry

Problem this solves

Readme's are verbose and rarely change, and we are duplicating the content for every release. We should store one file per version of it in the ecosystem-registry, and add references to it.

Proposed solution

Collector

We already are creating files with the hash included in the name, but right now we are creating a readme directory per release (ecosystem-registry/collector/contrib/v0.157.1-SNAPSHOT/component_readmes), so update to store them in the root of the distro instead: ecosystem-registry/collector/contrib/readmes and add a new readme reference in each component's metadata that has the file name

Java agent

Instead of storing them in ecosystem-registry/java/javaagent/v2.30.0/library_readmes, put them in ecosystem-registry/java/javaagent/library_readmes and similarly, add a readme link in the metadata

Acceptance criteria

We rebuild the java and collector registries and it deletes all the excess readme's and adds references to all components

The explorer-db-builder will need to be updated to handle this change as well to map the content back to the components

Additional context

No response

[Pittu-Sharma]

Hi @jaydeluca, I would like to work on this issue. Please assign it to me!

[Pittu-Sharma]

Hello @jaydeluca , I've gone deep down into the codebase to understand the problem and made an Implementation plan for this issue. I have decided to make 4 PRs for this issue so the it can be reviewed easily. Please go through the plan once and let me know if you like this plan.

What's the problem?

Right now, every time a new version is released, the README files for each component
get copied into that version's directory:

• Java: v{version}/library_readmes/
• Collector: v{version}/component_readmes/

The issue is that READMEs almost never change between releases, so we're storing a ton
of duplicate content in the registry for no reason.

What I'm thinking

The simplest fix is to store each README just once, in a shared folder per distribution,
and name the file based on a hash of its content — so if the same README appears across
10 versions, it's only stored once. Then each component just references the filename in
its metadata.

I'd break this into 4 PRs so each one is easier to review:

---

PR 1 — Move Java agent READMEs to a global folder

Instead of saving to v{version}/library_readmes/, save to a single
library_readmes/ folder at the javaagent root. File names stay the same pattern:
{safe_name}-{hash}.md.

The tricky part here is updating load_library_readme_map — right now it scans the
directory, but after this change it should just read the readme: field we'll be
adding to each entry in instrumentation.yaml. Also want to add a
prune_orphan_readmes() helper that cleans up any files that aren't referenced by any
version anymore.

Files: watcher-common/inventory_manager.py, instrumentation_sync.py

---

PR 2 — Add README extraction to the collector watcher

The collector watcher doesn't extract READMEs at all right now, so this is mostly
new code. During component scanning, if there's a README.md in the component folder,
we read it and carry the content through temporarily. When saving the inventory, we
hash it, write it to {distribution}/readmes/, and store the filename in the
component's metadata as metadata.readme.

Files: component_scanner.py, collector-watcher/inventory_manager.py

---

PR 3 — Update the DB builder to publish READMEs

Once the registry has the new structure, the DB builder needs to:

• Read the readme maps from the registry
• Write the .md files to public/data/collector/markdown/
• Add a markdown_hash field to each component JSON

For the Java side, the only change is updating load_library_readme_map to use the
new YAML-based lookup instead of scanning directories.

Files: collector_builder.py, collector_database_writer.py, main.py

Depends on PR 1 and 2 being merged first.

---

PR 4 — Add a README tab to the collector detail page

Last piece is the frontend. Pretty straightforward — add markdown_hash?: string to
the CollectorComponent type, add a loadComponentReadme function to the API layer
(same pattern as loadLibraryReadme for Java), hook it up to a custom hook, and
render it in a new README tab on the detail page using ReactMarkdown. Tab just
doesn't show if there's no hash.

Files: collector.ts, collector-data.ts, use-collector-data.ts,
collector-detail-page.tsx, locale files (en + es)

---

A few things I wanted to check with you

• Should prune_orphan_readmes run automatically on every sync, or is it safer to
  make it a manual step for now?
• For the collector readmes directory, should it be collector/{distribution}/readmes/
  or collector/readmes/{distribution}/? I went with the former since it feels more
  consistent with the rest of the layout, but open to the other way.
• If a component has no README, should we hide the tab entirely on the frontend, or
  show a small "No README available" message?

Let me know what you think — I can start with PR 1 once we're aligned on the approach.

[Pittu-Sharma]

Hi @jaydeluca , Please review this Implementation plan once, waiting for your approval.

[jaydeluca]

the collector watcher was recently updated to pull in readmes so there's some aspects of your plan that might already be done or need updating when you get to them, but the general idea of your approach looks good. feel free to proceed with PR 1 when you are ready

[Pittu-Sharma]

Hi @jaydeluca, I have made the PR-1 for this issue as mentioned in the Implementation Plan. Please review it once.