Add MLBF validation logic and corresponding management command

[KevinMind]

Fixes: mozilla/addons#15281

Description

• Implemented a validate method in the MLBF class to check for duplicate items in the cache.json, ensuring each item occurs only once and in a single data type.
• Added a new management command validate_mlbf to facilitate validation of MLBF directories from the command line.
• Created unit tests to verify the validation logic, ensuring it raises errors for duplicate items within a single data type and across multiple data types.

Context

We have discovered that MLBFs created since November 2024 are larger than expected. This validation is one step to ensure we do not see unchecked growth in the mlbf by verifying the underlying data set used to produce the filter.

Testing

• Download the cache.json from the (privately) linked g-drive URL
• move the mlbf directory into your "storage" directory in the repository
• Run the below command to validate the data

./manage.py validate_mlbf.py <id>

Where id is the timestamp directory name ./mlbf/<id>/.

Checklist

• Add #ISSUENUM at the top of your PR to an existing open issue in the mozilla/addons repository.
• Successfully verified the change locally.
• The change is covered by automated tests, or otherwise indicated why doing so is unnecessary/impossible.
• Add before and after screenshots (Only for changes that impact the UI).
• Add or update relevant docs reflecting the changes made.

[KevinMind]

Verified on production data that we do in fact have duplicates and the verification command catches them.

Now to find out how that happened in the first place and how to prevent.

FYI @Rob--W you're theory seems to be correct.

[Rob--W]

Now to find out how that happened in the first place and how to prevent.

FYI @Rob--W you're theory seems to be correct.

Note: although cache.json here does indeed have duplicates (which I had observed myself as well), it is not necessarily the root cause for the issues observed in #15261.

In my local testing that lead to mozilla/addons#15261 (comment), I generated a new MLBF solely based on the cache.json file, without de-duplication. This means that if there were any duplicates in cache.json, that it would directly contribute to the generated MLBF. The fact that my MLBF is still smaller than what was observed in production is a strong signal that there is additional duplicate data that is not stored in cache.json.

[KevinMind]

I generated a new MLBF solely based on the cache.json file, without de-duplication. This means that if there were any duplicates in cache.json, that it would directly contribute to the generated MLBF. The fact that my MLBF is still smaller than what was observed in production is a strong signal that there is additional duplicate data that is not stored in cache.json.

I'm not sure I follow this logic.

I generated a new MLBF solely based on the cache.json file, without de-duplication.

What do you mean by that? like what code did you actually run?

The fact that my MLBF is still smaller than what was observed in production is a strong signal that there is additional duplicate data that is not stored in cache.json.

That makes sense assuming the way you generated the filter is the same as the way production generated it. My understanding of that is based on the answer to the first question. But even if that were the case, if you did not de-duplicate the file and generated a filter that is smaller.. how did that happen? Isn't it even more odd if we get different sized filters on the same data set in different environments? I must be missing something from your comment.

[Rob--W]

I generated a new MLBF solely based on the cache.json file, without de-duplication. This means that if there were any duplicates in cache.json, that it would directly contribute to the generated MLBF. The fact that my MLBF is still smaller than what was observed in production is a strong signal that there is additional duplicate data that is not stored in cache.json.

I'm not sure I follow this logic.

The production file size is 931343 and linked from https://bugzilla.mozilla.org/show_bug.cgi?id=1938506#c5.
I generated a new MLBF from cache.json and got a smaller file. Conclusion: the actual MLBF generation on AMO prod includes more data than what is written to cache.json.

I generated a new MLBF solely based on the cache.json file, without de-duplication.

What do you mean by that? like what code did you actually run?

This is the code using filtercascade, similar to what generate_mlbf in mlbf.py does: https://gist.github.com/Rob--W/39940bf6266d5e5b2c3b7e65da67696b

I've attached the command output to https://mozilla-hub.atlassian.net/browse/AMOENG-1332?focusedCommentId=988314

[Rob--W]

Thanks for putting this fix up so quickly!

The fix of the primary issue looks good to me.

I'll defer to AMO engineers for the approving sign-off, in case they have thoughts on the tests or implementation details.

[eviljeff]

just some code nits. I'll defer to Rob/Will's opinions for whether this is doing the "right" thing.