Statistics subproject: the 150k cap

[chinmaypant21]

Hi @righettod and @riramar,
I've been reading through the OSHP repo to understand the project, and the statistics subproject caught my attention. I had a quick question I wanted to ask before going further.

From the README and the workflow, the current setup processes the first 150,000 entries of the Majestic Million list each month, with the cap explicitly tied to GitHub Actions' job time budget rather than to a statistical choice. I wanted to ask whether that cap is also methodologically sufficient, or whether a larger sample would be useful if it were feasible.

A few specific framings I was wondering about:

1. Is 150K considered sufficient for the kinds of aggregate adoption percentages the statistics page reports?
2. Would a larger sample size be beneficial for the statistics?

For context on why I'm asking: out of curiosity I ran some local benchmarks on gather_data.py and there appear to be paths to meaningfully reducing the execution time. But I don't want to spend more time on that if the existing cap is the enough for the project's needs.

Aware that the project is currently in a content freeze for the CMS migration, so no rush at all.

Thanks for the work you put into OSHP.

[righettod]

Hi @chinmaypant21

Thank you very much for your message and your feedback.

The number of 150k was found to stay under the processing time constraint of a GitHub Action (GHA) workflow step (set to 6 hours if i am not wrong) . Initially, it was set to 200k but the job sometimes fallen in case in which the processing time passed over the constraints so we moved back to 150k after several tests.

Regarding the consistency of 150k, to be fully honest and transparent, I'm not a statistician so I do not have a ground based answer to provide. When the scripts were developed, we decided to process the most entries possible in the provided GHA constraints. If a way exists to process the entire collection of domains then it is possible to move to such solution.

I totally share the fact that the gathering script can be enhanced/refactored. Indeed, currently it use requests with parallel threads. It should have better performance when moving to httpx + asyncio to make the parallel processing at I/O level.

The project is indeed frozen for the moment and I (I speak for myself) sincerely apologize for the slow down. We (OSHP team) must wait that the official new OWASP CMS site was released prior to perform the refactoring of the site. We know what we want to do but unfortunately we cannot start yet. However, we continue to update the content because frozing is mainly affecting the structure. We want to keep the OSHP content useful for the community and in sync with the reality.

Feel free to propose any PR for the gather_data.py script if you have an idea about it. All the content is open source and documented (input data and expected format for output data). If you use a coding assistant, we just ask that the code was not too verbose in order to be understand and maintained by us.

Feel free to come back to us, I we missed something according to your question.

[chinmaypant21]

Thank you for the detailed response @righettod, really helpful context.
I'll take a closer look at moving the script along the lines you suggested, and aim for something that comfortably handles 200K (and ideally more).

[righettod]

@chinmaypant21 You are welcome and thanks for the help 😉

[righettod]

Following exchange on #297

I reverted the PR but i kept the version refactored in the script gather_data_refactored.py.

We need to take time to test it locally prior to release it. Local test is possible as well as test into a GHA context.