GitHub - shekinahfire77/music_crawler: This is a resource-constrained, production-grade continuous hybrid roaming web crawler specifically optimized for Render's 0.5 vCPU and 512MB RAM limits. It's designed to crawl music-related websites efficiently while respecting robots.txt and maintaining politeness.

Branches Tags

Name		Name	Last commit message	Last commit date
Latest commit History 17 Commits
music-crawler		music-crawler
requirements.txt		requirements.txt
runtime.txt		runtime.txt

About

This is a resource-constrained, production-grade continuous hybrid roaming web crawler specifically optimized for Render's 0.5 vCPU and 512MB RAM limits. It's designed to crawl music-related websites efficiently while respecting robots.txt and maintaining politeness.