InSite Crawler

Deep-site research made simple. Crawl any website for specific keywords and get organized Markdown reports - perfect for competitive research, documentation discovery, and content analysis.

🎓 Academic Use Notice: This tool is designed for academic research, educational purposes, and legitimate competitive analysis. Please use responsibly and in compliance with applicable laws and website terms of service. See Ethical Crawling & Academic Use section for important guidelines.

🎯 What Does This Do?

InSite Crawler automatically visits web pages on a domain, searches for your keywords, and creates a detailed report showing exactly where those terms appear. Think of it as a super-powered "Find on Page" that works across entire websites.

Perfect for:

• 🔍 Competitive Research - Find pricing, features, or contact info across competitor sites
• 📚 Documentation Discovery - Locate API docs, tutorials, or specific technical information
• 📊 Content Analysis - Research how companies talk about specific topics
• 🎯 Lead Generation - Find contact pages, team info, or partnership opportunities

⚡ Quick Start

1. Install the Extension

• Download the .vsix file from releases
• In Cursor: Cmd+Shift+P → "Extensions: Install from VSIX" → Select the file
• Restart Cursor

2. Start Your First Search

1. Open Command Palette: Cmd+Shift+P (Mac) or Ctrl+Shift+P (Windows/Linux)
2. Type: "InSite: Start Research"
3. Enter Website: example.com (protocol optional - we'll add https:// automatically)
4. Enter Search: pricing or "contact us" or pricing AND enterprise
5. Wait: Watch the progress as we crawl and search
6. Get Results: Automatic Markdown report with all matches highlighted

3. Choose Where to Save

• With Workspace: Reports save to your open workspace folder
• Without Workspace: Folder picker appears - choose Desktop, Documents, etc.

🔍 Search Like a Pro

Simple Searches

pricing          → Find "pricing" anywhere
contact us       → Find "contact us" phrase
API              → Find "API" mentions

Advanced Boolean Searches

pricing AND enterprise     → Both terms must appear on same page
pricing OR cost OR fees    → Any of these terms
"enterprise pricing"       → Exact phrase only
(API OR SDK) AND tutorial  → Complex logic with parentheses

Real-World Examples

🏢 Competitor Research:
   "enterprise pricing" AND (contact OR demo)

📚 Documentation Hunt:
   (API OR SDK) AND (tutorial OR "getting started")

💼 Partnership Research:
   (partner OR integration) AND (program OR marketplace)

🎯 Contact Discovery:
   (contact OR team OR about) AND (email OR phone)

📊 What You Get

Rich Markdown Reports

• Smart Highlighting: Search terms pop with bright colors and larger text
• Full Context: Complete sentences around each match (not just snippets)
• Organized Results: Grouped by page with clickable table of contents
• Match Counts: See exactly how many times terms appear per page
• Error Tracking: Know which pages couldn't be crawled and why
• Robots.txt Compliance: See which pages were respectfully skipped

Example Report Structure

# InSite Research Report: enterprise pricing

## Research Metadata
- Search Term: enterprise pricing
- Target Domain: competitor.com
- Pages Scanned: 47
- Matches Found: 12
- Status: ✅ Complete

## Table of Contents
1. Enterprise Solutions - 5 matches
2. Pricing Page - 4 matches  
3. Contact Sales - 3 matches

## Results (12 matches)

### 1. Enterprise Solutions
**URL:** https://competitor.com/enterprise
**Matches:** 5

> Our **enterprise pricing** starts at $99/month for teams...
> Contact our sales team for custom **enterprise pricing**...

⚙️ Configuration

Access settings via Cmd+, → Search "InSite":

Crawling Behavior

• Max Depth (1-10): How deep to crawl from starting page (default: 2)
• Concurrency (1-20): Simultaneous page requests (default: 5)
• Timeout (1-30s): How long to wait per page (default: 5s)

Output Options

• Auto Preview: Automatically open report after generation (default: on)
• Respect Robots.txt: Follow website crawling rules (default: on, recommended)

Recommended Settings

{
  "insite.crawler.maxDepth": 2,        // Good balance of coverage vs speed
  "insite.crawler.concurrency": 5,     // Polite crawling
  "insite.crawler.respectRobotsTxt": true  // Be a good web citizen
}

🚀 Pro Tips

Performance Optimization

• Start Small: Begin with depth 1-2, increase if needed
• Use Specific URLs: Target /pricing or /docs instead of homepage
• Cancel Early: Found what you need? Hit cancel to get partial results faster

Search Strategy

• Broad First: Start with pricing then narrow to "enterprise pricing"
• Use Synonyms: Try cost OR price OR fee to catch variations
• Phrase Matching: Use quotes for exact phrases like "contact sales"
• Combine Terms: pricing AND (enterprise OR business) for targeted results

Workflow Integration

• Save to Project: Keep workspace open to save reports with your project
• Cursor AI Analysis: Ask Cursor to analyze, summarize, or compare reports
• Regular Research: Set up dedicated research workspace for ongoing competitive analysis

🛠️ Troubleshooting

Common Issues

"No matches found"

• ✅ Check spelling and try broader terms
• ✅ Verify the site actually contains your search terms
• ✅ Try synonyms or related terms

"Crawl takes too long"

• ✅ Reduce max depth setting (try 1-2)
• ✅ Increase timeout for slow sites
• ✅ Use cancel button for partial results

"Many pages blocked by robots.txt"

• ✅ This is normal and ethical - we respect website rules
• ✅ Try different starting URLs on the same domain
• ✅ Check the blocked pages list in your report

"Extension not appearing"

• ✅ Restart Cursor after installation
• ✅ Check Extensions panel for "InSite"
• ✅ Reinstall the .vsix file if needed

Getting Help

1. Check error messages in generated reports
2. Review Cursor Developer Console (Help → Toggle Developer Tools)
3. Adjust settings and retry
4. Try with a simpler website first

🔒 Ethical Crawling & Academic Use

InSite Crawler is designed to be respectful and is intended for academic, research, and educational purposes only:

• ✅ Respects robots.txt by default (can be disabled if needed)
• ✅ Rate limited to avoid overwhelming servers
• ✅ Same-domain only - won't follow external links
• ✅ Reasonable limits - stops at 500 URLs to prevent runaway crawls

🎓 ACADEMIC USE DISCLAIMER

This extension is designed for academic research, educational purposes, and legitimate competitive analysis only.

The robots.txt bypass and 403 error bypass features are provided for:

• 📚 Academic Research: University studies, thesis research, and scholarly analysis
• 🔬 Educational Purposes: Learning about web technologies, SEO analysis, and content research
• 📊 Legitimate Business Intelligence: Competitive analysis within legal and ethical boundaries
• 🛠️ Technical Research: Understanding web crawling, bot detection, and site architecture

⚠️ IMPORTANT: Bot Detection Bypass Feature

Version 0.5.0 includes an optional bot detection bypass feature (bypassBlocking setting). This feature is intended for academic and research purposes only.

Legal and Ethical Considerations:

• 🚨 Check Terms of Service: Bypassing bot detection may violate a website's terms of service
• ⚖️ Legal Liability: You are responsible for ensuring your use complies with applicable laws
• 🤝 Respect Website Owners: Just because you can bypass doesn't mean you should
• 📋 Academic Intent: Designed for legitimate research on sites that incorrectly block academic crawlers
• 🎓 Educational Use: Intended for learning about web technologies and research methodologies

When NOT to use bypass mode:

• ❌ Sites that explicitly disallow automated access in their ToS
• ❌ Commercial scraping for profit without permission
• ❌ Overwhelming servers with high concurrency
• ❌ Any illegal, unethical, or non-academic purpose
• ❌ Circumventing security measures for malicious purposes

When it MAY be appropriate (academic use only):

• ✅ University research projects with proper oversight
• ✅ Academic studies on web content and structure
• ✅ Educational analysis of publicly available information
• ✅ Legitimate competitive research within ethical boundaries
• ✅ Technical research on web crawling methodologies

This feature is disabled by default. Only enable it if you have a legitimate academic or research purpose and legal right to access the content.

📜 User Responsibility

By using InSite Crawler, especially with bypass features enabled, you acknowledge that:

• You will use this tool only for academic, educational, or legitimate research purposes
• You are responsible for complying with all applicable laws and website terms of service
• You will not use this tool for commercial scraping, data theft, or malicious purposes
• You understand that bypassing bot detection may violate website policies
• You will respect website owners' rights and server resources

📋 System Requirements

• Cursor IDE or VS Code 1.80.0+
• Node.js 20.0.0+ (usually bundled with Cursor)
• Internet connection for crawling
• Any OS: Windows, macOS, Linux

🆕 What's New

Version 0.5.0

• ⚠️ 403 Error Bypass System: Advanced bot detection evasion (use responsibly and ethically)
• Smart Retry Logic: Exponential backoff for blocked requests
• Enhanced Headers: Realistic browser fingerprints for better compatibility
• User Agent Rotation: Appears as legitimate browsers to avoid false positives

Version 0.4.1

• Enhanced Highlighting: Search terms now pop with vibrant colors and backgrounds
• Improved Visibility: Larger, bolder text with contrasting colors for better readability

Version 0.4.0

• Simplified Folder Selection: Streamlined experience when no workspace is open
• Cross-Platform Reliability: Works identically in VS Code and Cursor

Version 0.3.0

• Boolean Search Queries: Full AND/OR support with parentheses and quotes
• Advanced Search: Complex queries like (pricing OR cost) AND enterprise

📄 License

MIT License - Use freely for personal and commercial projects.

---

Ready to start researching? Install InSite Crawler and discover what your competitors are really saying about pricing, features, and more!