Skip to content

Crawl-Anywhere - Web Crawler and document processing pipeline with Solr integration.

License

Apache-2.0, Unknown licenses found

Licenses found

Apache-2.0
LICENSE.txt
Unknown
LICENSE-DEPENDENCIES.txt
Notifications You must be signed in to change notification settings

smrjans/crawl-anywhere

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Crawl-Anywhere

April 2013 - Starting version 4.0, Crawl-Anywhere becomes an open-source project. Current version is 4.0.0

Stable version 3.0.x is still available at http://www.crawl-anywhere.com/

Introduction

Crawl Anywhere is mainly a web crawler. However, Crawl-Anywhere includes all components in order to build a vertical search engine.

Crawl Anywhere includes :

Project home page : http://www.crawl-anywhere.com/

A web crawler is a program that discovers and read all HTML pages or documents (HTML, PDF, Office, ...) on a web site in order for example to index these data and build a search engine (like google). Wikipedia provides a great description of what is a Web crawler : http://en.wikipedia.org/wiki/Web_crawler.

Support

Build distribution

Pre-requisites :

  • Maven 3.0.0 or >
  • Oracle Java 7 or >

Steps :

Installation

Pre-requisites :

  • Oracle Java 7 or >
  • Apache 2.0 or >
  • PHP 5.2.x or 5.3.x or 5.4.x
  • MongoDB 64 bits 2.2 or >
  • Solr 4.3.0 or > (configuration files provided for Solr 4.3.0 and 4.10.0)

Steps :

Getting Started

See the User Manual at http://www.crawl-anywhere.com/getting-started/

History

  • release 4.0.0-alpha-1 : April, 28 2013
  • release 4.0.0-alpha-2 : May, 22 2013
  • release 4.0.0-alpha-3 : June, 21 2013
  • release 4.0.0-alpha-4 : June, 23 2013
  • release 4.0.0-beta-1 : August, 6 2013
  • release 4.0.0-release-candidate : October, 20 2013
  • release 4.0.0 final : December, 1, 2014

About

Crawl-Anywhere - Web Crawler and document processing pipeline with Solr integration.

Resources

License

Apache-2.0, Unknown licenses found

Licenses found

Apache-2.0
LICENSE.txt
Unknown
LICENSE-DEPENDENCIES.txt

Stars

Watchers

Forks

Packages

No packages published

Languages

  • PHP 78.1%
  • Java 15.8%
  • JavaScript 2.4%
  • XSLT 1.8%
  • Other 1.9%