Item Pipeline

Overview

Item Pipeline is one of the Antch core components to process a sequence of input values.

Item is an interface type, so you can passed any value into the item pipeline.

Customize Item Pipeline

Pipeline Interface:

type PipelineHandler interface {
	ServePipeline(Item)
}

type Pipeline func(PipelineHandler) PipelineHandler

type OutputPipeline struct {
	next antch.PipelineHandler
}

func (p *OutputPipeline) ServePipeline(v antch.Item) {
	fmt.Println(v)
	p.next.ServePipeline(v)
}

And then register you pipeline into the crawler via UsePipeline().

pipeline := func() antch.Pipeline {
    return func(next antch.PipelineHandler) antch.PipelineHandler {
        return &OutputPipeline{next}
    }
}
crawler := antch.NewCrawler()
crawler.UsePipeline(pipeline())