org.apache.nutch.crawl

Class Generator.Selector

java.lang.Object

org.apache.nutch.crawl.Generator.Selector

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, JobConfigurable, Mapper<Text,CrawlDatum,FloatWritable,Generator.SelectorEntry>, Partitioner<FloatWritable,Writable>, Reducer<FloatWritable,Generator.SelectorEntry,FloatWritable,Generator.SelectorEntry>

Enclosing class:

Generator

public static class Generator.Selector
extends java.lang.Object
implements Mapper<Text,CrawlDatum,FloatWritable,Generator.SelectorEntry>, Partitioner<FloatWritable,Writable>, Reducer<FloatWritable,Generator.SelectorEntry,FloatWritable,Generator.SelectorEntry>

Selects entries due for fetch.

Constructor Summary

Constructors

Constructor and Description
Selector()

Method Summary

Modifier and Type	Method and Description
void	close()
void	configure(JobConf job)
int	getPartition(FloatWritable key, Writable value, int numReduceTasks) Partition by host / domain or IP.
void	map(Text key, CrawlDatum value, OutputCollector<FloatWritable,Generator.SelectorEntry> output, Reporter reporter) Select and invert subset due for fetch.
void	open()
void	reduce(FloatWritable key, java.util.Iterator<Generator.SelectorEntry> values, OutputCollector<FloatWritable,Generator.SelectorEntry> output, Reporter reporter) Collect until limit is reached.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Selector

public Selector()

Method Detail

configure

public void configure(JobConf job)

Specified by:

configure in interface JobConfigurable

open

public void open()

close

public void close()

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

map

public void map(Text key,
                CrawlDatum value,
                OutputCollector<FloatWritable,Generator.SelectorEntry> output,
                Reporter reporter)
         throws java.io.IOException

Select and invert subset due for fetch.

Specified by:

map in interface Mapper<Text,CrawlDatum,FloatWritable,Generator.SelectorEntry>

Throws:

java.io.IOException

getPartition

public int getPartition(FloatWritable key,
                        Writable value,
                        int numReduceTasks)

Partition by host / domain or IP.

Specified by:

getPartition in interface Partitioner<FloatWritable,Writable>

reduce

public void reduce(FloatWritable key,
                   java.util.Iterator<Generator.SelectorEntry> values,
                   OutputCollector<FloatWritable,Generator.SelectorEntry> output,
                   Reporter reporter)
            throws java.io.IOException

Collect until limit is reached.

Specified by:

reduce in interface Reducer<FloatWritable,Generator.SelectorEntry,FloatWritable,Generator.SelectorEntry>

Throws:

java.io.IOException