org.apache.nutch.hostdb

Class UpdateHostDbMapper

java.lang.Object

org.apache.nutch.hostdb.UpdateHostDbMapper

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, JobConfigurable, Mapper<Text,Writable,Text,NutchWritable>

public class UpdateHostDbMapper
extends java.lang.Object
implements Mapper<Text,Writable,Text,NutchWritable>

Mapper ingesting HostDB and CrawlDB entries. Additionally it can also read host score info from a plain text key/value file generated by the Webgraph's NodeDumper tool.

Field Summary

Fields

Modifier and Type	Field and Description
protected java.lang.String[]	args
protected java.lang.String	buffer
protected CrawlDatum	crawlDatum
protected boolean	filter
protected URLFilters	filters
protected Text	host
protected HostDatum	hostDatum
protected boolean	normalize
protected URLNormalizers	normalizers
protected boolean	readingCrawlDb
protected java.lang.String	reprUrl

Constructor Summary

Constructors

Constructor and Description
UpdateHostDbMapper()

Method Summary

Modifier and Type	Method and Description
void	close()
void	configure(JobConf job)
protected java.lang.String	filterNormalize(java.lang.String url) Filters and or normalizes the input URL
void	map(Text key, Writable value, OutputCollector<Text,NutchWritable> output, Reporter reporter) Mapper ingesting records from the HostDB, CrawlDB and plaintext host scores file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

host

protected Text host

hostDatum

protected HostDatum hostDatum

crawlDatum

protected CrawlDatum crawlDatum

reprUrl

protected java.lang.String reprUrl

buffer

protected java.lang.String buffer

args

protected java.lang.String[] args

filter

protected boolean filter

normalize

protected boolean normalize

readingCrawlDb

protected boolean readingCrawlDb

filters

protected URLFilters filters

normalizers

protected URLNormalizers normalizers

Constructor Detail

UpdateHostDbMapper

public UpdateHostDbMapper()

Method Detail

close

public void close()

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

configure

public void configure(JobConf job)

Specified by:

configure in interface JobConfigurable

Parameters:

job -

filterNormalize

protected java.lang.String filterNormalize(java.lang.String url)

Filters and or normalizes the input URL

Parameters:

url -

Returns:

String

map

public void map(Text key,
                Writable value,
                OutputCollector<Text,NutchWritable> output,
                Reporter reporter)
         throws java.io.IOException

Mapper ingesting records from the HostDB, CrawlDB and plaintext host scores file. Statistics and scores are passed on.

Specified by:

map in interface Mapper<Text,Writable,Text,NutchWritable>

Parameters:

key -

value -

output -

reporter -

Throws:

java.io.IOException