org.apache.nutch.crawl

Class CrawlDbFilter

java.lang.Object

org.apache.hadoop.mapreduce.Mapper<Text,CrawlDatum,Text,CrawlDatum>

org.apache.nutch.crawl.CrawlDbFilter

public class CrawlDbFilter
extends Mapper<Text,CrawlDatum,Text,CrawlDatum>

This class provides a way to separate the URL normalization and filtering steps from the rest of CrawlDb manipulation code.

Author:

Andrzej Bialecki

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.Mapper

Mapper.Context

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	URL_FILTERING
static java.lang.String	URL_NORMALIZING
static java.lang.String	URL_NORMALIZING_SCOPE

Constructor Summary

Constructors

Constructor and Description
CrawlDbFilter()

Method Summary

Modifier and Type	Method and Description
void	close()
void	map(Text key, CrawlDatum value, Mapper.Context context)
void	setup(Mapper.Context context)

Methods inherited from class org.apache.hadoop.mapreduce.Mapper

cleanup, run

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

URL_FILTERING

public static final java.lang.String URL_FILTERING

See Also:

Constant Field Values

URL_NORMALIZING

public static final java.lang.String URL_NORMALIZING

See Also:

Constant Field Values

URL_NORMALIZING_SCOPE

public static final java.lang.String URL_NORMALIZING_SCOPE

See Also:

Constant Field Values

Constructor Detail

CrawlDbFilter

public CrawlDbFilter()

Method Detail

setup

public void setup(Mapper.Context context)

Overrides:

setup in class Mapper<Text,CrawlDatum,Text,CrawlDatum>

close

public void close()

map

public void map(Text key,
                CrawlDatum value,
                Mapper.Context context)
         throws java.io.IOException,
                java.lang.InterruptedException

Overrides:

map in class Mapper<Text,CrawlDatum,Text,CrawlDatum>

Throws:

java.io.IOException

java.lang.InterruptedException