org.apache.nutch.crawl

Class URLPartitioner

java.lang.Object

org.apache.nutch.crawl.URLPartitioner

All Implemented Interfaces:

Configurable

public class URLPartitioner
extends Object
implements Configurable

Partition urls by host, domain name or IP depending on the value of the parameter 'partition.url.mode' which can be 'byHost', 'byDomain' or 'byIP'

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	URLPartitioner.FetchEntryPartitioner
static class	URLPartitioner.SelectorEntryPartitioner

Field Summary

Fields

Modifier and Type	Field and Description
static String	PARTITION_MODE_DOMAIN
static String	PARTITION_MODE_HOST
static String	PARTITION_MODE_IP
static String	PARTITION_MODE_KEY
static String	PARTITION_URL_SEED

Constructor Summary

Constructors

Constructor and Description
URLPartitioner()

Method Summary

Methods

Modifier and Type	Method and Description
Configuration	getConf()
int	getPartition(String urlString, int numReduceTasks)
void	setConf(Configuration conf)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PARTITION_MODE_KEY

public static final String PARTITION_MODE_KEY

See Also:

Constant Field Values

PARTITION_MODE_HOST

public static final String PARTITION_MODE_HOST

See Also:

Constant Field Values

PARTITION_MODE_DOMAIN

public static final String PARTITION_MODE_DOMAIN

See Also:

Constant Field Values

PARTITION_MODE_IP

public static final String PARTITION_MODE_IP

See Also:

Constant Field Values

PARTITION_URL_SEED

public static final String PARTITION_URL_SEED

See Also:

Constant Field Values

Constructor Detail

URLPartitioner

public URLPartitioner()

Method Detail

getConf

public Configuration getConf()

Specified by:

getConf in interface Configurable

setConf

public void setConf(Configuration conf)

Specified by:

setConf in interface Configurable

getPartition

public int getPartition(String urlString,
               int numReduceTasks)