org.apache.nutch.crawl

Class URLPartitioner

java.lang.Object

org.apache.nutch.crawl.URLPartitioner

All Implemented Interfaces:

JobConfigurable, Partitioner<Text,Writable>

public class URLPartitioner
extends java.lang.Object
implements Partitioner<Text,Writable>

Partition urls by host, domain name or IP depending on the value of the parameter 'partition.url.mode' which can be 'byHost', 'byDomain' or 'byIP'

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	PARTITION_MODE_DOMAIN
static java.lang.String	PARTITION_MODE_HOST
static java.lang.String	PARTITION_MODE_IP
static java.lang.String	PARTITION_MODE_KEY

Constructor Summary

Constructors

Constructor and Description
URLPartitioner()

Method Summary

Modifier and Type	Method and Description
void	close()
void	configure(JobConf job)
int	getPartition(Text key, Writable value, int numReduceTasks) Hash by host or domain name or IP address.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PARTITION_MODE_KEY

public static final java.lang.String PARTITION_MODE_KEY

See Also:

Constant Field Values

PARTITION_MODE_HOST

public static final java.lang.String PARTITION_MODE_HOST

See Also:

Constant Field Values

PARTITION_MODE_DOMAIN

public static final java.lang.String PARTITION_MODE_DOMAIN

See Also:

Constant Field Values

PARTITION_MODE_IP

public static final java.lang.String PARTITION_MODE_IP

See Also:

Constant Field Values

Constructor Detail

URLPartitioner

public URLPartitioner()

Method Detail

configure

public void configure(JobConf job)

Specified by:

configure in interface JobConfigurable

close

public void close()

getPartition

public int getPartition(Text key,
                        Writable value,
                        int numReduceTasks)

Hash by host or domain name or IP address.

Specified by:

getPartition in interface Partitioner<Text,Writable>