org.apache.nutch.hostdb

Class UpdateHostDbReducer

java.lang.Object

org.apache.nutch.hostdb.UpdateHostDbReducer

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, JobConfigurable, Reducer<Text,NutchWritable,Text,HostDatum>

public class UpdateHostDbReducer
extends java.lang.Object
implements Reducer<Text,NutchWritable,Text,HostDatum>

Field Summary

Fields

Modifier and Type	Field and Description
protected static boolean	checkFailed
protected static boolean	checkKnown
protected static boolean	checkNew
protected java.util.concurrent.ThreadPoolExecutor	executor
protected static boolean	force
protected static long	now
protected static java.lang.String[]	numericFields
protected static Text[]	numericFieldWritables
protected java.lang.Integer	numResolverThreads
protected static int[]	percentiles
protected static java.lang.Integer	purgeFailedHostsThreshold
protected java.util.concurrent.BlockingQueue<java.lang.Runnable>	queue
protected static java.lang.Integer	recheckInterval
protected ResolverThread	resolverThread
protected static java.lang.String[]	stringFields
protected static Text[]	stringFieldWritables

Constructor Summary

Constructors

Constructor and Description
UpdateHostDbReducer()

Method Summary

Modifier and Type	Method and Description
void	close() Shut down all running threads and wait for completion.
void	configure(JobConf job) Configures the thread pool and prestarts all resolver threads.
protected boolean	isEligibleForCheck(HostDatum datum) Determines whether a record is eligible for recheck.
void	reduce(Text key, java.util.Iterator<NutchWritable> values, OutputCollector<Text,HostDatum> output, Reporter reporter)
protected boolean	shouldCheck(HostDatum datum) Determines whether a record should be checked.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

resolverThread

protected ResolverThread resolverThread

numResolverThreads

protected java.lang.Integer numResolverThreads

purgeFailedHostsThreshold

protected static java.lang.Integer purgeFailedHostsThreshold

recheckInterval

protected static java.lang.Integer recheckInterval

checkFailed

protected static boolean checkFailed

checkNew

protected static boolean checkNew

checkKnown

protected static boolean checkKnown

force

protected static boolean force

now

protected static long now

numericFields

protected static java.lang.String[] numericFields

stringFields

protected static java.lang.String[] stringFields

percentiles

protected static int[] percentiles

numericFieldWritables

protected static Text[] numericFieldWritables

stringFieldWritables

protected static Text[] stringFieldWritables

queue

protected java.util.concurrent.BlockingQueue<java.lang.Runnable> queue

executor

protected java.util.concurrent.ThreadPoolExecutor executor

Constructor Detail

UpdateHostDbReducer

public UpdateHostDbReducer()

Method Detail

configure

public void configure(JobConf job)

Configures the thread pool and prestarts all resolver threads.

Specified by:

configure in interface JobConfigurable

Parameters:

job -

reduce

public void reduce(Text key,
                   java.util.Iterator<NutchWritable> values,
                   OutputCollector<Text,HostDatum> output,
                   Reporter reporter)
            throws java.io.IOException

Specified by:

reduce in interface Reducer<Text,NutchWritable,Text,HostDatum>

Throws:

java.io.IOException

shouldCheck

protected boolean shouldCheck(HostDatum datum)

Determines whether a record should be checked.

Parameters:

datum -

Returns:

boolean

isEligibleForCheck

protected boolean isEligibleForCheck(HostDatum datum)

Determines whether a record is eligible for recheck.

Parameters:

datum -

Returns:

boolean

close

public void close()

Shut down all running threads and wait for completion.

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable