org.apache.nutch.crawl

Class Injector

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.nutch.util.NutchTool

org.apache.nutch.crawl.Injector

All Implemented Interfaces:

Configurable, Tool

public class Injector
extends NutchTool
implements Tool

Injector takes a flat text file of URLs (or a folder containing text files) and merges ("injects") these URLs into the CrawlDb. Useful for bootstrapping a Nutch crawl. The URL files contain one URL per line, optionally followed by custom metadata separated by tabs with the metadata key separated from the corresponding value by '='.

Note, that some metadata keys are reserved:

nutch.score

allows to set a custom score for a specific URL

nutch.fetchInterval

allows to set a custom fetch interval for a specific URL

nutch.fetchInterval.fixed

allows to set a custom fetch interval for a specific URL that is not changed by AdaptiveFetchSchedule

Example:

  http://www.nutch.org/ \t nutch.score=10 \t nutch.fetchInterval=2592000 \t userType=open_source
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Injector.InjectMapper InjectMapper reads the CrawlDb seeds are injected into the plain-text seed files and parses each line into the URL and metadata.
static class	Injector.InjectReducer Combine multiple new entries for a url.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	nutchFetchIntervalMDName metadata key reserved for setting a custom fetchInterval for a specific URL
static java.lang.String	nutchFixedFetchIntervalMDName metadata key reserved for setting a fixed custom fetchInterval for a specific URL
static java.lang.String	nutchScoreMDName metadata key reserved for setting a custom score for a specific URL
static java.lang.String	URL_FILTER_NORMALIZE_ALL property to pass value of command-line option -filterNormalizeAll to mapper

Fields inherited from class org.apache.nutch.util.NutchTool

currentJob, currentJobNum, numJobs, results, status

Constructor Summary

Constructors

Constructor and Description
Injector()
Injector(Configuration conf)

Method Summary

Modifier and Type	Method and Description
void	inject(Path crawlDb, Path urlDir)
void	inject(Path crawlDb, Path urlDir, boolean overwrite, boolean update)
void	inject(Path crawlDb, Path urlDir, boolean overwrite, boolean update, boolean normalize, boolean filter, boolean filterNormalizeAll)
static void	main(java.lang.String[] args)
java.util.Map<java.lang.String,java.lang.Object>	run(java.util.Map<java.lang.String,java.lang.Object> args, java.lang.String crawlId) Used by the Nutch REST service
int	run(java.lang.String[] args)
void	usage()

Methods inherited from class org.apache.nutch.util.NutchTool

getProgress, getStatus, killJob, stopJob

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

URL_FILTER_NORMALIZE_ALL

public static final java.lang.String URL_FILTER_NORMALIZE_ALL

property to pass value of command-line option -filterNormalizeAll to mapper

See Also:

Constant Field Values

nutchScoreMDName

public static java.lang.String nutchScoreMDName

metadata key reserved for setting a custom score for a specific URL

nutchFetchIntervalMDName

public static java.lang.String nutchFetchIntervalMDName

metadata key reserved for setting a custom fetchInterval for a specific URL

nutchFixedFetchIntervalMDName

public static java.lang.String nutchFixedFetchIntervalMDName

metadata key reserved for setting a fixed custom fetchInterval for a specific URL

Constructor Detail

Injector

public Injector()

Injector

public Injector(Configuration conf)

Method Detail

inject

public void inject(Path crawlDb,
                   Path urlDir)
            throws java.io.IOException,
                   java.lang.ClassNotFoundException,
                   java.lang.InterruptedException

Throws:

java.io.IOException

java.lang.ClassNotFoundException

java.lang.InterruptedException

inject

public void inject(Path crawlDb,
                   Path urlDir,
                   boolean overwrite,
                   boolean update)
            throws java.io.IOException,
                   java.lang.ClassNotFoundException,
                   java.lang.InterruptedException

Throws:

java.io.IOException

java.lang.ClassNotFoundException

java.lang.InterruptedException

inject

public void inject(Path crawlDb,
                   Path urlDir,
                   boolean overwrite,
                   boolean update,
                   boolean normalize,
                   boolean filter,
                   boolean filterNormalizeAll)
            throws java.io.IOException,
                   java.lang.ClassNotFoundException,
                   java.lang.InterruptedException

Throws:

java.io.IOException

java.lang.ClassNotFoundException

java.lang.InterruptedException

usage

public void usage()

main

public static void main(java.lang.String[] args)
                 throws java.lang.Exception

Throws:

java.lang.Exception

run

public int run(java.lang.String[] args)
        throws java.lang.Exception

Specified by:

run in interface Tool

Throws:

java.lang.Exception

run

public java.util.Map<java.lang.String,java.lang.Object> run(java.util.Map<java.lang.String,java.lang.Object> args,
                                                            java.lang.String crawlId)
                                                     throws java.lang.Exception

Used by the Nutch REST service

Specified by:

run in class NutchTool

Throws:

java.lang.Exception