org.apache.nutch.tools.arc

Class ArcSegmentCreator

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.nutch.tools.arc.ArcSegmentCreator

All Implemented Interfaces:

Configurable, Tool

public class ArcSegmentCreator
extends Configured
implements Tool

The ArcSegmentCreator is a replacement for fetcher that will take arc files as input and produce a nutch segment as output.

Arc files are tars of compressed gzips which are produced by both the internet archive project and the grub distributed crawler project.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ArcSegmentCreator.ArcSegmentCreatorMapper

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	URL_VERSION

Constructor Summary

Constructors

Constructor and Description
ArcSegmentCreator()
ArcSegmentCreator(Configuration conf) Constructor that sets the job configuration.

Method Summary

Modifier and Type	Method and Description
void	close()
void	createSegments(Path arcFiles, Path segmentsOutDir) Creates the arc files to segments job.
static java.lang.String	generateSegmentName() Generates a random name for the segments.
static void	main(java.lang.String[] args)
int	run(java.lang.String[] args)

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

URL_VERSION

public static final java.lang.String URL_VERSION

See Also:

Constant Field Values

Constructor Detail

ArcSegmentCreator

public ArcSegmentCreator()

ArcSegmentCreator

public ArcSegmentCreator(Configuration conf)

Constructor that sets the job configuration.

Parameters:

conf -

Method Detail

generateSegmentName

public static java.lang.String generateSegmentName()

Generates a random name for the segments.

Returns:

The generated segment name.

close

public void close()

createSegments

public void createSegments(Path arcFiles,
                           Path segmentsOutDir)
                    throws java.io.IOException,
                           java.lang.InterruptedException,
                           java.lang.ClassNotFoundException

Creates the arc files to segments job.

Parameters:

arcFiles - The path to the directory holding the arc files

segmentsOutDir - The output directory for writing the segments

Throws:

java.io.IOException - If an IO error occurs while running the job.

java.lang.InterruptedException

java.lang.ClassNotFoundException

main

public static void main(java.lang.String[] args)
                 throws java.lang.Exception

Throws:

java.lang.Exception

run

public int run(java.lang.String[] args)
        throws java.lang.Exception

Specified by:

run in interface Tool

Throws:

java.lang.Exception