org.apache.nutch.parse

Class ParseUtil

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.nutch.parse.ParseUtil

All Implemented Interfaces:

Configurable

public class ParseUtil
extends Configured

A Utility class containing methods to simply perform parsing utilities such as iterating through a preferred list of Parsers to obtain Parse objects.

Author:

mattmann, Jérôme Charron, Sébastien Le Callonnec

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor and Description
ParseUtil(Configuration conf)

Method Summary

Methods

Modifier and Type	Method and Description
Configuration	getConf()
Parse	parse(String url, WebPage page) Performs a parse by iterating through a List of preferred Parsers until a successful parse is performed and a Parse object is returned.
void	process(String key, WebPage page) Parses given web page and stores parsed content within page.
void	setConf(Configuration conf)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

ParseUtil

public ParseUtil(Configuration conf)

Parameters:

conf -

Method Detail

getConf

public Configuration getConf()

Specified by:

getConf in interface Configurable

Overrides:

getConf in class Configured

setConf

public void setConf(Configuration conf)

Specified by:

setConf in interface Configurable

Overrides:

setConf in class Configured

parse

public Parse parse(String url,
          WebPage page)
            throws ParserNotFound,
                   ParseException

Performs a parse by iterating through a List of preferred Parsers until a successful parse is performed and a Parse object is returned. If the parse is unsuccessful, a message is logged to the WARNING level, and an empty parse is returned.

Throws:

ParserNotFound - If there is no suitable parser found.

ParseException - If there is an error parsing.

process

public void process(String key,
           WebPage page)

Parses given web page and stores parsed content within page. Puts a meta-redirect to outlinks.

Parameters:

key -

page -