org.apache.nutch.parsefilter.naivebayes

Class NaiveBayesParseFilter

java.lang.Object

org.apache.nutch.parsefilter.naivebayes.NaiveBayesParseFilter

All Implemented Interfaces:

Configurable, HtmlParseFilter, Pluggable

public class NaiveBayesParseFilter
extends java.lang.Object
implements HtmlParseFilter

Html Parse filter that classifies the outlinks from the parseresult as relevant or irrelevant based on the parseText's relevancy (using a training file where you can give positive and negative example texts see the description of parsefilter.naivebayes.trainfile) and if found irrelevant it gives the link a second chance if it contains any of the words from the list given in parsefilter.naivebayes.wordlist. CAUTION: Set the parser.timeout to -1 or a bigger value than 30, when using this classifier.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DICTFILE_MODELFILTER
static java.lang.String	TRAINFILE_MODELFILTER

Fields inherited from interface org.apache.nutch.parse.HtmlParseFilter

X_POINT_ID

Constructor Summary

Constructors

Constructor and Description
NaiveBayesParseFilter()

Method Summary

Modifier and Type	Method and Description
boolean	classify(java.lang.String text)
boolean	containsWord(java.lang.String url, java.util.ArrayList<java.lang.String> wordlist)
ParseResult	filter(Content content, ParseResult parseResult, HTMLMetaTags metaTags, org.w3c.dom.DocumentFragment doc) Adds metadata or otherwise modifies a parse of HTML content, given the DOM tree of a page.
boolean	filterParse(java.lang.String text)
boolean	filterUrl(java.lang.String url)
Configuration	getConf()
void	setConf(Configuration conf)
void	train()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TRAINFILE_MODELFILTER

public static final java.lang.String TRAINFILE_MODELFILTER

See Also:

Constant Field Values

DICTFILE_MODELFILTER

public static final java.lang.String DICTFILE_MODELFILTER

See Also:

Constant Field Values

Constructor Detail

NaiveBayesParseFilter

public NaiveBayesParseFilter()

Method Detail

filterParse

public boolean filterParse(java.lang.String text)

filterUrl

public boolean filterUrl(java.lang.String url)

classify

public boolean classify(java.lang.String text)
                 throws java.io.IOException

Throws:

java.io.IOException

train

public void train()
           throws java.lang.Exception

Throws:

java.lang.Exception

containsWord

public boolean containsWord(java.lang.String url,
                            java.util.ArrayList<java.lang.String> wordlist)

setConf

public void setConf(Configuration conf)

Specified by:

setConf in interface Configurable

getConf

public Configuration getConf()

Specified by:

getConf in interface Configurable

filter

public ParseResult filter(Content content,
                          ParseResult parseResult,
                          HTMLMetaTags metaTags,
                          org.w3c.dom.DocumentFragment doc)

Description copied from interface: HtmlParseFilter

Adds metadata or otherwise modifies a parse of HTML content, given the DOM tree of a page.

Specified by:

filter in interface HtmlParseFilter