org.apache.nutch.scoring.similarity.util

Class LuceneTokenizer

java.lang.Object

org.apache.nutch.scoring.similarity.util.LuceneTokenizer

public class LuceneTokenizer
extends java.lang.Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	LuceneTokenizer.TokenizerType

Constructor Summary

Constructors

Constructor and Description
LuceneTokenizer(java.lang.String content, LuceneTokenizer.TokenizerType tokenizer, boolean useStopFilter, LuceneAnalyzerUtil.StemFilterType stemFilterType) Creates a tokenizer based on param values
LuceneTokenizer(java.lang.String content, LuceneTokenizer.TokenizerType tokenizer, java.util.List<java.lang.String> stopWords, boolean addToDefault, LuceneAnalyzerUtil.StemFilterType stemFilterType) Creates a tokenizer based on param values
LuceneTokenizer(java.lang.String content, LuceneTokenizer.TokenizerType tokenizer, LuceneAnalyzerUtil.StemFilterType stemFilterType, int mingram, int maxgram) Creates a tokenizer for the ngram model based on param values

Method Summary

Modifier and Type	Method and Description
TokenStream	getTokenStream() Returns the tokenStream created by the Tokenizer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LuceneTokenizer

public LuceneTokenizer(java.lang.String content,
                       LuceneTokenizer.TokenizerType tokenizer,
                       boolean useStopFilter,
                       LuceneAnalyzerUtil.StemFilterType stemFilterType)

Creates a tokenizer based on param values

Parameters:

content - - The text to tokenize

tokenizer - - the type of tokenizer to use CLASSIC or DEFAULT

useStopFilter - - if set to true the token stream will be filtered using default Lucene stopset

stemFilterType - - Type of stemming to perform

LuceneTokenizer

public LuceneTokenizer(java.lang.String content,
                       LuceneTokenizer.TokenizerType tokenizer,
                       java.util.List<java.lang.String> stopWords,
                       boolean addToDefault,
                       LuceneAnalyzerUtil.StemFilterType stemFilterType)

Creates a tokenizer based on param values

Parameters:

content - - The text to tokenize

tokenizer - - the type of tokenizer to use CLASSIC or DEFAULT

stopWords - - Provide a set of user defined stop words

addToDefault - - If set to true, the stopSet words will be added to the Lucene default stop set. If false, then only the user provided words will be used as the stop set

stemFilterType -

LuceneTokenizer

public LuceneTokenizer(java.lang.String content,
                       LuceneTokenizer.TokenizerType tokenizer,
                       LuceneAnalyzerUtil.StemFilterType stemFilterType,
                       int mingram,
                       int maxgram)

Creates a tokenizer for the ngram model based on param values

Parameters:

content - - The text to tokenize

tokenizer - - the type of tokenizer to use CLASSIC or DEFAULT

stemFilterType - - Type of stemming to perform

mingram - - Value of mingram for tokenizing

maxgram - - Value of maxgram for tokenizing

Method Detail

getTokenStream

public TokenStream getTokenStream()

Returns the tokenStream created by the Tokenizer

Returns: