org.apache.lucene.analysis.th

## Class ThaiTokenizer

- java.lang.Object
- - org.apache.lucene.util.AttributeSource
  - - org.apache.lucene.analysis.TokenStream
    - - org.apache.lucene.analysis.Tokenizer
      - - org.apache.lucene.analysis.util.SegmentingTokenizerBase
        - - org.apache.lucene.analysis.th.ThaiTokenizer

All Implemented Interfaces:
  Closeable, AutoCloseable

    public class ThaiTokenizer
    extends SegmentingTokenizerBase

Tokenizer that use `BreakIterator` to tokenize Thai text.

WARNING: this tokenizer may not be supported by all JREs. It is known to work with Sun/Oracle and Harmony JREs. If your application needs to be fully portable, consider using ICUTokenizer instead, which uses an ICU Thai BreakIterator that will always be available.

### Nested Class Summary

### Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

`AttributeSource.State`

### Field Summary

Fields
`static boolean` `DBBI_AVAILABLE` True if the JRE supports a working dictionary-based breakiterator for Thai.

### Fields inherited from class org.apache.lucene.analysis.util.SegmentingTokenizerBase

`buffer, BUFFERMAX, offset`

### Fields inherited from class org.apache.lucene.analysis.Tokenizer

`input`

### Fields inherited from class org.apache.lucene.analysis.TokenStream

`DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

### Constructor Summary

Constructors
Constructor and Description
`ThaiTokenizer()` Creates a new ThaiTokenizer
`ThaiTokenizer(AttributeFactory factory)` Creates a new ThaiTokenizer, supplying the AttributeFactory

### Method Summary

All Methods Instance Methods Concrete Methods
`protected boolean` `incrementWord()` Returns true if another word is available
`protected void` `setNextSentence(int sentenceStart, int sentenceEnd)` Provides the next input sentence for analysis

### Methods inherited from class org.apache.lucene.analysis.util.SegmentingTokenizerBase

`end, incrementToken, isSafeEnd, reset`

### Methods inherited from class org.apache.lucene.analysis.Tokenizer

`close, correctOffset, setReader`

### Methods inherited from class org.apache.lucene.util.AttributeSource

`addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

### Methods inherited from class java.lang.Object

`clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

### Field Detail

#### DBBI_AVAILABLE

    public static final boolean DBBI_AVAILABLE

True if the JRE supports a working dictionary-based breakiterator for Thai. If this is false, this tokenizer will not work at all!

### Constructor Detail

#### ThaiTokenizer

    public ThaiTokenizer()

Creates a new ThaiTokenizer

#### ThaiTokenizer

    public ThaiTokenizer(AttributeFactory factory)

Creates a new ThaiTokenizer, supplying the AttributeFactory

### Method Detail

#### setNextSentence

    protected void setNextSentence(int sentenceStart,
                                   int sentenceEnd)

Description copied from class: `SegmentingTokenizerBase`
Provides the next input sentence for analysis

Specified by:
  `setNextSentence` in class `SegmentingTokenizerBase`

#### incrementWord

    protected boolean incrementWord()

Description copied from class: `SegmentingTokenizerBase`
Returns true if another word is available

Specified by:
  `incrementWord` in class `SegmentingTokenizerBase`