org.apache.lucene.analysis.opennlp

## Class OpenNLPTokenizer

- java.lang.Object
- - org.apache.lucene.util.AttributeSource
  - - org.apache.lucene.analysis.TokenStream
    - - org.apache.lucene.analysis.Tokenizer
      - - org.apache.lucene.analysis.util.SegmentingTokenizerBase
        - - org.apache.lucene.analysis.opennlp.OpenNLPTokenizer

All Implemented Interfaces:
  Closeable, AutoCloseable

    public final class OpenNLPTokenizer
    extends SegmentingTokenizerBase

Run OpenNLP SentenceDetector and Tokenizer. The last token in each sentence is marked by setting the `EOS_FLAG_BIT` in the FlagsAttribute; following filters can use this information to apply operations to tokens one sentence at a time.

### Nested Class Summary

### Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

`AttributeSource.State`

### Field Summary

Fields
`static int` `EOS_FLAG_BIT`

### Fields inherited from class org.apache.lucene.analysis.util.SegmentingTokenizerBase

`buffer, BUFFERMAX, offset`

### Fields inherited from class org.apache.lucene.analysis.Tokenizer

`input`

### Fields inherited from class org.apache.lucene.analysis.TokenStream

`DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

### Constructor Summary

Constructors
Constructor and Description
`OpenNLPTokenizer(AttributeFactory factory, NLPSentenceDetectorOp sentenceOp, NLPTokenizerOp tokenizerOp)`

### Method Summary

All Methods Instance Methods Concrete Methods
`void` `close()`
`protected boolean` `incrementWord()`
`void` `reset()`
`protected void` `setNextSentence(int sentenceStart, int sentenceEnd)`

### Methods inherited from class org.apache.lucene.analysis.util.SegmentingTokenizerBase

`end, incrementToken, isSafeEnd`

### Methods inherited from class org.apache.lucene.analysis.Tokenizer

`correctOffset, setReader`

### Methods inherited from class org.apache.lucene.util.AttributeSource

`addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

### Methods inherited from class java.lang.Object

`clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

### Field Detail

#### EOS_FLAG_BIT

    public static int EOS_FLAG_BIT

### Constructor Detail

#### OpenNLPTokenizer

    public OpenNLPTokenizer(AttributeFactory factory,
                            NLPSentenceDetectorOp sentenceOp,
                            NLPTokenizerOp tokenizerOp)
                     throws IOException

Throws:
  `IOException`

### Method Detail

#### close

    public void close()
               throws IOException

Specified by:
  `close` in interface `Closeable`
Specified by:
  `close` in interface `AutoCloseable`
Overrides:
  `close` in class `Tokenizer`
Throws:
  `IOException`

#### setNextSentence

    protected void setNextSentence(int sentenceStart,
                                   int sentenceEnd)

Specified by:
  `setNextSentence` in class `SegmentingTokenizerBase`

#### incrementWord

    protected boolean incrementWord()

Specified by:
  `incrementWord` in class `SegmentingTokenizerBase`

#### reset

    public void reset()
               throws IOException

Overrides:
  `reset` in class `SegmentingTokenizerBase`
Throws:
  `IOException`