org.apache.lucene.analysis.pattern

## Class SimplePatternTokenizer

- java.lang.Object
- - org.apache.lucene.util.AttributeSource
  - - org.apache.lucene.analysis.TokenStream
    - - org.apache.lucene.analysis.Tokenizer
      - - org.apache.lucene.analysis.pattern.SimplePatternTokenizer

All Implemented Interfaces:
  Closeable, AutoCloseable

    public final class SimplePatternTokenizer
    extends Tokenizer

This tokenizer uses a Lucene `RegExp` or (expert usage) a pre-built determinized `Automaton`, to locate tokens. The regexp syntax is more limited than `PatternTokenizer`, but the tokenization is quite a bit faster. The provided regex should match valid token characters (not token separator characters, like `String.split`). The matching is greedy: the longest match at a given start point will be the next token. Empty string tokens are never produced.

WARNING: This API is experimental and might change in incompatible ways in the next release.

### Nested Class Summary

### Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

`AttributeSource.State`

### Field Summary

### Fields inherited from class org.apache.lucene.analysis.Tokenizer

`input`

### Fields inherited from class org.apache.lucene.analysis.TokenStream

`DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

### Constructor Summary

Constructors
Constructor and Description
`SimplePatternTokenizer(AttributeFactory factory, Automaton dfa)` Runs a pre-built automaton.
`SimplePatternTokenizer(AttributeFactory factory, String regexp, int maxDeterminizedStates)` See `RegExp` for the accepted syntax.
`SimplePatternTokenizer(Automaton dfa)` Runs a pre-built automaton.
`SimplePatternTokenizer(String regexp)` See `RegExp` for the accepted syntax.

### Method Summary

All Methods Instance Methods Concrete Methods
`void` `end()`
`boolean` `incrementToken()`
`void` `reset()`

### Methods inherited from class org.apache.lucene.analysis.Tokenizer

`close, correctOffset, setReader`

### Methods inherited from class org.apache.lucene.util.AttributeSource

`addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

### Methods inherited from class java.lang.Object

`clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

### Constructor Detail

#### SimplePatternTokenizer

    public SimplePatternTokenizer(String regexp)

See `RegExp` for the accepted syntax.

#### SimplePatternTokenizer

    public SimplePatternTokenizer(Automaton dfa)

Runs a pre-built automaton.

#### SimplePatternTokenizer

    public SimplePatternTokenizer(AttributeFactory factory,
                                  String regexp,
                                  int maxDeterminizedStates)

See `RegExp` for the accepted syntax.

#### SimplePatternTokenizer

    public SimplePatternTokenizer(AttributeFactory factory,
                                  Automaton dfa)

Runs a pre-built automaton.

### Method Detail

#### incrementToken

    public boolean incrementToken()
                           throws IOException

Specified by:
  `incrementToken` in class `TokenStream`
Throws:
  `IOException`

#### end

    public void end()
             throws IOException

Overrides:
  `end` in class `TokenStream`
Throws:
  `IOException`

#### reset

    public void reset()
               throws IOException

Overrides:
  `reset` in class `Tokenizer`
Throws:
  `IOException`