org.apache.lucene.analysis.pattern

## Class PatternTokenizer

- java.lang.Object
- - org.apache.lucene.util.AttributeSource
  - - org.apache.lucene.analysis.TokenStream
    - - org.apache.lucene.analysis.Tokenizer
      - - org.apache.lucene.analysis.pattern.PatternTokenizer

- All Implemented Interfaces:
    Closeable, AutoCloseable

      public final class PatternTokenizer
      extends Tokenizer

  This tokenizer uses regex pattern matching to construct distinct tokens for the input stream. It takes two arguments: "pattern" and "group".
  - "pattern" is the regular expression.
  - "group" says which group to extract into tokens.

  group=-1 (the default) is equivalent to "split". In this case, the tokens will be equivalent to the output from (without empty tokens): `String.split(java.lang.String)`

  Using group >= 0 selects the matching group as the token. For example, if you have:

        pattern = \'([^\']+)\'
        group = 0
        input = aaa 'bbb' 'ccc'

  the output will be two tokens: 'bbb' and 'ccc' (including the ' marks). With the same input but using group=1, the output would be: bbb and ccc (no ' marks)

  NOTE: This Tokenizer does not output tokens that are of zero length.

  See Also:
    `Pattern`

- - ### Nested Class Summary

    - ### Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

      `AttributeSource.State`

  - ### Field Summary

    - ### Fields inherited from class org.apache.lucene.analysis.Tokenizer

      `input`

    - ### Fields inherited from class org.apache.lucene.analysis.TokenStream

      `DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `PatternTokenizer(AttributeFactory factory, Pattern pattern, int group)` creates a new PatternTokenizer returning tokens from group (-1 for split functionality)
    `PatternTokenizer(Pattern pattern, int group)` creates a new PatternTokenizer returning tokens from group (-1 for split functionality)

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `close()`
    `void` | `end()`
    `boolean` | `incrementToken()`
    `void` | `reset()`

    - ### Methods inherited from class org.apache.lucene.analysis.Tokenizer

      `correctOffset, setReader`

    - ### Methods inherited from class org.apache.lucene.util.AttributeSource

      `addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

    - ### Methods inherited from class java.lang.Object

      `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- - ### Constructor Detail

    - #### PatternTokenizer

          public PatternTokenizer(Pattern pattern,
                                  int group)

      creates a new PatternTokenizer returning tokens from group (-1 for split functionality)

    - #### PatternTokenizer

          public PatternTokenizer(AttributeFactory factory,
                                  Pattern pattern,
                                  int group)

      creates a new PatternTokenizer returning tokens from group (-1 for split functionality)

  - ### Method Detail

    - #### incrementToken

          public boolean incrementToken()

      Specified by:
        `incrementToken` in class `TokenStream`

    - #### end

          public void end()
                   throws IOException

      Overrides:
        `end` in class `TokenStream`
      Throws:
        `IOException`

    - #### close

          public void close()
                     throws IOException

      Specified by:
        `close` in interface `Closeable`
      Specified by:
        `close` in interface `AutoCloseable`
      Overrides:
        `close` in class `Tokenizer`
      Throws:
        `IOException`

    - #### reset

          public void reset()
                     throws IOException

      Overrides:
        `reset` in class `Tokenizer`
      Throws:
        `IOException`

Copyright © 2000-2024 Apache Software Foundation. All Rights Reserved.