org.apache.lucene.analysis

## Class MockTokenizer

- java.lang.Object
- - org.apache.lucene.util.AttributeSource
  - - org.apache.lucene.analysis.TokenStream
    - - org.apache.lucene.analysis.Tokenizer
      - - org.apache.lucene.analysis.MockTokenizer

- All Implemented Interfaces:
    Closeable, AutoCloseable

      public class MockTokenizer
      extends Tokenizer

  Tokenizer for testing.

  This tokenizer is a replacement for `WHITESPACE`, `SIMPLE`, and `KEYWORD` tokenizers. If you are writing a component such as a TokenFilter, it's a great idea to test it wrapping this tokenizer instead for extra checks. This tokenizer has the following behavior:

  - An internal state-machine is used for checking consumer consistency. These checks can be disabled with `setEnableChecks(boolean)`.
  - For convenience, optionally lowercases terms that it outputs.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static int` | `DEFAULT_MAX_TOKEN_LENGTH`
    `static CharacterRunAutomaton` | `KEYWORD` Acts Similar to KeywordTokenizer.
    `static CharacterRunAutomaton` | `SIMPLE` Acts like LetterTokenizer.
    `static CharacterRunAutomaton` | `WHITESPACE` Acts Similar to WhitespaceTokenizer

    - ### Fields inherited from class org.apache.lucene.analysis.Tokenizer

      `input`

    - ### Fields inherited from class org.apache.lucene.analysis.TokenStream

      `DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `MockTokenizer()` Calls `MockTokenizer(Reader, WHITESPACE, true)`
    `MockTokenizer(AttributeFactory factory)` Calls `MockTokenizer(AttributeFactory, Reader, WHITESPACE, true)`
    `MockTokenizer(AttributeFactory factory, CharacterRunAutomaton runAutomaton, boolean lowerCase)`
    `MockTokenizer(AttributeFactory factory, CharacterRunAutomaton runAutomaton, boolean lowerCase, int maxTokenLength)`
    `MockTokenizer(CharacterRunAutomaton runAutomaton, boolean lowerCase)`
    `MockTokenizer(CharacterRunAutomaton runAutomaton, boolean lowerCase, int maxTokenLength)`

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `close()`
    `void` | `end()`
    `boolean` | `incrementToken()`
    `protected boolean` | `isTokenChar(int c)`
    `protected int` | `normalize(int c)`
    `protected int` | `readChar()`
    `protected int` | `readCodePoint()`
    `void` | `reset()`
    `void` | `setEnableChecks(boolean enableChecks)` Toggle consumer workflow checking: if your test consumes tokenstreams normally you should leave this enabled.

    - ### Methods inherited from class org.apache.lucene.analysis.Tokenizer

      `correctOffset, setReader`

    - ### Methods inherited from class org.apache.lucene.util.AttributeSource

      `addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

    - ### Methods inherited from class java.lang.Object

      `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- - ### Field Detail

    - #### WHITESPACE

          public static final CharacterRunAutomaton WHITESPACE

      Acts Similar to WhitespaceTokenizer

    - #### KEYWORD

          public static final CharacterRunAutomaton KEYWORD

      Acts Similar to KeywordTokenizer. TODO: Keyword returns an "empty" token for an empty reader...

    - #### SIMPLE

          public static final CharacterRunAutomaton SIMPLE

      Acts like LetterTokenizer.

    - #### DEFAULT_MAX_TOKEN_LENGTH

          public static final int DEFAULT_MAX_TOKEN_LENGTH

      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### MockTokenizer

          public MockTokenizer(AttributeFactory factory,
                               CharacterRunAutomaton runAutomaton,
                               boolean lowerCase,
                               int maxTokenLength)

    - #### MockTokenizer

          public MockTokenizer(CharacterRunAutomaton runAutomaton,
                               boolean lowerCase,
                               int maxTokenLength)

    - #### MockTokenizer

          public MockTokenizer(CharacterRunAutomaton runAutomaton,
                               boolean lowerCase)

    - #### MockTokenizer

          public MockTokenizer()

      Calls `MockTokenizer(Reader, WHITESPACE, true)`

    - #### MockTokenizer

          public MockTokenizer(AttributeFactory factory,
                               CharacterRunAutomaton runAutomaton,
                               boolean lowerCase)

    - #### MockTokenizer

          public MockTokenizer(AttributeFactory factory)

      Calls `MockTokenizer(AttributeFactory, Reader, WHITESPACE, true)`

  - ### Method Detail

    - #### incrementToken

          public final boolean incrementToken()
                                       throws IOException

      Specified by:
        `incrementToken` in class `TokenStream`
      Throws:
        `IOException`

    - #### readCodePoint

          protected int readCodePoint()
                               throws IOException

      Throws:
        `IOException`

    - #### readChar

          protected int readChar()
                          throws IOException

      Throws:
        `IOException`

    - #### isTokenChar

          protected boolean isTokenChar(int c)

    - #### normalize

          protected int normalize(int c)

    - #### reset

          public void reset()
                     throws IOException

      Overrides:
        `reset` in class `Tokenizer`
      Throws:
        `IOException`

    - #### close

          public void close()
                     throws IOException

      Specified by:
        `close` in interface `Closeable`
      Specified by:
        `close` in interface `AutoCloseable`
      Overrides:
        `close` in class `Tokenizer`
      Throws:
        `IOException`

    - #### end

          public void end()
                   throws IOException

      Overrides:
        `end` in class `TokenStream`
      Throws:
        `IOException`

    - #### setEnableChecks

          public void setEnableChecks(boolean enableChecks)

      Toggle consumer workflow checking: if your test consumes tokenstreams normally you should leave this enabled.

Copyright © 2000-2024 Apache Software Foundation. All Rights Reserved.