## Uses of Class
org.apache.lucene.analysis.Tokenizer

- Packages that use Tokenizer
  Package | Description
  --- | ---
  org.apache.lucene.analysis | Text analysis.
  org.apache.lucene.analysis.standard | Fast, general-purpose grammar-based tokenizer `StandardTokenizer` implements the Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.

- - ### Uses of Tokenizer in org.apache.lucene.analysis

    Fields in org.apache.lucene.analysis declared as Tokenizer
    Modifier and Type | Field and Description
    --- | ---
    `protected Tokenizer` | Analyzer.TokenStreamComponents.`source` Original source of the tokens.

    Methods in org.apache.lucene.analysis that return Tokenizer
    Modifier and Type | Method and Description
    --- | ---
    `Tokenizer` | Analyzer.TokenStreamComponents.`getTokenizer()` Returns the component's `Tokenizer`

    Constructors in org.apache.lucene.analysis with parameters of type Tokenizer
    Constructor and Description
    `TokenStreamComponents(Tokenizer source)` Creates a new `Analyzer.TokenStreamComponents` instance.
    `TokenStreamComponents(Tokenizer source, TokenStream result)` Creates a new `Analyzer.TokenStreamComponents` instance.

  - ### Uses of Tokenizer in org.apache.lucene.analysis.standard

    Subclasses of Tokenizer in org.apache.lucene.analysis.standard
    Modifier and Type | Class and Description
    --- | ---
    `class` | `StandardTokenizer` A grammar-based tokenizer constructed with JFlex.

Copyright © 2000-2019 Apache Software Foundation. All Rights Reserved.