## Uses of Class
org.apache.lucene.analysis.Tokenizer

- Packages that use Tokenizer
  Package | Description
  --- | ---
  org.apache.lucene.analysis | Text analysis.
  org.apache.lucene.analysis.standard | Fast, general-purpose grammar-based tokenizer `StandardTokenizer` implements the Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.

- - ### Uses of Tokenizer in org.apache.lucene.analysis

    Constructors in org.apache.lucene.analysis with parameters of type Tokenizer
    Constructor and Description
    `TokenStreamComponents(Tokenizer tokenizer)` Creates a new `Analyzer.TokenStreamComponents` from a Tokenizer
    `TokenStreamComponents(Tokenizer tokenizer, TokenStream result)` Creates a new `Analyzer.TokenStreamComponents` instance

  - ### Uses of Tokenizer in org.apache.lucene.analysis.standard

    Subclasses of Tokenizer in org.apache.lucene.analysis.standard
    Modifier and Type | Class and Description
    --- | ---
    `class` | `StandardTokenizer` A grammar-based tokenizer constructed with JFlex.

Copyright © 2000-2021 Apache Software Foundation. All Rights Reserved.