# Uses of Class
org.apache.lucene.analysis.Tokenizer

Packages that use Tokenizer
Package
Description
org.apache.lucene.analysis
Text analysis.
org.apache.lucene.analysis.standard
Fast, general-purpose grammar-based tokenizer `StandardTokenizer` implements the Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.

- ## Uses of Tokenizer in org.apache.lucene.analysis

  Methods in org.apache.lucene.analysis that return Tokenizer
  Modifier and Type
  Method
  Description
  `final Tokenizer`
  TokenizerFactory.`create()`
  Creates a TokenStream of the specified input using the default attribute factory.
  `abstract Tokenizer`
  TokenizerFactory.`create(AttributeFactory factory)`
  Creates a TokenStream of the specified input using the given AttributeFactory
  Constructors in org.apache.lucene.analysis with parameters of type Tokenizer
  Modifier
  Constructor
  Description
  ``
  `TokenStreamComponents(Tokenizer tokenizer)`
  Creates a new `Analyzer.TokenStreamComponents` from a Tokenizer
  ``
  `TokenStreamComponents(Tokenizer tokenizer, TokenStream result)`
  Creates a new `Analyzer.TokenStreamComponents` instance

- ## Uses of Tokenizer in org.apache.lucene.analysis.standard

  Subclasses of Tokenizer in org.apache.lucene.analysis.standard
  Modifier and Type
  Class
  Description
  `final class`
  `StandardTokenizer`
  A grammar-based tokenizer constructed with JFlex.