## Uses of Class
org.apache.lucene.analysis.Tokenizer

Packages that use Tokenizer
org.apache.lucene.analysis: Text analysis.
org.apache.lucene.analysis.standard: Fast, general-purpose grammar-based tokenizer `StandardTokenizer` implements the Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.

### Uses of Tokenizer in org.apache.lucene.analysis

Constructors in org.apache.lucene.analysis with parameters of type Tokenizer
Constructor and Description
`TokenStreamComponents(Tokenizer tokenizer)` Creates a new `Analyzer.TokenStreamComponents` from a Tokenizer
`TokenStreamComponents(Tokenizer tokenizer, TokenStream result)` Creates a new `Analyzer.TokenStreamComponents` instance

### Uses of Tokenizer in org.apache.lucene.analysis.standard

Subclasses of Tokenizer in org.apache.lucene.analysis.standard
`class` `StandardTokenizer` A grammar-based tokenizer constructed with JFlex.