# Package org.apache.lucene.analysis.standard

Fast, general-purpose grammar-based tokenizer `StandardTokenizer` implements the Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.

See: Description

- Class Summary
  Class | Description
  --- | ---
  StandardAnalyzer | Filters `StandardTokenizer` with `StandardFilter`, `LowerCaseFilter` and `StopFilter`, using a list of English stop words.
  StandardFilter | Normalizes tokens extracted with `StandardTokenizer`.
  StandardTokenizer | A grammar-based tokenizer constructed with JFlex.
  StandardTokenizerImpl | This class implements Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.

## Package org.apache.lucene.analysis.standard Description

Fast, general-purpose grammar-based tokenizer `StandardTokenizer` implements the Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29. Unlike `UAX29URLEmailTokenizer` from the analysis module, URLs and email addresses are not tokenized as single tokens, but are instead split up into tokens according to the UAX#29 word break rules.
`StandardAnalyzer` includes `StandardTokenizer`, `StandardFilter`, `LowerCaseFilter` and `StopFilter`.

Copyright © 2000-2018 Apache Software Foundation. All Rights Reserved.