# Uses of Class
org.apache.lucene.analysis.TokenStream

Packages that use TokenStream
Package
Description
org.apache.lucene.analysis
Text analysis.
org.apache.lucene.analysis.standard
Fast, general-purpose grammar-based tokenizer `StandardTokenizer` implements the Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.
org.apache.lucene.document
The logical representation of a `Document` for indexing and searching.
org.apache.lucene.index
Code to maintain and access indices.
org.apache.lucene.util
Some utility classes.
org.apache.lucene.util.graph
Utility classes for working with token streams as graphs.

- ## Uses of TokenStream in org.apache.lucene.analysis

  Subclasses of TokenStream in org.apache.lucene.analysis
  Modifier and Type
  Class
  Description
  `final class`
  `CachingTokenFilter`
  This class can be used if the token attributes of a TokenStream are intended to be consumed more than once.
  `class`
  `FilteringTokenFilter`
  Abstract base class for TokenFilters that may remove tokens.
  `class`
  `GraphTokenFilter`
  An abstract TokenFilter that exposes its input stream as a graph
  `class`
  `LowerCaseFilter`
  Normalizes token text to lower case.
  `class`
  `StopFilter`
  Removes stop words from a token stream.
  `class`
  `TokenFilter`
  A TokenFilter is a TokenStream whose input is another TokenStream.
  `class`
  `Tokenizer`
  A Tokenizer is a TokenStream whose input is a Reader.
  Fields in org.apache.lucene.analysis declared as TokenStream
  Modifier and Type
  Field
  Description
  `protected final TokenStream`
  TokenFilter.`input`
  The source of tokens for this filter.
  `protected final TokenStream`
  Analyzer.TokenStreamComponents.`sink`
  Sink tokenstream, such as the outer tokenfilter decorating the chain.
  Methods in org.apache.lucene.analysis that return TokenStream
  Modifier and Type
  Method
  Description
  `abstract TokenStream`
  TokenFilterFactory.`create(TokenStream input)`
  Transform the specified input TokenStream
  `TokenStream`
  Analyzer.TokenStreamComponents.`getTokenStream()`
  Returns the sink `TokenStream`
  `protected TokenStream`
  Analyzer.`normalize(String fieldName, TokenStream in)`
  Wrap the given `TokenStream` in order to apply normalization filters.
  `protected final TokenStream`
  AnalyzerWrapper.`normalize(String fieldName, TokenStream in)`
  `TokenStream`
  TokenFilterFactory.`normalize(TokenStream input)`
  Normalize the specified input TokenStream While the default implementation returns input unchanged, filters that should be applied at normalization time can delegate to `create` method.
  `final TokenStream`
  Analyzer.`tokenStream(String fieldName, Reader reader)`
  Returns a TokenStream suitable for `fieldName`, tokenizing the contents of `reader`.
  `final TokenStream`
  Analyzer.`tokenStream(String fieldName, String text)`
  Returns a TokenStream suitable for `fieldName`, tokenizing the contents of `text`.
  `static TokenStream`
  AutomatonToTokenStream.`toTokenStream(Automaton automaton)`
  converts an automaton into a TokenStream.
  `TokenStream`
  TokenFilter.`unwrap()`
  `protected TokenStream`
  AnalyzerWrapper.`wrapTokenStreamForNormalization(String fieldName, TokenStream in)`
  Wraps / alters the given TokenStream for normalization purposes, taken from the wrapped Analyzer, to form new components.
  `protected final TokenStream`
  DelegatingAnalyzerWrapper.`wrapTokenStreamForNormalization(String fieldName, TokenStream in)`
  Methods in org.apache.lucene.analysis with parameters of type TokenStream
  Modifier and Type
  Method
  Description
  `abstract TokenStream`
  TokenFilterFactory.`create(TokenStream input)`
  Transform the specified input TokenStream
  `protected TokenStream`
  Analyzer.`normalize(String fieldName, TokenStream in)`
  Wrap the given `TokenStream` in order to apply normalization filters.
  `protected final TokenStream`
  AnalyzerWrapper.`normalize(String fieldName, TokenStream in)`
  `TokenStream`
  TokenFilterFactory.`normalize(TokenStream input)`
  Normalize the specified input TokenStream While the default implementation returns input unchanged, filters that should be applied at normalization time can delegate to `create` method.
  `Automaton`
  TokenStreamToAutomaton.`toAutomaton(TokenStream in)`
  Pulls the graph (including `PositionLengthAttribute`) from the provided `TokenStream`, and creates the corresponding automaton where arcs are bytes (or Unicode code points if unicodeArcs = true) from each term.
  `protected TokenStream`
  AnalyzerWrapper.`wrapTokenStreamForNormalization(String fieldName, TokenStream in)`
  Wraps / alters the given TokenStream for normalization purposes, taken from the wrapped Analyzer, to form new components.
  `protected final TokenStream`
  DelegatingAnalyzerWrapper.`wrapTokenStreamForNormalization(String fieldName, TokenStream in)`
  Constructors in org.apache.lucene.analysis with parameters of type TokenStream
  Modifier
  Constructor
  Description
  ``
  `CachingTokenFilter(TokenStream input)`
  Create a new CachingTokenFilter around `input`.
  ``
  `FilteringTokenFilter(TokenStream in)`
  Create a new `FilteringTokenFilter`.
  ``
  `GraphTokenFilter(TokenStream input)`
  Create a new GraphTokenFilter
  ``
  `LowerCaseFilter(TokenStream in)`
  Create a new LowerCaseFilter, that normalizes token text to lower case.
  ``
  `StopFilter(TokenStream in, CharArraySet stopWords)`
  Constructs a filter which removes words from the input TokenStream that are named in the Set.
  `protected`
  `TokenFilter(TokenStream input)`
  Construct a token stream filtering the given input.
  ``
  `TokenStreamComponents(Consumer<Reader> source, TokenStream result)`
  Creates a new `Analyzer.TokenStreamComponents` instance.
  ``
  `TokenStreamComponents(Tokenizer tokenizer, TokenStream result)`
  Creates a new `Analyzer.TokenStreamComponents` instance

- ## Uses of TokenStream in org.apache.lucene.analysis.standard

  Subclasses of TokenStream in org.apache.lucene.analysis.standard
  Modifier and Type
  Class
  Description
  `final class`
  `StandardTokenizer`
  A grammar-based tokenizer constructed with JFlex.
  Methods in org.apache.lucene.analysis.standard that return TokenStream
  Modifier and Type
  Method
  Description
  `protected TokenStream`
  StandardAnalyzer.`normalize(String fieldName, TokenStream in)`
  Methods in org.apache.lucene.analysis.standard with parameters of type TokenStream
  Modifier and Type
  Method
  Description
  `protected TokenStream`
  StandardAnalyzer.`normalize(String fieldName, TokenStream in)`

- ## Uses of TokenStream in org.apache.lucene.document

  Methods in org.apache.lucene.document that return TokenStream
  Modifier and Type
  Method
  Description
  `TokenStream`
  FeatureField.`tokenStream(Analyzer analyzer, TokenStream reuse)`
  `TokenStream`
  Field.`tokenStream(Analyzer analyzer, TokenStream reuse)`
  `TokenStream`
  ShapeDocValuesField.`tokenStream(Analyzer analyzer, TokenStream reuse)`
  TokenStreams are not yet supported
  `TokenStream`
  Field.`tokenStreamValue()`
  The TokenStream for this field to be used when indexing, or null.
  Methods in org.apache.lucene.document with parameters of type TokenStream
  Modifier and Type
  Method
  Description
  `void`
  Field.`setTokenStream(TokenStream tokenStream)`
  Expert: sets the token stream to be used for indexing.
  `TokenStream`
  FeatureField.`tokenStream(Analyzer analyzer, TokenStream reuse)`
  `TokenStream`
  Field.`tokenStream(Analyzer analyzer, TokenStream reuse)`
  `TokenStream`
  ShapeDocValuesField.`tokenStream(Analyzer analyzer, TokenStream reuse)`
  TokenStreams are not yet supported
  Constructors in org.apache.lucene.document with parameters of type TokenStream
  Modifier
  Constructor
  Description
  ``
  `Field(String name, TokenStream tokenStream, IndexableFieldType type)`
  Create field with TokenStream value.
  ``
  `TextField(String name, TokenStream stream)`
  Creates a new un-stored TextField with TokenStream value.

- ## Uses of TokenStream in org.apache.lucene.index

  Methods in org.apache.lucene.index that return TokenStream
  Modifier and Type
  Method
  Description
  `TokenStream`
  IndexableField.`tokenStream(Analyzer analyzer, TokenStream reuse)`
  Creates the TokenStream used for indexing this field.
  Methods in org.apache.lucene.index with parameters of type TokenStream
  Modifier and Type
  Method
  Description
  `TokenStream`
  IndexableField.`tokenStream(Analyzer analyzer, TokenStream reuse)`
  Creates the TokenStream used for indexing this field.

- ## Uses of TokenStream in org.apache.lucene.util

  Methods in org.apache.lucene.util with parameters of type TokenStream
  Modifier and Type
  Method
  Description
  `protected Query`
  QueryBuilder.`analyzeBoolean(String field, TokenStream stream)`
  Creates simple boolean query from the cached tokenstream contents
  `protected Query`
  QueryBuilder.`analyzeGraphBoolean(String field, TokenStream source, BooleanClause.Occur operator)`
  Creates a boolean query from a graph token stream.
  `protected Query`
  QueryBuilder.`analyzeGraphPhrase(TokenStream source, String field, int phraseSlop)`
  Creates graph phrase query from the tokenstream contents
  `protected Query`
  QueryBuilder.`analyzeMultiBoolean(String field, TokenStream stream, BooleanClause.Occur operator)`
  Creates complex boolean query from the cached tokenstream contents
  `protected Query`
  QueryBuilder.`analyzeMultiPhrase(String field, TokenStream stream, int slop)`
  Creates complex phrase query from the cached tokenstream contents
  `protected Query`
  QueryBuilder.`analyzePhrase(String field, TokenStream stream, int slop)`
  Creates simple phrase query from the cached tokenstream contents
  `protected Query`
  QueryBuilder.`analyzeTerm(String field, TokenStream stream)`
  Creates simple term query from the cached tokenstream contents
  `protected Query`
  QueryBuilder.`createFieldQuery(TokenStream source, BooleanClause.Occur operator, String field, boolean quoted, int phraseSlop)`
  Creates a query from a token stream.

- ## Uses of TokenStream in org.apache.lucene.util.graph

  Methods in org.apache.lucene.util.graph that return types with arguments of type TokenStream
  Modifier and Type
  Method
  Description
  `Iterator<TokenStream>`
  GraphTokenStreamFiniteStrings.`getFiniteStrings()`
  Get all finite strings from the automaton.
  `Iterator<TokenStream>`
  GraphTokenStreamFiniteStrings.`getFiniteStrings(int startState, int endState)`
  Get all finite strings that start at `startState` and end at `endState`.
  Constructors in org.apache.lucene.util.graph with parameters of type TokenStream
  Modifier
  Constructor
  Description
  ``
  `GraphTokenStreamFiniteStrings(TokenStream in)`