org.apache.lucene.analysis

## Class TokenStreamToAutomaton

- java.lang.Object
- - org.apache.lucene.analysis.TokenStreamToAutomaton

    public class TokenStreamToAutomaton
    extends Object

Consumes a TokenStream and creates an `Automaton` where the transition labels are UTF8 bytes (or Unicode code points if unicodeArcs is true) from the `TermToBytesRefAttribute`. Between tokens we insert POS_SEP and for holes we insert HOLE.

WARNING: This API is experimental and might change in incompatible ways in the next release.

### Field Summary

Fields
`static int` `HOLE` We add this arc to represent a hole.
`static int` `POS_SEP` We create transition between two adjacent tokens.

### Constructor Summary

Constructors
Constructor and Description
`TokenStreamToAutomaton()` Sole constructor.

### Method Summary

All Methods Instance Methods Concrete Methods
`protected BytesRef` `changeToken(BytesRef in)` Subclass and implement this if you need to change the token (such as escaping certain bytes) before it's turned into a graph.
`void` `setFinalOffsetGapAsHole(boolean finalOffsetGapAsHole)` If true, any final offset gaps will result in adding a position hole.
`void` `setPreservePositionIncrements(boolean enablePositionIncrements)` Whether to generate holes in the automaton for missing positions, `true` by default.
`void` `setUnicodeArcs(boolean unicodeArcs)` Whether to make transition labels Unicode code points instead of UTF8 bytes, `false` by default
`Automaton` `toAutomaton(TokenStream in)` Pulls the graph (including `PositionLengthAttribute`) from the provided `TokenStream`, and creates the corresponding automaton where arcs are bytes (or Unicode code points if unicodeArcs = true) from each term.

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### POS_SEP

    public static final int POS_SEP

We create transition between two adjacent tokens.

See Also:
  Constant Field Values

#### HOLE

    public static final int HOLE

We add this arc to represent a hole.

See Also:
  Constant Field Values

### Constructor Detail

#### TokenStreamToAutomaton

    public TokenStreamToAutomaton()

Sole constructor.

### Method Detail

#### setPreservePositionIncrements

    public void setPreservePositionIncrements(boolean enablePositionIncrements)

Whether to generate holes in the automaton for missing positions, `true` by default.

#### setFinalOffsetGapAsHole

    public void setFinalOffsetGapAsHole(boolean finalOffsetGapAsHole)

If true, any final offset gaps will result in adding a position hole.

#### setUnicodeArcs

    public void setUnicodeArcs(boolean unicodeArcs)

Whether to make transition labels Unicode code points instead of UTF8 bytes, `false` by default

#### changeToken

    protected BytesRef changeToken(BytesRef in)

Subclass and implement this if you need to change the token (such as escaping certain bytes) before it's turned into a graph.

#### toAutomaton

    public Automaton toAutomaton(TokenStream in)
                          throws IOException

Pulls the graph (including `PositionLengthAttribute`) from the provided `TokenStream`, and creates the corresponding automaton where arcs are bytes (or Unicode code points if unicodeArcs = true) from each term.

Throws:
  `IOException`