org.apache.lucene.analysis

## Class Token

- java.lang.Object
- - org.apache.lucene.util.AttributeImpl
  - - org.apache.lucene.analysis.tokenattributes.CharTermAttributeImpl
    - - org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl
      - - org.apache.lucene.analysis.Token

All Implemented Interfaces:
  Appendable, CharSequence, Cloneable, CharTermAttribute, FlagsAttribute, OffsetAttribute, PayloadAttribute, PositionIncrementAttribute, PositionLengthAttribute, TermFrequencyAttribute, TermToBytesRefAttribute, TypeAttribute, Attribute

    public class Token
    extends PackedTokenAttributeImpl
    implements FlagsAttribute, PayloadAttribute

A Token is an occurrence of a term from the text of a field. It consists of a term's text, the start and end offset of the term in the text of the field, and a type string.

The start and end offsets permit applications to re-associate a token with its source text, e.g., to display highlighted query terms in a document browser, or to show matching text fragments in a KWIC display, etc.

The type is a string, assigned by a lexical analyzer (a.k.a. tokenizer), naming the lexical or syntactic class that the token belongs to. For example an end of sentence marker token might be implemented with type "eos". The default token type is "word".

A Token can optionally have metadata (a.k.a. payload) in the form of a variable length byte array. Use `PostingsEnum.getPayload()` to retrieve the payloads from the index. A few things to note:

- clear() initializes all of the fields to default values. This was changed in contrast to Lucene 2.4, but should affect no one.
- Because `TokenStreams` can be chained, one cannot assume that the `Token's` current type is correct.
- The startOffset and endOffset represent the start and offset in the source text, so be careful in adjusting them.
- When caching a reusable token, clone it. When injecting a cached token into a stream that can be reset, clone it again.

### Field Summary

Fields
`static AttributeFactory` `TOKEN_ATTRIBUTE_FACTORY` Convenience factory that returns `Token` as implementation for the basic attributes and return the default impl (with "Impl" appended) for all other attributes.

### Fields inherited from class org.apache.lucene.analysis.tokenattributes.CharTermAttributeImpl

`builder`

### Fields inherited from interface org.apache.lucene.analysis.tokenattributes.TypeAttribute

`DEFAULT_TYPE`

### Constructor Summary

Constructors
Constructor and Description
`Token()` Constructs a Token will null text.
`Token(CharSequence text, int start, int end)` Constructs a Token with the given term text, start and end offsets.
`Token(CharSequence text, int posInc, int start, int end)` Constructs a Token with the given term text, position increment, start and end offsets
`Token(CharSequence text, int posInc, int start, int end, int posLength)`

### Method Summary

All Methods Instance Methods Concrete Methods
`void` `clear()` Resets the term text, payload, flags, positionIncrement, positionLength, startOffset, endOffset and token type to default.
`Token` `clone()`
`void` `copyTo(AttributeImpl target)`
`boolean` `equals(Object obj)`
`int` `getFlags()`
`BytesRef` `getPayload()`
`int` `hashCode()`
`void` `reflectWith(AttributeReflector reflector)`
`void` `reinit(Token prototype)` Copy the prototype token's fields into this one.
`void` `setFlags(int flags)`
`void` `setPayload(BytesRef payload)`

### Methods inherited from class org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl

`end, endOffset, getPositionIncrement, getPositionLength, getTermFrequency, setOffset, setPositionIncrement, setPositionLength, setTermFrequency, setType, startOffset, type`

### Methods inherited from class org.apache.lucene.analysis.tokenattributes.CharTermAttributeImpl

`append, append, append, append, append, append, buffer, charAt, copyBuffer, getBytesRef, length, resizeBuffer, setEmpty, setLength, subSequence, toString`

### Methods inherited from class org.apache.lucene.util.AttributeImpl

`reflectAsString`

### Methods inherited from class java.lang.Object

`finalize, getClass, notify, notifyAll, wait, wait, wait`

### Methods inherited from interface java.lang.CharSequence

`chars, codePoints`

### Field Detail

#### TOKEN_ATTRIBUTE_FACTORY

    public static final AttributeFactory TOKEN_ATTRIBUTE_FACTORY

Convenience factory that returns `Token` as implementation for the basic attributes and return the default impl (with "Impl" appended) for all other attributes.

Since:
  3.0

### Constructor Detail

#### Token

    public Token()

Constructs a Token will null text.

#### Token

    public Token(CharSequence text,
                 int start,
                 int end)

Constructs a Token with the given term text, start and end offsets. The type defaults to "word." NOTE: for better indexing speed you should instead use the char[] termBuffer methods to set the term text.

Parameters:
  `text` - term text
  `start` - start offset in the source text
  `end` - end offset in the source text

#### Token

    public Token(CharSequence text,
                 int posInc,
                 int start,
                 int end)

Constructs a Token with the given term text, position increment, start and end offsets

#### Token

    public Token(CharSequence text,
                 int posInc,
                 int start,
                 int end,
                 int posLength)

### Method Detail

#### getFlags

    public int getFlags()

Specified by:
  `getFlags` in interface `FlagsAttribute`
See Also:
  `FlagsAttribute`

#### setFlags

    public void setFlags(int flags)

Specified by:
  `setFlags` in interface `FlagsAttribute`
See Also:
  `FlagsAttribute`

#### getPayload

    public BytesRef getPayload()

Specified by:
  `getPayload` in interface `PayloadAttribute`
See Also:
  `PayloadAttribute`

#### setPayload

    public void setPayload(BytesRef payload)

Specified by:
  `setPayload` in interface `PayloadAttribute`
See Also:
  `PayloadAttribute`

#### clear

    public void clear()

Resets the term text, payload, flags, positionIncrement, positionLength, startOffset, endOffset and token type to default.

Overrides:
  `clear` in class `PackedTokenAttributeImpl`

#### equals

    public boolean equals(Object obj)

Overrides:
  `equals` in class `PackedTokenAttributeImpl`

#### hashCode

    public int hashCode()

Overrides:
  `hashCode` in class `PackedTokenAttributeImpl`

#### clone

    public Token clone()

Overrides:
  `clone` in class `PackedTokenAttributeImpl`

#### reinit

    public void reinit(Token prototype)

Copy the prototype token's fields into this one. Note: Payloads are shared.

Parameters:
  `prototype` - source Token to copy fields from

#### copyTo

    public void copyTo(AttributeImpl target)

Overrides:
  `copyTo` in class `PackedTokenAttributeImpl`

#### reflectWith

    public void reflectWith(AttributeReflector reflector)

Overrides:
  `reflectWith` in class `PackedTokenAttributeImpl`