org.apache.lucene.analysis

## Class Token

- java.lang.Object
- - org.apache.lucene.util.AttributeImpl
  - - org.apache.lucene.analysis.tokenattributes.CharTermAttributeImpl
    - - org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl
      - - org.apache.lucene.analysis.Token

- All Implemented Interfaces:
    Appendable, CharSequence, Cloneable, CharTermAttribute, FlagsAttribute, OffsetAttribute, PayloadAttribute, PositionIncrementAttribute, PositionLengthAttribute, TermToBytesRefAttribute, TypeAttribute, Attribute

  Deprecated.
  This class is outdated and no longer used since Lucene 2.9. Nuke it finally!

      @Deprecated
      public class Token
      extends PackedTokenAttributeImpl
      implements FlagsAttribute, PayloadAttribute

  A Token is an occurrence of a term from the text of a field. It consists of a term's text, the start and end offset of the term in the text of the field, and a type string.

  The start and end offsets permit applications to re-associate a token with its source text, e.g., to display highlighted query terms in a document browser, or to show matching text fragments in a KWIC display, etc.

  The type is a string, assigned by a lexical analyzer (a.k.a. tokenizer), naming the lexical or syntactic class that the token belongs to. For example an end of sentence marker token might be implemented with type "eos". The default token type is "word".

  A Token can optionally have metadata (a.k.a. payload) in the form of a variable length byte array. Use `PostingsEnum.getPayload()` to retrieve the payloads from the index.

  NOTE: As of 2.9, Token implements all `Attribute` interfaces that are part of core Lucene and can be found in the `tokenattributes` subpackage. Even though it is not necessary to use Token anymore, with the new TokenStream API it can be used as convenience class that implements all `Attribute`s, which is especially useful to easily switch from the old to the new TokenStream API. A few things to note:

  - clear() initializes all of the fields to default values. This was changed in contrast to Lucene 2.4, but should affect no one.
  - Because `TokenStreams` can be chained, one cannot assume that the `Token's` current type is correct.
  - The startOffset and endOffset represent the start and offset in the source text, so be careful in adjusting them.
  - When caching a reusable token, clone it. When injecting a cached token into a stream that can be reset, clone it again.

  Please note: With Lucene 3.1, the `toString()` method had to be changed to match the `CharSequence` interface introduced by the interface `CharTermAttribute`. This method now only prints the term text, no additional information anymore.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static AttributeFactory` | `TOKEN_ATTRIBUTE_FACTORY` Deprecated. Convenience factory that returns `Token` as implementation for the basic attributes and return the default impl (with "Impl" appended) for all other attributes.

    - ### Fields inherited from class org.apache.lucene.analysis.tokenattributes.CharTermAttributeImpl

      `builder`

    - ### Fields inherited from interface org.apache.lucene.analysis.tokenattributes.TypeAttribute

      `DEFAULT_TYPE`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `Token()` Deprecated. Constructs a Token will null text.
    `Token(CharSequence text, int start, int end)` Deprecated. Constructs a Token with the given term text, start and end offsets.

  - ### Method Summary

    All Methods Instance Methods Concrete Methods Deprecated Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `clear()` Deprecated. Resets the term text, payload, flags, positionIncrement, positionLength, startOffset, endOffset and token type to default.
    `Token` | `clone()` Deprecated. In most cases the clone is, and should be, deep in order to be able to properly capture the state of all attributes.
    `void` | `copyTo(AttributeImpl target)` Deprecated. Copies the values from this Attribute into the passed-in target attribute.
    `boolean` | `equals(Object obj)` Deprecated.
    `int` | `getFlags()` Deprecated. Get the bitset for any bits that have been set.
    `BytesRef` | `getPayload()` Deprecated. Returns this Token's payload.
    `int` | `hashCode()` Deprecated.
    `void` | `reflectWith(AttributeReflector reflector)` Deprecated. This method is for introspection of attributes, it should simply add the key/values this attribute holds to the given `AttributeReflector`.
    `void` | `reinit(Token prototype)` Deprecated. Copy the prototype token's fields into this one.
    `void` | `setFlags(int flags)` Deprecated. Set the flags to a new bitset.
    `void` | `setPayload(BytesRef payload)` Deprecated. Sets this Token's payload.

    - ### Methods inherited from class org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl

      `end, endOffset, getPositionIncrement, getPositionLength, setOffset, setPositionIncrement, setPositionLength, setType, startOffset, type`

    - ### Methods inherited from class org.apache.lucene.analysis.tokenattributes.CharTermAttributeImpl

      `append, append, append, append, append, append, buffer, charAt, copyBuffer, getBytesRef, length, resizeBuffer, setEmpty, setLength, subSequence, toString`

    - ### Methods inherited from class org.apache.lucene.util.AttributeImpl

      `reflectAsString`

    - ### Methods inherited from class java.lang.Object

      `finalize, getClass, notify, notifyAll, wait, wait, wait`

    - ### Methods inherited from interface java.lang.CharSequence

      `chars, codePoints`

- - ### Field Detail

    - #### TOKEN_ATTRIBUTE_FACTORY

          public static final AttributeFactory TOKEN_ATTRIBUTE_FACTORY

      Deprecated.
      Convenience factory that returns `Token` as implementation for the basic attributes and return the default impl (with "Impl" appended) for all other attributes.
      Since:
        3.0

  - ### Constructor Detail

    - #### Token

          public Token()

      Deprecated.
      Constructs a Token will null text.

    - #### Token

          public Token(CharSequence text,
                       int start,
                       int end)

      Deprecated.
      Constructs a Token with the given term text, start and end offsets. The type defaults to "word." NOTE: for better indexing speed you should instead use the char[] termBuffer methods to set the term text.
      Parameters:
        `text` - term text
        `start` - start offset in the source text
        `end` - end offset in the source text

  - ### Method Detail

    - #### getFlags

          public int getFlags()

      Deprecated.
      Get the bitset for any bits that have been set.
      Specified by:
        `getFlags` in interface `FlagsAttribute`
      Returns:
        The bits
      See Also:
        `FlagsAttribute`

    - #### setFlags

          public void setFlags(int flags)

      Deprecated.
      Set the flags to a new bitset.
      Specified by:
        `setFlags` in interface `FlagsAttribute`
      See Also:
        `FlagsAttribute`

    - #### getPayload

          public BytesRef getPayload()

      Deprecated.
      Returns this Token's payload.
      Specified by:
        `getPayload` in interface `PayloadAttribute`
      See Also:
        `PayloadAttribute`

    - #### setPayload

          public void setPayload(BytesRef payload)

      Deprecated.
      Sets this Token's payload.
      Specified by:
        `setPayload` in interface `PayloadAttribute`
      See Also:
        `PayloadAttribute`

    - #### clear

          public void clear()

      Deprecated.
      Resets the term text, payload, flags, positionIncrement, positionLength, startOffset, endOffset and token type to default.
      Overrides:
        `clear` in class `PackedTokenAttributeImpl`

    - #### equals

          public boolean equals(Object obj)

      Deprecated.
      Overrides:
        `equals` in class `PackedTokenAttributeImpl`

    - #### hashCode

          public int hashCode()

      Deprecated.
      Overrides:
        `hashCode` in class `PackedTokenAttributeImpl`

    - #### clone

          public Token clone()

      Deprecated.
      Description copied from class: `AttributeImpl`
      In most cases the clone is, and should be, deep in order to be able to properly capture the state of all attributes.
      Overrides:
        `clone` in class `PackedTokenAttributeImpl`

    - #### reinit

          public void reinit(Token prototype)

      Deprecated.
      Copy the prototype token's fields into this one. Note: Payloads are shared.
      Parameters:
        `prototype` - source Token to copy fields from

    - #### copyTo

          public void copyTo(AttributeImpl target)

      Deprecated.
      Description copied from class: `AttributeImpl`
      Copies the values from this Attribute into the passed-in target attribute. The target implementation must support all the Attributes this implementation supports.
      Overrides:
        `copyTo` in class `PackedTokenAttributeImpl`

    - #### reflectWith

          public void reflectWith(AttributeReflector reflector)

      Deprecated.
      Description copied from class: `AttributeImpl`
      This method is for introspection of attributes, it should simply add the key/values this attribute holds to the given `AttributeReflector`.

      Implementations look like this (e.g. for a combined attribute implementation):

             public void reflectWith(AttributeReflector reflector) {
               reflector.reflect(CharTermAttribute.class, "term", term());
               reflector.reflect(PositionIncrementAttribute.class, "positionIncrement", getPositionIncrement());
             }

      If you implement this method, make sure that for each invocation, the same set of `Attribute` interfaces and keys are passed to `AttributeReflector.reflect(java.lang.Class<? extends org.apache.lucene.util.Attribute>, java.lang.String, java.lang.Object)` in the same order, but possibly different values. So don't automatically exclude e.g. `null` properties!

      Overrides:
        `reflectWith` in class `PackedTokenAttributeImpl`
      See Also:
        `AttributeImpl.reflectAsString(boolean)`

Copyright © 2000-2019 Apache Software Foundation. All Rights Reserved.