org.apache.lucene.search.highlight

## Class TokenSources

- java.lang.Object
- - org.apache.lucene.search.highlight.TokenSources

-     public class TokenSources
      extends Object

  Convenience methods for obtaining a `TokenStream` for use with the `Highlighter` - can obtain from term vectors with offsets and positions or from an Analyzer re-parsing the stored content.
  See Also:
    `TokenStreamFromTermVector`

- - ### Method Summary

    All Methods Static Methods Concrete Methods Deprecated Methods
    Modifier and Type | Method and Description
    --- | ---
    `static TokenStream` | `getAnyTokenStream(IndexReader reader, int docId, String field, Analyzer analyzer)` Deprecated.
    `static TokenStream` | `getAnyTokenStream(IndexReader reader, int docId, String field, Document document, Analyzer analyzer)` Deprecated.
    `static TokenStream` | `getTermVectorTokenStreamOrNull(String field, Fields tvFields, int maxStartOffset)` Get a token stream by un-inverting the term vector.
    `static TokenStream` | `getTokenStream(Document doc, String field, Analyzer analyzer)` Deprecated.
    `static TokenStream` | `getTokenStream(IndexReader reader, int docId, String field, Analyzer analyzer)` Deprecated.
    `static TokenStream` | `getTokenStream(String field, Fields tvFields, String text, Analyzer analyzer, int maxStartOffset)` Get a token stream from either un-inverting a term vector if possible, or by analyzing the text.
    `static TokenStream` | `getTokenStream(String field, String contents, Analyzer analyzer)` Deprecated.
    `static TokenStream` | `getTokenStream(Terms tpv)` Deprecated.
    `static TokenStream` | `getTokenStream(Terms vector, boolean tokenPositionsGuaranteedContiguous)` Deprecated.
    `static TokenStream` | `getTokenStreamWithOffsets(IndexReader reader, int docId, String field)` Deprecated.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Method Detail

    - #### getTokenStream

          public static TokenStream getTokenStream(String field,
                                                   Fields tvFields,
                                                   String text,
                                                   Analyzer analyzer,
                                                   int maxStartOffset)
                                            throws IOException

      Get a token stream from either un-inverting a term vector if possible, or by analyzing the text. WARNING: Don't call this if there is more than one value for this field. If there are, and if there are term vectors, then there is a single tokenstream with offsets suggesting all the field values were concatenated.
      Parameters:
        `field` - The field to either get term vectors from or to analyze the text from.
        `tvFields` - from `IndexReader.getTermVectors(int)`. Possibly null. For performance, this instance should be re-used for the same document (e.g. when highlighting multiple fields).
        `text` - the text to analyze, failing term vector un-inversion
        `analyzer` - the analyzer to analyze `text` with, failing term vector un-inversion
        `maxStartOffset` - Terms with a startOffset greater than this aren't returned. Use -1 for no limit. Suggest using `Highlighter.getMaxDocCharsToAnalyze()` - 1.
      Returns:
        a token stream from either term vectors, or from analyzing the text. Never null.
      Throws:
        `IOException`

    - #### getTermVectorTokenStreamOrNull

          public static TokenStream getTermVectorTokenStreamOrNull(String field,
                                                                   Fields tvFields,
                                                                   int maxStartOffset)
                                                            throws IOException

      Get a token stream by un-inverting the term vector. This method returns null if `tvFields` is null or if the field has no term vector, or if the term vector doesn't have offsets. Positions are recommended on the term vector but it isn't strictly required.
      Parameters:
        `field` - The field to get term vectors from.
        `tvFields` - from `IndexReader.getTermVectors(int)`. Possibly null. For performance, this instance should be re-used for the same document (e.g. when highlighting multiple fields).
        `maxStartOffset` - Terms with a startOffset greater than this aren't returned. Use -1 for no limit. Suggest using `Highlighter.getMaxDocCharsToAnalyze()` - 1
      Returns:
        a token stream from term vectors. Null if no term vectors with the right options.
      Throws:
        `IOException`

    - #### getAnyTokenStream

          @Deprecated
          public static TokenStream getAnyTokenStream(IndexReader reader,
                                                                  int docId,
                                                                  String field,
                                                                  Document document,
                                                                  Analyzer analyzer)
                                                           throws IOException

      Deprecated.
      A convenience method that tries to first get a `TokenStreamFromTermVector` for the specified docId, then, falls back to using the passed in `Document` to retrieve the TokenStream. This is useful when you already have the document, but would prefer to use the vector first.
      Parameters:
        `reader` - The `IndexReader` to use to try and get the vector from
        `docId` - The docId to retrieve.
        `field` - The field to retrieve on the document
        `document` - The document to fall back on
        `analyzer` - The analyzer to use for creating the TokenStream if the vector doesn't exist
      Returns:
        The `TokenStream` for the `IndexableField` on the `Document`
      Throws:
        `IOException` - if there was an error loading

    - #### getAnyTokenStream

          @Deprecated
          public static TokenStream getAnyTokenStream(IndexReader reader,
                                                                  int docId,
                                                                  String field,
                                                                  Analyzer analyzer)
                                                           throws IOException

      Deprecated.
      A convenience method that tries a number of approaches to getting a token stream. The cost of finding there are no termVectors in the index is minimal (1000 invocations still registers 0 ms). So this "lazy" (flexible?) approach to coding is probably acceptable
      Returns:
        null if field not stored correctly
      Throws:
        `IOException` - If there is a low-level I/O error

    - #### getTokenStream

          @Deprecated
          public static TokenStream getTokenStream(Terms vector,
                                                               boolean tokenPositionsGuaranteedContiguous)
                                                        throws IOException

      Deprecated.
      Simply calls `getTokenStream(org.apache.lucene.index.Terms)` now.
      Throws:
        `IOException`

    - #### getTokenStream

          @Deprecated
          public static TokenStream getTokenStream(Terms tpv)
                                                        throws IOException

      Deprecated.
      Returns a token stream generated from a `Terms`. This can be used to feed the highlighter with a pre-parsed token stream. The `Terms` must have offsets available. If there are no positions available, all tokens will have position increments reflecting adjacent tokens, or coincident when terms share a start offset. If there are stopwords filtered from the index, you probably want to ensure term vectors have positions so that phrase queries won't match across stopwords.
      Throws:
        `IllegalArgumentException` - if no offsets are available
        `IOException`

    - #### getTokenStreamWithOffsets

          @Deprecated
          public static TokenStream getTokenStreamWithOffsets(IndexReader reader,
                                                                          int docId,
                                                                          String field)
                                                                   throws IOException

      Deprecated.
      Returns a `TokenStream` with positions and offsets constructed from field termvectors. If the field has no termvectors or offsets are not included in the termvector, return null. See `getTokenStream(org.apache.lucene.index.Terms)` for an explanation of what happens when positions aren't present.
      Parameters:
        `reader` - the `IndexReader` to retrieve term vectors from
        `docId` - the document to retrieve termvectors for
        `field` - the field to retrieve termvectors for
      Returns:
        a `TokenStream`, or null if offsets are not available
      Throws:
        `IOException` - If there is a low-level I/O error
      See Also:
        `getTokenStream(org.apache.lucene.index.Terms)`

    - #### getTokenStream

          @Deprecated
          public static TokenStream getTokenStream(IndexReader reader,
                                                               int docId,
                                                               String field,
                                                               Analyzer analyzer)
                                                        throws IOException

      Deprecated.
      Throws:
        `IOException`

    - #### getTokenStream

          @Deprecated
          public static TokenStream getTokenStream(Document doc,
                                                               String field,
                                                               Analyzer analyzer)

      Deprecated.

    - #### getTokenStream

          @Deprecated
          public static TokenStream getTokenStream(String field,
                                                               String contents,
                                                               Analyzer analyzer)

      Deprecated.

Copyright © 2000-2024 Apache Software Foundation. All Rights Reserved.