org.apache.lucene.codecs.blocktree

## Class BlockTreeTermsWriter

- java.lang.Object
- - org.apache.lucene.codecs.FieldsConsumer
  - - org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter

- All Implemented Interfaces:
    Closeable, AutoCloseable

      public final class BlockTreeTermsWriter
      extends FieldsConsumer

  Block-based terms index and dictionary writer.

  Writes terms dict and index, block-encoding (column stride) each term's metadata for each set of terms between two index terms.

  Files:

  - `.tim`: Term Dictionary
  - `.tip`: Term Index

  ### Term Dictionary

  The .tim file contains the list of terms in each field along with per-term statistics (such as docfreq) and per-term metadata (typically pointers to the postings list for that term in the inverted index).

  The .tim is arranged in blocks: with blocks containing a variable number of entries (by default 25-48), where each entry is either a term or a reference to a sub-block.

  NOTE: The term dictionary can plug into different postings implementations: the postings writer/reader are actually responsible for encoding and decoding the Postings Metadata and Term Metadata sections.

  - TermsDict (.tim) --> Header, PostingsHeader, NodeBlockNumBlocks, FieldSummary, DirOffset, Footer
  - NodeBlock --> (OuterNode | InnerNode)
  - OuterNode --> EntryCount, SuffixLength, ByteSuffixLength, StatsLength, < TermStats >EntryCount, MetaLength, <TermMetadata>EntryCount
  - InnerNode --> EntryCount, SuffixLength[,Sub?], ByteSuffixLength, StatsLength, < TermStats ? >EntryCount, MetaLength, <TermMetadata ? >EntryCount
  - TermStats --> DocFreq, TotalTermFreq
  - FieldSummary --> NumFields, <FieldNumber, NumTerms, RootCodeLength, ByteRootCodeLength, SumTotalTermFreq?, SumDocFreq, DocCount, LongsSize, MinTerm, MaxTerm>NumFields
  - Header --> `CodecHeader`
  - DirOffset --> `Uint64`
  - MinTerm,MaxTerm --> `VInt` length followed by the byte[]
  - EntryCount,SuffixLength,StatsLength,DocFreq,MetaLength,NumFields, FieldNumber,RootCodeLength,DocCount,LongsSize --> `VInt`
  - TotalTermFreq,NumTerms,SumTotalTermFreq,SumDocFreq --> `VLong`
  - Footer --> `CodecFooter`

  Notes:

  - Header is a `CodecHeader` storing the version information for the BlockTree implementation.
  - DirOffset is a pointer to the FieldSummary section.
  - DocFreq is the count of documents which contain the term.
  - TotalTermFreq is the total number of occurrences of the term. This is encoded as the difference between the total number of occurrences and the DocFreq.
  - FieldNumber is the fields number from `FieldInfos`. (.fnm)
  - NumTerms is the number of unique terms for the field.
  - RootCode points to the root block for the field.
  - SumDocFreq is the total number of postings, the number of term-document pairs across the entire field.
  - DocCount is the number of documents that have at least one posting for this field.
  - LongsSize records how many long values the postings writer/reader record per term (e.g., to hold freq/prox/doc file offsets).
  - MinTerm, MaxTerm are the lowest and highest term in this field.
  - PostingsHeader and TermMetadata are plugged into by the specific postings implementation: these contain arbitrary per-file data (such as parameters or versioning information) and per-term data (such as pointers to inverted files).
  - For inner nodes of the tree, every entry will steal one bit to mark whether it points to child nodes(sub-block). If so, the corresponding TermStats and TermMetaData are omitted

  ### Term Index

  The .tip file contains an index into the term dictionary, so that it can be accessed randomly. The index is also used to determine when a given term cannot exist on disk (in the .tim file), saving a disk seek.

  - TermsIndex (.tip) --> Header, FSTIndexNumFields <IndexStartFP>NumFields, DirOffset, Footer
  - Header --> `CodecHeader`
  - DirOffset --> `Uint64`
  - IndexStartFP --> `VLong`
  - FSTIndex --> `FST<byte[]>`
  - Footer --> `CodecFooter`

  Notes:

  - The .tip file contains a separate FST for each field. The FST maps a term prefix to the on-disk block that holds all terms starting with that prefix. Each field's IndexStartFP points to its FST.
  - DirOffset is a pointer to the start of the IndexStartFPs for all fields
  - It's possible that an on-disk block would contain too many terms (more than the allowed maximum (default: 48)). When this happens, the block is sub-divided into new blocks (called "floor blocks"), and then the output in the FST for the block's prefix encodes the leading byte of each sub-block, and its file pointer.
  See Also:
    `BlockTreeTermsReader`
  WARNING: This API is experimental and might change in incompatible ways in the next release.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static int` | `DEFAULT_MAX_BLOCK_SIZE` Suggested default value for the `maxItemsInBlock` parameter to `BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.
    `static int` | `DEFAULT_MIN_BLOCK_SIZE` Suggested default value for the `minItemsInBlock` parameter to `BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `BlockTreeTermsWriter(SegmentWriteState state, PostingsWriterBase postingsWriter, int minItemsInBlock, int maxItemsInBlock)` Create a new writer.

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `close()`
    `static void` | `validateSettings(int minItemsInBlock, int maxItemsInBlock)` Throws `IllegalArgumentException` if any of these settings is invalid.
    `void` | `write(Fields fields, NormsProducer norms)` Write all fields, terms and postings.

    - ### Methods inherited from class org.apache.lucene.codecs.FieldsConsumer

      `merge`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### DEFAULT_MIN_BLOCK_SIZE

          public static final int DEFAULT_MIN_BLOCK_SIZE

      Suggested default value for the `minItemsInBlock` parameter to `BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.
      See Also:
        Constant Field Values

    - #### DEFAULT_MAX_BLOCK_SIZE

          public static final int DEFAULT_MAX_BLOCK_SIZE

      Suggested default value for the `maxItemsInBlock` parameter to `BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.
      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### BlockTreeTermsWriter

          public BlockTreeTermsWriter(SegmentWriteState state,
                                      PostingsWriterBase postingsWriter,
                                      int minItemsInBlock,
                                      int maxItemsInBlock)
                               throws IOException

      Create a new writer. The number of items (terms or sub-blocks) per block will aim to be between minItemsPerBlock and maxItemsPerBlock, though in some cases the blocks may be smaller than the min.
      Throws:
        `IOException`

  - ### Method Detail

    - #### validateSettings

          public static void validateSettings(int minItemsInBlock,
                                              int maxItemsInBlock)

      Throws `IllegalArgumentException` if any of these settings is invalid.

    - #### write

          public void write(Fields fields,
                            NormsProducer norms)
                     throws IOException

      Description copied from class: `FieldsConsumer`
      Write all fields, terms and postings. This the "pull" API, allowing you to iterate more than once over the postings, somewhat analogous to using a DOM API to traverse an XML tree.

      Notes:

      - You must compute index statistics, including each Term's docFreq and totalTermFreq, as well as the summary sumTotalTermFreq, sumTotalDocFreq and docCount.
      - You must skip terms that have no docs and fields that have no terms, even though the provided Fields API will expose them; this typically requires lazily writing the field or term until you've actually seen the first term or document.
      - The provided Fields instance is limited: you cannot call any methods that return statistics/counts; you cannot pass a non-null live docs when pulling docs/positions enums.
      Specified by:
        `write` in class `FieldsConsumer`
      Throws:
        `IOException`

    - #### close

          public void close()
                     throws IOException

      Specified by:
        `close` in interface `Closeable`
      Specified by:
        `close` in interface `AutoCloseable`
      Specified by:
        `close` in class `FieldsConsumer`
      Throws:
        `IOException`

Copyright © 2000-2024 Apache Software Foundation. All Rights Reserved.