Package org.apache.lucene.codecs.lucene90.blocktree

# Class Lucene90BlockTreeTermsWriter

java.lang.Object
org.apache.lucene.codecs.FieldsConsumer
org.apache.lucene.codecs.lucene90.blocktree.Lucene90BlockTreeTermsWriter

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`

public final class Lucene90BlockTreeTermsWriter extends FieldsConsumer
Block-based terms index and dictionary writer.

Writes terms dict and index, block-encoding (column stride) each term's metadata for each set of terms between two index terms.

Files:

- `.tim`: Term Dictionary
- `.tmd`: Term Metadata
- `.tip`: Term Index

## Term Dictionary

The .tim file contains the list of terms in each field along with per-term statistics (such as docfreq) and per-term metadata (typically pointers to the postings list for that term in the inverted index).

The .tim is arranged in blocks: with blocks containing a variable number of entries (by default 25-48), where each entry is either a term or a reference to a sub-block.

NOTE: The term dictionary can plug into different postings implementations: the postings writer/reader are actually responsible for encoding and decoding the Postings Metadata and Term Metadata sections.

- TermsDict (.tim) --> Header, FieldDictNumFields, Footer
- FieldDict --> PostingsHeader, NodeBlockNumBlocks
- NodeBlock --> (OuterNode | InnerNode)
- OuterNode --> EntryCount, SuffixLength, ByteSuffixLength, StatsLength, < TermStats >EntryCount, MetaLength, <TermMetadata>EntryCount
- InnerNode --> EntryCount, SuffixLength[,Sub?], ByteSuffixLength, StatsLength, < TermStats ? >EntryCount, MetaLength, <TermMetadata ? >EntryCount
- TermStats --> DocFreq, TotalTermFreq
- Header --> `CodecHeader`
- EntryCount,SuffixLength,StatsLength,DocFreq,MetaLength --> `VInt`
- TotalTermFreq --> `VLong`
- Footer --> `CodecFooter`

Notes:

- Header is a `CodecHeader` storing the version information for the BlockTree implementation.
- DocFreq is the count of documents which contain the term.
- TotalTermFreq is the total number of occurrences of the term. This is encoded as the difference between the total number of occurrences and the DocFreq.
- PostingsHeader and TermMetadata are plugged into by the specific postings implementation: these contain arbitrary per-file data (such as parameters or versioning information) and per-term data (such as pointers to inverted files).
- For inner nodes of the tree, every entry will steal one bit to mark whether it points to child nodes(sub-block). If so, the corresponding TermStats and TermMetaData are omitted.

## Term Metadata

The .tmd file contains the list of term metadata (such as FST index metadata) and field level statistics (such as sum of total term freq).

- TermsMeta (.tmd) --> Header, NumFields, <FieldStats>NumFields, TermIndexLength, TermDictLength, Footer
- FieldStats --> FieldNumber, NumTerms, RootCodeLength, ByteRootCodeLength, SumTotalTermFreq?, SumDocFreq, DocCount, MinTerm, MaxTerm, IndexStartFP, FSTHeader, FSTMetadata
- Header,FSTHeader --> `CodecHeader`
- TermIndexLength, TermDictLength --> `Uint64`
- MinTerm,MaxTerm --> `VInt` length followed by the byte[]
- NumFields,FieldNumber,RootCodeLength,DocCount --> `VInt`
- NumTerms,SumTotalTermFreq,SumDocFreq,IndexStartFP --> `VLong`
- Footer --> `CodecFooter`

Notes:

- FieldNumber is the fields number from `FieldInfos`. (.fnm)
- NumTerms is the number of unique terms for the field.
- RootCode points to the root block for the field.
- SumDocFreq is the total number of postings, the number of term-document pairs across the entire field.
- DocCount is the number of documents that have at least one posting for this field.
- MinTerm, MaxTerm are the lowest and highest term in this field.

## Term Index

The .tip file contains an index into the term dictionary, so that it can be accessed randomly. The index is also used to determine when a given term cannot exist on disk (in the .tim file), saving a disk seek.

- TermsIndex (.tip) --> Header, FSTIndexNumFieldsFooter
- Header --> `CodecHeader`
- FSTIndex --> `FST<byte[]>`
- Footer --> `CodecFooter`

Notes:

- The .tip file contains a separate FST for each field. The FST maps a term prefix to the on-disk block that holds all terms starting with that prefix. Each field's IndexStartFP points to its FST.
- It's possible that an on-disk block would contain too many terms (more than the allowed maximum (default: 48)). When this happens, the block is sub-divided into new blocks (called "floor blocks"), and then the output in the FST for the block's prefix encodes the leading byte of each sub-block, and its file pointer.

See Also:
  - `Lucene90BlockTreeTermsReader`
WARNING: This API is experimental and might change in incompatible ways in the next release.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final int`
  `DEFAULT_MAX_BLOCK_SIZE`
  Suggested default value for the `maxItemsInBlock` parameter to `Lucene90BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.
  `static final int`
  `DEFAULT_MIN_BLOCK_SIZE`
  Suggested default value for the `minItemsInBlock` parameter to `Lucene90BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `Lucene90BlockTreeTermsWriter(SegmentWriteState state, PostingsWriterBase postingsWriter, int minItemsInBlock, int maxItemsInBlock)`
  Create a new writer.
  `Lucene90BlockTreeTermsWriter(SegmentWriteState state, PostingsWriterBase postingsWriter, int minItemsInBlock, int maxItemsInBlock, int version)`
  Expert constructor that allows configuring the version, used for bw tests.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `close()`
  `static void`
  `validateSettings(int minItemsInBlock, int maxItemsInBlock)`
  Throws `IllegalArgumentException` if any of these settings is invalid.
  `void`
  `write(Fields fields, NormsProducer norms)`
  Write all fields, terms and postings.

  ### Methods inherited from class org.apache.lucene.codecs.FieldsConsumer

  `merge`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### DEFAULT_MIN_BLOCK_SIZE

    public static final int DEFAULT_MIN_BLOCK_SIZE
    Suggested default value for the `minItemsInBlock` parameter to `Lucene90BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.
    See Also:
      - Constant Field Values

  - ### DEFAULT_MAX_BLOCK_SIZE

    public static final int DEFAULT_MAX_BLOCK_SIZE
    Suggested default value for the `maxItemsInBlock` parameter to `Lucene90BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### Lucene90BlockTreeTermsWriter

    public Lucene90BlockTreeTermsWriter(SegmentWriteState state, PostingsWriterBase postingsWriter, int minItemsInBlock, int maxItemsInBlock) throws IOException
    Create a new writer. The number of items (terms or sub-blocks) per block will aim to be between minItemsPerBlock and maxItemsPerBlock, though in some cases the blocks may be smaller than the min.
    Throws:
      `IOException`

  - ### Lucene90BlockTreeTermsWriter

    public Lucene90BlockTreeTermsWriter(SegmentWriteState state, PostingsWriterBase postingsWriter, int minItemsInBlock, int maxItemsInBlock, int version) throws IOException
    Expert constructor that allows configuring the version, used for bw tests.
    Throws:
      `IOException`

- ## Method Details

  - ### validateSettings

    public static void validateSettings(int minItemsInBlock, int maxItemsInBlock)
    Throws `IllegalArgumentException` if any of these settings is invalid.

  - ### write

    public void write(Fields fields, NormsProducer norms) throws IOException
    Description copied from class: `FieldsConsumer`
    Write all fields, terms and postings. This the "pull" API, allowing you to iterate more than once over the postings, somewhat analogous to using a DOM API to traverse an XML tree.

    Notes:

    - You must compute index statistics, including each Term's docFreq and totalTermFreq, as well as the summary sumTotalTermFreq, sumTotalDocFreq and docCount.
    - You must skip terms that have no docs and fields that have no terms, even though the provided Fields API will expose them; this typically requires lazily writing the field or term until you've actually seen the first term or document.
    - The provided Fields instance is limited: you cannot call any methods that return statistics/counts; you cannot pass a non-null live docs when pulling docs/positions enums.
    Specified by:
      `write` in class `FieldsConsumer`
    Throws:
      `IOException`

  - ### close

    public void close() throws IOException
    Specified by:
      `close` in interface `AutoCloseable`
    Specified by:
      `close` in interface `Closeable`
    Specified by:
      `close` in class `FieldsConsumer`
    Throws:
      `IOException`