org.apache.lucene.index

## Class CheckIndex

- java.lang.Object
- - org.apache.lucene.index.CheckIndex

- All Implemented Interfaces:
    Closeable, AutoCloseable

      public final class CheckIndex
      extends Object
      implements Closeable

  Basic tool and API to check the health of an index and write a new segments file that removes reference to problematic segments.

  As this tool checks every byte in the index, on a large index it can take quite a long time to run.

  WARNING: This API is experimental and might change in incompatible ways in the next release.
    Please make a complete backup of your index before using this to exorcise corrupted documents from your index!

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class and Description
    --- | ---
    `static class` | `CheckIndex.CheckIndexException` The marker RuntimeException used by CheckIndex APIs when index integrity failure is detected.
    `static class` | `CheckIndex.Options` Run-time configuration options for CheckIndex commands.
    `static class` | `CheckIndex.Status` Returned from `checkIndex()` detailing the health and status of the index.
    `static class` | `CheckIndex.VerifyPointsVisitor` Walks the entire N-dimensional points space, verifying that all points fall within the last cell's boundaries.

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `CheckIndex(Directory dir)` Create a new CheckIndex on the directory.
    `CheckIndex(Directory dir, Lock writeLock)` Expert: create a directory with the specified lock.

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `static boolean` | `assertsOn()` Check whether asserts are enabled or not.
    `CheckIndex.Status` | `checkIndex()` Returns a `CheckIndex.Status` instance detailing the state of the index.
    `CheckIndex.Status` | `checkIndex(List<String> onlySegments)` Returns a `CheckIndex.Status` instance detailing the state of the index.
    `CheckIndex.Status` | `checkIndex(List<String> onlySegments, ExecutorService executorService)` Returns a `CheckIndex.Status` instance detailing the state of the index.
    `void` | `close()`
    `int` | `doCheck(CheckIndex.Options opts)` Actually perform the index check
    `boolean` | `doSlowChecks()` See `setDoSlowChecks(boolean)`.
    `void` | `exorciseIndex(CheckIndex.Status result)` Repairs the index using previously returned result from `checkIndex()`.
    `boolean` | `getChecksumsOnly()` See `getChecksumsOnly()`.
    `boolean` | `getFailFast()` See `setFailFast(boolean)`.
    `static void` | `main(String[] args)` Command-line interface to check and exorcise corrupt segments from an index.
    `static CheckIndex.Options` | `parseOptions(String[] args)` Parse command line args into fields
    `void` | `setChecksumsOnly(boolean v)` If true, only validate physical integrity for all files.
    `void` | `setDoSlowChecks(boolean v)` If true, additional slow checks are performed.
    `void` | `setFailFast(boolean v)` If true, just throw the original exception immediately when corruption is detected, rather than continuing to iterate to other segments looking for more corruption.
    `void` | `setInfoStream(PrintStream out)` Set infoStream where messages should go.
    `void` | `setInfoStream(PrintStream out, boolean verbose)` Set infoStream where messages should go.
    `void` | `setThreadCount(int tc)` Set threadCount used for parallelizing index integrity checking.
    `static CheckIndex.Status.DocValuesStatus` | `testDocValues(CodecReader reader, PrintStream infoStream, boolean failFast)` Test docvalues.
    `static CheckIndex.Status.FieldInfoStatus` | `testFieldInfos(CodecReader reader, PrintStream infoStream, boolean failFast)` Test field infos.
    `static CheckIndex.Status.FieldNormStatus` | `testFieldNorms(CodecReader reader, PrintStream infoStream, boolean failFast)` Test field norms.
    `static CheckIndex.Status.LiveDocStatus` | `testLiveDocs(CodecReader reader, PrintStream infoStream, boolean failFast)` Test live docs.
    `static CheckIndex.Status.PointsStatus` | `testPoints(CodecReader reader, PrintStream infoStream, boolean failFast)` Test the points index
    `static CheckIndex.Status.TermIndexStatus` | `testPostings(CodecReader reader, PrintStream infoStream)` Test the term index.
    `static CheckIndex.Status.TermIndexStatus` | `testPostings(CodecReader reader, PrintStream infoStream, boolean verbose, boolean doSlowChecks, boolean failFast)` Test the term index.
    `static CheckIndex.Status.IndexSortStatus` | `testSort(CodecReader reader, Sort sort, PrintStream infoStream, boolean failFast)` Tests index sort order.
    `static CheckIndex.Status.StoredFieldStatus` | `testStoredFields(CodecReader reader, PrintStream infoStream, boolean failFast)` Test stored fields.
    `static CheckIndex.Status.TermVectorStatus` | `testTermVectors(CodecReader reader, PrintStream infoStream)` Test term vectors.
    `static CheckIndex.Status.TermVectorStatus` | `testTermVectors(CodecReader reader, PrintStream infoStream, boolean verbose, boolean doSlowChecks, boolean failFast)` Test term vectors.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### CheckIndex

          public CheckIndex(Directory dir)
                     throws IOException

      Create a new CheckIndex on the directory.
      Throws:
        `IOException`

    - #### CheckIndex

          public CheckIndex(Directory dir,
                            Lock writeLock)

      Expert: create a directory with the specified lock. This should really not be used except for unit tests!!!! It exists only to support special tests (such as TestIndexWriterExceptions*), that would otherwise be more complicated to debug if they had to close the writer for each check.

  - ### Method Detail

    - #### close

          public void close()
                     throws IOException

      Specified by:
        `close` in interface `Closeable`
      Specified by:
        `close` in interface `AutoCloseable`
      Throws:
        `IOException`

    - #### setDoSlowChecks

          public void setDoSlowChecks(boolean v)

      If true, additional slow checks are performed. This will likely drastically increase time it takes to run CheckIndex!

    - #### doSlowChecks

          public boolean doSlowChecks()

      See `setDoSlowChecks(boolean)`.

    - #### setFailFast

          public void setFailFast(boolean v)

      If true, just throw the original exception immediately when corruption is detected, rather than continuing to iterate to other segments looking for more corruption.

    - #### getFailFast

          public boolean getFailFast()

      See `setFailFast(boolean)`.

    - #### getChecksumsOnly

          public boolean getChecksumsOnly()

      See `getChecksumsOnly()`.

    - #### setChecksumsOnly

          public void setChecksumsOnly(boolean v)

      If true, only validate physical integrity for all files. Note that the returned nested status objects (e.g. storedFieldStatus) will be null.

    - #### setThreadCount

          public void setThreadCount(int tc)

      Set threadCount used for parallelizing index integrity checking.

    - #### setInfoStream

          public void setInfoStream(PrintStream out,
                                    boolean verbose)

      Set infoStream where messages should go. If null, no messages are printed. If verbose is true then more details are printed.

    - #### setInfoStream

          public void setInfoStream(PrintStream out)

      Set infoStream where messages should go. See `setInfoStream(PrintStream,boolean)`.

    - #### checkIndex

          public CheckIndex.Status checkIndex()
                                       throws IOException

      Returns a `CheckIndex.Status` instance detailing the state of the index.

      As this method checks every byte in the index, on a large index it can take quite a long time to run.

      WARNING: make sure you only call this when the index is not opened by any writer.

      Throws:
        `IOException`

    - #### checkIndex

          public CheckIndex.Status checkIndex(List<String> onlySegments)
                                       throws IOException

      Returns a `CheckIndex.Status` instance detailing the state of the index.
      Parameters:
        `onlySegments` - list of specific segment names to check

        As this method checks every byte in the specified segments, on a large index it can take quite a long time to run.

      Throws:
        `IOException`

    - #### checkIndex

          public CheckIndex.Status checkIndex(List<String> onlySegments,
                                              ExecutorService executorService)
                                       throws IOException

      Returns a `CheckIndex.Status` instance detailing the state of the index.

      This method allows caller to pass in customized ExecutorService to speed up the check.

      WARNING: make sure you only call this when the index is not opened by any writer.

      Throws:
        `IOException`

    - #### testSort

          public static CheckIndex.Status.IndexSortStatus testSort(CodecReader reader,
                                                                   Sort sort,
                                                                   PrintStream infoStream,
                                                                   boolean failFast)
                                                            throws IOException

      Tests index sort order.
      Throws:
        `IOException`
      WARNING: This API is experimental and might change in incompatible ways in the next release.

    - #### testLiveDocs

          public static CheckIndex.Status.LiveDocStatus testLiveDocs(CodecReader reader,
                                                                     PrintStream infoStream,
                                                                     boolean failFast)
                                                              throws IOException

      Test live docs.
      Throws:
        `IOException`
      WARNING: This API is experimental and might change in incompatible ways in the next release.

    - #### testFieldInfos

          public static CheckIndex.Status.FieldInfoStatus testFieldInfos(CodecReader reader,
                                                                         PrintStream infoStream,
                                                                         boolean failFast)
                                                                  throws IOException

      Test field infos.
      Throws:
        `IOException`
      WARNING: This API is experimental and might change in incompatible ways in the next release.

    - #### testFieldNorms

          public static CheckIndex.Status.FieldNormStatus testFieldNorms(CodecReader reader,
                                                                         PrintStream infoStream,
                                                                         boolean failFast)
                                                                  throws IOException

      Test field norms.
      Throws:
        `IOException`
      WARNING: This API is experimental and might change in incompatible ways in the next release.

    - #### testPostings

          public static CheckIndex.Status.TermIndexStatus testPostings(CodecReader reader,
                                                                       PrintStream infoStream)
                                                                throws IOException

      Test the term index.
      Throws:
        `IOException`
      WARNING: This API is experimental and might change in incompatible ways in the next release.

    - #### testPostings

          public static CheckIndex.Status.TermIndexStatus testPostings(CodecReader reader,
                                                                       PrintStream infoStream,
                                                                       boolean verbose,
                                                                       boolean doSlowChecks,
                                                                       boolean failFast)
                                                                throws IOException

      Test the term index.
      Throws:
        `IOException`
      WARNING: This API is experimental and might change in incompatible ways in the next release.

    - #### testPoints

          public static CheckIndex.Status.PointsStatus testPoints(CodecReader reader,
                                                                  PrintStream infoStream,
                                                                  boolean failFast)
                                                           throws IOException

      Test the points index
      Throws:
        `IOException`
      WARNING: This API is experimental and might change in incompatible ways in the next release.

    - #### testStoredFields

          public static CheckIndex.Status.StoredFieldStatus testStoredFields(CodecReader reader,
                                                                             PrintStream infoStream,
                                                                             boolean failFast)
                                                                      throws IOException

      Test stored fields.
      Throws:
        `IOException`
      WARNING: This API is experimental and might change in incompatible ways in the next release.

    - #### testDocValues

          public static CheckIndex.Status.DocValuesStatus testDocValues(CodecReader reader,
                                                                        PrintStream infoStream,
                                                                        boolean failFast)
                                                                 throws IOException

      Test docvalues.
      Throws:
        `IOException`
      WARNING: This API is experimental and might change in incompatible ways in the next release.

    - #### testTermVectors

          public static CheckIndex.Status.TermVectorStatus testTermVectors(CodecReader reader,
                                                                           PrintStream infoStream)
                                                                    throws IOException

      Test term vectors.
      Throws:
        `IOException`
      WARNING: This API is experimental and might change in incompatible ways in the next release.

    - #### testTermVectors

          public static CheckIndex.Status.TermVectorStatus testTermVectors(CodecReader reader,
                                                                           PrintStream infoStream,
                                                                           boolean verbose,
                                                                           boolean doSlowChecks,
                                                                           boolean failFast)
                                                                    throws IOException

      Test term vectors.
      Throws:
        `IOException`
      WARNING: This API is experimental and might change in incompatible ways in the next release.

    - #### exorciseIndex

          public void exorciseIndex(CheckIndex.Status result)
                             throws IOException

      Repairs the index using previously returned result from `checkIndex()`. Note that this does not remove any of the unreferenced files after it's done; you must separately open an `IndexWriter`, which deletes unreferenced files when it's created.

      WARNING: this writes a new segments file into the index, effectively removing all documents in broken segments from the index. BE CAREFUL.

      Throws:
        `IOException`

    - #### assertsOn

          public static boolean assertsOn()

      Check whether asserts are enabled or not.
      Returns:
        true iff asserts are enabled

    - #### main

          public static void main(String[] args)
                           throws IOException,
                                  InterruptedException

      Command-line interface to check and exorcise corrupt segments from an index.

      Run it like this:

              java -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex pathToIndex [-exorcise] [-verbose] [-segment X] [-segment Y]

      - `-exorcise`: actually write a new segments_N file, removing any problematic segments. *LOSES DATA*
      - `-segment X`: only check the specified segment(s). This can be specified multiple times, to check more than one segment, eg `-segment _2 -segment _a`. You can't use this with the -exorcise option.

      WARNING: `-exorcise` should only be used on an emergency basis as it will cause documents (perhaps many) to be permanently removed from the index. Always make a backup copy of your index before running this! Do not run this tool on an index that is actively being written to. You have been warned!

      Run without -exorcise, this tool will open the index, report version information and report any exceptions it hits and what action it would take if -exorcise were specified. With -exorcise, this tool will remove any segments that have issues and write a new segments_N file. This means all documents contained in the affected segments will be removed.

      This tool exits with exit code 1 if the index cannot be opened or has any corruption, else 0.

      Throws:
        `IOException`
        `InterruptedException`

    - #### parseOptions

          public static CheckIndex.Options parseOptions(String[] args)

      Parse command line args into fields
      Parameters:
        `args` - The command line arguments
      Returns:
        An Options struct
      Throws:
        `IllegalArgumentException` - if any of the CLI args are invalid

    - #### doCheck

          public int doCheck(CheckIndex.Options opts)
                      throws IOException,
                             InterruptedException

      Actually perform the index check
      Parameters:
        `opts` - The options to use for this check
      Returns:
        0 iff the index is clean, 1 otherwise
      Throws:
        `IOException`
        `InterruptedException`

Copyright © 2000-2024 Apache Software Foundation. All Rights Reserved.