Package org.apache.lucene.index

# Class CheckIndex

java.lang.Object
org.apache.lucene.index.CheckIndex

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`

public final class CheckIndex extends Object implements Closeable
Basic tool and API to check the health of an index and write a new segments file that removes reference to problematic segments.

As this tool checks every byte in the index, on a large index it can take quite a long time to run.

WARNING: This API is experimental and might change in incompatible ways in the next release.
  Please make a complete backup of your index before using this to exorcise corrupted documents from your index!

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static class`
  `CheckIndex.CheckIndexException`
  The marker RuntimeException used by CheckIndex APIs when index integrity failure is detected.
  `static class`
  `CheckIndex.Level`
  Class with static variables with information about CheckIndex's -level parameter.
  `static class`
  `CheckIndex.Options`
  Run-time configuration options for CheckIndex commands.
  `static class`
  `CheckIndex.Status`
  Returned from `checkIndex()` detailing the health and status of the index.
  `static class`
  `CheckIndex.VerifyPointsVisitor`
  Walks the entire N-dimensional points space, verifying that all points fall within the last cell's boundaries.

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `CheckIndex(Directory dir)`
  Create a new CheckIndex on the directory.
  `CheckIndex(Directory dir, Lock writeLock)`
  Expert: create a directory with the specified lock.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `static boolean`
  `assertsOn()`
  Check whether asserts are enabled or not.
  `CheckIndex.Status`
  `checkIndex()`
  Returns a `CheckIndex.Status` instance detailing the state of the index.
  `CheckIndex.Status`
  `checkIndex(List<String> onlySegments)`
  Returns a `CheckIndex.Status` instance detailing the state of the index.
  `CheckIndex.Status`
  `checkIndex(List<String> onlySegments, ExecutorService executorService)`
  Returns a `CheckIndex.Status` instance detailing the state of the index.
  `void`
  `close()`
  `int`
  `doCheck(CheckIndex.Options opts)`
  Actually perform the index check
  `void`
  `exorciseIndex(CheckIndex.Status result)`
  Repairs the index using previously returned result from `checkIndex()`.
  `boolean`
  `getFailFast()`
  See `setFailFast(boolean)`.
  `int`
  `getLevel()`
  See `setLevel(int)`.
  `static void`
  `main(String[] args)`
  Command-line interface to check and exorcise corrupt segments from an index.
  `static CheckIndex.Options`
  `parseOptions(String[] args)`
  Parse command line args into fields
  `void`
  `setFailFast(boolean v)`
  If true, just throw the original exception immediately when corruption is detected, rather than continuing to iterate to other segments looking for more corruption.
  `void`
  `setInfoStream(PrintStream out)`
  Set infoStream where messages should go.
  `void`
  `setInfoStream(PrintStream out, boolean verbose)`
  Set infoStream where messages should go.
  `void`
  `setLevel(int v)`
  Sets Level, the higher the value, the more additional checks are performed.
  `void`
  `setThreadCount(int tc)`
  Set threadCount used for parallelizing index integrity checking.
  `static CheckIndex.Status.DocValuesStatus`
  `testDocValues(CodecReader reader, PrintStream infoStream, boolean failFast)`
  Test docvalues.
  `static CheckIndex.Status.FieldInfoStatus`
  `testFieldInfos(CodecReader reader, PrintStream infoStream, boolean failFast)`
  Test field infos.
  `static CheckIndex.Status.FieldNormStatus`
  `testFieldNorms(CodecReader reader, PrintStream infoStream, boolean failFast)`
  Test field norms.
  `static CheckIndex.Status.LiveDocStatus`
  `testLiveDocs(CodecReader reader, PrintStream infoStream, boolean failFast)`
  Test live docs.
  `static CheckIndex.Status.PointsStatus`
  `testPoints(CodecReader reader, PrintStream infoStream, boolean failFast)`
  Test the points index.
  `static CheckIndex.Status.TermIndexStatus`
  `testPostings(CodecReader reader, PrintStream infoStream)`
  Test the term index.
  `static CheckIndex.Status.TermIndexStatus`
  `testPostings(CodecReader reader, PrintStream infoStream, boolean verbose, int level, boolean failFast)`
  Test the term index.
  `static CheckIndex.Status.IndexSortStatus`
  `testSort(CodecReader reader, Sort sort, PrintStream infoStream, boolean failFast)`
  Tests index sort order.
  `static CheckIndex.Status.StoredFieldStatus`
  `testStoredFields(CodecReader reader, PrintStream infoStream, boolean failFast)`
  Test stored fields.
  `static CheckIndex.Status.TermVectorStatus`
  `testTermVectors(CodecReader reader, PrintStream infoStream)`
  Test term vectors.
  `static CheckIndex.Status.TermVectorStatus`
  `testTermVectors(CodecReader reader, PrintStream infoStream, boolean verbose, int level, boolean failFast)`
  Test term vectors.
  `static CheckIndex.Status.VectorValuesStatus`
  `testVectors(CodecReader reader, PrintStream infoStream, boolean failFast)`
  Test the vectors index.

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Constructor Details

  - ### CheckIndex

    public CheckIndex(Directory dir) throws IOException
    Create a new CheckIndex on the directory.
    Throws:
      `IOException`

  - ### CheckIndex

    public CheckIndex(Directory dir, Lock writeLock)
    Expert: create a directory with the specified lock. This should really not be used except for unit tests!!!! It exists only to support special tests (such as TestIndexWriterExceptions*), that would otherwise be more complicated to debug if they had to close the writer for each check.

- ## Method Details

  - ### close

    public void close() throws IOException
    Specified by:
      `close` in interface `AutoCloseable`
    Specified by:
      `close` in interface `Closeable`
    Throws:
      `IOException`

  - ### setLevel

    public void setLevel(int v)
    Sets Level, the higher the value, the more additional checks are performed. This will likely drastically increase time it takes to run CheckIndex! See `CheckIndex.Level`

  - ### getLevel

    public int getLevel()
    See `setLevel(int)`.

  - ### setFailFast

    public void setFailFast(boolean v)
    If true, just throw the original exception immediately when corruption is detected, rather than continuing to iterate to other segments looking for more corruption.

  - ### getFailFast

    public boolean getFailFast()
    See `setFailFast(boolean)`.

  - ### setThreadCount

    public void setThreadCount(int tc)
    Set threadCount used for parallelizing index integrity checking.

  - ### setInfoStream

    public void setInfoStream(PrintStream out, boolean verbose)
    Set infoStream where messages should go. If null, no messages are printed. If verbose is true then more details are printed.

  - ### setInfoStream

    public void setInfoStream(PrintStream out)
    Set infoStream where messages should go. See `setInfoStream(PrintStream,boolean)`.

  - ### checkIndex

    public CheckIndex.Status checkIndex() throws IOException
    Returns a `CheckIndex.Status` instance detailing the state of the index.

    As this method checks every byte in the index, on a large index it can take quite a long time to run.

    WARNING: make sure you only call this when the index is not opened by any writer.

    Throws:
      `IOException`

  - ### checkIndex

    public CheckIndex.Status checkIndex(List<String> onlySegments) throws IOException
    Returns a `CheckIndex.Status` instance detailing the state of the index.
    Parameters:
      `onlySegments` - list of specific segment names to check

      As this method checks every byte in the specified segments, on a large index it can take quite a long time to run.

    Throws:
      `IOException`

  - ### checkIndex

    public CheckIndex.Status checkIndex(List<String> onlySegments, ExecutorService executorService) throws IOException
    Returns a `CheckIndex.Status` instance detailing the state of the index.

    This method allows caller to pass in customized ExecutorService to speed up the check.

    WARNING: make sure you only call this when the index is not opened by any writer.

    Throws:
      `IOException`

  - ### testSort

    public static CheckIndex.Status.IndexSortStatus testSort(CodecReader reader, Sort sort, PrintStream infoStream, boolean failFast) throws IOException
    Tests index sort order.
    Throws:
      `IOException`

  - ### testLiveDocs

    public static CheckIndex.Status.LiveDocStatus testLiveDocs(CodecReader reader, PrintStream infoStream, boolean failFast) throws IOException
    Test live docs.
    Throws:
      `IOException`

  - ### testFieldInfos

    public static CheckIndex.Status.FieldInfoStatus testFieldInfos(CodecReader reader, PrintStream infoStream, boolean failFast) throws IOException
    Test field infos.
    Throws:
      `IOException`

  - ### testFieldNorms

    public static CheckIndex.Status.FieldNormStatus testFieldNorms(CodecReader reader, PrintStream infoStream, boolean failFast) throws IOException
    Test field norms.
    Throws:
      `IOException`

  - ### testPostings

    public static CheckIndex.Status.TermIndexStatus testPostings(CodecReader reader, PrintStream infoStream) throws IOException
    Test the term index.
    Throws:
      `IOException`

  - ### testPostings

    public static CheckIndex.Status.TermIndexStatus testPostings(CodecReader reader, PrintStream infoStream, boolean verbose, int level, boolean failFast) throws IOException
    Test the term index.
    Throws:
      `IOException`

  - ### testPoints

    public static CheckIndex.Status.PointsStatus testPoints(CodecReader reader, PrintStream infoStream, boolean failFast) throws IOException
    Test the points index.
    Throws:
      `IOException`

  - ### testVectors

    public static CheckIndex.Status.VectorValuesStatus testVectors(CodecReader reader, PrintStream infoStream, boolean failFast) throws IOException
    Test the vectors index.
    Throws:
      `IOException`

  - ### testStoredFields

    public static CheckIndex.Status.StoredFieldStatus testStoredFields(CodecReader reader, PrintStream infoStream, boolean failFast) throws IOException
    Test stored fields.
    Throws:
      `IOException`

  - ### testDocValues

    public static CheckIndex.Status.DocValuesStatus testDocValues(CodecReader reader, PrintStream infoStream, boolean failFast) throws IOException
    Test docvalues.
    Throws:
      `IOException`

  - ### testTermVectors

    public static CheckIndex.Status.TermVectorStatus testTermVectors(CodecReader reader, PrintStream infoStream) throws IOException
    Test term vectors.
    Throws:
      `IOException`

  - ### testTermVectors

    public static CheckIndex.Status.TermVectorStatus testTermVectors(CodecReader reader, PrintStream infoStream, boolean verbose, int level, boolean failFast) throws IOException
    Test term vectors.
    Throws:
      `IOException`

  - ### exorciseIndex

    public void exorciseIndex(CheckIndex.Status result) throws IOException
    Repairs the index using previously returned result from `checkIndex()`. Note that this does not remove any of the unreferenced files after it's done; you must separately open an `IndexWriter`, which deletes unreferenced files when it's created.

    WARNING: this writes a new segments file into the index, effectively removing all documents in broken segments from the index. BE CAREFUL.

    Throws:
      `IOException`

  - ### assertsOn

    public static boolean assertsOn()
    Check whether asserts are enabled or not.
    Returns:
      true iff asserts are enabled

  - ### main

    public static void main(String[] args) throws IOException, InterruptedException
    Command-line interface to check and exorcise corrupt segments from an index.

    Run it like this:

         java -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex pathToIndex [-exorcise] [-verbose] [-segment X] [-segment Y]

    - `-exorcise`: actually write a new segments_N file, removing any problematic segments. *LOSES DATA*
    - `-segment X`: only check the specified segment(s). This can be specified multiple times, to check more than one segment: `-segment _2 * -segment _a`. You can't use this with the -exorcise option.

    WARNING: `-exorcise` should only be used on an emergency basis as it will cause documents (perhaps many) to be permanently removed from the index. Always make a backup copy of your index before running this! Do not run this tool on an index that is actively being written to. You have been warned!

    Run without -exorcise, this tool will open the index, report version information and report any exceptions it hits and what action it would take if -exorcise were specified. With -exorcise, this tool will remove any segments that have issues and write a new segments_N file. This means all documents contained in the affected segments will be removed.

    This tool exits with exit code 1 if the index cannot be opened or has any corruption, else 0.

    Throws:
      `IOException`
      `InterruptedException`

  - ### parseOptions

    public static CheckIndex.Options parseOptions(String[] args)
    Parse command line args into fields
    Parameters:
      `args` - The command line arguments
    Returns:
      An Options struct
    Throws:
      `IllegalArgumentException` - if any of the CLI args are invalid

  - ### doCheck

    public int doCheck(CheckIndex.Options opts) throws IOException, InterruptedException
    Actually perform the index check
    Parameters:
      `opts` - The options to use for this check
    Returns:
      0 iff the index is clean, 1 otherwise
    Throws:
      `IOException`
      `InterruptedException`