org.apache.lucene.index

## Class ConcurrentMergeScheduler

- java.lang.Object
- - org.apache.lucene.index.MergeScheduler
  - - org.apache.lucene.index.ConcurrentMergeScheduler

- All Implemented Interfaces:
    Closeable, AutoCloseable

      public class ConcurrentMergeScheduler
      extends MergeScheduler

  A `MergeScheduler` that runs each merge using a separate thread.

  Specify the max number of threads that may run at once, and the maximum number of simultaneous merges with `setMaxMergesAndThreads(int, int)`.

  If the number of merges exceeds the max number of threads then the largest merges are paused until one of the smaller merges completes.

  If more than `getMaxMergeCount()` merges are requested then this class will forcefully throttle the incoming threads by pausing until one more merges complete.

  This class attempts to detect whether the index is on rotational storage (traditional hard drive) or not (e.g. solid-state disk) and changes the default max merge and thread count accordingly. This detection is currently Linux-only, and relies on the OS to put the right value into /sys/block/<dev>/block/rotational. For all other operating systems it currently assumes a rotational disk for backwards compatibility. To enable default settings for spinning or solid state disks for such operating systems, use `setDefaultMaxMergesAndThreads(boolean)`.

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class and Description
    --- | ---
    `protected class` | `ConcurrentMergeScheduler.MergeThread` Runs a merge thread to execute a single merge, then exits.

  - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static int` | `AUTO_DETECT_MERGES_AND_THREADS` Dynamic default for `maxThreadCount` and `maxMergeCount`, used to detect whether the index is backed by an SSD or rotational disk and set `maxThreadCount` accordingly.
    `static String` | `DEFAULT_CPU_CORE_COUNT_PROPERTY` Used for testing.
    `static String` | `DEFAULT_SPINS_PROPERTY` Used for testing.
    `protected int` | `mergeThreadCount` How many `ConcurrentMergeScheduler.MergeThread`s have kicked off (this is use to name them).
    `protected List<ConcurrentMergeScheduler.MergeThread>` | `mergeThreads` List of currently active `ConcurrentMergeScheduler.MergeThread`s.
    `protected double` | `targetMBPerSec` Current IO writes throttle rate

    - ### Fields inherited from class org.apache.lucene.index.MergeScheduler

      `infoStream`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `ConcurrentMergeScheduler()` Sole constructor, with all settings set to default values.

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `close()` Close this MergeScheduler.
    `void` | `disableAutoIOThrottle()` Turn off auto IO throttling.
    `protected void` | `doMerge(IndexWriter writer, MergePolicy.OneMerge merge)` Does the actual merge, by calling `IndexWriter.merge(org.apache.lucene.index.MergePolicy.OneMerge)`
    `protected void` | `doStall()` Called from `maybeStall(org.apache.lucene.index.IndexWriter)` to pause the calling thread for a bit.
    `void` | `enableAutoIOThrottle()` Turn on dynamic IO throttling, to adaptively rate limit writes bytes/sec to the minimal rate necessary so merges do not fall behind.
    `boolean` | `getAutoIOThrottle()` Returns true if auto IO throttling is currently enabled.
    `double` | `getForceMergeMBPerSec()` Get the per-merge IO throttle rate for forced merges.
    `double` | `getIORateLimitMBPerSec()` Returns the currently set per-merge IO writes rate limit, if `enableAutoIOThrottle()` was called, else `Double.POSITIVE_INFINITY`.
    `int` | `getMaxMergeCount()` See `setMaxMergesAndThreads(int, int)`.
    `int` | `getMaxThreadCount()` Returns `maxThreadCount`.
    `protected ConcurrentMergeScheduler.MergeThread` | `getMergeThread(IndexWriter writer, MergePolicy.OneMerge merge)` Create and return a new MergeThread
    `protected void` | `handleMergeException(Directory dir, Throwable exc)` Called when an exception is hit in a background merge thread
    `protected boolean` | `maybeStall(IndexWriter writer)` This is invoked by `merge(org.apache.lucene.index.IndexWriter, org.apache.lucene.index.MergeTrigger, boolean)` to possibly stall the incoming thread when there are too many merges running or pending.
    `void` | `merge(IndexWriter writer, MergeTrigger trigger, boolean newMergesFound)` Run the merges provided by `IndexWriter.getNextMerge()`.
    `int` | `mergeThreadCount()` Returns the number of merge threads that are alive, ignoring the calling thread if it is a merge thread.
    `void` | `setDefaultMaxMergesAndThreads(boolean spins)` Sets max merges and threads to proper defaults for rotational or non-rotational storage.
    `void` | `setForceMergeMBPerSec(double v)` Set the per-merge IO throttle rate for forced merges (default: `Double.POSITIVE_INFINITY`).
    `void` | `setMaxMergesAndThreads(int maxMergeCount, int maxThreadCount)` Expert: directly set the maximum number of merge threads and simultaneous merges allowed.
    `void` | `sync()` Wait for any running merge threads to finish.
    `protected void` | `targetMBPerSecChanged()` Subclass can override to tweak targetMBPerSec.
    `String` | `toString()`
    `protected void` | `updateMergeThreads()` Called whenever the running merges have changed, to set merge IO limits.
    `Directory` | `wrapForMerge(MergePolicy.OneMerge merge, Directory in)` Wraps the incoming `Directory` so that we can merge-throttle it using `RateLimitedIndexOutput`.

    - ### Methods inherited from class org.apache.lucene.index.MergeScheduler

      `message, verbose`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

- - ### Field Detail

    - #### AUTO_DETECT_MERGES_AND_THREADS

          public static final int AUTO_DETECT_MERGES_AND_THREADS

      Dynamic default for `maxThreadCount` and `maxMergeCount`, used to detect whether the index is backed by an SSD or rotational disk and set `maxThreadCount` accordingly. If it's an SSD, `maxThreadCount` is set to `max(1, min(4, cpuCoreCount/2))`, otherwise 1. Note that detection only currently works on Linux; other platforms will assume the index is not on an SSD.
      See Also:
        Constant Field Values

    - #### DEFAULT_CPU_CORE_COUNT_PROPERTY

          public static final String DEFAULT_CPU_CORE_COUNT_PROPERTY

      Used for testing.
      See Also:
        Constant Field Values
      NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.

    - #### DEFAULT_SPINS_PROPERTY

          public static final String DEFAULT_SPINS_PROPERTY

      Used for testing.
      See Also:
        Constant Field Values
      NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.

    - #### mergeThreads

          protected final List<ConcurrentMergeScheduler.MergeThread> mergeThreads

      List of currently active `ConcurrentMergeScheduler.MergeThread`s.

    - #### mergeThreadCount

          protected int mergeThreadCount

      How many `ConcurrentMergeScheduler.MergeThread`s have kicked off (this is use to name them).

    - #### targetMBPerSec

          protected double targetMBPerSec

      Current IO writes throttle rate

  - ### Constructor Detail

    - #### ConcurrentMergeScheduler

          public ConcurrentMergeScheduler()

      Sole constructor, with all settings set to default values.

  - ### Method Detail

    - #### setMaxMergesAndThreads

          public void setMaxMergesAndThreads(int maxMergeCount,
                                             int maxThreadCount)

      Expert: directly set the maximum number of merge threads and simultaneous merges allowed.
      Parameters:
        `maxMergeCount` - the max # simultaneous merges that are allowed. If a merge is necessary yet we already have this many threads running, the incoming thread (that is calling add/updateDocument) will block until a merge thread has completed. Note that we will only run the smallest `maxThreadCount` merges at a time.
        `maxThreadCount` - the max # simultaneous merge threads that should be running at once. This must be <= `maxMergeCount`

    - #### setDefaultMaxMergesAndThreads

          public void setDefaultMaxMergesAndThreads(boolean spins)

      Sets max merges and threads to proper defaults for rotational or non-rotational storage.
      Parameters:
        `spins` - true to set defaults best for traditional rotatational storage (spinning disks), else false (e.g. for solid-state disks)

    - #### setForceMergeMBPerSec

          public void setForceMergeMBPerSec(double v)

      Set the per-merge IO throttle rate for forced merges (default: `Double.POSITIVE_INFINITY`).

    - #### getForceMergeMBPerSec

          public double getForceMergeMBPerSec()

      Get the per-merge IO throttle rate for forced merges.

    - #### enableAutoIOThrottle

          public void enableAutoIOThrottle()

      Turn on dynamic IO throttling, to adaptively rate limit writes bytes/sec to the minimal rate necessary so merges do not fall behind. By default this is enabled.

    - #### disableAutoIOThrottle

          public void disableAutoIOThrottle()

      Turn off auto IO throttling.
      See Also:
        `enableAutoIOThrottle()`

    - #### getAutoIOThrottle

          public boolean getAutoIOThrottle()

      Returns true if auto IO throttling is currently enabled.

    - #### getIORateLimitMBPerSec

          public double getIORateLimitMBPerSec()

      Returns the currently set per-merge IO writes rate limit, if `enableAutoIOThrottle()` was called, else `Double.POSITIVE_INFINITY`.

    - #### getMaxThreadCount

          public int getMaxThreadCount()

      Returns `maxThreadCount`.
      See Also:
        `setMaxMergesAndThreads(int, int)`

    - #### getMaxMergeCount

          public int getMaxMergeCount()

      See `setMaxMergesAndThreads(int, int)`.

    - #### wrapForMerge

          public Directory wrapForMerge(MergePolicy.OneMerge merge,
                                        Directory in)

      Description copied from class: `MergeScheduler`
      Wraps the incoming `Directory` so that we can merge-throttle it using `RateLimitedIndexOutput`.
      Overrides:
        `wrapForMerge` in class `MergeScheduler`

    - #### updateMergeThreads

          protected void updateMergeThreads()

      Called whenever the running merges have changed, to set merge IO limits. This method sorts the merge threads by their merge size in descending order and then pauses/unpauses threads from first to last -- that way, smaller merges are guaranteed to run before larger ones.

    - #### close

          public void close()

      Description copied from class: `MergeScheduler`
      Close this MergeScheduler.
      Specified by:
        `close` in interface `Closeable`
      Specified by:
        `close` in interface `AutoCloseable`
      Specified by:
        `close` in class `MergeScheduler`

    - #### sync

          public void sync()

      Wait for any running merge threads to finish. This call is not interruptible as used by `close()`.

    - #### mergeThreadCount

          public int mergeThreadCount()

      Returns the number of merge threads that are alive, ignoring the calling thread if it is a merge thread. Note that this number is ≤ `mergeThreads` size.
      NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.

    - #### merge

          public void merge(IndexWriter writer,
                            MergeTrigger trigger,
                            boolean newMergesFound)
                     throws IOException

      Description copied from class: `MergeScheduler`
      Run the merges provided by `IndexWriter.getNextMerge()`.
      Specified by:
        `merge` in class `MergeScheduler`
      Parameters:
        `writer` - the `IndexWriter` to obtain the merges from.
        `trigger` - the `MergeTrigger` that caused this merge to happen
        `newMergesFound` - `true` iff any new merges were found by the caller otherwise `false`
      Throws:
        `IOException`

    - #### maybeStall

          protected boolean maybeStall(IndexWriter writer)

      This is invoked by `merge(org.apache.lucene.index.IndexWriter, org.apache.lucene.index.MergeTrigger, boolean)` to possibly stall the incoming thread when there are too many merges running or pending. The default behavior is to force this thread, which is producing too many segments for merging to keep up, to wait until merges catch up. Applications that can take other less drastic measures, such as limiting how many threads are allowed to index, can do nothing here and throttle elsewhere. If this method wants to stall but the calling thread is a merge thread, it should return false to tell caller not to kick off any new merges.

    - #### doStall

          protected void doStall()

      Called from `maybeStall(org.apache.lucene.index.IndexWriter)` to pause the calling thread for a bit.

    - #### doMerge

          protected void doMerge(IndexWriter writer,
                                 MergePolicy.OneMerge merge)
                          throws IOException

      Does the actual merge, by calling `IndexWriter.merge(org.apache.lucene.index.MergePolicy.OneMerge)`
      Throws:
        `IOException`

    - #### getMergeThread

          protected ConcurrentMergeScheduler.MergeThread getMergeThread(IndexWriter writer,
                                                                        MergePolicy.OneMerge merge)
                                                                 throws IOException

      Create and return a new MergeThread
      Throws:
        `IOException`

    - #### handleMergeException

          protected void handleMergeException(Directory dir,
                                              Throwable exc)

      Called when an exception is hit in a background merge thread

    - #### toString

          public String toString()

      Overrides:
        `toString` in class `Object`

    - #### targetMBPerSecChanged

          protected void targetMBPerSecChanged()

      Subclass can override to tweak targetMBPerSec.

Copyright © 2000-2020 Apache Software Foundation. All Rights Reserved.