Package org.apache.datasketches.quantiles

Class ItemsSketch<T>

  • Type Parameters:
    T - type of item
    public final class ItemsSketch<T>
extends Object
    This is a stochastic streaming sketch that enables near-real time analysis of the approximate distribution of comparable items from a very large stream in a single pass. The analysis is obtained using a getQuantiles(*) function or its inverse functions the Probability Mass Function from getPMF(*) and the Cumulative Distribution Function from getCDF(*).

    The documentation for DoublesSketch applies here except that the size of an ItemsSketch is very dependent on the Items input into the sketch, so there is no comparable size table as for the DoublesSketch.

    There is more documentation available on datasketches.apache.org.

    Author:
    Kevin Lang, Alexander Saydakov

    • Field Summary

      Fields 
      Modifier and Type Field Description
      static Random rand
      Setting the seed makes the results of the sketch deterministic if the input values are received in exactly the same order.

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      ItemsSketch<T> downSample​(int newK)
      From an existing sketch, this creates a new sketch that can have a smaller value of K.
      double[] getCDF​(T[] splitPoints)
      Returns an approximation to the Cumulative Distribution Function (CDF), which is the cumulative analog of the PMF, of the input stream given a set of splitPoints (values).
      static <T> ItemsSketch<T> getInstance​(int k, Comparator<? super T> comparator)
      Obtains a new instance of an ItemsSketch.
      static <T> ItemsSketch<T> getInstance​(Comparator<? super T> comparator)
      Obtains a new instance of an ItemsSketch using the DEFAULT_K.
      static <T> ItemsSketch<T> getInstance​(org.apache.datasketches.memory.Memory srcMem, Comparator<? super T> comparator, ArrayOfItemsSerDe<T> serDe)
      Heapifies the given srcMem, which must be a Memory image of a ItemsSketch
      int getK()
      Returns the configured value of K
      static int getKFromEpsilon​(double epsilon, boolean pmf)
      Gets the approximate value of k to use given epsilon, the normalized rank error.
      T getMaxValue()
      Returns the max value of the stream
      T getMinValue()
      Returns the min value of the stream
      long getN()
      Returns the length of the input stream so far.
      double getNormalizedRankError​(boolean pmf)
      Gets the approximate rank error of this sketch normalized as a fraction between zero and one.
      static double getNormalizedRankError​(int k, boolean pmf)
      Gets the normalized rank error given k and pmf.
      double[] getPMF​(T[] splitPoints)
      Returns an approximation to the Probability Mass Function (PMF) of the input stream given a set of splitPoints (values).
      T getQuantile​(double fraction)
      This returns an approximation to the value of the data item that would be preceded by the given fraction of a hypothetical sorted version of the input stream so far.
      T getQuantileLowerBound​(double fraction)
      Gets the lower bound of the value interval in which the true quantile of the given rank exists with a confidence of at least 99%.
      T[] getQuantiles​(double[] fRanks)
      This is a more efficient multiple-query version of getQuantile().
      T[] getQuantiles​(int evenlySpaced)
      This is also a more efficient multiple-query version of getQuantile() and allows the caller to specify the number of evenly spaced fractional ranks.
      T getQuantileUpperBound​(double fraction)
      Gets the upper bound of the value interval in which the true quantile of the given rank exists with a confidence of at least 99%.
      double getRank​(T value)
      Returns an approximation to the normalized (fractional) rank of the given value from 0 to 1 inclusive.
      int getRetainedItems()
      Computes the number of retained entries (samples) in the sketch
      boolean isDirect()  
      boolean isEmpty()
      Returns true if this sketch is empty
      boolean isEstimationMode()  
      ItemsSketchIterator<T> iterator()  
      void putMemory​(org.apache.datasketches.memory.WritableMemory dstMem, ArrayOfItemsSerDe<T> serDe)
      Puts the current sketch into the given Memory if there is sufficient space.
      void reset()
      Resets this sketch to a virgin state, but retains the original value of k.
      byte[] toByteArray​(boolean ordered, ArrayOfItemsSerDe<T> serDe)
      Serialize this sketch to a byte array form.
      byte[] toByteArray​(ArrayOfItemsSerDe<T> serDe)
      Serialize this sketch to a byte array form.
      String toString()
      Returns summary information about this sketch.
      String toString​(boolean sketchSummary, boolean dataDetail)
      Returns summary information about this sketch.
      static String toString​(byte[] byteArr)
      Returns a human readable string of the preamble of a byte array image of an ItemsSketch.
      static String toString​(org.apache.datasketches.memory.Memory mem)
      Returns a human readable string of the preamble of a Memory image of an ItemsSketch.
      void update​(T dataItem)
      Updates this sketch with the given double data item

    • Field Detail

      • rand

        public static final Random rand
        Setting the seed makes the results of the sketch deterministic if the input values are received in exactly the same order. This is only useful when performing test comparisons, otherwise is not recommended.

    • Method Detail

      • getInstance

        public static <T> ItemsSketch<T> getInstance​(Comparator<? super T> comparator)
        Obtains a new instance of an ItemsSketch using the DEFAULT_K.
        Type Parameters:
        T - type of item
        Parameters:
        comparator - to compare items
        Returns:
        a GenericQuantileSketch

      • getInstance

        public static <T> ItemsSketch<T> getInstance​(int k,
                                             Comparator<? super T> comparator)
        Obtains a new instance of an ItemsSketch.
        Type Parameters:
        T - type of item
        Parameters:
        k - Parameter that controls space usage of sketch and accuracy of estimates. Must be greater than 2 and less than 65536 and a power of 2.
        comparator - to compare items
        Returns:
        a GenericQuantileSketch

      • getInstance

        public static <T> ItemsSketch<T> getInstance​(org.apache.datasketches.memory.Memory srcMem,
                                             Comparator<? super T> comparator,
                                             ArrayOfItemsSerDe<T> serDe)
        Heapifies the given srcMem, which must be a Memory image of a ItemsSketch
        Type Parameters:
        T - type of item
        Parameters:
        srcMem - a Memory image of a sketch. See Memory
        comparator - to compare items
        serDe - an instance of ArrayOfItemsSerDe
        Returns:
        a ItemsSketch on the Java heap.

      • update

        public void update​(T dataItem)
        Updates this sketch with the given double data item
        Parameters:
        dataItem - an item from a stream of items. NaNs are ignored.

      • getQuantile

        public T getQuantile​(double fraction)
        This returns an approximation to the value of the data item that would be preceded by the given fraction of a hypothetical sorted version of the input stream so far.

        We note that this method has a fairly large overhead (microseconds instead of nanoseconds) so it should not be called multiple times to get different quantiles from the same sketch. Instead use getQuantiles(). which pays the overhead only once.

        Parameters:
        fraction - the specified fractional position in the hypothetical sorted stream. These are also called normalized ranks or fractional ranks. If fraction = 0.0, the true minimum value of the stream is returned. If fraction = 1.0, the true maximum value of the stream is returned.
        Returns:
        the approximation to the value at the above fraction

      • getQuantileUpperBound

        public T getQuantileUpperBound​(double fraction)
        Gets the upper bound of the value interval in which the true quantile of the given rank exists with a confidence of at least 99%.
        Parameters:
        fraction - the given normalized rank as a fraction
        Returns:
        the upper bound of the value interval in which the true quantile of the given rank exists with a confidence of at least 99%. Returns NaN if the sketch is empty.

      • getQuantileLowerBound

        public T getQuantileLowerBound​(double fraction)
        Gets the lower bound of the value interval in which the true quantile of the given rank exists with a confidence of at least 99%.
        Parameters:
        fraction - the given normalized rank as a fraction
        Returns:
        the lower bound of the value interval in which the true quantile of the given rank exists with a confidence of at least 99%. Returns NaN if the sketch is empty.

      • getQuantiles

        public T[] getQuantiles​(double[] fRanks)
        This is a more efficient multiple-query version of getQuantile().

        This returns an array that could have been generated by using getQuantile() with many different fractional ranks, but would be very inefficient. This method incurs the internal set-up overhead once and obtains multiple quantile values in a single query. It is strongly recommend that this method be used instead of multiple calls to getQuantile().

        If the sketch is empty this returns null.

        Parameters:
        fRanks - the given array of fractional (or normalized) ranks in the hypothetical sorted stream of all the input values seen so far. These fRanks must all be in the interval [0.0, 1.0] inclusively.
        Returns:
        array of approximate quantiles of the given fRanks in the same order as in the given fRanks array.

      • getQuantiles

        public T[] getQuantiles​(int evenlySpaced)
        This is also a more efficient multiple-query version of getQuantile() and allows the caller to specify the number of evenly spaced fractional ranks.
        Parameters:
        evenlySpaced - an integer that specifies the number of evenly spaced fractional ranks. This must be a positive integer greater than 1. A value of 2 will return the min and the max value. A value of 3 will return the min, the median and the max value, etc.
        Returns:
        array of approximations to the given fractions in the same order as given fractions array.

      • getRank

        public double getRank​(T value)
        Returns an approximation to the normalized (fractional) rank of the given value from 0 to 1 inclusive.

        The resulting approximation has a probabilistic guarantee that be obtained from the getNormalizedRankError(false) function.

        If the sketch is empty this returns NaN.

        Parameters:
        value - to be ranked
        Returns:
        an approximate rank of the given value

      • getPMF

        public double[] getPMF​(T[] splitPoints)
        Returns an approximation to the Probability Mass Function (PMF) of the input stream given a set of splitPoints (values).

        The resulting approximations have a probabilistic guarantee that be obtained from the getNormalizedRankError(true) function.

        If the sketch is empty this returns null.

        Parameters:
        splitPoints - an array of m unique, monotonically increasing item values that divide the ordered space into m+1 consecutive disjoint intervals. The definition of an "interval" is inclusive of the left splitPoint (or minimum value) and exclusive of the right splitPoint, with the exception that the last interval will include the maximum value. It is not necessary to include either the min or max values in these splitpoints.
        Returns:
        an array of m+1 doubles each of which is an approximation to the fraction of the input stream values (the mass) that fall into one of those intervals. The definition of an "interval" is inclusive of the left splitPoint and exclusive of the right splitPoint, with the exception that the last interval will include maximum value.

      • getCDF

        public double[] getCDF​(T[] splitPoints)
        Returns an approximation to the Cumulative Distribution Function (CDF), which is the cumulative analog of the PMF, of the input stream given a set of splitPoints (values).

        The resulting approximations have a probabilistic guarantee that be obtained from the getNormalizedRankError(false) function.

        If the sketch is empty this returns null.

        Parameters:
        splitPoints - an array of m unique, monotonically increasing item values that divide the ordered space into m+1 consecutive disjoint intervals. The definition of an "interval" is inclusive of the left splitPoint (or minimum value) and exclusive of the right splitPoint, with the exception that the last interval will include the maximum value. It is not necessary to include either the min or max values in these splitpoints.
        Returns:
        an array of m+1 double values, which are a consecutive approximation to the CDF of the input stream given the splitPoints. The value at array position j of the returned CDF array is the sum of the returned values in positions 0 through j of the returned PMF array.

      • getK

        public int getK()
        Returns the configured value of K
        Returns:
        the configured value of K

      • getMinValue

        public T getMinValue()
        Returns the min value of the stream
        Returns:
        the min value of the stream

      • getMaxValue

        public T getMaxValue()
        Returns the max value of the stream
        Returns:
        the max value of the stream

      • getN

        public long getN()
        Returns the length of the input stream so far.
        Returns:
        the length of the input stream so far

      • getNormalizedRankError

        public double getNormalizedRankError​(boolean pmf)
        Gets the approximate rank error of this sketch normalized as a fraction between zero and one.
        Parameters:
        pmf - if true, returns the "double-sided" normalized rank error for the getPMF() function. Otherwise, it is the "single-sided" normalized rank error for all the other queries.
        Returns:
        if pmf is true, returns the normalized rank error for the getPMF() function. Otherwise, it is the "single-sided" normalized rank error for all the other queries.

      • getNormalizedRankError

        public static double getNormalizedRankError​(int k,
                                            boolean pmf)
        Gets the normalized rank error given k and pmf. Static method version of the getNormalizedRankError(boolean).
        Parameters:
        k - the configuation parameter
        pmf - if true, returns the "double-sided" normalized rank error for the getPMF() function. Otherwise, it is the "single-sided" normalized rank error for all the other queries.
        Returns:
        if pmf is true, the normalized rank error for the getPMF() function. Otherwise, it is the "single-sided" normalized rank error for all the other queries.

      • getKFromEpsilon

        public static int getKFromEpsilon​(double epsilon,
                                  boolean pmf)
        Gets the approximate value of k to use given epsilon, the normalized rank error.
        Parameters:
        epsilon - the normalized rank error between zero and one.
        pmf - if true, this function returns the value of k assuming the input epsilon is the desired "double-sided" epsilon for the getPMF() function. Otherwise, this function returns the value of k assuming the input epsilon is the desired "single-sided" epsilon for all the other queries.
        Returns:
        the value of k given a value of epsilon.

      • isEmpty

        public boolean isEmpty()
        Returns true if this sketch is empty
        Returns:
        true if this sketch is empty

      • isDirect

        public boolean isDirect()
        Returns:
        true if this sketch is off-heap

      • isEstimationMode

        public boolean isEstimationMode()
        Returns:
        true if in estimation mode

      • reset

        public void reset()
        Resets this sketch to a virgin state, but retains the original value of k.

      • toByteArray

        public byte[] toByteArray​(ArrayOfItemsSerDe<T> serDe)
        Serialize this sketch to a byte array form.
        Parameters:
        serDe - an instance of ArrayOfItemsSerDe
        Returns:
        byte array of this sketch

      • toByteArray

        public byte[] toByteArray​(boolean ordered,
                          ArrayOfItemsSerDe<T> serDe)
        Serialize this sketch to a byte array form.
        Parameters:
        ordered - if true the base buffer will be ordered (default == false).
        serDe - an instance of ArrayOfItemsSerDe
        Returns:
        this sketch in a byte array form.

      • toString

        public String toString()
        Returns summary information about this sketch.
        Overrides:
        toString in class Object

      • toString

        public String toString​(boolean sketchSummary,
                       boolean dataDetail)
        Returns summary information about this sketch. Used for debugging.
        Parameters:
        sketchSummary - if true includes sketch summary
        dataDetail - if true includes data detail
        Returns:
        summary information about the sketch.

      • toString

        public static String toString​(byte[] byteArr)
        Returns a human readable string of the preamble of a byte array image of an ItemsSketch.
        Parameters:
        byteArr - the given byte array
        Returns:
        a human readable string of the preamble of a byte array image of an ItemsSketch.

      • toString

        public static String toString​(org.apache.datasketches.memory.Memory mem)
        Returns a human readable string of the preamble of a Memory image of an ItemsSketch.
        Parameters:
        mem - the given Memory
        Returns:
        a human readable string of the preamble of a Memory image of an ItemsSketch.

      • downSample

        public ItemsSketch<T> downSample​(int newK)
        From an existing sketch, this creates a new sketch that can have a smaller value of K. The original sketch is not modified.
        Parameters:
        newK - the new value of K that must be smaller than current value of K. It is required that this.getK() = newK * 2^(nonnegative integer).
        Returns:
        the new sketch.

      • getRetainedItems

        public int getRetainedItems()
        Computes the number of retained entries (samples) in the sketch
        Returns:
        the number of retained entries (samples) in the sketch

      • putMemory

        public void putMemory​(org.apache.datasketches.memory.WritableMemory dstMem,
                      ArrayOfItemsSerDe<T> serDe)
        Puts the current sketch into the given Memory if there is sufficient space. Otherwise, throws an error.
        Parameters:
        dstMem - the given memory.
        serDe - an instance of ArrayOfItemsSerDe