org.apache.nutch.indexer.solr

## Class SolrDeleteDuplicates

- java.lang.Object
- - org.apache.hadoop.mapreduce.Reducer<Text,SolrDeleteDuplicates.SolrRecord,Text,SolrDeleteDuplicates.SolrRecord>
  - - org.apache.nutch.indexer.solr.SolrDeleteDuplicates

- All Implemented Interfaces:
    Configurable, Tool

      public class SolrDeleteDuplicates
      extends Reducer<Text,SolrDeleteDuplicates.SolrRecord,Text,SolrDeleteDuplicates.SolrRecord>
      implements Tool

  Utility class for deleting duplicate documents from a solr index. The algorithm goes like follows: Preparation:
  1. Query the solr server for the number of documents (say, N)
  2. Partition N among M map tasks. For example, if we have two map tasks the first map task will deal with solr documents from 0 - (N / 2 - 1) and the second will deal with documents from (N / 2) to (N - 1).
  MapReduce:
  - Map: Identity map where keys are digests and values are `SolrDeleteDuplicates.SolrRecord` instances(which contain id, boost and timestamp)
  - Reduce: After map, `SolrDeleteDuplicates.SolrRecord`s with the same digest will be grouped together. Now, of these documents with the same digests, delete all of them except the one with the highest score (boost field). If two (or more) documents have the same score, then the document with the latest timestamp is kept. Again, every other is deleted from solr index.
  Note that we assume that two documents in a solr index will never have the same URL. So this class only deals with documents with different URLs but the same digest.

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class and Description
    --- | ---
    `static class` | `SolrDeleteDuplicates.SolrInputFormat`
    `static class` | `SolrDeleteDuplicates.SolrInputSplit`
    `static class` | `SolrDeleteDuplicates.SolrRecord`
    `static class` | `SolrDeleteDuplicates.SolrRecordReader`

    - ### Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.Reducer

      `Reducer.Context`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `SolrDeleteDuplicates()`

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `cleanup(Reducer.Context context)`
    `boolean` | `dedup(java.lang.String solrUrl)`
    `Configuration` | `getConf()`
    `static void` | `main(java.lang.String[] args)`
    `void` | `reduce(Text key, java.lang.Iterable<SolrDeleteDuplicates.SolrRecord> values, Reducer.Context context)`
    `int` | `run(java.lang.String[] args)`
    `void` | `setConf(Configuration conf)`
    `void` | `setup(Reducer.Context job)`

    - ### Methods inherited from class org.apache.hadoop.mapreduce.Reducer

      `run`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### SolrDeleteDuplicates

          public SolrDeleteDuplicates()

  - ### Method Detail

    - #### getConf

          public Configuration getConf()

      Specified by:
        `getConf` in interface `Configurable`

    - #### setConf

          public void setConf(Configuration conf)

      Specified by:
        `setConf` in interface `Configurable`

    - #### setup

          public void setup(Reducer.Context job)
                     throws java.io.IOException

      Overrides:
        `setup` in class `Reducer<Text,SolrDeleteDuplicates.SolrRecord,Text,SolrDeleteDuplicates.SolrRecord>`
      Throws:
        `java.io.IOException`

    - #### cleanup

          public void cleanup(Reducer.Context context)
                       throws java.io.IOException

      Overrides:
        `cleanup` in class `Reducer<Text,SolrDeleteDuplicates.SolrRecord,Text,SolrDeleteDuplicates.SolrRecord>`
      Throws:
        `java.io.IOException`

    - #### reduce

          public void reduce(Text key,
                             java.lang.Iterable<SolrDeleteDuplicates.SolrRecord> values,
                             Reducer.Context context)
                      throws java.io.IOException

      Overrides:
        `reduce` in class `Reducer<Text,SolrDeleteDuplicates.SolrRecord,Text,SolrDeleteDuplicates.SolrRecord>`
      Throws:
        `java.io.IOException`

    - #### dedup

          public boolean dedup(java.lang.String solrUrl)
                        throws java.io.IOException,
                               java.lang.InterruptedException,
                               java.lang.ClassNotFoundException

      Throws:
        `java.io.IOException`
        `java.lang.InterruptedException`
        `java.lang.ClassNotFoundException`

    - #### run

          public int run(java.lang.String[] args)
                  throws java.io.IOException,
                         java.lang.InterruptedException,
                         java.lang.ClassNotFoundException

      Specified by:
        `run` in interface `Tool`
      Throws:
        `java.io.IOException`
        `java.lang.InterruptedException`
        `java.lang.ClassNotFoundException`

    - #### main

          public static void main(java.lang.String[] args)
                           throws java.lang.Exception

      Throws:
        `java.lang.Exception`

Copyright © 2019 The Apache Software Foundation