org.apache.nutch.crawl

## Class Generator

- java.lang.Object
- - org.apache.hadoop.conf.Configured
  - - org.apache.nutch.util.NutchTool
    - - org.apache.nutch.crawl.Generator

- All Implemented Interfaces:
    Configurable, Tool

      public class Generator
      extends NutchTool
      implements Tool

  Generates a subset of a crawl db to fetch. This version allows to generate fetchlists for several segments in one go. Unlike in the initial version (OldGenerator), the IP resolution is done ONLY on the entries which have been selected for fetching. The URLs are partitioned by IP, domain or host within a segment. We can chose separately how to count the URLS i.e. by domain or host to limit the entries.

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class and Description
    --- | ---
    `static class` | `Generator.CrawlDbUpdater` Update the CrawlDB so that the next generate won't include the same URLs.
    `static class` | `Generator.DecreasingFloatComparator`
    `static class` | `Generator.HashComparator` Sort fetch lists by hash of URL.
    `static class` | `Generator.PartitionReducer`
    `static class` | `Generator.Selector` Selects entries due for fetch.
    `static class` | `Generator.SelectorEntry`
    `static class` | `Generator.SelectorInverseMapper`
    `static class` | `Generator.SelectorMapper` Select and invert subset due for fetch.
    `static class` | `Generator.SelectorReducer` Collect until limit is reached.

  - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static java.lang.String` | `GENERATE_UPDATE_CRAWLDB`
    `static java.lang.String` | `GENERATOR_COUNT_MODE`
    `static java.lang.String` | `GENERATOR_COUNT_VALUE_DOMAIN`
    `static java.lang.String` | `GENERATOR_COUNT_VALUE_HOST`
    `static java.lang.String` | `GENERATOR_CUR_TIME`
    `static java.lang.String` | `GENERATOR_DELAY`
    `static java.lang.String` | `GENERATOR_EXPR`
    `static java.lang.String` | `GENERATOR_FETCH_DELAY_EXPR`
    `static java.lang.String` | `GENERATOR_FILTER`
    `static java.lang.String` | `GENERATOR_HOSTDB`
    `static java.lang.String` | `GENERATOR_MAX_COUNT`
    `static java.lang.String` | `GENERATOR_MAX_COUNT_EXPR`
    `static java.lang.String` | `GENERATOR_MAX_NUM_SEGMENTS`
    `static java.lang.String` | `GENERATOR_MIN_INTERVAL`
    `static java.lang.String` | `GENERATOR_MIN_SCORE`
    `static java.lang.String` | `GENERATOR_NORMALISE`
    `static java.lang.String` | `GENERATOR_RESTRICT_STATUS`
    `static java.lang.String` | `GENERATOR_TOP_N`
    `protected static org.slf4j.Logger` | `LOG`

    - ### Fields inherited from class org.apache.nutch.util.NutchTool

      `currentJob, currentJobNum, numJobs, results, status`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `Generator()`
    `Generator(Configuration conf)`

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `Path[]` | `generate(Path dbDir, Path segments, int numLists, long topN, long curTime)`
    `Path[]` | `generate(Path dbDir, Path segments, int numLists, long topN, long curTime, boolean filter, boolean force)` old signature used for compatibility - does not specify whether or not to normalise and set the number of segments to 1
    `Path[]` | `generate(Path dbDir, Path segments, int numLists, long topN, long curTime, boolean filter, boolean norm, boolean force, int maxNumSegments, java.lang.String expr)`
    `Path[]` | `generate(Path dbDir, Path segments, int numLists, long topN, long curTime, boolean filter, boolean norm, boolean force, int maxNumSegments, java.lang.String expr, java.lang.String hostdb)` Generate fetchlists in one or more segments.
    `static java.lang.String` | `generateSegmentName()`
    `static void` | `main(java.lang.String[] args)` Generate a fetchlist from the crawldb.
    `java.util.Map<java.lang.String,java.lang.Object>` | `run(java.util.Map<java.lang.String,java.lang.Object> args, java.lang.String crawlId)` Runs the tool, using a map of arguments.
    `int` | `run(java.lang.String[] args)`

    - ### Methods inherited from class org.apache.nutch.util.NutchTool

      `getProgress, getStatus, killJob, stopJob`

    - ### Methods inherited from class org.apache.hadoop.conf.Configured

      `getConf, setConf`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

    - ### Methods inherited from interface org.apache.hadoop.conf.Configurable

      `getConf, setConf`

- - ### Field Detail

    - #### LOG

          protected static final org.slf4j.Logger LOG

    - #### GENERATE_UPDATE_CRAWLDB

          public static final java.lang.String GENERATE_UPDATE_CRAWLDB

      See Also:
        Constant Field Values

    - #### GENERATOR_MIN_SCORE

          public static final java.lang.String GENERATOR_MIN_SCORE

      See Also:
        Constant Field Values

    - #### GENERATOR_MIN_INTERVAL

          public static final java.lang.String GENERATOR_MIN_INTERVAL

      See Also:
        Constant Field Values

    - #### GENERATOR_RESTRICT_STATUS

          public static final java.lang.String GENERATOR_RESTRICT_STATUS

      See Also:
        Constant Field Values

    - #### GENERATOR_FILTER

          public static final java.lang.String GENERATOR_FILTER

      See Also:
        Constant Field Values

    - #### GENERATOR_NORMALISE

          public static final java.lang.String GENERATOR_NORMALISE

      See Also:
        Constant Field Values

    - #### GENERATOR_MAX_COUNT

          public static final java.lang.String GENERATOR_MAX_COUNT

      See Also:
        Constant Field Values

    - #### GENERATOR_COUNT_MODE

          public static final java.lang.String GENERATOR_COUNT_MODE

      See Also:
        Constant Field Values

    - #### GENERATOR_COUNT_VALUE_DOMAIN

          public static final java.lang.String GENERATOR_COUNT_VALUE_DOMAIN

      See Also:
        Constant Field Values

    - #### GENERATOR_COUNT_VALUE_HOST

          public static final java.lang.String GENERATOR_COUNT_VALUE_HOST

      See Also:
        Constant Field Values

    - #### GENERATOR_TOP_N

          public static final java.lang.String GENERATOR_TOP_N

      See Also:
        Constant Field Values

    - #### GENERATOR_CUR_TIME

          public static final java.lang.String GENERATOR_CUR_TIME

      See Also:
        Constant Field Values

    - #### GENERATOR_DELAY

          public static final java.lang.String GENERATOR_DELAY

      See Also:
        Constant Field Values

    - #### GENERATOR_MAX_NUM_SEGMENTS

          public static final java.lang.String GENERATOR_MAX_NUM_SEGMENTS

      See Also:
        Constant Field Values

    - #### GENERATOR_EXPR

          public static final java.lang.String GENERATOR_EXPR

      See Also:
        Constant Field Values

    - #### GENERATOR_HOSTDB

          public static final java.lang.String GENERATOR_HOSTDB

      See Also:
        Constant Field Values

    - #### GENERATOR_MAX_COUNT_EXPR

          public static final java.lang.String GENERATOR_MAX_COUNT_EXPR

      See Also:
        Constant Field Values

    - #### GENERATOR_FETCH_DELAY_EXPR

          public static final java.lang.String GENERATOR_FETCH_DELAY_EXPR

      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### Generator

          public Generator()

    - #### Generator

          public Generator(Configuration conf)

  - ### Method Detail

    - #### generate

          public Path[] generate(Path dbDir,
                                 Path segments,
                                 int numLists,
                                 long topN,
                                 long curTime)
                          throws java.io.IOException,
                                 java.lang.InterruptedException,
                                 java.lang.ClassNotFoundException

      Throws:
        `java.io.IOException`
        `java.lang.InterruptedException`
        `java.lang.ClassNotFoundException`

    - #### generate

          public Path[] generate(Path dbDir,
                                 Path segments,
                                 int numLists,
                                 long topN,
                                 long curTime,
                                 boolean filter,
                                 boolean force)
                          throws java.io.IOException,
                                 java.lang.InterruptedException,
                                 java.lang.ClassNotFoundException

      old signature used for compatibility - does not specify whether or not to normalise and set the number of segments to 1
      Throws:
        `java.io.IOException`
        `java.lang.InterruptedException`
        `java.lang.ClassNotFoundException`

    - #### generate

          public Path[] generate(Path dbDir,
                                 Path segments,
                                 int numLists,
                                 long topN,
                                 long curTime,
                                 boolean filter,
                                 boolean norm,
                                 boolean force,
                                 int maxNumSegments,
                                 java.lang.String expr)
                          throws java.io.IOException,
                                 java.lang.InterruptedException,
                                 java.lang.ClassNotFoundException

      Throws:
        `java.io.IOException`
        `java.lang.InterruptedException`
        `java.lang.ClassNotFoundException`

    - #### generate

          public Path[] generate(Path dbDir,
                                 Path segments,
                                 int numLists,
                                 long topN,
                                 long curTime,
                                 boolean filter,
                                 boolean norm,
                                 boolean force,
                                 int maxNumSegments,
                                 java.lang.String expr,
                                 java.lang.String hostdb)
                          throws java.io.IOException,
                                 java.lang.InterruptedException,
                                 java.lang.ClassNotFoundException

      Generate fetchlists in one or more segments. Whether to filter URLs or not is read from the "generate.filter" property set for the job from command-line. If the property is not found, the URLs are filtered. Same for the normalisation.
      Parameters:
        `dbDir` - Crawl database directory
        `segments` - Segments directory
        `numLists` - Number of reduce tasks
        `topN` - Number of top URLs to be selected
        `curTime` - Current time in milliseconds
      Returns:
        Path to generated segment or null if no entries were selected
      Throws:
        `java.io.IOException` - When an I/O error occurs
        `java.lang.InterruptedException`
        `java.lang.ClassNotFoundException`

    - #### generateSegmentName

          public static java.lang.String generateSegmentName()

    - #### main

          public static void main(java.lang.String[] args)
                           throws java.lang.Exception

      Generate a fetchlist from the crawldb.
      Throws:
        `java.lang.Exception`

    - #### run

          public int run(java.lang.String[] args)
                  throws java.lang.Exception

      Specified by:
        `run` in interface `Tool`
      Throws:
        `java.lang.Exception`

    - #### run

          public java.util.Map<java.lang.String,java.lang.Object> run(java.util.Map<java.lang.String,java.lang.Object> args,
                                                                      java.lang.String crawlId)
                                                               throws java.lang.Exception

      Description copied from class: `NutchTool`
      Runs the tool, using a map of arguments. May return results, or null.
      Specified by:
        `run` in class `NutchTool`
      Throws:
        `java.lang.Exception`

Copyright © 2019 The Apache Software Foundation