org.apache.nutch.util

## Class SitemapProcessor

- java.lang.Object
- - org.apache.hadoop.conf.Configured
  - - org.apache.nutch.util.SitemapProcessor

- All Implemented Interfaces:
    Configurable, Tool

      public class SitemapProcessor
      extends Configured
      implements Tool

  Performs Sitemap processing by fetching sitemap links, parsing the content and merging the urls from Sitemap (with the metadata) with the existing crawldb.

  There are two use cases supported in Nutch's Sitemap processing:

  1. Sitemaps are considered as "remote seed lists". Crawl administrators can prepare a list of sitemap links and get only those sitemap pages. This suits well for targeted crawl of specific hosts.
  2. For open web crawl, it is not possible to track each host and get the sitemap links manually. Nutch would automatically get the sitemaps for all the hosts seen in the crawls and inject the urls from sitemap to the crawldb.

  For more details see: https://cwiki.apache.org/confluence/display/NUTCH/SitemapFeature

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static String` | `CURRENT_NAME`
    `static String` | `LOCK_NAME`
    `static org.slf4j.Logger` | `LOG`
    `static SimpleDateFormat` | `sdf`
    `static String` | `SITEMAP_ALWAYS_TRY_SITEMAPXML_ON_ROOT`
    `static String` | `SITEMAP_OVERWRITE_EXISTING`
    `static String` | `SITEMAP_REDIR_MAX`
    `static String` | `SITEMAP_SIZE_MAX`
    `static String` | `SITEMAP_STRICT_PARSING`
    `static String` | `SITEMAP_URL_FILTERING`
    `static String` | `SITEMAP_URL_NORMALIZING`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `SitemapProcessor()`

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `static void` | `main(String[] args)`
    `int` | `run(String[] args)`
    `void` | `sitemap(Path crawldb, Path hostdb, Path sitemapUrlDir, boolean strict, boolean filter, boolean normalize, int threads)`
    `static void` | `usage()`

    - ### Methods inherited from class org.apache.hadoop.conf.Configured

      `getConf, setConf`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

    - ### Methods inherited from interface org.apache.hadoop.conf.Configurable

      `getConf, setConf`

- - ### Field Detail

    - #### LOG

          public static final org.slf4j.Logger LOG

    - #### sdf

          public static final SimpleDateFormat sdf

    - #### CURRENT_NAME

          public static final String CURRENT_NAME

      See Also:
        Constant Field Values

    - #### LOCK_NAME

          public static final String LOCK_NAME

      See Also:
        Constant Field Values

    - #### SITEMAP_STRICT_PARSING

          public static final String SITEMAP_STRICT_PARSING

      See Also:
        Constant Field Values

    - #### SITEMAP_URL_FILTERING

          public static final String SITEMAP_URL_FILTERING

      See Also:
        Constant Field Values

    - #### SITEMAP_URL_NORMALIZING

          public static final String SITEMAP_URL_NORMALIZING

      See Also:
        Constant Field Values

    - #### SITEMAP_ALWAYS_TRY_SITEMAPXML_ON_ROOT

          public static final String SITEMAP_ALWAYS_TRY_SITEMAPXML_ON_ROOT

      See Also:
        Constant Field Values

    - #### SITEMAP_OVERWRITE_EXISTING

          public static final String SITEMAP_OVERWRITE_EXISTING

      See Also:
        Constant Field Values

    - #### SITEMAP_REDIR_MAX

          public static final String SITEMAP_REDIR_MAX

      See Also:
        Constant Field Values

    - #### SITEMAP_SIZE_MAX

          public static final String SITEMAP_SIZE_MAX

      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### SitemapProcessor

          public SitemapProcessor()

  - ### Method Detail

    - #### sitemap

          public void sitemap(Path crawldb,
                              Path hostdb,
                              Path sitemapUrlDir,
                              boolean strict,
                              boolean filter,
                              boolean normalize,
                              int threads)
                       throws Exception

      Throws:
        `Exception`

    - #### main

          public static void main(String[] args)
                           throws Exception

      Throws:
        `Exception`

    - #### usage

          public static void usage()

    - #### run

          public int run(String[] args)
                  throws Exception

      Specified by:
        `run` in interface `Tool`
      Throws:
        `Exception`

Copyright © 2021 The Apache Software Foundation