org.apache.nutch.net

## Class URLNormalizers

- java.lang.Object
- - org.apache.nutch.net.URLNormalizers

-     public final class URLNormalizers
      extends Object

  This class uses a "chained filter" pattern to run defined normalizers. Different lists of normalizers may be defined for different "scopes", or contexts where they are used (note however that they need to be activated first through `plugin.include` property).

  There is one global scope defined by default, which consists of all active normalizers. The order in which these normalizers are executed may be defined in "urlnormalizer.order" property, which lists space-separated implementation classes (if this property is missing normalizers will be run in random order). If there are more normalizers activated than explicitly named on this list, the remaining ones will be run in random order after the ones specified on the list are executed.

  You can define a set of contexts (or scopes) in which normalizers may be called. Each scope can have its own list of normalizers (defined in "urlnormalizer.scope.<scope_name>" property) and its own order (defined in "urlnormalizer.order.<scope_name>" property). If any of these properties are missing, default settings are used for the global scope.

  In case no normalizers are required for any given scope, a `org.apache.nutch.net.urlnormalizer.pass.PassURLNormalizer` should be used.

  Each normalizer may further select among many configurations, depending on the scope in which it is called, because the scope name is passed as a parameter to each normalizer. You can also use the same normalizer for many scopes.

  Several scopes have been defined, and various Nutch tools will attempt using scope-specific normalizers first (and fall back to default config if scope-specific configuration is missing).

  Normalizers may be run several times, to ensure that modifications introduced by normalizers at the end of the list can be further reduced by normalizers executed at the beginning. By default this loop is executed just once - if you want to ensure that all possible combinations have been applied you may want to run this loop up to the number of activated normalizers. This loop count can be configured through `urlnormalizer.loop.count` property. As soon as the url is unchanged the loop will stop and return the result.

  Author:
    Andrzej Bialecki

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static String` | `SCOPE_CRAWLDB` Scope used when updating the CrawlDb with new URLs.
    `static String` | `SCOPE_DEFAULT` Default scope.
    `static String` | `SCOPE_FETCHER` Scope used by `Fetcher` when processing redirect URLs.
    `static String` | `SCOPE_GENERATE_HOST_COUNT` Scope used by `Generator`.
    `static String` | `SCOPE_INDEXER` Scope used when indexing URLs.
    `static String` | `SCOPE_INJECT` Scope used by `Injector`.
    `static String` | `SCOPE_LINKDB` Scope used when updating the LinkDb with new URLs.
    `static String` | `SCOPE_OUTLINK` Scope used when constructing new `Outlink` instances.
    `static String` | `SCOPE_PARTITION` Scope used by `URLPartitioner`.

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `URLNormalizers(Configuration conf, String scope)`

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `String` | `normalize(String urlString, String scope)` Normalize

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### SCOPE_DEFAULT

          public static final String SCOPE_DEFAULT

      Default scope. If no scope properties are defined then the configuration for this scope will be used.
      See Also:
        Constant Field Values

    - #### SCOPE_PARTITION

          public static final String SCOPE_PARTITION

      Scope used by `URLPartitioner`.
      See Also:
        Constant Field Values

    - #### SCOPE_GENERATE_HOST_COUNT

          public static final String SCOPE_GENERATE_HOST_COUNT

      Scope used by `Generator`.
      See Also:
        Constant Field Values

    - #### SCOPE_FETCHER

          public static final String SCOPE_FETCHER

      Scope used by `Fetcher` when processing redirect URLs.
      See Also:
        Constant Field Values

    - #### SCOPE_CRAWLDB

          public static final String SCOPE_CRAWLDB

      Scope used when updating the CrawlDb with new URLs.
      See Also:
        Constant Field Values

    - #### SCOPE_LINKDB

          public static final String SCOPE_LINKDB

      Scope used when updating the LinkDb with new URLs.
      See Also:
        Constant Field Values

    - #### SCOPE_INJECT

          public static final String SCOPE_INJECT

      Scope used by `Injector`.
      See Also:
        Constant Field Values

    - #### SCOPE_OUTLINK

          public static final String SCOPE_OUTLINK

      Scope used when constructing new `Outlink` instances.
      See Also:
        Constant Field Values

    - #### SCOPE_INDEXER

          public static final String SCOPE_INDEXER

      Scope used when indexing URLs.
      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### URLNormalizers

          public URLNormalizers(Configuration conf,
                                String scope)

  - ### Method Detail

    - #### normalize

          public String normalize(String urlString,
                                  String scope)
                           throws MalformedURLException

      Normalize
      Parameters:
        `urlString` - The URL string to normalize.
        `scope` - The given scope.
      Returns:
        A normalized String, using the given `scope`
      Throws:
        `MalformedURLException` - If the given URL string is malformed.

Copyright © 2021 The Apache Software Foundation