org.apache.nutch.net.urlnormalizer.regex

## Class RegexURLNormalizer

- java.lang.Object
- - org.apache.hadoop.conf.Configured
  - - org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer

- All Implemented Interfaces:
    Configurable, URLNormalizer

      public class RegexURLNormalizer
      extends Configured
      implements URLNormalizer

  Allows users to do regex substitutions on all/any URLs that are encountered, which is useful for stripping session IDs from URLs.

  This class uses the `urlnormalizer.regex.file` property. It should be set to the file name of an xml file which should contain the patterns and substitutions to be done on encountered URLs.

  This class also supports different rules depending on the scope. Please see the javadoc in `URLNormalizers` for more details.

  Author:
    Luke Baker, Andrzej Bialecki

- - ### Field Summary

    - ### Fields inherited from interface org.apache.nutch.net.URLNormalizer

      `X_POINT_ID`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `RegexURLNormalizer()` The default constructor which is called from UrlNormalizerFactory (normalizerClass.newInstance()) in method: getNormalizer()*
    `RegexURLNormalizer(Configuration conf)`
    `RegexURLNormalizer(Configuration conf, java.lang.String filename)` Constructor which can be passed the file name, so it doesn't look in the configuration files for it.

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `java.util.HashMap<java.lang.String,java.util.List<org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer.Rule>>` | `getScopedRules()`
    `static void` | `main(java.lang.String[] args)` Spits out patterns and substitutions that are in the configuration file.
    `java.lang.String` | `normalize(java.lang.String urlString, java.lang.String scope)`
    `java.lang.String` | `regexNormalize(java.lang.String urlString, java.lang.String scope)` This function does the replacements by iterating through all the regex patterns.
    `void` | `setConf(Configuration conf)`

    - ### Methods inherited from class org.apache.hadoop.conf.Configured

      `getConf`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

    - ### Methods inherited from interface org.apache.hadoop.conf.Configurable

      `getConf`

- - ### Constructor Detail

    - #### RegexURLNormalizer

          public RegexURLNormalizer()

      The default constructor which is called from UrlNormalizerFactory (normalizerClass.newInstance()) in method: getNormalizer()*

    - #### RegexURLNormalizer

          public RegexURLNormalizer(Configuration conf)

    - #### RegexURLNormalizer

          public RegexURLNormalizer(Configuration conf,
                                    java.lang.String filename)
                             throws java.io.IOException,
                                    java.util.regex.PatternSyntaxException

      Constructor which can be passed the file name, so it doesn't look in the configuration files for it.
      Throws:
        `java.io.IOException`
        `java.util.regex.PatternSyntaxException`

  - ### Method Detail

    - #### getScopedRules

          public java.util.HashMap<java.lang.String,java.util.List<org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer.Rule>> getScopedRules()

    - #### setConf

          public void setConf(Configuration conf)

      Specified by:
        `setConf` in interface `Configurable`
      Overrides:
        `setConf` in class `Configured`

    - #### regexNormalize

          public java.lang.String regexNormalize(java.lang.String urlString,
                                                 java.lang.String scope)

      This function does the replacements by iterating through all the regex patterns. It accepts a string url as input and returns the altered string.

    - #### normalize

          public java.lang.String normalize(java.lang.String urlString,
                                            java.lang.String scope)
                                     throws java.net.MalformedURLException

      Specified by:
        `normalize` in interface `URLNormalizer`
      Throws:
        `java.net.MalformedURLException`

    - #### main

          public static void main(java.lang.String[] args)
                           throws java.util.regex.PatternSyntaxException,
                                  java.io.IOException

      Spits out patterns and substitutions that are in the configuration file.
      Throws:
        `java.util.regex.PatternSyntaxException`
        `java.io.IOException`

Copyright © 2019 The Apache Software Foundation