org.apache.nutch.hostdb

## Class UpdateHostDbMapper

- java.lang.Object
- - org.apache.hadoop.mapreduce.Mapper<Text,Writable,Text,NutchWritable>
  - - org.apache.nutch.hostdb.UpdateHostDbMapper

-     public class UpdateHostDbMapper
      extends Mapper<Text,Writable,Text,NutchWritable>

  Mapper ingesting HostDB and CrawlDB entries. Additionally it can also read host score info from a plain text key/value file generated by the Webgraph's NodeDumper tool.

- - ### Nested Class Summary

    - ### Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.Mapper

      `Mapper.Context`

  - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected String[]` | `args`
    `protected String` | `buffer`
    `protected CrawlDatum` | `crawlDatum`
    `protected boolean` | `filter`
    `protected URLFilters` | `filters`
    `protected Text` | `host`
    `protected HostDatum` | `hostDatum`
    `protected boolean` | `normalize`
    `protected URLNormalizers` | `normalizers`
    `protected boolean` | `readingCrawlDb`
    `protected String` | `reprUrl`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `UpdateHostDbMapper()`

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `protected String` | `filterNormalize(String url)` Filters and or normalizes the input URL
    `void` | `map(Text key, Writable value, Mapper.Context context)` Mapper ingesting records from the HostDB, CrawlDB and plaintext host scores file.
    `void` | `setup(Mapper.Context context)`

    - ### Methods inherited from class org.apache.hadoop.mapreduce.Mapper

      `cleanup, run`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### host

          protected Text host

    - #### hostDatum

          protected HostDatum hostDatum

    - #### crawlDatum

          protected CrawlDatum crawlDatum

    - #### reprUrl

          protected String reprUrl

    - #### buffer

          protected String buffer

    - #### args

          protected String[] args

    - #### filter

          protected boolean filter

    - #### normalize

          protected boolean normalize

    - #### readingCrawlDb

          protected boolean readingCrawlDb

    - #### filters

          protected URLFilters filters

    - #### normalizers

          protected URLNormalizers normalizers

  - ### Constructor Detail

    - #### UpdateHostDbMapper

          public UpdateHostDbMapper()

  - ### Method Detail

    - #### setup

          public void setup(Mapper.Context context)

      Overrides:
        `setup` in class `Mapper<Text,Writable,Text,NutchWritable>`

    - #### filterNormalize

          protected String filterNormalize(String url)

      Filters and or normalizes the input URL
      Parameters:
        `url` -
      Returns:
        String

    - #### map

          public void map(Text key,
                          Writable value,
                          Mapper.Context context)
                   throws IOException,
                          InterruptedException

      Mapper ingesting records from the HostDB, CrawlDB and plaintext host scores file. Statistics and scores are passed on.
      Overrides:
        `map` in class `Mapper<Text,Writable,Text,NutchWritable>`
      Parameters:
        `key` -
        `value` -
        `context` -
      Throws:
        `IOException`
        `InterruptedException`

Copyright © 2021 The Apache Software Foundation