org.apache.nutch.crawl

## Class CrawlDbReader

- java.lang.Object
- - org.apache.hadoop.conf.Configured
  - - org.apache.nutch.util.AbstractChecker
    - - org.apache.nutch.crawl.CrawlDbReader

- All Implemented Interfaces:
    Closeable, AutoCloseable, Configurable, Tool

      public class CrawlDbReader
      extends AbstractChecker
      implements Closeable

  Read utility for the CrawlDB.
  Author:
    Andrzej Bialecki

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class and Description
    --- | ---
    `static class` | `CrawlDbReader.CrawlDatumCsvOutputFormat`
    `static class` | `CrawlDbReader.CrawlDatumJsonOutputFormat`
    `static class` | `CrawlDbReader.CrawlDbDumpMapper`
    `static class` | `CrawlDbReader.CrawlDbStatMapper`
    `static class` | `CrawlDbReader.CrawlDbStatReducer`
    `static class` | `CrawlDbReader.CrawlDbTopNMapper`
    `static class` | `CrawlDbReader.CrawlDbTopNReducer`
    `static class` | `CrawlDbReader.JsonIndenter`

  - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected String` | `crawlDb`

    - ### Fields inherited from class org.apache.nutch.util.AbstractChecker

      `keepClientCnxOpen, stdin, tcpPort, usage`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `CrawlDbReader()`

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `close()`
    `CrawlDatum` | `get(String crawlDb, String url, Configuration config)`
    `static void` | `main(String[] args)`
    `protected int` | `process(String line, StringBuilder output)`
    `void` | `processDumpJob(String crawlDb, String output, Configuration config, String format, String regex, String status, Integer retry, String expr, Float sample)`
    `void` | `processStatJob(String crawlDb, Configuration config, boolean sort)`
    `void` | `processTopNJob(String crawlDb, long topN, float min, String output, Configuration config)`
    `Object` | `query(Map<String,String> args, Configuration conf, String type, String crawlId)`
    `void` | `readUrl(String crawlDb, String url, Configuration config, StringBuilder output)`
    `int` | `run(String[] args)`

    - ### Methods inherited from class org.apache.nutch.util.AbstractChecker

      `getProtocolOutput, parseArgs, processSingle, processStdin, processTCP, run`

    - ### Methods inherited from class org.apache.hadoop.conf.Configured

      `getConf, setConf`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

    - ### Methods inherited from interface org.apache.hadoop.conf.Configurable

      `getConf, setConf`

- - ### Field Detail

    - #### crawlDb

          protected String crawlDb

  - ### Constructor Detail

    - #### CrawlDbReader

          public CrawlDbReader()

  - ### Method Detail

    - #### close

          public void close()

      Specified by:
        `close` in interface `Closeable`
      Specified by:
        `close` in interface `AutoCloseable`

    - #### processStatJob

          public void processStatJob(String crawlDb,
                                     Configuration config,
                                     boolean sort)
                              throws IOException,
                                     InterruptedException,
                                     ClassNotFoundException

      Throws:
        `IOException`
        `InterruptedException`
        `ClassNotFoundException`

    - #### get

          public CrawlDatum get(String crawlDb,
                                String url,
                                Configuration config)
                         throws IOException

      Throws:
        `IOException`

    - #### process

          protected int process(String line,
                                StringBuilder output)
                         throws Exception

      Specified by:
        `process` in class `AbstractChecker`
      Throws:
        `Exception`

    - #### readUrl

          public void readUrl(String crawlDb,
                              String url,
                              Configuration config,
                              StringBuilder output)
                       throws IOException

      Throws:
        `IOException`

    - #### processDumpJob

          public void processDumpJob(String crawlDb,
                                     String output,
                                     Configuration config,
                                     String format,
                                     String regex,
                                     String status,
                                     Integer retry,
                                     String expr,
                                     Float sample)
                              throws IOException,
                                     ClassNotFoundException,
                                     InterruptedException

      Throws:
        `IOException`
        `ClassNotFoundException`
        `InterruptedException`

    - #### processTopNJob

          public void processTopNJob(String crawlDb,
                                     long topN,
                                     float min,
                                     String output,
                                     Configuration config)
                              throws IOException,
                                     ClassNotFoundException,
                                     InterruptedException

      Throws:
        `IOException`
        `ClassNotFoundException`
        `InterruptedException`

    - #### run

          public int run(String[] args)
                  throws IOException,
                         InterruptedException,
                         ClassNotFoundException,
                         Exception

      Specified by:
        `run` in interface `Tool`
      Throws:
        `IOException`
        `InterruptedException`
        `ClassNotFoundException`
        `Exception`

    - #### main

          public static void main(String[] args)
                           throws Exception

      Throws:
        `Exception`

    - #### query

          public Object query(Map<String,String> args,
                              Configuration conf,
                              String type,
                              String crawlId)
                       throws Exception

      Throws:
        `Exception`

Copyright © 2021 The Apache Software Foundation