Harvest

The dkan_harvest module provides integration with the harvest library. This enables you to use the public feed or API of another data portal and import items from that portal’s catalog into your own.

For example, Data.gov harvests all of its datasets from the data.json files of hundreds of U.S. federal, state and local data portals.

A "harvest" is the execution of a harvest plan.

Drush Commands

Command	Args	Notes
dkan-harvest:list	n/a	Lists avaialble harvests.
dkan-harvest:register	$config	Register a new harvest, file saved to the dkan_harvest/plans directory.
dkan-harvest:deregister	$identifier	Deletes the cached plan file.
dkan-harvest:cache	$identifier	This will fetch the source data, apply the source configuration and cache the data to a local file.
dkan-harvest:run	$identifier	This will harvest the current cache for the selected sources.
dkan-harvest:revert	$identifier	Reverts harvest.

Harvest Plan schema

The harvest plan is the configuration used to import data into your catalog. https://github.com/GetDKAN/harvest/blob/master/schema/schema.json

  {
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "harvest-plan",
  "type": "object",
  "title": "Harvest Plan",
  "required": [
    "identifier",
    "extract",
    "load"
  ],
  "properties": {
    "identifier": {
      "type": "string",
      "title": "The plan's identifier",
      "pattern": "^(.*)$"
    },
    "extract": {
      "type": "object",
      "title": "Extract",
      "required": [
        "type",
        "uri"
      ],
      "properties": {
        "type": {
          "type": "string",
          "title": "Class utilized to extract the data from the source."
        },
        "uri": {
          "type": "string",
          "title": "The URL or Location of the Source",
          "examples": [
            "http://demo.getdkan.com/data.json"
          ]
        }
      }
    },
    "transforms": {
      "type": "array",
      "title": "The Transforms for the Harvest",
      "additionalProperties": false,
      "items": {
        "type": "string"
      }
    },
    "load": {
      "type": "object",
      "title": "The Load settings for the Harvest",
      "required": [
        "type"
      ],
      "properties": {
        "type": {
          "type": "string",
          "title": "Class utilized to load the harvested data."
        }
      }
    }
  }
}

Setup

Currently there is no UI for creating or running a harvest, use the drush commands above to harvest data into your catalog.

1. Register a new harvest source, passing in the harvest plan configuration as JSON, wrapped in single quotes, do not add spaces between elements:
   • Create a unique name as the identifier
   • Provide the extract object with type and uri values: type being the class that matches the data structure, (most likely "\\Harvest\\ETL\\Extract\\DataJson"), and the full URI for the source endpoint (such as http://source/data.json or file://source/data.json)
   • Provide the load object that defines the type of content you want to create, most likely datasets, so use: "\\Drupal\\dkan_harvest\\Load\\Dataset"
2. If you would also like to make changes to the data you are harvesting, you can create custom transforms that will modify the data before saving to your catalog. Add multiple transforms as an array. Learn more here.

Example

drush dkan-harvest:register '{"identifier":"example","extract":{"type":"\\Harvest\\ETL\\Extract\\DataJson","uri":"https://source/data.json"},"transforms":[],"load":{"type":"\\Drupal\\dkan_harvest\\Load\\Dataset"}}'

Run the harvest

Once you have registered a harvest source, run the import, passing in the identifier as an arguement drush dkan-harvest:run example

How to create transforms

Transforms allow you to modify what you are harvesting. Let's say you want to harvest from a Socrata catalog and only want to import the distributions with csv files. Create a custom module with the following structure: custom_module/src/Harvest/Transform/Socrata.php

<?php
 
namespace Drupal\custom_module\Harvest\Transform;
 
use Harvest\ETL\Transform\Transform;
 
class Socrata extends Transform {
 
  private $counter = 0;
 
  public function run($item) {
    $counter = 0;
    $identifier = $item->identifier;
    $item->identifier = $this->getIdentifier($identifier);
    foreach ($item->distribution as $key => $dist) {
      if ($dist->mediaType != "text/csv") {
        unset($item->distribution[$key]);
      }
      else {
        $dist->title = "distribution_{$this->counter}_{$counter}";
        $item->distribution[$key] = $dist;
      }
      $counter++;
    }
 
    return $item;
  }
 
  private function getIdentifier($identifier) {
    $path = parse_url($identifier, PHP_URL_PATH);
    $path = str_replace('/api/views/', "", $path);
    return $path;
  }
}

Example with new transform

drush dkan-harvest:register '{"identifier":"example","extract":{"type":"\\Harvest\\ETL\\Extract\\DataJson","uri":"https://source/data.json"},"transforms":["\\Drupal\\custom_module\\Transform\\Socrata"],"load":{"type":"\\Drupal\\dkan_harvest\\Load\\Dataset"}}'