Koop Provider Hub Search

This is a Koop provider that extracts datasets from the ArcGIS Hub Search API.

Use - Streaming All Datasets

This provider plugin currently only supports streaming ALL datasets matching a given search query. It performs searches using the searchContent function from @esri/hub.js and requests all pages from the API in order.

Define Search Parameters

To perform a search from an output plugin, attach an IContentSearchRequest to the response.

req.res.locals.searchRequest = {
  options: {
    site: 'my-site.hub.arcgis.com'
  },
  filter: {
    term: 'some search terms'
  }
};

Notes

A search request must be scoped to include at least one of the following:

• an "id" (filter: { id: '<id here>' })
• at least one "group" (filter: { group: ['<group id here>'] })
• an "orgid" (filter: { orgid: '<or id here>' })
• a "site" (options: { site: '<site here>' })

If none of the above are provided, an error will be returned. Importantly, if only a site is provided, it is still possible for an error to be returned if the site does not have an organization or group specified as part of its catalog.

Pull the Readable Stream

Then pass the request object to this.model.pullStream.

const docStream = await this.model.pullStream(req);

Full Example

What you have now is a Node.js Readable stream. You can pipe the datasets from the readable through a transform stream in order to format them into some other type of output before sending them back to the browser by piping them to the Express.js response. The following simple example also uses the through2 library to conveniently define a transform stream.

async handleRequest (req, res) {
  req.res.locals.searchRequest = { /* some search params */ };

  const docStream = await this.model.pullStream(req);

  docStream
    .pipe(through2.obj(function (dataset, enc, callback) {
      const transformed = someTranformFunc(dataset);

      // the Express.js "res" Writable only accepts strings or Buffers
      this.push(JSON.stringify(transformed));
      callback();
    }))
    .pipe(res);
}

Develop

# clone and install dependencies
git clone https://github.com/koopjs/koop-output-dcat-ap-201
cd koop-output-dcat-ap-201
npm i

# starts the example Koop app found in ./example-app.
npm run dev

Test

Run the npm t commmand to spin up the automated tests.