Bulk API

Design notes for the Gitana Bulk API

Usage

var gitana = require("gitana");

  gitana.createTransaction()
    .for("branch://<platformId>/<repositoryId>/<branchId>") // using a reference string
    //.for(branch) // using an object
    .insert({
      "title": "My first article",
      "_type": "custom:article"
    })
    .insert({
      "title": "My second page",
      "_type": "custom:page"
    })
    .remove("GUID1")
    .remove({
      "_doc": "GUID2"
    })
    .read("GUID4") // could we provide a way to bulk read objects?
    .retryCount(3)
    .fail(function(result) {
      // called if the commit fails
    })
    .success(function(result) {
      // called if the commit succeeds
    })
    .commit();

As a first pass, the created transaction should store all JSON objects in-memory until commit() is called. When commit is called, the transaction can run through all objects and send single requests. For N number of objects, there could be N requests to add items to the transaction.

As a second pass, the created transaction could still store all JSON objects in-memory until commit() is called. However, when commit() is called, the objects could be grouped and sent in packages of 10, 100 or more depending on the total size of the JSON payload.

As a third pass, the created transaction could start chunking sends as things are being added. This would remove the requirement to hold things in memory. Local Storage could also be used as a way to queue things. This is not required for the moment, but an intended optimization down the road.

HTTP/REST

The REST API supports four methods for creating, canceling and committing a transaction as well as populating the transaction with objects to commit.

Create a Transaction

Creates a transaction. The only required parameter is a reference against which the transaction will be run. In the case of a transaction containing nodes, the transaction should be referenced to the branch. All references are written out using the reference syntax:

<type>://<platformId>//<datastoreId>[//objectId1][//objectId2]

The method is invoked like this:

POST /transactions?reference=branch://{platformId}/{repositoryId}/{branchId}

The response is:

{
   "_doc": "<transactionId>",
   "container-reference": "<reference>"
}

Add objects to a Transaction

Adds one or more objects to an existing transaction

POST /transactions/{transactionId}/add

The request should look like this:

{
   "objects": [{
      "header": {
         "type": "node",
         "operation": "write" (or "delete")
      },
      "data": {
         ... data fields ...
      }
   }, ... more items ...]
}

And the response looks like:

{
   "results": [{
      "_doc": "<objectId>",
      "transactionId": "<id of the transaction instance>",
      "operation": "write" (or "delete"),
      "type": "node"
   }, ... more results ...]
}

Rollback a Transaction

Deletes a transaction and removes any objects associated with it

DELETE /transactions/{transactionId}

Commit a Transaction

Commits all of the objects within a transaction to the database

POST /transactions/{transactionId}/commit

The response looks like this:

{
   "transactionId": "<id of the transaction instance>",
   "startTime": <transaction commit start time in ms>,
   "endTime": <transaction commit end time in ms>,
   "totalCount": <total number of objects committed>,
   "errorCount": <total number of object commit failures>,
   "successCount": <total number of object commit successes>,
   "results": {
      "<transactionObjectId>": {
         "ok": true,
         "startTime": <time when object commit started>,
         "endTime": <time when object commit ended>,
         "dataId": "<id of the object written or deleted>",
         "error": {
            "message": "[error message]"
         }
      }
   }
}