Adding a new loader

You can add new loaders to lodex. Loaders are added in the packages/workers/src/loaders directory. A loader receives a config and the uploaded file as a stream, and returns the modified stream. Example of a csv parser:

# packages/workers/src/loaders/csv.ini
append = pack
label = csv

# load some plugins to activate some statements
[use]
plugin = basics

[CSVParse]

[CSVObject]

# Ensures that each object contains an identification key (required by lodex)
[swing]
test = pick(['URI', 'uri']).pickBy(_.identity).isEmpty()
[swing/identify]

# Ignore objects with duplicate URI
[dedupe]
ignore = true

# Prevent keys from containing dot path notation (which is forbidden by nodejs mongoDB driver)
[OBJFlatten]
separator = fix('.')
reverse = true
safe = true

# Uncomment to see each data sent to the database
#[debug]

# Add contextual metadata related to the import
[assign]
path = lodexStamp.importedDate
value = fix(new Date()).thru(d => d.toDateString())
path = lodexStamp.usedParser
value = env('parser')
path = lodexStamp.uploadedFilename
value = env('source')
path = uri
value = get('uri').trim()

Notice how the key will determine the name of the loader. This name must match the extension of the target file. This is how we determine which loader to use. Thus, a loader for .csv file must be exported as csv.

The config is taken from config.json, in loader.<file extension>, and allow to configure your loader on an instance basis. For example for the loader csv:

...
    "loaders": [
        ...
        "csv"
        ...
    ]