Skip to content

Idea: ability to pass extra variables to --convert scripts #492

Open

Description

@simonw

Got this idea from this example in https://jeqo.github.io/notes/2022-09-24-ingest-logs-sqlite/

sqlite-utils insert /tmp/kafka-logs.db logs server.log.2022-09-24-21 --text --convert "
import re
r = re.compile(r'^\[(?P<datetime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})\] (?P<level>\w+) (?P<log>(.+(\n(?\!\[).+|)+))', re.MULTILINE)
def convert(text):
    rows = [m.groupdict() for m in r.finditer(text)]
    for row in rows:
        row.update({'server': 'localhost'})
        row.update({'component': 'broker'})
    return rows
"

And the accompanying note:

The row.update allows to label rows as I’m planning to ingest logs from different hosts and potentially different components.

This made me think: it might be neat if you could inject additional variable values into that script with extra command-line options, to make this kind of reuse easier. Something like this:

sqlite-utils insert /tmp/kafka-logs.db logs server.log.2022-09-24-21 --text --convert "
import re
r = re.compile(r'^\[(?P<datetime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})\] (?P<level>\w+) (?P<log>(.+(\n(?\!\[).+|)+))', re.MULTILINE)
def convert(text):
    rows = [m.groupdict() for m in r.finditer(text)]
    for row in rows:
        row.update({'server': server})
        row.update({'component': component})
    return rows
" --var server "localhost" --var component "broker"

Activity

  1. simonw commented on Sep 26, 2022

    @simonw
    OwnerAuthor

    This idea would make more sense if there was a good mechanism to say "run the conversion script held in this file" as opposed to passing it as an option. This would also make having to remember bash escaping rules (see tip) much easier!

    shot-scraper has that for --javascript, using the --input option: https://shot-scraper.datasette.io/en/stable/javascript.html#shot-scraper-javascript-help

    Maybe --convert-script would work here? Or --convert-file? It should accept - for stdin too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions