You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Idea: ability to pass extra variables to --convert scripts #492
sqlite-utils insert /tmp/kafka-logs.db logs server.log.2022-09-24-21 --text --convert "import rer = re.compile(r'^\[(?P<datetime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})\] (?P<level>\w+) (?P<log>(.+(\n(?\!\[).+|)+))', re.MULTILINE)def convert(text): rows = [m.groupdict() for m in r.finditer(text)] for row in rows: row.update({'server': 'localhost'}) row.update({'component': 'broker'}) return rows"
And the accompanying note:
The row.update allows to label rows as I’m planning to ingest logs from different hosts and potentially different components.
This made me think: it might be neat if you could inject additional variable values into that script with extra command-line options, to make this kind of reuse easier. Something like this:
sqlite-utils insert /tmp/kafka-logs.db logs server.log.2022-09-24-21 --text --convert "import rer = re.compile(r'^\[(?P<datetime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})\] (?P<level>\w+) (?P<log>(.+(\n(?\!\[).+|)+))', re.MULTILINE)def convert(text): rows = [m.groupdict() for m in r.finditer(text)] for row in rows: row.update({'server': server}) row.update({'component': component}) return rows" --var server "localhost" --var component "broker"
This idea would make more sense if there was a good mechanism to say "run the conversion script held in this file" as opposed to passing it as an option. This would also make having to remember bash escaping rules (see tip) much easier!
Activity
simonw commentedon Sep 26, 2022
This idea would make more sense if there was a good mechanism to say "run the conversion script held in this file" as opposed to passing it as an option. This would also make having to remember bash escaping rules (see tip) much easier!
shot-scraperhas that for--javascript, using the--inputoption: https://shot-scraper.datasette.io/en/stable/javascript.html#shot-scraper-javascript-helpMaybe
--convert-scriptwould work here? Or--convert-file? It should accept-for stdin too.