shell

A mini Nim DSL to execute shell commands more conveniently.

Usage

With this macro you can simply write

shell:
  touch foo
  mv foo bar
  rm bar

which is then rewritten to something equivalent to:

execShell("touch foo")
execShell("mv foo bar")
execShell("rm bar")

where execShell is a proc around startProcess for normal compilation and gorgeEx when using NimScript.

Note: When using NimScript the given command is prepended by

&"cd {getCurrentDir()} && "

in order to switch the evaluation into the directory of the shell call. The same is achieved on the compiled backend by the poEvalCommand argument to startProcess.

See Full expansion of the macro below for more details and how to read the exit code of executed commands.

Most simple things should work as expected. See below for some known quirks.

one and pipe

By default each line in the shell macro will be handled by a different call to execShell. If you need several commands, which depend on the state of the previous, you may do so via the one command like so:

shell:
  one:
    mkdir foo
    cd foo
    touch bar
    cd ".."
    rm foo/bar

Similar to the one command, the pipe command exists. This concats the command via the shell pipe |:

shell:
  pipe:
    cat test.txt
    head -3

will produce:

execShell("cat test.txt | head -3")

Both of these can even be combined!

shell:
  one:
    mkdir foo
    pushd foo
    echo "Hallo\nWorld" > test.txt
    pipe:
      cat test.txt
      grep H
    popd
    rm foo/test.txt
    rmdir foo

will work just as expected, echoing Hallo in the shell.

Handling programs that require user input

Some terminal commands will require user input. Starting from version v0.5.0, basic support for a expect / send feature is available. It allows for functionality similar to the =expect(1)= program.

Let’s consider two simple examples. Assuming we have a script that reads from stdin such as:

tExpect.sh:

#!/bin/sh

echo "Hello world. Your name?"
read foo
echo "Your name is" $foo

calling this script using the shell macro would normally

import shell
shell:
  ./tExpect.sh

cause the nim program to simply stop upon encountering the read line.

Now we can do just this:

shell:
  ./tExpect.sh
  send: "BaFu"

This goes full rogue mode and just sends “BaFu”, regardless what the question is.

You can also specifiy what answer you send to which question. By using expect / send we can handle it this way:

import shell
shell:
  ./tExpect.sh
  expect: "Your name?"
  send: "BaFu"

As you notice the expect line only contains the second part of the last printed line of the shell script. That is because the expect functionality tries to match the shell output line by line using one of the following:

• the expect line matches exactly
• the line starts with the expect line
• the line ends with the expect line (useful for long outputs that end with "y/N" like constructs)

(this may become configurable in the future)

Another example would be installing a nimble package and dealing with the possibility of having to upgrade / overwrite package:

import shell
shell:
  nimble install foo
  expect: "Overwrite? [y/N]"
  send: "y"

which handles such situations programatically.

Note: You may have multiple expect / send constructs in a single shell call. But keep in mind that every expect must have a send.

Nim symbol quoting

NOTE: In a previous version this was done via accented quotes `. For the old behavior compile with -d:oldQuote.

Another important feature to make this library useful is quoting of Nim symbols.

This is handled via parenthesis () (if you need to run something in a subshell unfortunately that will have to be done with an explicit string now). Any tree in () is subject to quoting. That means if an identifier within () is preceded by a $, the symbol is unquoted. Note however that for the moment only a single variable may be quoted in each ().

The simplest case would be:

let name = "Vindaar"
shell:
  echo Hello from ($name)

which will perform the call:

execShell(&"echo Hello from {name}!")

and after the call to strformat.&:

execShell("echo Hello from Vindaar!")

Appending to a Nim identifier

Assuming we have a filename identifier and we want to convert some image from png to jpg with image magick. The simplest command should look like:

convert myimage.png myimage.jpg

This can be done in several ways.

Using dot expressions and no string literals:

let fname = "myimage"
shell:
  convert ($fname).png ($fname).jpg

Note that this is a special case. Continuing after a () quote without literal strings will only work for dot expressions. For instance:

let fname = "myimage"
shell:
  convert ($fname)".png" ($fname)".jpg"

will wrongly be converted to:

convert myimage .png myimage .jpg

which is obviously not what one would expect.

Using string literals:

let fname = "myimage"
shell:
  convert ($fname".pdf") ($fname".png")

In contrast to the wrong example shown above, this will work as expected.

This is especially useful for cases without dot expressions after the quoted nim identifier.

Appending a Nim identifier to a string literal

The other example would be appending a Nim identifier to a literal string. For instance in case we have a filename, which we create at run time and we wish to hand it to some command which takes an argument, which is must be given without a space like:

./myBin input --out=output

In this case one of the following ways works:

using () after a string literal:

let outfile = "myoutput.txt"
shell:
  ./myBin input "--out="($outfile)

If the () appears after the literal we can correctly generate the string without a space (in comparison to the case presented above when a string literal follows a ()).

For more predictable behavior, put the string literal also into

():

let outfile = "myoutput.txt"
shell:
  ./myBin input ("--out="$outfile)

General remark on predictability

NOTE: previously this section said to handle quoting + concatenation with strings both in the case of with and without space with () for the most predictable behavior. But that was a bad idea from my side! If you need spaces, simply put it outside the () and use a space!

The doAssert below is to be understood in the context of the shell macro. To summarize the above then:

let outfile = "myoutput.txt"
doAssert ("--out="$outfile) == &"--out={outfile}" # <- without space, ident after
doAssert "--out" ($outfile) == &"--out {outfile}" # <- with space, ident after
let fname = "myimage"
doAssert ($outfile".jpg") == &"{fname}.jpg" # <- without space, ident first
doAssert ($outfile) "image2" == &"{outfile} image2" # <- without space, ident first

NOTE 2: For the moment however, the () usage is restricted to a single string literal (or something that is convertible to a string via the stringify proc) and a single Nim identifier! This restriction will maybe be removed in the future.

This syntax also works for more complicated Nim expressions than a simple identifier:

const t = (a: "name", b: 5.5)
doAssert ("--out="$(t.a))
doAssert ("--out="$t.a)

both work. Of course t needn’t be a tuple. It can also be an object or even a function call, like for instance extracting a filename within a call:

import os, shell
let path = "/some/user/path/toAFile.txt"
shell:
  ./myBin ("--inputFile="$(path.extractFilename))

should produce:

./myBin --inputFile=toAFile.txt

Accented quotes

NOTE: In a previous version accented quotes were also used to quote Nim identifiers. That use case is now handled via parentheses. For the old behavior compile with -d:oldQuote.

Accented quotes allow you to hand raw strings.

Note: this has the downside of disallowing ` as a token to be handed to the shell. If you want to use the shell’s `, you need to put the appropriate command into quotation marks.

Raw strings

If you want to hand a literal string to the shell, you may do so by putting it into accented quotes:

echo `hello`

will be rewritten to

execShell("echo \"hello\"")

For a string consisting of multiple commands / words, put quotation marks around it:

echo `"Hello from Nim!"`

which will then also be rewritten to:

execShell("echo \"Hello from Nim!\"")

Assignment of results to Nim variables

Also useful is assignment of the result of a shell call to a Nim string. This can be done with the shellAssign macro. It is a little special compared to the shell and shellEcho macros. It only supports a single statement (*), which needs to be an assignment of a shell call of the syntax presented above to a Nim variable, such as:

var name = ""
shellAssign:
  name = echo Araq
assert name == "Araq"

Here the left name is the Nim variable (note: this is an exception of the Nim symbol quoting mentioned above!), whereas the right hand side is an arbitrary shell call, in this case a simple call to echo. The Nim variable will be assigned the result of the shell call, by being rewritten to:

var name = ""
name = asgnShell("echo Araq")
assert name == "Araq"

asgnShell is internally called by execShell mentioned above. asgnShell itself performs the calls to execCmdEx (or exec for NimScript).

(*): a single statement is not entirely precise, because the one and pipe operators can be used in combination with the assignment! For example the following is also possible:

var res = ""
shellAssign:
  res = pipe:
    seq 0 1 10
    tail -3
assert res == "8\n9\n10"

NimScript & Nim compile time (nimvm)

This macro can also be used in NimScript as well as at regular Nim compile time (i.e. in nimvm contexts). It uses gorgeEx in this case. Some features may not work perfectly. Instead of using the current working directory, it always starts at the path of the current project being compiled! So you may need to prepend your own cd <foo> depending on where you need to run your commands.

Known issues

Certain things unfortunately have to go into quotation marks. As seen in the one example above, the simple .. is not allowed.

Variable assignments in the shell need to be handed via a string literal:

shell:
  one:
    "a=`echo hello`"
    echo $a

Also if you need assignment via `:`, you need to also put it into quotation marks, as the Nim tree is not unambiguous enough to properly parse the resulting command. Say you wish to compile a Nim program, you might want to do:

shell:
  nim c "--out:noTest" test.nim

A slightly different case is assignment via `=`. A single `=` assignment in a command is possible, but not more. That is due to a Nim parser limitation (as you cannot have “nested” `nnkAsgn` nodes).

So this:

shel:
  foo --bar --option=value --more

is fine, but this:

shell:
  foo --bar --option=value --secondOption=value2

is not valid Nim syntax. Fortunately, in most cases the `=` sign is optional anyway and you may just drop (one or both):

shel:
  foo --bar --option value --secondOption value2

is valid and means the same for most programs.

In general, if in doubt you can just write strings or triple string (to pass a " to the shell).

Full expansion of the macro

As mentioned at the top of the README, the expansion shown is simplified (as a matter of fact it was as simple once, but has since become more complex).

The full expansion of the first example is:

discard block:
  var outputStr381052 = ""
  var exitCode381051: int
  if exitCode381051 ==
      0:
    let tmp381063 = execShell("touch foo")
    outputStr381052 = outputStr381052 &
        tmp381063[0]
    exitCode381051 = tmp381063[1]
  else:
    echo "Skipped command `" & "touch foo" &
        "` due to failure in previous command!"
  if exitCode381051 ==
      0:
    let tmp381064 = execShell("mv foo bar")
    outputStr381052 = outputStr381052 &
        tmp381064[0]
    exitCode381051 = tmp381064[1]
  else:
    echo "Skipped command `" & "mv foo bar" &
        "` due to failure in previous command!"
  if exitCode381051 ==
      0:
    let tmp381065 = execShell("rm bar")
    outputStr381052 = outputStr381052 &
        tmp381065[0]
    exitCode381051 = tmp381065[1]
  else:
    echo "Skipped command `" & "rm bar" &
        "` due to failure in previous command!"
  (outputStr381052, exitCode381051)

As can be seen from the expansion above, successive commands are only run, if the exit code of the previous command was 0, while the output is appended to the previous command’s output.

The normal shell command discards the return value of the block. If you want to keep it, use the shellVerbose macro:

let res = shellVerbose:
  someCommand

where res will be of type tuple[output: string, exitCode: string] according to the expansion above.

Debugging

In order to see what’s going on, you can either compile your program with the -d:debugShell flag, which will then echo the rewritten commands during compilation. Alternatively in order to avoid calling the commands immediately, you may use the shellEcho macro instead. It simply echoes the commands that would otherwise be run.

Error reporting

By default shell prints output messages to stdout:

import shell

shell:
  ls

shellCmd: ls
shell> nim.cfg
shell> README.org
shell> shell
shell> shell.nim
shell> shell.nim.bin
shell> shell.nimble
shell> tests

What is printed to stdout can be configured by using defines:

shellNoDebugOutput

Do not print command output

shellNoDebugError

Do not print error output

shellNoDebugCommand

Do not print command being executed

shellNoDebugRuntime

When error occurs do not print failed command

By default these are disabled - to enable use either -d:shellNoDebug* or use the {.define(shellNoDebug*).} pragma

{.define(shellNoDebugOutput).}

import shell

shell:
  ls

shellCmd: ls

The default shellVerbose command combines stderr and stdout into single result. To get stdout, stderr and the return code separately use shellVerboseErr. Both of these templates have an overload that takes set[DebugOutputKind] to control printing settings:

import shell

let (res, err, code) = shellVerboseErr {dokCommand}:
  echo "test"

echo "Returned string: '", res, "' with exit code ", code

shellCmd: echo test
Returned string: 'test' with exit code 0

Printing errors directly into stdout is good solution for most of the use cases, but sometimes it is necessary to provide more sophisticated error handing - throwing an exception when the command failed. To switch to exceptions use -d:shellThrowException. It will automatically disable all other output types in the default configuration.

{.define(shellThrowException).}

import shell, strutils

try:
  shell:
    ls -l
    ls -z
except ShellExecError:
  let e = cast[ShellExecError](getCurrentException())
  echo e.msg # Error message describing what happened
  echo "command was: ", e.cmd # Original command string
  assert e.cmd == "ls -z"
  echo "return code: ", e.retcode # Return code
  echo "regular out: ", e.outstr # Stdout from command
  echo "error outpt: "
  for l in e.errstr.split('\n'): # Stderr from the command
    echo "  ", l

Command ls -z exited with non-zero code
command was: ls -z
return code: 2
regular out:
error outpt:
  ls: invalid option -- 'z'
  Try 'ls --help' for more information.

On command failure ShellExecError is raised.

Note that some commands output error messages into stdout rather than into stderr - it might be necessary to check both. In this particular example content of the stderr is largely meaningless: actual reason for error was printed into stdout.

{.define(shellThrowException).}

import shell, strutils

try:
  shell:
    ngspice -b "/tmp/ngpsice-simulation/zzz.netkRs8jE"
except ShellExecError:
  let e = cast[ShellExecError](getCurrentException())
  echo e.msg # Error message describing what happened
  echo "command was: ", e.cmd # Original command string
  echo "exec direct: ", e.cwd # 
  echo "return code: ", e.retcode # Return code
  echo "regular out: \n====\n", e.outstr # Stdout from command
  echo "====\nerror outpt: \n====\n", e.errstr # Stderr from the command

Command ngspice -b /tmp/ngpsice-simulation/zzz.netkRs8jE exited with non-zero code
command was: ngspice -b /tmp/ngpsice-simulation/zzz.netkRs8jE
exec direct: /home/test/workspace/git-sandbox/shell
return code: 1
regular out: 
====

====
error outpt: 
====
/tmp/ngpsice-simulation/zzz.netkRs8jE: No such file or directory