How do I merge stdout and stderr of a child process?

[Darksonn]

If I open a child process with tokio::process::Command, the ChildStderr and ChildStdout types are two separate streams of data. I would like to merge the two streams so I can read the output line-by-line from both streams in an interleaved manner.

[Darksonn]

The general approach for merging the two streams is the following:

1. Put stdout and stderr into tokio::io::BufReader.
2. Put the buffered streams into tokio::io::Lines.
3. Wrap it in tokio_stream::wrappers::LinesStream.
4. Merge them with tokio_stream::StreamExt::merge.

This gives the following code:

use tokio::io::{AsyncBufReadExt, BufReader};
use tokio_stream::StreamExt;
use tokio_stream::wrappers::LinesStream;

// Variable of type tokio::process::Child.
let child = ...;

// Take ownership of stdout and stderr from child.
let stdout = child.stdout.take().unwrap();
let stderr = child.stderr.take().unwrap();

// Wrap them up and merge them.
let stdout = LinesStream::new(BufReader::new(stdout).lines());
let stderr = LinesStream::new(BufReader::new(stderr).lines());
let mut merged = StreamExt::merge(stdout, stderr);

// Iterate through the stream line-by-line.
while let Some(line) = merged.next().await {
    // Since reading a line may fail, we use the question-mark to unwrap the line.
    let line = line?;
    println!("{}", line);
}

[nayeemrmn]

Thanks for looking into this, Deno has a need for this feature.

I would like to merge the two streams so I can read the output line-by-line from both streams in an interleaved manner.

Systems which support combining pipes can do better than buffering line-by-line. Running the following script should show that shells multiplex the two streams more granularly:

#!/bin/bash
printf "abc"
printf "def" 1>&2
printf "ghi"
printf "jkl" 1>&2

abcdefghijkl

The following shows that Python 3's pipe combination can do the same:

#!/usr/bin/env python3
import subprocess
print(subprocess.run(["./temp.sh"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT))

CompletedProcess(args=['./temp.sh'], returncode=0, stdout=b'abcdefghijkl')

Ultimately, I think we need a mirror of https://docs.rs/os_pipe/0.9.2/os_pipe/ where https://docs.rs/os_pipe/0.9.2/os_pipe/struct.PipeReader.html can be asynchronously read.

[andrewbaxter]

Just went down this rabbit hole. IIUC to state the above differently, with one fd there's only one buffer in the child process, whereas when combining in rust there could be two child buffers leading to interleaving issues unrelated to using line buffering in the rust parent process.

Tokio has an async pipe type created with tokio::net::unix::pipe::pipe() but they don't implement Stdio so they can't be passed to Command functions. Additionally, they don't implement try_clone or whatever the suggested solution is with the standard library parallels.

I thought maybe I could work around it by turning them into owned fds, cloning, then turning them back but then hit the Stdio issue.

I couldn't find a 3rd party async pipe library.

That said, I just looked at the Python source and afaict the buffers on stdout/stderr are set up independently? So I'm not sure if even sharing an fd would resolve interleaving issues, although maybe it'd be more deterministic.

One other idea - for as popular a request as this is, maybe instead of exposing a pipe + conversion methods, having a Command or Stdtio constant to combine the streams would be generally sufficient.