Alexandre Ferrieux$Revision: 1.15 $Tcl exec process subprocess pipeline channel

Currently, it is not easy to get both (separate) dataflows from the stdout and stderr of a child. BLT's bgexec does this in an extension, and could be added to the core. But the point of this TIP is to show that a much smaller code addition can provide a lower-level primitive with much more potential than bgexec's: a standalone pipe creation tool like TclX's pipe command.

Getting back both stdout and stderr from a child has long been an FAQ on , to the point that bgexec has been offered in an extension, BLT, whose main job is very remote from IPC. Now this has been a problem for many, who didn't want to have the problems of distributing a script depending on an extension. Moreover, bgexec does not scale up, in that it cannot bring back the separate stderrs of all four children in: CXNldCBmZiBbb3BlbiAifGEgfCBiIHwgYyB8IGQiIHJd A popular workaround for script-only purists is to spawn an external "pump" like cat in an [open ... r+], and redirect the wanted stderr to the write side of the pump. Its output can then be monitored through the read side: CXNldCBwdW1wIFtvcGVuICJ8Y2F0IiByK10=CXNldCBmMSBbb3BlbiAifGNtZCBhcmdzIDI+QCAkcHVtcCIgcl0=CWZpbGVldmVudCAkZjEgcmVhZGFibGUgZ290X3N0ZG91dA==CWZpbGVldmVudCAkcHVtcCByZWFkYWJsZSBnb3Rfc3RkZXJy Now this is all but elegant of course, difficult to deploy on Windows (where you need an extra cat.exe), and not especially efficient since the "pump" consumes context switches and memory bandwidth only to emulate a single OS pipe when Tcl is forced to create two of them via [open ... r+]. For this latter performance issue, a better alternative is a named pipe. But it is even harder to create on Windows, and it is a nightmare to handle its lifecycle properly (it doesn't die automagically with the creating process; blocks on open() if other side is not ready).

All this points to the obvious solution: steal TclX's pipe command, which wraps the OS's pipe()/CreatePipe() syscall, yielding two Tcl channels wrapping the read and write side of the underlying pipe. TclX's pipe command allows two syntaxes: CXBpcGUgcHIgcHc=CWZvcmVhY2gge3ByIHB3fSBbcGlwZV0gYnJlYWs= Its application to following stderr is straightfoward: CXBpcGUgcHIgcHc=CXNldCBmMSBbb3BlbiAifGNtZCBhcmdzIDI+QCAkcHciIHJdCWZpbGVldmVudCAkZjEgcmVhZGFibGUgZ290X3N0ZG91dA==CWZpbGVldmVudCAkcHIgcmVhZGFibGUgZ290X3N0ZGVycg== The basic functionality is to return a pair of channels, so the script-level API does just that: chan pipe Create a pipe, and return a list of two channels wrapping either side of the pipe. The first element is the side opened for reading, the second for writing: CWxhc3NpZ24gW2NoYW4gcGlwZV0gcHIgcHc= Subsequently, anything written to $pw will be readable on $pr. We purposefully drop the other [pipe pr pw] syntax from TclX, for the sake of minimality. A counterpart to chan pipe is added to Tcl's C API: int Tcl_CreatePipe(Tcl_Interp *interp, Tcl_Channel *rchan, Tcl_Channel *wchan, int flags) The interp argument is used for reporting errors and registering the channels created, the rchan and wchan arguments point to variables into which to place the channels for each end of the pipeline, and the flags argument is reserved for future expansion.

It has been in TclX for years, so it is fireproof. The code can be directly copied from there, and is dead simple, just calling Tcl_MakeFileChannel and Tcl_RegisterChannel on both fds/handles. Just the low-level syscall changes between unix and windows. It should also be noted that this primitive allows to deprecate the use of [open "|..."] and pid themselves ! CWxhc3NpZ24gW2NoYW4gcGlwZV0gcDFyIHAxdw==CWxhc3NpZ24gW2NoYW4gcGlwZV0gcDJyIHAydw==CXNldCBwaWRzIFtleGVjIGNtZCAkYXJncyA+QCAkcDF3IDI+QCAkcDJ3ICZdCWZpbGVldmVudCAkcDFyIHJlYWRhYmxlIGdvdF9zdGRvdXQ=CWZpbGVldmVudCAkcDJyIHJlYWRhYmxlIGdvdF9zdGRlcnI= Taking this idea one step further, one can even deprecate the "|" in exec, with: CWV4ZWMgYSB8IGIgfCBjICY= being rewritten as: CWV4ZWMgYSA+QCAkcDF3ICY=CWV4ZWMgYiA8QCAkcDFyID5AICRwMncgJg==CWV4ZWMgYyA8QCAkcDJyICY= (I'm not crazy about actually deprecating ol'good and concise idioms in favour of the tedious lines above; however as an orthogonality worshipper I like to think it would be theoretically possible to reimplement them in pure Tcl thanks to the added power of one tiny extra primitive) A more serious advantage, is that it allows the "half-close" of a command pipeline (detailed in which the current TIP makes partly obsolete). Indeed CWxhc3NpZ24gW2NoYW4gcGlwZV0gcHIgcHc=CWxhc3NpZ24gW2NoYW4gcGlwZV0gcXIgcXc=CWV4ZWMgYSA8QCRwciA+QCRxdyAm allows to close just the read side with CWNsb3NlICRwdw== and still read all the output of the command up to its EOF: CXdoaWxlIHtbZ2V0cyAkcXIgbGluZV0+PTB9IHsuLi59 which is notoriously impossible with CXNldCBwcSBbb3BlbiB8YSByK10=

On unix, there's no reason to limit the use of unnamed pipes to stdout and stderr. An arbitrary pipe topology can be set up between several childs with descriptors 3,4,5... For this we could imagine a natural extension of exec's 2>@ to 3>@, 4<@, etc. CWV4ZWMgZm9vID5AICRwIDM+QCAkcSA0PEAgJHIgJg==CWV4ZWMgYmFyIDxAICRxID5AICRyIDQ8QCAkcCAm Of course, such uses are extreme, but useful in complicated IPC setups to achieve much better performance (through direct point-to-point pipes) than would a simpler "star" or "blackboard" topology (where all children write back to a central message routing process).

A patch against the Tcl HEAD (8.6) is located at

This document has been placed in the public domain.