Steve Bold$Revision: 1.14 $Tcl

This TIP proposes some new commands through which Tcl scripts can create and monitor child processes.

This TIP is intended to overcome the following limitations of the existing exec and open commands: While the stderr stream of a child process can be redirected to a file, it cannot be directed to a pipe and so cannot be captured progressively as the process runs. has partially addressed this issue but only for the case where the child's stderr stream is directed to the same pipe as its stdout stream. Independent progressive capture of both stdout and stderr is still not possible.In the (admittedly rare) case that a program has a significant delay between closing its standard streams and the process itself terminating, a Tcl script running that program as a background process cannot determine the exit status without blocking until the process terminates.The existing exec and open commands impose a special interpretation on the characters <>|&. This causes two kinds of problems:scripts wishing to invoke a command on a remote computer using an rsh or similar command will sometimes wish to have characters such as <>|& interpreted on the remote machinescripts may pass a user entered string as an argument to exec. Such scripts may break unexpectedly if the comment string contains one of the special characters. Such problems could be considered a security weakness in Tcl.Multiple child processes can be launched together with pipes used to link the streams of adjacent processes. However, little flexibility is provided in such cases, for example you can only capture the exit status of the last process in the pipeline. A more general problem is that each process related command is a separate top-level command. This is inconsistent with much else in Tcl, makes it harder to find the related commands in some forms of documentation and increases the risk of name clashes as new process related commands are introduced. The BLT toolkit contains the command bgexec which addresses items (1) and (2) in the above list. However, the resulting implementation is complex and does not appear easy to transfer to the Tcl core. In addition, it is not clear to the author how bgexec could be extended to address items (3) and (4). A variety of other approaches to addressing these problems are listed on the Wiki []. This suggests that it may be difficult to achieve a consensus on what the ideal command(s) for launching processes should look like. This TIP provides a basis through which many of these approaches could be implemented in pure Tcl. The commands specified in this TIP map easily onto the existing low level process related functions in the Tcl core, so the implementation cost is low.

There shall be a new ensemble command, process, with at least four subcommands. The sub-command invoke takes 4 arguments and invokes a sub-process, returning the process id of the child process. The arguments are (in order):a list containing the program name invoke and its argumentsa channel to be connected to the stdin stream of the child process (or an empty string if the channel is to be disconnected in the child process).a channel to be connected to the stdout stream of the child process (or an empty string if the channel is to be disconnected in the child process).a channel to be connected to the stderr stream of the child process (or an empty string if the channel is to be disconnected in the child process).The sub-command pipe takes no arguments and returns a two element list containing the input and output channels of the pipe in that order.The sub-command status takes a single argument which is a process id and returns a two element list. The first element is either running or completed The second element is the exit status of the process.A process will report an arbitrary exit status of zero while it is running.The sub-command wait is similar to status but blocks until the child process has completed.

The following shows how the commands proposed here can be used to produce a bgexec like command in pure Tcl. Not all the bgexec options are included and the implementation lacks the error handling needed for a robust implementation. cHJvYyBiZ0V4ZWNDbG9zZUhhbmRsZXIge3BpZCBjbWR9IHs=ICAgbGFzc2lnbiBbcHJvY2VzcyBzdGF0dXMgJHBpZF0gc3RhdHVzIGV4aXRDb2RlICAgaWYgeyRzdGF0dXMgZXEgInJ1bm5pbmcifSB7ICAgICAgcHV0cyAiLi4uIGRlZmVycmluZyBjbG9zZSBoYW5kbGluZyBmb3IgJHBpZCI=ICAgICAgYWZ0ZXIgMTAwMCBbbGlzdCBiZ0V4ZWNDbG9zZUhhbmRsZXIgJHBpZCAkY21kXQ==ICAgfSBlbHNlIHs=ICAgICAgaWYgeyRjbWQgbmUgIiJ9IHs=ICAgICAgICAge2V4cGFuZH0kY21kICRwaWQgJGV4aXRDb2RlICAgICAgfQ==ICAgfQ==fQ==cHJvYyBiZ0V4ZWNSZWFkSGFuZGxlciB7Y2hhbiBjbWR9IHs=ICAgaWYge1tnZXRzICRjaGFuIGxpbmVdID09IC0xfSB7ICAgICAgY2xvc2UgJGNoYW4=ICAgICAgICAgICAgaWYge1tpbmZvIGV4aXN0cyA6OmJnRXhlY0Nsb3NlSW5mbygkY2hhbildfSB7ICAgICAgICAgbGFzc2lnbiAkOjpiZ0V4ZWNDbG9zZUluZm8oJGNoYW4pIHBpZCBjbWQ=ICAgICAgICAgYWZ0ZXIgMCBiZ0V4ZWNDbG9zZUhhbmRsZXIgJHBpZCAkY21kICAgICAgICAgdW5zZXQgOjpiZ0V4ZWNDbG9zZUluZm8oJGNoYW4pICAgICAgfQ==ICAgfSBlbHNlIHs=ICAgICAge2V4cGFuZH0kY21kICRsaW5lICAgfQ==fQ==cHJvYyBiZ0V4ZWNMaWtlIHthcmdzfSB7ICAgc2V0IG91dENoYW4gIiI7IHNldCBlcnJDaGFuICIiICAgc2V0IGkgMA==ICAgc2V0IGV4aXRDbWQgIiI7IHNldCBwYXJlbnRPdXRDaGFuICIiICAgd2hpbGUgeyRpICE9IFtsbGVuZ3RoICRhcmdzXX0gew==ICAgICAgc2V0IGFyZyBbbGluZGV4ICRhcmdzICRpXQ==ICAgICAgc3dpdGNoIC1nbG9iIC0tICRhcmcgew==ICAgICAgCQ==ICAgICAgCS1vbm91dHB1dCB7ICAgICAgCSAgIGluY3IgaQ==ICAgICAgCSAgIHNldCBjbWQgW2xpbmRleCAkYXJncyAkaV0=ICAgICAgCSAgIGxhc3NpZ24gW3Byb2Nlc3MgcGlwZV0gcGFyZW50Q2hhbiBvdXRDaGFuICAgICAgCSAgIGZpbGVldmVudCAkcGFyZW50Q2hhbiByZWFkYWJsZSBbbGlzdCBcICAgICAgICAgICAgICAgICAgYmdFeGVjUmVhZEhhbmRsZXIgJHBhcmVudENoYW4gJGNtZF0=ICAgICAgCSAgIHNldCBvdXRDbWQgJGNtZA==ICAgICAgCSAgIHNldCBwYXJlbnRPdXRDaGFuICRwYXJlbnRDaGFuICAgICAgCX0=ICAgICAgCQ==ICAgICAgCS1vbmVycm9yIHs=ICAgICAgCSAgIGluY3IgaQ==ICAgICAgCSAgIHNldCBjbWQgW2xpbmRleCAkYXJncyAkaV0=ICAgICAgCSAgIGxhc3NpZ24gW3Byb2Nlc3MgcGlwZV0gcGFyZW50Q2hhbiBlcnJDaGFuICAgICAgCSAgIGZpbGVldmVudCAkcGFyZW50Q2hhbiByZWFkYWJsZSBbbGlzdCBcICAgICAgICAgICAgICAgICAgYmdFeGVjUmVhZEhhbmRsZXIgJHBhcmVudENoYW4gJGNtZF0=ICAgICAgCX0=ICAgICAgCQ==ICAgICAgCS1vbmV4aXQgew==ICAgICAgCSAgIGluY3IgaQ==ICAgICAgCSAgIHNldCBleGl0Q21kIFtsaW5kZXggJGFyZ3MgJGldICAgICAgCX0=ICAgICAgCQ==ICAgICAgCS0qIHs=ICAgICAgCSAgIGVycm9yICJVbmtub3duIHN3aXRjaCAkYXJnIg==ICAgICAgCX0=ICAgICAgCQ==ICAgICAgCSogew==ICAgICAgCSAgIGJyZWFrICAgICAgCX0=ICAgICAgfQ==ICAgICAgaW5jciBpICAgfQ==ICAgICAgc2V0IGNtZExpbmUgW2xyYW5nZSAkYXJncyAkaSBlbmRdICAgIyBwdXRzIFtsaXN0IHByb2Nlc3MgaW52b2tlICRjbWRMaW5lICIiICRvdXRDaGFuICRlcnJDaGFuXQ==ICAgc2V0IHBpZCBbcHJvY2VzcyBpbnZva2UgJGNtZExpbmUgIiIgJG91dENoYW4gJGVyckNoYW5dICAgIyBDbG9zZSB0aGUgY2hpbGQgZW5kIG9mIHRoZSBwaXBlcyAtIGlmIHdlIG9wZW5lZCB0aGVtLg==ICAgZm9yZWFjaCB2YXIge291dENoYW4gZXJyQ2hhbn0gew==ICAgICAgaWYge1tzZXQgJHZhcl0gbmUgIiJ9IHs=ICAgICAgICAgY2xvc2UgW3NldCAkdmFyXQ==ICAgICAgfQ==ICAgfQ==ICAgICAgaWYgeyRwYXJlbnRPdXRDaGFuIGVxICIifSB7ICAgICAgIyBQb2xsIGZvciBjaGlsZCBwcm9jZXNzIGV4aXQgdGhlbiBub3RpZnkgY2xpZW50LCBvciBhdCBsZWFzdA==ICAgICAgIyBjbGVhbiB1cCB0aGUgem9tYmllLg==ICAgICAgYWZ0ZXIgMCBiZ0V4ZWNDbG9zZUhhbmRsZXIgJHBpZCAkZXhpdENtZA==ICAgfSBlbHNlIHs=ICAgICAgIyBXZSBjb3B5IEJMVCdzIHRyaWNrIG9mIGRlZmVycmluZyBwb2xsaW5nIHRpbGwgdGhlIHN0ZG91dCBwaXBlICAgICAgIyBjbG9zZXMuIFRoaXMgaXMgbWFyZ2luYWxseSBtb3JlIGVmZmljaWVudCwgbW9yZSBpbXBvcnRhbnRseQ==ICAgICAgIyBpdCBzdG9wcyBjbGllbnRzIGJlaW5nIG5vdGlmaWVkIG9mIHRoZWlyIHByb2Nlc3MgdW50aWwgYXQgc3Rkb3V0ICAgICAgIyBjaGFubmVsIGhhcyBjbG9zZWQuICAgICAgc2V0IDo6YmdFeGVjQ2xvc2VJbmZvKCRwYXJlbnRPdXRDaGFuKSBbbGlzdCAkcGlkICRleGl0Q21kXQ==ICAgfSAgIA==fQ==IyBub3cgc2hvdyBiZ0V4ZWNMaWtlIGluIGFjdGlvbiAuLi4=cHJvYyBzaG93RXhpdCB7cGlkIGNvZGV9IHs=ICAgcHV0cyAiJHBpZCB0ZXJtaW5hdGVkIHdpdGggY29kZSAkY29kZSI=fQ==cHJvYyBzaG93TGluZSB7Y2hhbm5lbCBsaW5lfSB7ICAgcHV0cyAiJGNoYW5uZWw6ICRsaW5lIg==fQ==cHJvYyBydW5McyB7YXJnc30gew==ICAgcHV0cyAiaW52b2tpbmcgbHMgb24gJGFyZ3MiICAgYmdFeGVjTGlrZSAtb25vdXRwdXQgInNob3dMaW5lIHN0ZG91dCIgLW9uZXJyb3IgInNob3dMaW5lIHN0ZGVyciIgXA==ICAgICAgICAgICAtb25leGl0IHNob3dFeGl0IGxzIHtleHBhbmR9JGFyZ3M=fQ==IyBTYW1wbGUgaW52b2NhdGlvbnM6IG5vdGUgd2hlbiBydW5uaW5nIHVuZGVyIHRjbHNoLCB0aGVyZSBpcyBubyBldmVudCBsb29wLA==IyB1c2UgJ3VwZGF0ZScgdG8gc2VlIHRoZSBvdXRwdXQgdG8gc2VlIHdoYXQncyBoYXBwZW5pbmcuIyBzdWNjZXNzZnVsIGxpc3Rpbmc=cnVuTHMgLg==IyBVbnN1Y2Nlc3NmdWwgbGlzdGluZw==cnVuTHMgbm90LWZvdW5kIyBMaXN0aW5nIG9mIChub24gZXhpc3RlbnQpIGZpbGVzIGNvbnRhaW5pbmcgZXhlYy9vcGVuIG1ldGEgY2hhcmFjdGVycw==cnVuTHMgPCA+IHwgJg==

For convenient use, the functionality proposed here needs to be supplemented with additional commands providing a higher level interface, perhaps one of them being similar to the bgExecLike example given previously. The author has decided to omit this feature from the TIP because:such commands can be implemented in pure Tcl using the commands described herethe exact nature of the high level commands may produce lengthy discussionsit could even be argued that such commands are more appropriate in tcllib rather than the Tcl coreAs with the current implementation of exec, each channel passed to process invoke must have a valid underlying OS file handle. Consequently when running on Windows:use of a wish standard channel will be immediately rejecteduse of a socket will be accepted but will trigger an error in the child process.Efficiency - The author has not yet attempted a detailed performance study, but this proposal does have some theoretical inefficiencies when compared to a pure C implementation, such as bgexec:an intermediate Tcl procedure is used to capture output from a pipeeach end of each pipe has to be wrapped in a CommandChannel before it can be passed back to the calling script, even if the pipe is just going to be used to link together two processes in a pipeline.

For Windows, an important limitation that is not addressed by this TIP, is the lack of control over the console window settings when invoking a process. This will require changes to TclpCreateProcess.A kill subcommand would be a useful addition to the process ensemble. On Windows, the ability to kill a child console process cleanly is related to the choice of console mode, so this issue would ideally be addressed in conjunction with item (1) above.A command to categorise an exit status obtained from process status or process wait along similar lines to the data placed in $errorCode by TclCleanupChildren().Some aspects of the existing exec command depend on use of temporary files. Since this TIP transfers the high level implementation of process launching into Tcl scripts, support for creation of uniquely named temporary files, as proposed in , would be useful.The wish console on Windows could be improved, using this mechanism, so that program names typed interactively will run in the background, allowing output to be seen before the process completes.The ability to define argv[0], independently from the program name, would occasionally be useful. For example, some UNIX shells run as login shells when argv[0] begins with a dash.Public C functions for invoking a process and creating a pipe wrapped in command channels.Support for detaching process and for reaping detached processes.The existing exit command could be duplicated in the process ensemble.The basic form of the existing pid command, which obtains the process id of the current process, could be added to the process ensemble.In some cases, it is more appropriate to run a child process with its streams connected to a null file rather than disconnected. Since the name of the null file is platform specific, it would be helpful to have a platform independent way of accessing the name.An option to obtain full status information. On Windows, process exit codes are 32 bit. On UNIX, higher bits of a waitpid() status value distinguish termination via exit() from termination via an uncaught signal.

Submitted as patch 1315115 []

This document has been placed in the public domain.