Don PorterDonal K. Fellows$Revision: 1.39 $

This TIP analyzes existing limitations on the coding of control structure commands as procs, and presents expanded forms of catch and return to remove those limitations.

It is a distinguishing feature of Tcl that everything is a command, including control structure functionality that in many other languages are part of the language itself, such as if, for, and switch. The command interface of Tcl, including both a return code and a result, allows extensions to create their own control structure commands. Control structure commands have the feature that one or more of their arguments is a script, often called a body, meant to be evaluated in the caller's context. The control structure command exists to control whether, when, in what context, or how many times that script is evaluated. When the body is evaluated, however, it is intended to behave as if it were interpreted directly in the place of the control structure command. The built-in commands of Tcl provide the ability for scripts themselves to define new commands. Notably, the proc command makes this possible. In addition, other commands such as catch, return, uplevel, and upvar offer enough control and access to the caller's context that it is possible to create new control structure commands for Tcl, entirely at the script level. Almost. There is one limitation that separates control structure commands created by proc from those created in C by a direct call to Tcl_Create(Obj)Command. It is most easily seen in the following example that compares the built-in command while to the command control::do created by proc in the control package of tcllib. ICAlIHBhY2thZ2UgcmVxdWlyZSBjb250cm9sICAlIHByb2MgYSB7fSB7d2hpbGUgMSB7cmV0dXJuIC1jb2RlIGVycm9yfX0=ICAlIHByb2MgYiB7fSB7Y29udHJvbDo6ZG8ge3JldHVybiAtY29kZSBlcnJvcn0gd2hpbGUgMX0=ICAlIGNhdGNoIGE=ICAxICAlIGNhdGNoIGI=ICAw The control structure command control::do fails to evaluate return -code error in such a way that it acts the same as if return -code error was evaluated directly within proc b.

There are two deficiencies in Tcl's built-in commands that lead to this incapacity in control structure commands defined by proc. First, catch is not able to capture the information. Consider: ICAgJSAgc2V0IGNvZGUgW2NhdGNoIHs=ICAgICAgICAgIHJldHVybiAtY29kZSBlcnJvciAtZXJyb3JpbmZvIGZvbyAtZXJyb3Jjb2RlIGJhciBiYXo=ICAgICAgfSBtZXNzYWdlXQ== After evaluation, code contains "2" (TCL_RETURN), and message contains "baz", but the other values are locked away in internal fields of the Tcl_Interp structure as interp->returnCode, interp->errorCode, and interp->errorInfo. The "-errorcode" and "-errorinfo" values will be copied to the global variables "::errorCode" and "::errorInfo", respectively, but there will be no way at the script level to get at the interp->returnCode value which was the value of the original "-code" option. Second, even if the information were available, there is no built-in command in Tcl that can be evaluated within the body of a proc to make the proc itself act as if it were the command return -code. Stated another way, it is not possible to create a command with proc that behaves exactly the same as return -code. Because of that, it is also not possible to create a command with proc that behaves exactly the same as while, if, etc. - any command that evaluates any of its arguments as a script in the caller's context. This is a curious, and likely unintentional, limitation. Tcl goes to great lengths to be sure I can create my own break replacement with proc. IHByb2MgbXlCcmVhayB7fSB7cmV0dXJuIC1jb2RlIGJyZWFrfQ== It would be a welcome completion of Tcl's set of built-in commands to be able to create a replacement for every one of them using proc.

The return command shall have syntax: IHJldHVybiA/b3B0aW9uIHZhbHVlIC4uLj8gP3Jlc3VsdD8= There can be any number of option value pairs, and any value at all is acceptable for an option argument. The legal values of a value argument are limited for some options, as follows: the value after a "-code" must be either an integer (32-bit only), or one of the strings, "ok", "error", "return", "break", or "continue", just as in the 8.4 spec for return. The default value for the "-code" option is "0". the value after a "-level" must be a non-negative integer. The default value for the "-level" option is "1". the value after a "-options" must be a dictionary (). The default value for the "-options" option is an empty dictionary. The keys and values in the dictionary value of the "-options" option are pulled out and treated as additional option value arguments to the return command. Note that this "-options" option for option expansion is offered only because Tcl itself has no syntax for argument expansion, as observed many, many times before (for example, ). The result argument, if any, is stored in the interp as the result of the return command. In default operation, this becomes the result of the procedure in which the return command is evaluated. The return code of the return command is determined by the values of the "-code" and "-level" options. If the value of the "-level" option is non-zero, then the return code of return is TCL_RETURN. If the value of the "-level" option is "0", then the return code of return is the value of the "-code" option, translated from string, as needed. In this way, IHJldHVybiAtbGV2ZWwgMCAtY29kZSBicmVhaw== is a synonym for IGJyZWFr while IHJldHVybiAtY29kZSBicmVhaw== spelled out with defaults filled in as: IHJldHVybiAtbGV2ZWwgMSAtY29kZSBicmVhaw== continues to function as before, causing the procedure in which the return is evaluated to return the TCL_BREAK return code. All option value arguments to return are stored in a return options dictionary kept in the interp, just as the result argument gets stored in the result of the interp. The TclUpdateReturnInfo() function is modified, so that each level of procedure returning decrements the value of the "-level" key in the return options dictionary. When the value of the "-level" key reaches "0", the return code from the current procedure will be the value of the "-code" key in the return options dictionary. Otherwise, the return code of the current procedure will be TCL_RETURN. In this way, IHJldHVybiAtbGV2ZWwgMiAtY29kZSBvaw== is equivalent to IHJldHVybiAtY29kZSByZXR1cm4= and should (absent some intervening catch) cause a normal return to the caller's caller. Likewise, IHJldHVybiAtbGV2ZWwgMyAtY29kZSBvaw== would cause a normal return to the caller's caller's caller (again absent an intervening catch), something that can't currently be accomplished. The catch command shall have syntax: IGNhdGNoIHNjcmlwdCA/cmVzdWx0VmFyPyA/b3B0aW9uc1Zhcj8= The new argument optionsVar, if present, will be the name of a variable in which a dictionary of return options should be stored. The return options stored in that dictionary are exactly those needed so that the evaluation of IGNhdGNoICRzY3JpcHQgcmVzdWx0IG9wdGlvbnM=IHJldHVybiAtb3B0aW9ucyAkb3B0aW9ucyAkcmVzdWx0 is completely indistinguishable (except for the existence and values of variables "result" and "options") from the direct evaluation of $script by the interpreter. In particular, any values of the "::errorCode" and "::errorInfo" variables are the same as if there were never a catch in the first place. In addition, when the result of catch is TCL_ERROR, the value in the errorLine field of the Interp struct will be stored as the value of the "-errorline" key in the return options dictionary. This specification may seem a bit complex, but it makes possible very simple solutions to the problems posed above.

First lets revisit the analysis: ICAgJSAgc2V0IGNvZGUgW2NhdGNoIHs=ICAgICAgICAgIHJldHVybiAtY29kZSBlcnJvciAtZXJyb3JpbmZvIGZvbyAtZXJyb3Jjb2RlIGJhciBiYXo=ICAgICAgfSBtZXNzYWdlIG9wdGlvbnNd After evaluation, code contains "2" (TCL_RETURN), message contains "baz", and now options contains: IC1lcnJvcmNvZGUgYmFyIC1lcnJvcmluZm8gZm9vIC1jb2RlIDEgLWxldmVsIDE= So, the options variable now contains the information that was previously inaccessible. We can now IHJldHVybiAtb3B0aW9ucyAkb3B0aW9ucyAkbWVzc2FnZQ== to get the same results as if the catch had never been there in the first place. In 8.4 Tcl, it is not possible to implement a replacement for the return command as a proc. After this proposal, such a replacement is: IHByb2MgbXlSZXR1cm4gYXJncyB7ICAgICBzZXQgcmVzdWx0ICIiICAgICBpZiB7W2xsZW5ndGggJGFyZ3NdICUgMn0gew==ICAgICAgICAgc2V0IHJlc3VsdCBbbGluZGV4ICRhcmdzIGVuZF0=ICAgICAgICAgc2V0IGFyZ3MgW2xyYW5nZSAkYXJncyAwIGVuZC0xXQ==ICAgICB9ICAgICBzZXQgb3B0aW9ucyBbZXZhbCBbbGlzdCBkaWN0IGNyZWF0ZSAtbGV2ZWwgMV0gJGFyZ3NdICAgICBkaWN0IGluY3Igb3B0aW9ucyAtbGV2ZWw=ICAgICByZXR1cm4gLW9wdGlvbnMgJG9wdGlvbnMgJHJlc3VsdA==IH0= In every way myReturn should be an equivalent to return. The new ability to exactly reproduce stack traces makes a catch of large scripts more attractive. For example, a procedure that allocates some resource, then performs operations, and finally frees the resource before returning. In order to be sure the resource is freed, we must catch any errors that might cause the procedure to return before the freeing of the resource. The solution looks like: IHByb2MgZG9Tb21ldGhpbmcge30gew==ICAgICBzZXQgcmVzb3VyY2UgW2FsbG9jYXRlXQ==ICAgICBjYXRjaCB7ICAgICAgICAgICMgQXJiaXRyYXJpbHkgbG9uZyBzY3JpcHQgb2Ygb3BlcmF0aW9ucw==ICAgICB9IHJlc3VsdCBvcHRpb25zICAgICBkZWFsbG9jYXRlICRyZXNvdXJjZQ==ICAgICByZXR1cm4gLW9wdGlvbnMgJG9wdGlvbnMgJHJlc3VsdA==IH0= With that structure, we are confident the resource is always freed, but any error or exception will be presented to the caller exactly as if it had never been caught in the first place. Here are two examples of how to use the new features in a control structure proc. The essence of a control structure command is its ability to evaluate a script in the caller's context, preserving the illusion that no additional stack frame was ever used. So, a proc replacement for eval illustrates the technique. The first approach assumes one knows the internal details of how the uplevel command adds to the stack trace. This is straightforward, but will require a rewrite if uplevel ever changes how it manipulates the stack trace. IHByb2MgbXlFdmFsIHNjcmlwdCB7ICAgICBpZiB7W2NhdGNoIHt1cGxldmVsIDEgJHNjcmlwdH0gcmVzdWx0IG9wdGlvbnNdID09IDF9IHs=ICAgICAgICAgc2V0IHN0YWNrIFtkaWN0IGdldCAkb3B0aW9ucyAtZXJyb3JpbmZvXQ==ICAgICAgICAgcmVnc3ViIHtccytpbnZva2VkIGZyb20gd2l0aGluXHMrInVwbGV2ZWwgMSBcJHNjcmlwdCIkfSAkc3RhY2sge30gc3RhY2s=ICAgICAgICAgcmVnc3ViIHtcKCJ1cGxldmVsIiBib2R5IGxpbmUgKFxkKylcKSR9ICRzdGFjayBbc3Vic3QgLW5vYmFja3NsYXNoZXMgXA==ICAgICAgICAgICAgICAgICB7KCJbbGluZGV4IFtpbmZvIGxldmVsIDBdIDBdIiBib2R5IGxpbmUgXDEpfV0gc3RhY2s=ICAgICAgICAgZGljdCBzZXQgb3B0aW9ucyAtZXJyb3JpbmZvICRzdGFjaw==ICAgICB9ICAgICBkaWN0IGluY3Igb3B0aW9ucyAtbGV2ZWw=ICAgICByZXR1cm4gLW9wdGlvbnMgJG9wdGlvbnMgJHJlc3VsdA==IH0= A second, more robust solution is possible, but requires a bit more context gymnastics. IG5hbWVzcGFjZSBldmFsIGNvbnRyb2wgew==ICAgICBwcm9jIGV2YWwgc2NyaXB0IHs=ICAgICAgICAgdmFyaWFibGUgcmVzdWx0ICAgICAgICAgdmFyaWFibGUgb3B0aW9ucw==ICAgICAgICAgc2V0IGNvZGUgW3VwbGV2ZWwgMSBcICAgICAgICAgICAgICAgICBbbGlzdCA6OmNhdGNoICRzY3JpcHQgW25hbWVzcGFjZSB3aGljaCAtdmFyaWFibGUgcmVzdWx0XSBcICAgICAgICAgICAgICAgICAgICAgICAgIFtuYW1lc3BhY2Ugd2hpY2ggLXZhcmlhYmxlIG9wdGlvbnNdXV0=ICAgICAgICAgaWYgeyRjb2RlID09IDF9IHs=ICAgICAgICAgICAgIHNldCBsaW5lIFtkaWN0IGdldCAkb3B0aW9ucyAtZXJyb3JsaW5lXQ==ICAgICAgICAgICAgIGRpY3QgYXBwZW5kIG9wdGlvbnMgLWVycm9yaW5mbyBcICAgICAgICAgICAgICAgICAgICAgIlxuICAgIChcIltsaW5kZXggW2luZm8gbGV2ZWwgMF0gMF1cIiBib2R5IGxpbmUgJGxpbmUpIg==ICAgICAgICAgfQ==ICAgICAgICAgZGljdCBpbmNyIG9wdGlvbnMgLWxldmVsICAgICAgICAgcmV0dXJuIC1vcHRpb25zICRvcHRpb25zICRyZXN1bHQ=ICAgICB9IH0= Note that in the second solution we did not have to strip away the contributions of uplevel to the stack trace, because we captured the stack trace before uplevel added anything. Then we could add our own information (drawing in part on the new "-errorline" value available to us now at the script level). We confirm that either approach solves the original problem: ICUgcHJvYyBhIHt9IHtldmFsIHtyZXR1cm4gLWNvZGUgZXJyb3J9fQ==ICUgcHJvYyBiIHt9IHtteUV2YWwge3JldHVybiAtY29kZSBlcnJvcn19ICUgcHJvYyBjIHt9IHtjb250cm9sOjpldmFsIHtyZXR1cm4gLWNvZGUgZXJyb3J9fQ==ICUgY2F0Y2ggYQ==IDE=ICUgY2F0Y2ggYg==IDE=ICUgY2F0Y2ggYw==IDE= Finally, the new features make possible a utility command that can be of use to people making simple control structure commands, or doing simple wrapping, where there is no need to augment the stack trace, or to treat any return codes in a special way: IG5hbWVzcGFjZSBldmFsIGNvbnRyb2wgew==ICAgICBwcm9jIGFzY2FsbGVyIHNjcmlwdCB7ICAgICAgICAgaWYge1tpbmZvIGxldmVsXSA8IDJ9IHs=ICAgICAgICAgICAgIHJldHVybiAtY29kZSBlcnJvciBcICAgICAgICAgICAgICAgICAgICAgIltsaW5kZXggW2luZm8gbGV2ZWwgMF0gMF0gY2FsbGVkIG91dHNpZGUgYSBwcm9jIg==ICAgICAgICAgfQ==ICAgICAgICAgdmFyaWFibGUgcmVzdWx0ICAgICAgICAgdmFyaWFibGUgb3B0aW9ucw==ICAgICAgICAgc2V0IGNvZGUgW3VwbGV2ZWwgMiBcICAgICAgICAgICAgICAgICBbbGlzdCA6OmNhdGNoICRzY3JpcHQgICBbbmFtZXNwYWNlIHdoaWNoIC12YXJpYWJsZSByZXN1bHRdIFw=ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbbmFtZXNwYWNlIHdoaWNoIC12YXJpYWJsZSBvcHRpb25zXV1dICAgICAgICAgaWYgeyRjb2RlID09IDB9IHs=ICAgICAgICAgICAgIHJldHVybiAkcmVzdWx0ICAgICAgICAgfQ==ICAgICAgICAgZGljdCBpbmNyIG9wdGlvbnMgLWxldmVsIDI=ICAgICAgICAgcmV0dXJuIC1vcHRpb25zICRvcHRpb25zICRyZXN1bHQ=ICAgICB9IH0= Within a proc, ascaller $script will take care of all aspects of evaluating $script in the caller context, and exiting as appropriate for all non-TCL_OK return codes.

The return -code command has always accepted any integer value as a valid argument, allowing package and application authors to define their own new return codes as needed by their own control structure commands. Now that return will accept any option argument, and catch can capture all option value argument pairs passed to the caught return command, package and application authors now have the ability to augment their custom return codes with additional data. Some prefix convention should be established to avoid key name conflicts in the return options dictionary.

Reviewers of drafts of this TIP wondered whether the new "-level" option to return raised the possibility of trouble with an attempt to return more levels than beyond the top of the call stack. It should be understood that return -level N does not take any shortcut past the intervening levels. Each level of the call stack gets a TCL_RETURN return code, and a "-level" value, dropping by one each step up the stack. Any level in the stack might choose to catch the TCL_RETURN and treat it as it wishes. This is exactly the way the existing return -code return is handled. Normally, it would cause a normal return to the caller's caller, but if the caller chooses to 'catch' it, then the caller has control. At the toplevel we run out of callers. Then the question becomes how is a TCL_RETURN code at toplevel handled? ICUgcmV0dXJuIC1sZXZlbCAwICAgICAgIDsjIHNhbWUgYXMgYSBUQ0xfT0sgYXQgdG9wbGV2ZWw=ICUgcmV0dXJuIC1sZXZlbCAxICAgICAgIDsjIHNhbWUgYXMgW3JldHVybl0=ICUgcmV0dXJuIC1sZXZlbCAyICAgICAgIDsjIHNhbWUgYXMgW3JldHVybiAtY29kZSByZXR1cm5dIGNvbW1hbmQgcmV0dXJuZWQgYmFkIGNvZGU6IDI= From the C level, Tcl_AllowExceptions() can be used to modify this toplevel behavior. The following proc will produce the same results as above, but from any level in the call stack (absent an intervening catch): ICUgcHJvYyBlc2NhcGUgbGV2ZWwgew==ICAgICAgIHNldCB4IFtpbmZvIGxldmVsXQ==ICAgICAgIGluY3IgeCAkbGV2ZWw=ICAgICAgIHJldHVybiAtbGV2ZWwgJHg=ICAgfQ==ICUgZXNjYXBlIDA=ICUgZXNjYXBlIDE=ICUgZXNjYXBlIDI=IGNvbW1hbmQgcmV0dXJuZWQgYmFkIGNvZGU6IDI= Another concern was whether this proposal gave slave interpreters any new powers over their masters. The return code from evaluation of an untrusted script in a slave interpreter should always be wrapped in a catch already, lest a TCL_ERROR in the script blow the stack. Given that, the only thing this proposal does is give the catch command more information to use to decide how to handle the misbehaving script.

It is the author's belief that this proposal is completely compatible with prior Tcl 8.X releases. Any error-free script that ran before, should continue to run with the same results. At the C level, only internal changes are made, and no new interfaces are defined. Any extension or embedding C program that sticks to the public stubs interface should see no visible change.

This proposal is implemented by Tcl Patch 531640 at SourceForge. The prototype covers all described functionality, but might be further improved with more substantial bytecompiling of [return].

The main reason the global variables ::errorInfo and ::errorCode exist is to give the script level access to stack and error code information following the catch of a script that raises an error. After this proposal, the catch command itself provides access to that information, so the global variables are not required. One can imagine deprecating them, asking users of Tcl 8.5 to stop writing code that accesses them. They could still have apparent existence, to satisfy the needs of scripts written for earlier Tcl 8.X releases, by means of read traces. In time, Tcl 9 could either continue the read trace scheme, or not provide these global variables at all. One part of Tcl itself that currently makes use of the ::errorCode and ::errorInfo variables is the bgerror command. Currently, bgerror accepts exactly one argument, the error message. To make use of stack or error code information, bgerror must retrieve them from the global variables. The proper values of these global variables are re-set by Tcl_BackgroundError() prior to evaluation of bgerror. As an alternative, Tcl_BackgroundError() could first attempt to call bgerror with two arguments, first the message, then a dictionary of options. If that call returned TCL_ERROR, then a second attempt could be made with a single message argument. In that way, cleaner bgerror commands that get all data from arguments could be supported, while still keeping support for those bgerror commands that were defined for single argument use. It has been noted several times that the processing of the value of ::errorInfo is rather difficult because it is an arbitrary string with no documented structure. A different, more structured way of representing stack trace information would be an improvement. This proposal does not propose an alternative, but because it offers an extensible dictionary for storing arbitrary return options data, it does provide an infrastructure where such approaches might be tried out.

This proposal is a synthesis of ideas from many sources. As best I can recall, major contributions came from Joe English, Andreas Leitgeb, Reinhard Max, and Kevin Kenny. If you like the idea, give them some credit; it you don't, blame me for combining the ideas badly.

Documentation for tcllib's control package:

This document has been placed in the public domain.