Tcl HomeTcl Home Hosted by
ActiveState

Google SiteSearch

New Regular Expression Features in Tcl 8.1

Tcl 8.1 now handles advanced regular expressions (REs). Previous regular expression handling is almost unchanged except that clumsy handling of escapes like \n has been much improved, and a few escapes that were previously legal (but useless) now won't work.

Note that a few advanced features aren't useful yet but are ready for future Tcl releases. That's because Tcl 8.1 (apart from the regular expression engine) implements only the Unicode locale (where all characters sort in Unicode order, there are no multi-character collating elements and no equivalence classes).

This document has an overview of the new regular expression features. For exact semantics and more details, see the new re_syntax(n) reference page. (The re_syntax(n) page was split from the 8.1 regexp(n) reference page, which used to cover RE syntax for all Tcl commands.) This howto document covers:

1. Regular Expression Overview

2. Regular Expressions in Tcl 8.1

3. Summary: Regular Expression changes in Tcl 8.1

Part 1. Regular Expression Overview

This Part describes regular expressions (REs), explains REs from Tcl 8.0 and before, and describes the Tcl regexp and regsub commands. Part Two describes the new Tcl 8.1 REs.

What are Regular Expressions?

A regular expression, or RE, describes strings of characters (words or phrases or any arbitrary text). It's a pattern that matches certain strings and doesn't match others. For example, you could write an RE to tell you if a string contains a URL (World Wide Web Uniform Resource Locator, such as http://somehost/somefile.html). Regular expressions can be either broad and general or focused and precise.

A regular expression uses metacharacters (characters that assume special meaning for matching other characters) such as *, [], $ and .. For example, the RE [Hh]ello!* would match Hello and hello and Hello! (and hello!!!!!). The RE [Hh](ello|i)!* would match Hello and Hi and Hi! (and so on). A backslash (\) disables the special meaning of the following character, so you could match the string [Hello] with the RE \[Hello\].

Regular Expressions in Tcl 8.0 and Before

Regular expressions in Tcl 8.0 and before had the following metacharacters:
.Match any single character (e.g., m.d matches mad, mod, m3d, etc.)
[]Bracket expression: Match any one of the enclosed characters (e.g., [a-z0-9_] matches a lowercase ASCII letter, a digit, or an underscore)
^Start-of-string anchor: Match only at the start of a string (e.g., ^hi matches hi and his but not this)
$End-of-string anchor: Match only at the end of a string (e.g., hi$ matches hi and chi but not this)
*Zero-or-more quantifier: makes the previous part of the RE match zero or more times (e.g., M.*D matches MD, MAD, MooD, M.D, etc.)
?Zero-or-one quantifier: makes the previous part of the RE match zero or one time (e.g., hi!? matches hi or hi!)
+One-or-more quantifier: makes the previous part of the RE match one or more times (e.g., hi!+ matches hi! or hi!! or hi!!! or ...)
|Alternation (vertical bar): Match just one alternative (e.g., this|that matches this or that)
()Subpattern: Group part of the RE. Many uses, such as:
  • Makes a quantifier apply to a group of text (e.g., ([0-9A-F][0-9A-F])+ matches groups of two hexadecimal digits: A9 or AB03 or 8A6E00, but not A or A2C).
  • Set limits for alternation (e.g., "Eat (this|that)!" matches "Eat this!" or "Eat that!").
  • Used for subpattern matching in the regexp and regsub commands.
\Escape: Disables meaning of the following metacharacter (e.g., a\.* matches a or a. or a.. or etc.). Note that \ also has special meaning to the Tcl interpreter (and to applications, such as C compilers).

The syntax above is supported in Tcl 8.1. Tcl 8.1 also supports advanced regular expressions (AREs). These powerful expressions are introduced in more detail in Part Two. Briefly, though, AREs support backreferences, lookahead, non-greedy matching, many escapes, features that are useful for internationalization (handling collation elements, equivalence classes and character classes), and much more.

The Tcl 8.1 regular expression engine almost always interprets 8.0-style REs correctly. In the few cases that it doesn't, and when the problem is too difficult to fix, the 8.1 engine has an option to select 8.0 ("ERE") interpretation.

Overview of regexp and regsub

The Tcl commands regexp and regsub use regular expressions:
  • regexp compares a string to an RE. It returns a value of 1 if the RE matches part or all of the string or 0 if there's no match. Optionally, it stores the matched part of the string in a variable (and also can store subparts of a string in multiple variables). For example, to compare the string in $line against the RE [Hh]ello!*, you would write:
    regexp {[Hh]ello!*} $line match

    If part or all of the line variable matches the RE, regexp stores the matching part in the match variable and returns a value of 1.

  • regsub substitutes part of a string that matches an RE. For instance, the following command edits the string from $in_line to replace all space or tab characters with a single space character; the edited line is stored in the out_line variable:
    regsub -all {[ \t]+} $in_line { } out_line

    Please also read the following section about backslash processing.

Backslash Processing

If you've used Tcl, you probably recognize the \t in the previous example as a character-entry escape that stands for a tab character.

We actually used the 8.1 syntax above; the example wouldn't have worked under 8.0! In Tcl 8.0 and before, you had to surround the regular expression with double quotes so the Tcl backslash processor could convert the \t to a literal tab character. The square brackets had to be hidden from the backslash processor by adding backslashes before them, which made code harder to read and possibly more error-prone. Here's the previous example rewritten for Tcl 8.0 and before:

regsub -all "\[ \t\]+" $in_line { } out_line

For more about the simplified 8.1 syntax, see the section Backslash Escapes.

Part 2. Regular Expressions in Tcl 8.1

Tcl 8.1 regular expressions are basically a superset of 8.0 REs. This howto document has an overview of the new features. Please see the re_syntax(n) reference page for exact semantics and more details.

Non-Greedy Quantifiers

A quantifier specifies "how many." For example, the quantifier * in the RE z* matches zero or more zs. By default, regular expression quantifiers are greedy: they match as much text as they can. Tcl 8.1 REs also have non-greedy quantifiers, which match the least text they can. To make a non-greedy quantifier, add a question mark (?) at the end.

Let's start by storing some HTML text in a variable, then using two regexp commands to match it. The first RE is greedy, and the second is non-greedy:


% set x {<EM>He</EM> sits, but <EM>she</EM> stands.}
<EM>He</EM> sits, but <EM>she</EM> stands.
% regexp {<EM>.*</EM>} $x match; set match
<EM>He</EM> sits, but <EM>she</EM>
% regexp {<EM>.*?</EM>} $x match; set match
<EM>He</EM>
The first RE <EM>.*</EM> is "greedy." It matches from the first <EM> to the last </EM>. The second RE <EM>.*?</EM>, with a question mark (?) after the * quantifier, is non-greedy: it matches as little text as possible after the first <EM>. Could you write a greedy RE that works like the non-greedy version? It isn't easy! A greedy RE like <EM>[^<]*</EM> would do it in this case -- but it wouldn't work if there were other HTML tags (with a < character) between the pair of <EM> tags in the $x string.

Here are a new string and another pair of REs to match it:


% set y {123zzz456}
123zzz456
% regexp {3z*} $y match; set match
3zzz
% regexp {3z*?} $y match; set match
3
The greedy RE 3z* matches all the zs it can (three) under its "zero or more" rule. The non-greedy RE 3z*? matches just 3 because it matches the fewest zs it can under its "zero or more" rule.

To review, the greedy quantifiers from Tcl 8.0 are: *, +, and ?. So the non-greedy quantifiers (added in Tcl 8.1) are: *?, +?, and ??. Tcl 8.1 also has the new quantifiers {m}, {m,}, and {m,n}, as well as the non-greedy versions {m}?, {m,}?, and {m,n}?. The section on bounds explains -- and has more examples of non-greedy matching.

Backslash Escapes

A backslash (\) disables the metacharacter after it. For example, a\* matches the character a followed by a literal asterisk (*) character. In Tcl 8.0 and before, it was legal to put a backslash before a non-metacharacter -- for instance, regexp {\p} matched the character p. (Note that regexp {\n} matched the character n, which was a source of confusion. To get a newline character into an RE before version 8.1, you had to write regexp "\n" so Tcl processing inside double quotes would convert the \n to a newline.)

The Tcl 8.1 regular expression engine interprets backslash escapes itself. So now regexp {\n} matches a newline, not the character n. REs are simpler to write in 8.1 because of this. (You can still write regexp "\n" -- and let Tcl conversion happen inside the double quotes -- so most old code will still work.)

One of the most important changes in 8.1 is that a backslash inside a bracket expression is treated as the start of an escape. In 8.0 and before, a backslash inside brackets was treated as a literal backslash character. For example, in 8.0 and before, regexp {[a\n]} would match the characters a, \, or n. But in 8.1, regexp {[a\n]} would match the characters a or newline (because \n is the backslash escape for "newline").

Tcl 8.1 has also added many new backslash escapes. For instance, \d matches a digit. Some of these are listed below, and the re_syntax(n) reference page has the whole list.

In Tcl 8.1 regular expressions (but not in other parts of the language), it's illegal to use a backslash before a non-metacharacter unless it makes a valid escape. So regexp {\p} is now an error. If you have code that (for some bizarre reason) has regular expressions with a backslash before a non-metacharacter, like regexp {\p}, you'll need to fix it.

As explained above, the Tcl 8.1 regular expression engine now interprets backslash sequences like \n to mean "newline". It also has four new kinds of escapes: character entry escapes, class shorthand escapes, constraint escapes, and back references. Here's an introduction. (The re_syntax(n) page has full details.)

  • A character entry escape is a convenient way to enter a non-printing or other difficult character. For instance, \n represents a newline character. \uwxyz (where wxyz is hexadecimal) represents the Unicode character U+wxyz.
  • Class shorthand escapes are shorthand for common character classes. For example, \d stands for [[:digit:]], which means "any single digit."
  • A constraint escape constrains an RE to match only at a certain place. For example, the constraint escape \m matches only at the start of a word -- so the RE \mhi will match the third word in the string he said hi but won't match he said thigh.
  • A back reference matches the same string that was matched by a previous parenthesized subexpression. (This works like subexpressions in regsub, but it's used for matching instead of extracting.) For example, (X.*Y)\1 matches any doubled string that starts with X and ends with Y, such as XYXY, XabcYXabcY, X--YX--Y, etc.

Finally, remember that (as in Tcl 8.0 and before) some applications, such as C compilers, interpret these backslash sequences themselves before the regular expression engine sees them. You may need to double (or quadruple, etc.) the number of backslashes for these applications. Still, in straight Tcl 8.1 code, writing backslash escapes is now both simpler and more powerful than in 8.0 and before.

Bounds

You've seen the quantifiers *, +, and ?. They specify "how many" (respectively, zero or more, one or more, and zero or one). Tcl 8.1 added new quantifiers that let you choose exactly how many matches: the bounds operators, {}.

These operators come in three greedy forms: {m}, {m,}, and {m,n}. The corresponding non-greedy forms are {m}?, {m,}?, and {m,n}?.

  • The {m} quantifier matches exactly m occurrences. So does {m}?. For example, either #{70} or #{70}? match a string of exactly 70 # characters.
  • The {m,} quantifier matches at least m occurrences. Here's a demo of the greedy and non-greedy versions:
    
    % set x {a##b#######c}
    a##b#######c
    % regexp {#{4,}} $x match; set match
    #######
    % regexp {#{4,}?} $x match; set match
    ####
    
    Notice that the first two number signs (##) in the string are never matched because there aren't at least four of them.
  • The {m,n} quantifier matches at least m but no more than n occurrences.

    For example, the RE http://([^/]+/?){1,3} would match Web URLs that have 3 components (like http://xyz.fr/euro/billets.htm), or with 2 components (like http://xyz.fr/euro/, or with just 1 component (like http://xyz.fr). The RE matches a final slash (/) if there is one. As always, a greedy match will match as long a string as possible: it would try for 3 matches.

    A non-greedy quantifier would try to match the least (1 match). But be careful: http://([^/]+/?){1,3}? won't match all the way to a possible slash because it matches the fewest characters possible! (With input http://xyz.fr/, that RE would match just http://x.) This brings up one of the many subtleties in these advanced regular expressions: that the outer non-greedy quantifier overrides the inner greedy quantifiers and makes all quantifiers non-greedy! There's an explanation in re_syntax(n) reference page section named Matching.

Character Classes

A character class is a name for one or more characters. For example, punct stands for the "punctuation" characters. A character class is always written as part of a bracket expression, which is a list of characters enclosed in [].

For instance, the character class named digit stands for any of the digits 0-9 (zero through nine). The character class is written with the class name inside a set of brackets and colons, like this: [[:digit:]]. The old familiar expression for digits is written as a range: [0-9]. When you compare the new character class to the old range version, you can see that the outer square brackets are the same in both. So a character class is written [:classname:].

The table below describes the Tcl 8.1 character classes.
alphaA letter (includes many non-ASCII characters).
upperAn upper-case letter.
lowerA lower-case letter.
digitA decimal digit.
xdigitA hexadecimal digit.
alnumAn alphanumeric (letter or digit).
printAn alphanumeric. (Same as alnum.)
blankA space or tab character.
spaceA character producing white space in displayed text. (Includes en-space, hair space, many others.)
punctA punctuation character.
graphA character with a visible representation.
cntrlA control character.

You can use more than one character class in a bracket expression. You can also mix character classes with ranges and single characters. For instance, [[:digit:]a-cx-z] would match a digit (0-9), a, b, c, x, y, or z -- and [^[:digit:]a-cx-z] would match any character except those. This syntax can take some time to get familiar with! The key is to look for the character class (here, [:digit:]) inside the bracket expression.

The advantage of character classes (like [:alpha:]) over explicit ranges in brackets (like [a-z]) is that character classes include characters that aren't easy to type on ASCII keyboards. For example, the Spanish language includes the character ñ. It doesn't fall into the range [a-z], but it is in the Tcl 8.1 character class [:alpha:]. In the same way, the Spanish punctuation character ¡ isn't in a list of punctuation characters like [.!?,], but it is part of [:punct:].

Tcl 8.1 has a standard set of character classes that are defined in the source code file generic/regc_locale.c. Tcl 8.1 has one locale defined: the Unicode locale. It may support other locales (and other character classes) in the future.

Collating Elements

A collating symbol lets you represent other characters unambiguously. A collating symbol is written surrounded by brackets and dots, like [.number-sign.] Collating symbols must be written in a bracket expression (inside []). So [[.number-sign.]] will match the character #, as you can see here:

% regexp {[[.number-sign.]]+} {123###456} match
1
% set match
###
Tcl 8.1 has a standard set of collating symbols that are defined in the source code file generic/regc_locale.c. Note: Tcl 8.1 does not implement multi-character collating elements like ch (which is the fourth character in the Spanish alphabet a, b, c, ch, d, e, f, g, h, i...) So the examples below are not supported in Tcl 8.1, but are here for completeness. (Future versions of Tcl may have multi-character collating elements.)

Suppose ch and c sort next to each other in your dialect, and ch is treated as an atomic character. The example bracket expression below uses two collating symbols. It matches one or more of ch and c. But it doesn't match an h standing alone:


% set input "cchchh"
cchchh
% regexp {[[.ch.][.c.]]+} $input match; set match
cchch
Here's one tricky and surprising thing about collating symbols. A caret at the start of a bracket expression ([^...) means that, in a locale with multi-character collating elements, the symbol can match more than one character. For instance, the RE in the example below matches any character other than c, followed by the character b. So the expression matches all of chb:

% set input chb
% regexp {[^[.c.]]b} $input match; set match
chb
Again, the two previous examples are not supported in Tcl 8.1, but are here for completeness.

Equivalence Classes

An equivalence class is written as part of a bracket expression, like [[=c=]]. It's any collating element that has the same relative order in the collating sequence as c.

Note: Tcl 8.1 only implements the Unicode locale. It doesn't define any equivalence classes. So, although the Tcl regular expression engine supports equivalence classes, the examples below are not supported in Tcl 8.1. (Future versions of Tcl may define equivalence classes.)

Let's imagine that both of the characters A and a fall at the same place in the collating sequence; they belong to the same equivalence class. In that case, both of the bracket expressions [[=A=]b] and [[=a=]b] are equivalent to writing [Aab]. As another example, if o and ô are members of an equivalence class, then all of the bracket expressions [[=o=]], [[=ô=]], and [oô] match those same two characters.

Noncapturing Subpatterns

There are two reasons to put parentheses around all or part of an RE. One is to make a quantifier (like * or +) apply to the parenthesized part. For instance, the RE Oh,( no!)+ would match Oh, no! as well as Oh, no! no! and so on. The other reason to use parentheses is that they capture the matched text. Captured text is used in back references, in "matching" variables in the regexp command, as well as in the regsub command.

If you don't want parentheses to capture text, add ?: after the opening parenthesis. For instance, in the example below, the subexpression (?:http|ftp) matches either http or ftp but doesn't capture it. So the back reference \1 will hold the end of the URL (from the second set of parentheses):


% set x http://www.activestate.com
http://www.activestate.com
% regsub {(?:http|ftp)://(.*)} $x {The hostname is \1} answer
1
% set answer
The hostname is www.activestate.com

Lookahead Assertions

There are times you'd like to be able to test for a pattern without including that text in the match. For instance, you might want to match the protocol in a URL (like http or ftp), but only if that URL ends with .com. Or maybe you want to match the protocol only if the URL does not end with .edu. In cases like those, you'd like to "look ahead" and see how the URL ends. A lookahead assertion is handy here.

A positive lookahead has the form (?=re). It matches at any place ahead where there's a substring like re. A negative lookahead has the form (?!re). It matches at any point where the regular expression re does not match. Let's see some examples:


% set x http://www.activestate.com
http://www.activestate.com
% regexp {^[^:]+(?=.*\.com$)} $x match
1
% set match
http
% regexp {^[^:]+(?=.*\.edu$)} $x match
0
% regexp {^[^:]*(?!.*\.edu$)} $x match
1
% set match
http
The regular expressions above may seem complicated, but they're really not bad! Find the lookahead expression in the first regexp command above; it starts with (?= and ends at the corresponding parenthesis. The "guts" of this lookahead expression is .*\.com$, which stands for "a string that ends with .com". So the first regexp command above matches any string containing non-colon (:) characters, as long as the rest of the string ends with .com. The second regexp is similar but looks for a string ending with .edu. Because regexp returns 0, you can see that this doesn't match. The third regexp looks for a string not ending with .edu. It matches because $x ends with .com.

Tcl 8.1 lets you document complex regular expressions by embedding comments. See the next section.

Switches

Tcl 8.1 added command switches to regexp and regsub. For a complete list, see the commands' reference pages. Let's look at two of the most important changes.

Complex REs can be difficult to document. The -expanded switch sets expanded syntax, which lets you add comments within a regular expression. Comments start with a # character; whitespace is ignored. This is mostly for scripting -- but you can also use it on a command line, as we'll do in the example below. Let's look the same RE twice: first in the standard compact syntax, and second in expanded syntax:


% set x http://www.activestate.com
http://www.activestate.com
% regexp {^[^:]+(?=.*\.com$)} $x match
1
% set match
http
% regexp -expanded {
  ^             # beginning of string
  [^:]+         # all characters to the first colon
  (?=           # begin positive lookahead
    .*\.com$    # for a trailing .com
  )             # end positive lookahead
} $x match
1
% set match
http
In expanded syntax, you can use space and tab characters to indent and make your code clear. To enter actual space and tab characters into your RE, use the escapes \s and \t, respectively.

The other important new switch we'll cover here is -line. It enables newline-sensitive matching. By default (without -line), Tcl regular expressions have always treated newlines as an ordinary character. For example, if a string contains several lines (separated by newline characters), the end-of-string anchor $ wouldn't match at any of the embedded newlines. To write code that matched line-by-line, you had to read input lines one by one and do separate matches against each line.

With the -line switch, the metacharacters ^, $, ., and [] treat a newline as the end of a "line." So, for example, the regular expression ^San Jose matches the second line of input below:


% set x {Dolores Sanchez
San Jose, CA}
Dolores Sanchez
San Jose, CA
% regexp {^San Jose} $x match
0
% regexp -line {^San Jose} $x match
1
% set match
San Jose
The -line switch actually enables two other switches. You can set part of the features from -line by choosing one of these switches instead: The -lineanchor switch makes ^ and $ match at the beginning and end of a line. The -linestop switch makes . and [] stop matching at a newline character.

Options, Directors

This section introduces two more features from Tcl 8.1. Details are in the re_syntax(n) reference page.

An 8.1 RE can start with embedded options. These look like (?xyz), where xyz are one or more option letters. For instance, (?i)ouch matches OUCH because i is the "case-insensitive matching" option. Other options include (?e), which marks the rest of the regular expression as an 8.0-style RE -- to let you avoid confusion with the new 8.1 syntax.

An RE can also start with three asterisks, which is a director. For example, ***= is the director that says the rest of the regular expression is literal text. So the RE ***=(?i)ouch matches exactly (?i)ouch; the (?i) isn't treated as an option.

Part 3. Summary: Regular Expression Changes in Tcl 8.1

Tcl 8.1 added advanced regular expression syntax. The new re_syntax(n) reference page has details.

This table below summarizes the new syntax:
{m}Matches m instances of the previous pattern item
{m}?Matches m instances of the previous pattern item. Non-greedy.
{m,}Matches m or more instances of the previous pattern item.
{m,}?Matches m or more instances of the previous pattern item. Non-greedy.
{m,n}Matches m through n instances of the previous pattern item.
{m,n}?Matches m through n instances of the previous pattern item. Non-greedy.
*?Matches zero or more of the previous pattern item. Non-greedy.
+?Matches one or more of the previous pattern item. Non-greedy.
??Matches zero or one of the previous pattern item. Non-greedy.
(?:re)Groups a subpattern, re, but does not capture the result.
(?=re)Positive lookahead. Matches the point where re begins.
(?!re)Negative lookahead. Matches any point where re does not begin.
\cOne of many backslash escapes.
[. .]Delimits a collating element within a bracketed expression.
[= =]Delimits an equivalence class within a bracketed expression.
[: :]Delimits a character class within a bracketed expression.
(?abc)Embedded options a, b, and c
***Director

Some of the new switches for regexp and regsub are:
-expandedEnable expanded syntax (for comments)
-lineEnable newline-sensitive matching
-linestopMake [] and . stop at newlines.
-lineanchorMake ^ and $ match the start and end of a line.
Add link to comments for /doc/howto/regexp81.html