PowerShell in Depth, Second Edition (2015)

Part 2. PowerShell management

Chapter 13. Regular expressions

This chapter covers

· Regular expression syntax

· The –match operator

· Regular expressions in Select-String

· Regular expressions in the switch statement

· The REGEX object

Regular expressions are a powerful and complex language you can use to describe data patterns. Most pattern-based data, including email addresses, phone numbers, IP addresses, and more, can be represented as a “regex,” as they’re also known.

We need to set some limits for this chapter, because regular expressions are complex enough to deserve their own book. PowerShell uses industry-standard regex syntax and supports the full .NET regular expression library, but we’ll only be covering the basics. Most of our focus will be on how PowerShell uses a regex once you’ve written one. If you’d like to explore regular expressions further, write more complex expressions, or look for a regex that meets a particular need, we recommend one of these two books as a starting place: Mastering Regular Expressions by Jeffrey E.F. Friedl (a favorite), and Regular Expression Pocket Reference by Tony Stubblebine, both from O’Reilly. You can also visit http://RegExLib.com, a free community repository of regular expressions for specific tasks.

13.1. Basic regular expression syntax

A simple regular expression is a literal string. For example, the regular expression “Don” will match any instance of “Don,” although in PowerShell such matching is case insensitive by default. Where regular expressions show their power is when you start using wildcards, character classes, and placeholders instead of literal strings. Table 13.1 shows some of the basic syntax, which is also covered in PowerShell’s about_regular_expressions help file. We’ve included some additional comments and clarified a few points.

Table 13.1. Basic regular expression syntax

Regular expressions work around the concept of matching: Either the pattern described by the regex matches a given string or it doesn’t. If PowerShell has a match, it’ll give you $True. We typically use regex matches in IF statements and Where-Object script blocks. You’ve already seen matching in a simplistic form when you used wildcards. In the filesystem, for example, the wildcard pattern “*.tmp” would match all files ending in “.tmp,” including “this.tmp” and “that.tmp.” Those filesystem wildcards are, in essence, a super-simplified kind of regex.

Typing out all the characters you might want in a match can be tedious, even when using ranges. For that reason, regex syntax includes character classes, several of which are shown in table 13.2. (Although the table isn’t a comprehensive list, they’re the ones you’re most likely to use in PowerShell.) These classes stand in for any character contained within that class.

Table 13.2. Regular expression classes

The three examples for the \S regex symbol are where PowerShell is case sensitive, without being explicitly told to behave that way. The uppercase versions of those classes represent the “opposite,” that is, anything that isn’t contained in the class.

You can see from these class examples that it’s not always sufficient to know that some portion of a string has matched your regex; you may also need to know that no portion of the string failed to match, or you may need to know which part of the string matched your expression. Those are somewhat trickier tasks, and we’ll analyze them in some of the examples later in this chapter.

One way to be more precise with your expressions is to use quantifiers. The major ones are listed in table 13.3.

Table 13.3. Basic regular expression quantifiers

Parentheses can also be used to group subexpressions. For example, “(\d{1,3}\.){3}” would match “192.168.12.” because the group contains one to three digits followed by a period repeated three times. There’d be no match on “192.168.” But be careful because “192.168.12.1.1” will also match unless you use an anchor like “^(\d{1,3}\.){3}$.” where you force the match to occur at the beginning of the string.

13.2. The –match operator

PowerShell’s –match operator, its case-sensitive –cmatch version, and the logical opposites –notmatch and –cnotmatch (case-insensitive versions -imatch and –inotmatch also exist but aren’t used much because PowerShell is case insensitive by default) all use regular expressions. The left side of the operator is a string to test, whereas the right side is the regex to test against. If you match a single string, the result is $True if there’s a match and $False if there’s no match. In addition, PowerShell autopopulates a collection called $matches with the matches it’s detected. Using that collection, you can see exactly what PowerShell matched against, for example:

PS C:\> 'don' -match '\w'

True

PS C:\> $matches

Name Value

---- -----

d

In the previous example, you can see that “don” does match the regex “\w” and, more specifically, that it was the first letter, “d,” that made the match.

You can also match multiple strings, in which case the result of the operator is a collection of the strings that matched:

PS C:\> 'one','two','three' -match '\w{3}'

one

two

three

When used in this fashion, $matches isn’t populated. Looking at that example carefully, you might be surprised to see “three” in the output. There’s the tricky bit of a regex. It was the “thr” that matched the regex, which in turn caused the entire string to output. If you want to have three characters and then the end of the string, use the $ anchor:

PS C:\> 'one','two','three' -match '\w{3}$'

one

two

three

Whoops—that time it matched the “ree” in “three.” Let’s also add the ^ anchor:

PS C:\> 'one','two','three' -match '^\w{3}$'

one

two

There you go. This illustrates the difficulty of working with regular expressions—you have to be careful to make sure they’re rejecting the right data, as well as matching the right data. For example, see if you think this is a legitimate regex for a Universal Naming Convention (UNC) path such as \\Server\Share\Folder\File.txt:

PS C:\> '\\server\share\folder\file.txt' -match '\\\\\w+([\w\.]\\+)'

True

Seems fine, but let’s try a deliberately malformed UNC:

PS C:\> 'x\\server\share\folder\file.txt' -match '\\\\\w+([\w\.]\\+)'

True

That shouldn’t have matched, so you clearly need to tweak it:

PS C:\> 'x\\server\share\folder\file.txt' -match '^\\\\\w+([\w\.]\\+)'

False

And now you can make sure the original, valid UNC still works:

PS C:\> '\\server\share\folder\file.txt' -match '^\\\\\w+([\w\.]\\+)'

True

Again, these can be tricky. Can you think of any other ways to fool this regex or to improve it? Let’s break down what it’s doing in table 13.4.

Table 13.4. Deconstructing a regex

Breaking it down like that, we suspect other problems may exist with it. You won’t always end a UNC in a backslash, for example. Maybe the following would be better?

PS C:\> '\\server\share\folder\file.txt' -match '^\\\\\w+([\w\.\\]+)+'

True

We’ll leave it to you to decipher and decide if you can improve on it. For what it’s worth, the online regex library we use documents at least five regular expressions designed to detect various kinds of file paths, including UNCs: http://regexlib.com/Search.aspx?k=unc&c=2&m=-1&ps=20. That reinforces how tricky a regex can be to write.

Tip

Even if you decide to write your own regex, it’s always worth checking regexlib.com to determine if there’s a more efficient way.

13.3. The Select-String cmdlet

The Select-String cmdlet is designed to look through a bunch of data, finding data that matches a pattern you provide. You can pipe in strings or, better yet, point the command to a directory full of files. Here’s one example, which searches for all files containing the word “shell”:

PS C:\Windows\System32\WindowsPowerShell\v1.0\en-US> Select-String -Pattern

"shell" -SimpleMatch -Path *.txt -List

about_Aliases.help.txt:6: PowerShell.

about_Arithmetic_Operators.help.txt:5: Describes the operators that

perform arithmetic in Windows PowerShell.

about_Arrays.help.txt:9: of the same type. Windows PowerShell supports

data elements, such as

about_Assignment_Operators.help.txt:13: Windows PowerShell supports

the following assignment operators.

<display truncated for brevity>

This example used the command’s –SimpleMatch parameter, which tells it to treat the pattern not like a regex but like a text string. As you can see, it lists all files with a match, the matching line number, and the matching line itself. Now try this with a regex:

PS C:\Windows\System32\WindowsPowerShell\v1.0\en-US> Select-String -Pattern

"\sGet-\w{4,8}\s" -Path *.txt -List

about_Aliases.help.txt:89: get-help about_profile

about_Arithmetic_Operators.help.txt:397:C:\PS> get-date + $day

about_Arrays.help.txt:50: Microsoft .NET Framework. For example, the

objects that Get-Process

about_Assignment_Operators.help.txt:113: $a = get-service | sort

name

<display truncated for brevity>

You asked for matches on a pattern that specified a space, “Get-,” and four to eight characters followed by another space. These are only partial results; run the command on your own—in the same directory used here—to see the full list.

A variety of other parameters are available on the command to customize its behavior further, and it’s a great way to (for example) scan through text log files looking for a particular pattern of characters.

13.4. The Switch statement

You can also use regular expressions in the Switch statement. You’ll use the –regex parameter to indicate this. Consider the example in the following listing.

Listing 13.1. A regex Switch example

'abcd', 'Abcd', 'abc1', '123a', '!>@#' |

foreach {

switch -regex -case ($_){

"[a-z]{4}" {"[a-z]{4} matched $_"}

"\d" {"\d matched $_"}

"\d{3}" {"\d{3} matched $_"}

"\W" {"\W matched $_"}

default {"Didn't match $_"}

}

}

Which of the input strings didn’t match?

The tests can be decoded as follows:

· “[a-z]{4}” means match four characters, each of which is a letter in the range a to z.

· “\d” means match a digit.

· “\d{3}” means match three digits.

· “\W” means match a non-word character.

When we tested this code we received the following output:

[a-z]{4} matched abcd

Didn't match Abcd

\d matched abc1

\d matched 123a

\d{3} matched 123a

\W matched !>@#

Why didn’t “Abcd” match? At a PowerShell prompt it works:

PS C:\> "Abcd" -match "[a-z]{4}"

True

So why did it not match in our code snippet? The catch is that we also used the -case parameter in the Switch statement. It makes the matches case sensitive, which is the equivalent of doing the following:

PS C:\> "Abcd" -cmatch "[a-z]{4}"

False

The next listing shows one more example that might be a bit more practical.

Listing 13.2. Another regular expression Switch example

$Computername="LON-DC01"

Switch -regex ($Computername) {

"^SYR" {

#run Syracuse location code

}

"^LAS" {

#run Las Vegas location code

}

"^LON" {

#run London location code

}

"DC" {

#run Domain controller specific code

}

"FP" {

#run file/print specific code

}

"WEB" {

#run IIS specific code

}

Default {Write-Warning "No code found $computername"}

}

Listing 13.2 contains a code excerpt from something you might want to do. The Switch statement will evaluate the $computername variable using a series of regular expression patterns. Wherever there’s a match, the corresponding script block will execute. In the example, this would mean PowerShell would run any code specific to the London location and the domain controller role, assuming our servers follow a standard naming convention.

13.5. The REGEX object

Most of the time simple matching or not matching is sufficient. But regular expressions have some special capabilities such as finding all the matches in a string or replacing matched text. To accomplish these tasks you need to create a REGEX object. The easiest approach is to use the [regex] type accelerator with a regular expression pattern. Start with something simple:

PS C:\> [regex]$rx="Don"

Piping the object to Get-Member reveals some interesting possibilities:

PS C:\> $rx | get-member

TypeName: System.Text.RegularExpressions.Regex

Name MemberType Definition

---- ---------- ----------

Equals Method bool Equals(System.Object obj)

GetGroupNames Method string[] GetGroupNames()

GetGroupNumbers Method int[] GetGroupNumbers()

GetHashCode Method int GetHashCode()

GetType Method type GetType()

GroupNameFromNumber Method string GroupNameFromNumber(int i)

GroupNumberFromName Method int GroupNumberFromName(string name)

IsMatch Method bool IsMatch(string input), bool IsMatch(...

Match Method System.Text.RegularExpressions.Match Matc...

Matches Method System.Text.RegularExpressions.MatchColle...

Replace Method string Replace(string input, string repla...

Split Method string[] Split(string input), string[] Sp...

ToString Method string ToString()

Options Property System.Text.RegularExpressions.RegexOptio...

RightToLeft Property System.Boolean RightToLeft {get;}

Don’t use the REGEX object with –Match. Instead, invoke the Match() method. With this object, case matters:

PS C:\> $rx.Match("don")

Groups : {}

Success : False

Captures : {}

Index : 0

Length : 0

Value :

PS C:\> $rx.Match("Don")

Groups : {Don}

Success : True

Captures : {Don}

Index : 0

Length : 3

Value : Don

PS C:\> $rx.Match("Richard")

Groups : {}

Success : False

Captures : {}

Index : 0

Length : 0

Value :

The object writes a new object to the pipeline. The Match() method returns an object only on the first match. Look at the following:

PS C:\> $rx.Match("Let me introduce Don Jones. Don is a PowerShell MVP")

Groups : {Don}

Success : True

Captures : {Don}

Index : 17

Length : 3

Value : Don

If you want to identify all of the matches of “Don,” use the Matches() method:

PS C:\> $rx.Matches("Let me introduce Don Jones. Don is a PowerShell MVP")

Groups : {Don}

Success : True

Captures : {Don}

Index : 17

Length : 3

Value : Don

Groups : {Don}

Success : True

Captures : {Don}

Index : 28

Length : 3

Value : Don

This code will write a Match object for each match.

13.5.1. Replacing with REGEX

Where this gets even more interesting is when it comes time to replace matched text:

PS C:\> $rx.Replace("Let me introduce Don Jones. Don is a PowerShell

MVP","Donald")

Let me introduce Donald Jones. Donald is a PowerShell MVP

The REGEX object replaces all the matching instances of the pattern “Don” in the string with “Donald.”

The Replace() method from the String object as well as the –Replace operator are using regular expressions under the hood. You can use regular expression patterns:

PS C:\> "172.16.10.12" -replace "\d{3}","XXX"

XXX.16.10.12

13.5.2. Splitting with REGEX

Another technique we want to show with the REGEX object is splitting data on the regular expression match. Let’s say you want to parse the contents of the Windows Update log. The log contains lines like the following:

$s="2012-03-14 18:57:35:321 1196 13f8 PT Server URL =

http://172.16.10.1/SimpleAuthWebService/SimpleAuth.asmx"

Suppose you want to split this line on the time stamp value (18:57:35:321) so you construct a REGEX object:

[regex]$r="\d{2}:\d{2}:\d{2}:\d{3}"

In order to split, it has to match, so you’ll test:

PS C:\> $r.match($s)

Groups : {18:57:35:321}

Success : True

Captures : {18:57:35:321}

Index : 11

Length : 12

Value : 18:57:35:321

Excellent. Now you can split the string on the pattern match:

PS C:\> $r.split($s)

2012-03-14

1196 13f8 PT Server URL =

http://172.16.10.1/SimpleAuthWebService/SimpleAuth.Asmx

The second line in the previous example is wrapping, but it worked. Notice that the matching data is gone. That happens because you used that as the split “character” so all you’re left with is everything before it, the date, and everything after it. Depending on your pattern, you may have extra spaces to take care of, but that’s simple to fix:

PS C:\> $a=$r.split($s) | foreach {$_.trim()}

PS C:\> $a[1]

1196 13f8 PT Server URL =

http://172.16.10.1/SimpleAuthWebService/SimpleAuth.aSmx

The Trim() method deletes any leading or trailing spaces from the string. Extending this to the entire file, you can split each line and save the second part to a variable:

PS C:\> $data=get-content C:\windows\WindowsUpdate.log |

foreach {$r.split($_)[1].Trim()}

You can then do whatever you want with $data. We’ll come back to the Windows Update log in a moment.

13.6. Subexpressions and named captures

The last item we want to demonstrate with regular expressions and PowerShell is the ability to use subexpressions and named captures. Consider this simple example:

PS C:\> "aaa 123 bbb 456" -match "\d{3}"

True

PS C:\> $matches

Name Value

---- -----

0 123

The –Match operator found the first three-digit match. A subexpression, at its simplest, is the pattern wrapped in parentheses:

PS C:\> "aaa 123 bbb 456" -match "(\d{3})"

True

PS C:\> $matches

Name Value

---- -----

1 123

0 123

One subtle but important change is that the Match object has two values now. One of them is for the subexpression:

PS C:\> "aaa 123 bbb 456" -match "(\d{3}) (\w{3})"

True

PS C:\> $matches

Name Value

---- -----

2 bbb

1 123

0 123 bbb

Now we have subexpression matches for numbers and text. Let’s take the next step and use a REGEX object so we can capture everything:

PS C:\> [regex]$rx = "(\d{3}) (\w{3})"

PS C:\> $m =$rx.Matches("aaa 123 bbb 456")

PS C:\> $m

Groups : {123 bbb, 123, bbb}

Success : True

Captures : {123 bbb}

Index : 4

Length : 7

Value : 123 bbb

PS C:\> $m.groups

Groups : {123 bbb, 123, bbb}

Success : True

Captures : {123 bbb}

Index : 4

Length : 7

Value : 123 bbb

Success : True

Captures : {123}

Index : 4

Length : 3

Value : 123

Success : True

Captures : {bbb}

Index : 8

Length : 3

Value : bbb

If we wanted to get the match of just the letters or numbers, that gets a little tricky. Fortunately, the solution is to use a named capture. All you need to do is to define a name for your capture like this:

PS C:\> "aaa 123 bbb 456" -match "(?<word>\w{3}) (?<num>\d{3})"

True

Now for the cool part. The Match object now shows these names:

PS C:\> $matches

Name Value

---- -----

word aaa

num 123

0 aaa 123

We can get the name we want like this:

PS C:\> $matches.num

123

PS C:\> $matches.word

aaa

When a REGEX object is used, you have to access the matches using the defined names:

PS C:\> [regex]$rx="(?<word>\w{3}) (?<num>\d{3})"

PS C:\> $m = $rx.Matches("aaa 123 bbb 456")

The tricky part is that $m contains several types of objects:

PS C:\> $m | foreach {$_.groups["num"].value}

123

456

PS C:\> $m | foreach {$_.groups["word"].value}

aaa

bbb

Here’s a more practical example. Suppose you get a string that looks like a UNC and you want to get the server and share name:

PS C:\> $p = "\\server01\data"

You could split the string on the last \ into an array and know that the server name would be element 0 and the share name element 1. Or you could use a named capture:

PS C:\> $p -match "^\\\\(?<server>\w+)\\(?<share>\w+$)"

True

PS C:\> $matches.server

server01

PS C:\> $matches.share

Data

Let’s end this section by returning to the Windows Update log. Earlier we showed you how to split each line. Because the log file has a known and consistent layout, you can use a regular expression named capture and “objectify” the log file.

Note

We realize there are other ways to import the log into PowerShell, but this is a log everyone has and serves as a good learning tool.

We’ll walk through the process using a sample line from the log:

$t = "2014-01-07 10:23:48:211 880 a9c IdleTmr Incremented PDC

RefCount for Network to 2"

The layout uses the format DATE TIME PID TID COMPONENT TEXT, so you need to define a set of regular expression patterns to capture the various elements:

$t -match "(?<Date>\d{4}-\d{2}-\d{2})\s+(?<Time>(\d{2}:)+\d{3})\s+

(?<PID>\d+)\s+(?<TID>\w+)\s+(?<Component>\w+)\s+(?<Message>[.*)"

The match result shows success:

PS C:\> $matches

Name Value

---- -----

Message Incremented PDC RefCount for Network to 2

Time 10:23:48:211

Date 2014-01-07

Component IdleTmr

PID 880

TID a9c

1 48:

0 2014-01-07 10:23:48:211 880 a9c...

Now let’s use the REGEX object:

[regex]$rx = "(?<Date>\d{4}-\d{2}-\d{2})\s+(?<Time>(\d{2}:)+\d{3})\s+

(?<PID>\d+)\s+(?<TID>\w+)\s+(?<Component>\w+)\s+(?<Message>.*)"

An advantage with this approach is that you can identify the names:

PS C:\> $rx.GetGroupNames()

0

1

Date

Time

PID

TID

Component

Message

The first two aren’t our names, so they can be filtered out:

$names = $rx.GetGroupNames() | where {$_ -match "\w{2}"}

For our proof of concept, we’ll get the first five lines of the log file:

$t = get-content C:\windows\WindowsUpdate.log | select -first 5

The variable $t is a collection of five lines that need to be “decoded” individually:

$data = $t | foreach {

$rx.Matches($_) | foreach {

$match = $_

$names | foreach -begin {$hash=[ordered]@{}} -process {

$hash.Add($_,$match.groups["$_"].value)

} -end { [pscustomobject]$hash}

}

}

In the code we get all the matches using the REGEX object. Each match is then processed using a nested ForEach-Object loop to get the corresponding named match. Each match is added to an ordered hash table that’s using the Name as the key and Match as the value. After processing all names, PowerShell writes a custom object to the pipeline using the hash table. Everything is saved to $data.

PS C:\> $data

Date : 2014-01-20

Time : 00:45:08:210

PID : 1112

TID : 24b4

Component : Report

Message : WARNING: CSerializationHelper:: InitSerialize failed :

0x80070002

Date : 2014-01-20

Time : 00:45:08:211

PID : 1112

TID : 24b4

Component : Report

Message : CWERReporter::Init succeeded

Date : 2014-01-20

Time : 00:45:08:211

PID : 1112

TID : 24b4

Component : Agent

Message : *********** Agent: Initializing Windows Update Agent

***********

Date : 2014-01-20

Time : 00:45:08:211

PID : 1112

TID : 24b4

Component : Agent

Message : * Found 2 persisted download calls to restore

Date : 2014-01-20

Time : 00:45:08:212

PID : 1112

TID : 24b4

Component : DnldMgr

Message : Download manager restoring 1 downloads

Because everything is now an object, the log file can be viewed any way you want:

PS C:\> $data | where component -eq 'agent' |

select Date,Time,Message | format-table –AutoSize -Wrap

Date Time Message

---- ---- -------

2014-01-20 00:45:08:211 *********** Agent: Initializing Windows

Update Agent ***********

2014-01-20 00:45:08:211 * Found 2 persisted download calls to restore

You could use these techniques for just about any log file.

13.7. Summary

Remember, the REGEX object is based on a regular expression pattern that can be as complicated as necessary. Be careful to watch how matches are made and be sure to test for failures as well when developing a regular expression.

Tip

Don’t try to reinvent the wheel. Visit sites like Regexlib.com to look for a pattern that someone has already developed. In most cases you can use the pattern as is in your PowerShell scripting.

Regular expressions are, no question, a complex language all their own. We’ve briefly touched on the main ways in which PowerShell uses them, as well as on the basics of regex syntax, including the REGEX object. We know there’s more to regex than we could cover in this chapter, and if you’d like to explore them further, please check out the resources we mentioned in the beginning of the chapter.