Detecting and Resolving Performance Issues - Windows PowerShell for Administration: The Personal Trainer (2015)

Windows PowerShell for Administration: The Personal Trainer (2015)

Chapter 12. Detecting and Resolving Performance Issues

In previous chapters, I discussed techniques for monitoring and optimizing Windows systems. Monitoring is the process by which systems are regularly checked for problems. Optimization is the process of fine-tuning system performance to maintain or achieve its optimal capacity. Now that you know the essentials for monitoring and optimization, let’s dig deeper and look at techniques that you can use to detect and resolve performance issues.

Understanding Performance Monitoring Commands

Performance monitoring helps you watch for adverse conditions and take appropriate action to resolve them. Windows PowerShell has several commands for this purpose, and in this section, we’ll look at the ones you’ll use the most.

Commands you can use to monitor performance include:

· Get-Counter Gets objects representing real-time performance counter data directly from the performance monitoring instrumentation in Windows. You can list the performance counter sets and the counters that they contain, set the sample size and interval, and specify the credentials of users with permission to collect performance data.

Get-Counter [-MaxSamples NumSamples] [-Counter] CounterPaths
[-SampleInterval Interval] {AddtlParams}

Get-Counter -ListSet SetNames {AddtlParams}

{AddtlParams}
[-Credential CredentialObject] [-ComputerName ComputerNames]

· Export-Counter Exports performance counter data to log files in BLG (binary performance log, the default), CSV (comma-separated), or TSV (tab-separated) format. This cmdlet is designed to export data that is returned by the Get-Counter and Import-Counter cmdlets.

Export-Counter [-FileFormat Format] [-Path] SavePath
-InputObject PerformanceCounterSampleSets {AddtlParams}

{AddtlParams}
[-Force {$True | $False}] [-Circular {$True | $False}]
[-MaxSize MaxSizeInBytes]

· Import-Counter Imports performance counter data from performance counter log files and creates objects for each counter sample in the file. The objects created are identical to those that Get-Counter returns when it collects performance counter data. You can import data from BLG, CSV, and TSV formatted log files. When you are using BLG, you can import up to 32 files in each command. To get a subset of data from a file, use the parameters of Import-Counter to filter the data that you import.

Import-Counter [-Path] FilePaths {AddlParams}
Import-Counter -ListSet SetNames [-Path] FilePaths
Import-Counter [-Summary {$True | $False}]

{AddtlParams}
[-Counter CountersToInclude] [-MaxSamples NumSamples]
[-StartTime DateTime] [-EndTime DateTime]

Get-Counter is designed to track and display performance information in real time. It gathers information on any performance parameters you’ve configured for monitoring and presents it as output. Each performance item you want to monitor is defined by the following three components:

· Performance object Represents any system component that has a set of measurable properties. A performance object can be a physical part of the operating system, such as the memory, the processor, or the paging file; a logical component, such as a logical disk or print queue; or a software element, such as a process or a thread.

· Performance object instance Represents single occurrences of performance objects. If a particular object has multiple instances, such as when a computer has multiple processors or multiple disk drives, you can use an object instance to track a specific occurrence of that object. You can also elect to track all instances of an object, such as whether you want to monitor all processors on a system.

· Performance counter Represents measurable properties of performance objects. For example, with a paging file, you can measure the percentage utilization using the %Usage counter.

In a standard installation of Windows, many performance objects are available for monitoring. As you add services, applications, and components, additional performance objects can become available. For example, when you install the Domain Name System (DNS) on a server, the DNS object becomes available for monitoring on that computer.

Tracking Performance Data

Using Get-Counter, you can write performance data to the output or to a log file. The key to using Get-Counter is to identify the path names of the performance counters you want to track. The performance counter path has the following syntax:

\\ComputerName\ObjectName\ObjectCounter

where ComputerName is the computer name or IP address of the local or remote computer you want to work with, ObjectName is the name of a counter object, and ObjectCounter is the name of the object counter to use. For example, if you want to track the available memory on Dbserver79, you type the following, and the output would be similar to that shown:

get-counter "\\dbs79\memory\available mbytes"

Timestamp CounterSamples
--------- --------------
2/27/2015 4:26:54 PM \\dbs79\memory\available mbytes : 16375
2/27/2015 4:26:55 PM \\dbs79\memory\available mbytes : 16072

NOTE Enclosing the counter path in double quotation marks is required in this example because the counter path includes spaces. Although double quotation marks aren’t always required, it is good form to always use them.

Specifying the computer name as part of the counter path is optional. If you don’t specify the computer name in the counter path, Get-Counter uses the values you specify in the –ComputerName parameter to set the full path for you. If you don‘t specify computer names, the local computer name is used. Although this allows you to easily work with multiple computers, you should familiarize yourself with the full path format because this is what is recorded in performance traces and performance logs. Without the computer name in the path, the abbreviated path becomes

\ObjectName\ObjectCounter

In the following example, you check the available memory on multiple computers by using the –ComputerName parameter:

get-counter –computername fileserver12, dbserver18, dcserver21
"\memory\available mbytes"

When you are working with a remote computer, you might need to provide alternative credentials. You can do this as shown in the following example:

$cred = get-credential
get-counter –computername fileserver12, dbserver18, dcserver21
"\memory\available mbytes" –credential $cred

When you use Get-Credential, Windows PowerShell prompts you for a user name and password and then stores the credentials provided in the $cred variable. These credentials are then passed to the remote computers for authentication.

You can easily track all counters for an object by using an asterisk (*) as the counter name, such as in the following example:

get-counter "\\dbserver79\Memory\*"

Here, you track all counters for the Memory object.

When objects have multiple instances, such as with the Processor or LogicalDisk object, you must specify the object instance you want to work with. The full syntax for this is as follows:

\\ComputerName\ObjectName(ObjectInstance)\ObjectCounter

Here, you follow the object name with the object instance in parentheses. When an object has multiple instances, you can work with all instances of that object using _Total as the instance name. You can work with a specific instance of an object by using its instance identifier. For example, if you want to examine the Processor\% Processor Time counter, you can use this command to work with all processor instances:

get-counter "\\dbserver79\Processor(_Total)\% Processor Time"

Or you use this command to work with a specific processor instance:

get-counter "\\dbserver79\Processor(0)\% Processor Time"

Here, Processor(0) identifies the first processor on the system.

Get-Counter has several parameters. –MaxSamples sets the number of samples to collect. –SampleInterval sets the time between samples where the default is 1 second. –ListSet lists installed counters for the specified objects.

Get-Counter writes its output to the prompt by default. You can redirect the output to a performance log by sending the output to Export-Counter. By default, Export-Counter exports performance counter data to log files in binary performance log format. Using the –FileFormat parameter, you can set the format as CSV for a comma-delimited text file, TSV for a tab-delimited text file, or BLG for a binary file. Consider the following example:

get-counter "\\dbserver79\Memory\*" | export-counter –fileformat
tsv –path .\dbserver79.txt

Here, you track all counters for the Memory object and write the output to a tab-delimited text file called Dbserver79.txt in the current working directory. When you want to work with this data later, you use Import-Counter. Typeimport-counter -path followed by the path to the performance log to view the performance data. Type import-counter -summary -path followed by the path to the performance log to get a summary view of the data. Typeimport-counter -listset * -path followed by the path to the performance log to see what counters were tracked. Optionally, use –StartTime and –EndTime to specify a datetime range to review. Consider the following example:

$startdate = (get-date).adddays(-1)
$enddate = (get-date)
import-counter –path .\data.txt -starttime $startdate –endtime $enddate

Here, you examine the performance data in a file in the current directory, called Data.txt. You review the performance details from yesterday at the current time to today at the current time.

If you need help determining how an object can be used and what its counters are, type get-counter -listset followed by the object name for which you want to view counters. The following example and sample output show how this can be used to get all the Memory-related counters:

get-counter -listset Memory

Counter: {\Memory\Page Faults/sec, \Memory\Available Bytes,
\Memory\Committed Bytes, \Memory\Commit Limit...}
CounterSetName : Memory
MachineName : EngPC85
CounterSetType : SingleInstance
Description : The Memory performance object consists of counters that describe the behavior of physical and virtual memory on the computer. Physical memory is the amount of random access memory on the computer. Virtual memory consists of the space in physical memory and on disk. Many of the memory counters monitor paging, which is the movement of pages of code and data between disk and physical memory. Excessive paging, a symptom of a memory shortage, can cause delays which interfere with all system processes.
Paths: {\Memory\Page Faults/sec, \Memory\Available Bytes,
\Memory\Committed Bytes, \Memory\Commit Limit...}
PathsWithInstances : {}

As with all results returned in PowerShell, the output is returned as an object that you can manipulate. To get a complete list of counter paths, you can reference the Paths property as shown in the following example and partial output:

$c = get-counter -listset Memory
$c.paths

\Memory\Page Faults/sec
\Memory\Available Bytes
\Memory\Committed Bytes
\Memory\Commit Limit

Alternatively, you can get the same result using the following code:

get-counter -listset memory | foreach ($a) {$_.paths}

If an object has multiple instances, you can list the installed counters with instances by using the PathsWithInstances property. An example and partial output follow:

$d = get-counter -listset PhysicalDisk
$d.pathswithinstances

\PhysicalDisk(0 E: C:)\Current Disk Queue Length
\PhysicalDisk(1 W:)\Current Disk Queue Length
\PhysicalDisk(2 D:)\Current Disk Queue Length
\PhysicalDisk(3 I:)\Current Disk Queue Length
\PhysicalDisk(4 J:)\Current Disk Queue Length
\PhysicalDisk(5 K:)\Current Disk Queue Length
\PhysicalDisk(6 L:)\Current Disk Queue Length
\PhysicalDisk(7 N:)\Current Disk Queue Length
\PhysicalDisk(8 O:)\Current Disk Queue Length
\PhysicalDisk(9 P:)\Current Disk Queue Length
\PhysicalDisk(10 Q:)\Current Disk Queue Length
\PhysicalDisk(_Total)\Current Disk Queue Length

Alternatively, you can get the same result using the following code:

get-counter -listset PhysicalDisk | foreach ($a)
{$_.pathswithinstances}

Either way, the output is a long list of available counters arranged according to their object instances. You can write the output to a text file, such as in the following example:

get-counter -listset PhysicalDisk > disk-counters.txt

Then edit the text file so that only the counters you want to track are included. You can then use the file to determine which performance counters are tracked, as shown in the following example:

get-counter (get-content .\disk-counters.txt) | export-counter
–path c:\perflogs\disk-check.blg

Here, Get-Counter reads the list of counters to track from Disk-Counters.txt, and then it writes the performance data in binary format to the Disk-Check.blg file in the C:\Perflogs directory.

By default, Get-Counter samples data once every second until you tell it to stop by pressing Ctrl+C. This might be okay when you are working at the PowerShell prompt and actively monitoring the output. However, it doesn’t work so well when you have other things to do and can’t actively monitor the output—which is probably most of the time. Therefore, you’ll usually want to control the sampling interval and duration.

To control the sampling interval and set how long to sample, you can use the –SampleInterval and –MaxSamples parameters, respectively. For example, if you want Get-Counter to sample every 120 seconds and stop logging after 100 samples, you can enter the following command:

get-counter (get-content .\disk-counters.txt) –sampleinterval
120 –maxsamples 100 | export-counter –path
c:\perflogs\disk-check.blg

Monitoring System Resource Usage and Processes

Get-Process and Get-Counter provide everything you need for detecting and resolving most performance issues. However, you’ll often need to dig deep to determine whether a problem exists and, if so, what is causing the problem.

When you are working with processes, you’ll often want to get a snapshot of system resource usage, which will show you exactly how memory is being used. One way to get a snapshot is to use the Get-Counter command to display current values for key counters of the memory object. As discussed previously, the Memory object is one of many performance objects available, and you can list its related performance counters by typing the following command at the PowerShell prompt:

get-counter -listset memory | foreach ($a) {$_.paths}

The Memory object has many counters you can work with. Most counters of the Memory object display the last observed value or the current percentage value rather than an average.

Sample 12-1 provides an example of how you can use Get-Counter to get a snapshot of memory usage. In this example, you use a counter file called Perf.txt to specify the counters you want to track. You collect five samples with an interval of 30 seconds between samples and save the output in a file called SaveMemData.txt. If you import the data into a spreadsheet or convert it to a table in a Word document, you can make better sense of the output and gain a better understanding of how the computer is using the page file and paging to disk.

I chose to track these counters because they give you a good overall snapshot of memory usage. If you save the command line as a script, you can run the script as a scheduled job to get a snapshot of memory usage at various times of the day.

SAMPLE 12-1 Getting a Snapshot of Memory Usage

Commands

get-counter (get-content .\perf.txt) -maxsamples 5
-sampleinterval 30 > SaveMemData.txt

Source for Perf.txt

\memory\% Committed Bytes In Use
\memory\Available MBytes
\memory\Cache Bytes
\memory\Cache Bytes Peak
\memory\Committed Bytes
\memory\Commit Limit
\memory\Page Faults/sec
\memory\Pool Nonpaged Bytes
\memory\Pool Paged Bytes

Sample output

Timestamp CounterSamples
--------- --------------
2/28/2015 5:04:37 PM \\techpc22\memory\% committed bytes in use:
22.9519764760423
\\techpc22\memory\available mbytes :
1734
\\techpc22\memory\cache bytes :
390168576
\\techpc22\memory\cache bytes peak :
390688768
\\techpc22\memory\committed bytes :
1650675712
\\techpc22\memory\commit limit :
7191867392
\\techpc22\memory\page faults/sec :
3932.45999649944
\\techpc22\memory\pool nonpaged bytes :
70017024
\\techpc22\memory\pool paged bytes :
154710016

2/28/2015 5:05:07 PM \\techpc22\memory\% committed bytes in use:
23.2283134955779
\\techpc22\memory\available mbytes :
1714
\\techpc22\memory\cache bytes :
389664768
\\techpc22\memory\cache bytes peak :
390701056
\\techpc22\memory\committed bytes :
1670549504
\\techpc22\memory\commit limit :
7191867392
\\techpc22\memory\page faults/sec :
617.601067565369
\\techpc22\memory\pool nonpaged bytes :
70008832
\\techpc22\memory\pool paged bytes :
154791936

If you suspect there is a problem with memory usage, you can obtain detailed information about running processes by using Get-Process. At a PowerShell prompt, you can view important statistics for all processes by typing the following command:

get-process | format-table –property ProcessName,
BasePriority, HandleCount, Id, NonpagedSystemMemorySize,
PagedSystemMemorySize, PeakPagedMemorySize,
PeakVirtualMemorySize, PeakWorkingSet, SessionId, Threads,
TotalProcessorTime, VirtualMemorySize, WorkingSet, CPU, Path

The order of the properties in the comma-separated list determines the display order. If you want to change the display order, simply move a property to a different position in the list. If desired, you can redirect the output to a file as shown in the following example:

get-process | format-table –property ProcessName,
BasePriority, HandleCount, Id, NonpagedSystemMemorySize,
PagedSystemMemorySize, PeakPagedMemorySize,
PeakVirtualMemorySize, PeakWorkingSet, SessionId, Threads,
TotalProcessorTime, VirtualMemorySize, WorkingSet, CPU, Path >
savedata.txt

Whether you write output to the prompt or to a file, modify the properties of the PowerShell prompt and set the width to at least 180 characters. This ensures you can read the output.

Monitoring Memory Paging and Paging to Disk

Often, you’ll want to get detailed information on hard and soft page faults that are occurring. A page fault occurs when a process requests a page in memory and the system can’t find it at the requested location. If the requested page is elsewhere in memory, the fault is called a soft page fault. If the requested page must be retrieved from disk, the fault is called a hard page fault.

To see page faults that are occurring in real time, type the following at the command line:

get-counter "\memory\Page Faults/sec" –sampleinterval 5

Timestamp CounterSamples
--------- --------------
6/18/2015 6:00:01 PM \\techpc22\memory\page faults/sec :
172.023153991804
6/18/2015 6:00:06 PM \\techpc22\memory\page faults/sec :
708.944308818821
6/18/2015 6:00:11 PM \\techpc22\memory\page faults/sec :
14.5375722784541

Here, you check memory page faults every 5 seconds. To stop Get-Counter, press Ctrl+C. Page faults are shown according to the number of hard and soft faults occurring per second. Other counters of the Memory object that you can use for tracking page faults include the following:

· Cache Faults/sec

· Demand Zero Faults/sec

· Page Reads/sec

· Page Writes/sec

· Write Copies/sec

· Transition Faults/sec

· Transition Pages RePurposed/sec

Pay particular attention to the Page Reads/sec and Page Writes/sec, which provide information on hard faults. Although developers will be interested in the source of page faults, administrators are more interested in how many page faults are occurring. Most processors can handle large numbers of soft faults. A soft fault simply means the system had to look elsewhere in memory for the requested memory page. With a hard fault, on the other hand, the requested memory page must be retrieved from disk, which can cause significant delays. If you are seeing a lot of hard faults, you might need to increase the amount of memory or reduce the amount of memory being cached by the system and applications.

In addition to counters of the Memory object discussed previously, you can use other objects and counters to check for disk paging issues. If a particular object has multiple instances, such as when a computer has multiple physical disks or multiple paging files, you can use an object instance to track a specific occurrence of that object. You can also elect to track all instances of an object, such as whether you want to monitor all physical disks on a system. Specify _Total to work with all counter instances, or specify individual counter instances to monitor.

Sample 12-2 provides an example of how you can use Get-Counter to get a snapshot of disk paging. In this example, you use a counter file called PagePerf.txt to specify the counters you want to track. You collect five samples with an interval of 30 seconds between samples and save the output in a file called SavePageData.txt. If you import the data into a spreadsheet or convert it to a table in a Word document, you can make better sense of the output and gain a better understanding of how the computer is using the page file and paging to disk.

SAMPLE 12-2 Checking Disk Paging

Commands

get-counter (get-content .\pageperf.txt) -maxsamples 5
-sampleinterval 30 > SavePageData.txt

Source for PagePerf.txt

\memory\Pages/Sec
\Paging File(_Total)\% Usage
\Paging File(_Total)\% Usage Peak
\PhysicalDisk(_Total)\% Disk Time
\PhysicalDisk(_Total)\Avg. Disk Queue Length

Monitoring Memory Usage and the Working Memory Set for Individual Processes

You can use Get-Process to get basic memory usage for a process. The syntax you can use is

get-process –id ProcessID

where ProcessID is the ID number of the process you want to work with. The output from Get-Process shows you how much memory the process is currently using. For example, if you were tracking process ID 1072, your output might look like the following:

Handles NPM(K) PM(K) WS(K) VM(M) CPU(s) Id ProcessName
------- ------ ----- ----- ----- ------ -- -----------
493 13 15520 13452 77 1072 svchost

In this example, the process is using 13,452 KB of memory. By watching the memory usage over time, you can determine whether the memory usage is increasing. If memory usage is increasing compared to a typical baseline, the process might have a memory-related problem.

Sample 12-3 provides the source for a PowerShell script that checks the memory usage of a process over a timed interval. The script expects the process ID you want to work with to be passed as the first parameter. If you do not supply a process ID, error text is written to the output.

SAMPLE 12-3 Viewing Memory Usage at the PowerShell Prompt

MemUsage.ps1

$p = read-host "Enter process id to track"
$n = read-host "Enter number of minutes to track"
for ($c=1; $c -le $n; $c++) {get-process –id $p; start-sleep -seconds 60}

Sample output

Enter process id to track: 1072
Enter number of minutes to track: 1

Handles NPM(K) PM(K) WS(K) VM(M) CPU(s) Id ProcessName
------- ------ ----- ----- ----- ------ -- -----------
497 13 15548 13464 77 1072 svchost
494 13 15520 13452 77 1072 svchost
495 13 15520 13452 77 1072 svchost
493 13 15520 13452 77 1072 svchost
495 13 15548 13464 77 1072 svchost
495 13 15520 13452 77 1072 svchost

In Sample 12-3, the process’s memory usage shows small variances over time, but there isn’t a trend of increasing memory usage over time. Because of this, it is unlikely the process has a memory leak, but to be sure you’d need to sample over a longer period.

You can use Get-Process to track detailed memory usage for individual processes as well. The syntax you can use is

get-process ProcessName | format-table –property
NonpagedSystemMemorySize, PagedSystemMemorySize,
VirtualMemorySize, PeakVirtualMemorySize, MinWorkingSet,
WorkingSet, PeakWorkingSet

where ProcessName is the name of the process without the .exe or .dll. In a PowerShell script, such as the one shown as Sample 12-4, you can combine Get-Process and Start-Sleep to view the memory usage for a process at timed intervals.

SAMPLE 12-4 Viewing Detailed Memory Usage

DetMemUsage.ps1

$p = read-host "Enter process name to track"
$n = read-host "Enter number of minutes to track"

for ($c=1; $c -le $n; $c++) { get-process $p | format-table
–property NonpagedSystemMemorySize, PagedSystemMemorySize,
VirtualMemorySize, PeakVirtualMemorySize, MinWorkingSet,
WorkingSet, PeakWorkingSet
start-sleep -seconds 60}

Sample output

Nonpaged Paged Virtual Peak Min Working Peak
System System MemorySize Virtual WorkingSet Set WorkingSet
MemorySize MemorySize MemorySize
--------- ---------- -------- --------- --------- --------- ---------
4776 96368 52891648 161480704 6946816 7282688
8424 137056 61505536 161480704 8986624 9039872
13768 121136 137351168 161480704 38670336 73658368
13792 128904 82386944 161480704 13889536 73658368
14320 167912 187904000 258859008 74432512 138919936
44312 235704 221302784 429953024 65249280 278482944
25288 156520 91754496 429953024 15536128 278482944
30112 159376 123875328 429953024 23248896 278482944
25296 118424 86568960 429953024 20758528 278482944
2248 48088 24174592 429953024 2924544 278482944
7112 105160 55832576 429953024 6393856 278482944
5368 110960 63991808 429953024 7655424 278482944
1472 30400 15618048 429953024 2330624 278482944

The Get-Counter properties examined in Sample 12-4 provide the following information:

· NonPagedSystemMemorySize Shows the amount of allocated memory that can’t be written to disk

· PagedSystemMemorySize Shows the amount of allocated memory that is allowed to be paged to the hard disk

· VirtualMemorySize Shows the amount of virtual memory allocated to and reserved for a process

· PeakVirtualMemorySize Shows the peak amount of paged memory used by the process

· WorkingSet Shows the amount of memory allocated to the process by the operating system

· PeakWorkingSet Shows the peak amount of memory used by the process

When you focus on these properties, you are zeroing in on the memory usage of a specific process. The key aspect to monitor is the working memory set. The working set of memory shows how much memory is allocated to the process by the operating system. If the working set increases over time and doesn’t eventually go back to baseline usage, the process might have a memory leak. With a memory leak, the process isn’t properly releasing memory that it’s using, which can lead to reduced performance of the entire system.

In Sample 12-4, the process’s memory usage changes substantially over the sampled interval. Although it is most likely the process is simply actively being used by users or the computer itself, the process should eventually return to a baseline memory usage. If this doesn’t happen, the process might have a memory-related problem.

Because memory is usually the primary performance bottleneck on both workstations and servers, I’ve discussed many techniques previously in this chapter that you can use to help identify problems with memory. Memory is the resource you should examine first to try to determine why a system isn’t performing as expected. However, memory isn’t the only bottleneck. Processors, hard disks, and networking components can also cause bottlenecks.