Pro Exchange 2013 SP1 PowerShell Administration: For Exchange On-Premises and Office 365 (2014)
Chapter 7. Backup, Restore, and Disaster Recovery in Exchange Server 2013
One of the things a typical Exchange Server administrator doesn’t want to talk about is restoring information or disaster recovery, because those can be so very difficult to do in Exchange 2013. The first matter, backing up data, is not that difficult; you just install a backup application and have it run on a regular basis. So far, no need to worry.
But what happens if a Mailbox database crashes and you’ve got users complaining they can’t work? What do you do then? What’s a good time to start looking at tools like ESEUTIL? Or, when do you decide to restore a mailbox database from your backup?
Even worse, what happens when an entire Exchange 2013 server crashes and is completely lost? Do you rebuild the server? Do you restore it from backup? Or, maybe you rely on snapshot technology, which you’ve been told is a good thing?
In this chapter we’re going to explore backup technologies in Exchange 2013. In particular, we’ll cover:
· VSS backups, or Volume Shadowcopy service, sometimes also referred to as “snapshot backup.” We’ll explore the default Windows Server Backup (WSB) and a low-level tool called DiskShadow, which shows you what’s happening when you’re creating a VSS backup.
· Restoring techniques, to both a standard location and an alternative location.
· Recovery techniques, using ESEUTIL.
· Disaster recovery techniques, to rebuild an entire server.
· Exchange Native Data Protection, sometimes also referred to as “backupless environment.”
When your Exchange 2013 server crashes dramatically, it’s time to recover your Exchange Server 2013, and we’re going to explore your options there as well. In short, we’re going to rebuild and recover the entire Exchange 2013 server, but we’ll also have a look at rebuilding and recovering a mailbox database.
The last topic that’s explored in this chapter is a new technology introduced in Exchange 2010 (so it’s not so new), called Exchange Native Data Protection. This is also sometimes referred to as a “backupless” Exchange environment. This backupless thing scares a lot of IT administrators, but in fact it is possible to recover from all major outages without having a backup. The only thing you have to do is evaluate the requirements for implementing this Exchange Native Data Protection solution to determine if it fits your needs.
Let’s get started.
Backing up Exchange 2013
Backing up Exchange 2013 is a process of storing your Exchange server data, like the mailbox databases, on another medium. This medium, in turn, can be stored on another location as a safeguard if you face the loss of an entire location.
But not only mailbox databases need to be backed up; you can also back up information like your transport log files or protocol log files, SSL certificates, or maybe even the entire Exchange 2013 Server.
Before we dive into backing up Exchange 2013, especially the mailbox databases, though, we have to take a closer look at the mailbox databases themselves and the database internals. Only this way can you fully understand what’s happening when a backup of a mailbox database is created; this knowledge is necessary to successfully recover a mailbox database after a failure.
A Refresher on Mailbox Database Technologies
In Chapter 4, I explained the basic principles of the Exchange Server mailbox database and its database engine. The database engine in Exchange 2013 is based on ESE, or extensible storage engine. This is a database engine that’s highly optimized for use with Exchange Server.
A mailbox database consists of several parts:
· The mailbox database, where the actual data is stored in transactions.
· The transaction log files, where the transactions are stored as soon as processing of a transaction is finished in memory.
· The checkpoint file, which keeps track of which transactions are stored in the transaction log files but have not yet been flushed to the mailbox database.
· Reserved transaction log files, which are stored as reserved transaction log files in case a disk where these transaction log files are stored becomes full.
Transactions are processed in memory, and as soon as the processing of a transaction has finished, the transaction is stored immediately in a transaction log file. At this time the transaction is not committed to the mailbox database file but is kept in memory; since the transaction is stored safely in a transaction log file, data that suffers a server crash is fully recoverable.
Later on these transactions are flushed into the mailbox database. To keep track of which transactions are flushed into the mailbox database, ESE uses a checkpoint file. This checkpoint file points to the location in the transaction log files of the latest transaction that has been flushed into the mailbox database. Newer transactions than are located in this checkpoint file are not in the mailbox databases, while older transactions are.
It’s important to remember is that all transactions are always stored in the transaction log files before they are flushed into the mailbox database. More detailed information regarding the mailbox database and its database engine can be found in Chapter 4. You need to know the workings of the mailbox database and its database technologies so as to understand the backup and restore technologies used in Exchange 2013, but also as might be useful for recovery purposes.
The Backup Technologies
The Exchange 2013 mailbox databases are backed up using snapshots. Microsoft utilizes a framework for this called the Volume Shadowcopy service (VSS). Let’s look at what VSS is and how it works with Exchange 2013.
VSS Backup
Windows Server 2003 was the first Microsoft operating system capable of creating snapshot backups using the Volume Shadowcopy service (VSS) framework. Unfortunately, in those days it was not possible to create VSS backups of Exchange Server itself. Exchange Server 2007 running on Windows Server 2008 was the first version of Exchange Server capable of using the VSS technology, and VSS backups have been in use in Exchange Server ever since.
There are two kinds of snapshot backups:
· Clone This is a full copy or split mirror. In this scenario, a complete copy, or mirror image, is maintained until an application or administrator effectively “breaks” the mirror. From this point on, the original and the clone are fully independent of each other. The mirror copy is effectively frozen in time.
· Copy on Write This is a differential copy. A shadow copy is created that is different from a full copy of the original data, made before the original data has been overwritten. Effectively, the backup copy consists of the data in the shadow copy combined with the data in the original location. Both copies need to be available to reconstruct the original data.
The VSS consists of the VSS service itself, as well as requestors, writers, and providers. The central part is the VSS running on the computer. It is responsible for coordinating all activities concerning backups and restores.
The requestor typically is the backup application. This can be the default out-of-the-box Windows Server Backup, or it can be any third-party backup solution or a Windows tool like DiskShadow.
The writer is the application-specific part of VSS. Writers exist for Microsoft Exchange, Active Directory, IIS, NTFS, SQL Server, and so on. The Exchange writer is responsible for coordinating all Exchange-related activities, such as flushing data to the mailbox database, freezing the mailbox database during the VSS snapshot, and so on.
In Exchange Server 2010, the writer had two parts: one writer inside the Microsoft Information Store (store.exe) and one inside the Microsoft Exchange Replication service (MSExchangeRepl.exe). In Exchange 2013, the functions of the Information Store writer have been moved to the Microsoft Exchange Replication service, thereby forming one new writer, called Exchange writer. This new writer is used by Exchange-aware backup applications, which include the out-of-the-box Windows Server Backup, to create snapshot backups of both the active copy and the passive copies of a mailbox database.
The provider works with storage. It can be the Windows provider, which can create copy-on-write snapshots of a disk, or it can be a vendor-specific hardware provider.
Thus, the VSS is at the core, as can be seen in Figure 7-1. The arrows show the communication paths within the VSS function.
Figure 7-1. VSS with its requestor, writers, and providers
For a VSS backup to be created, the following steps occur sequentially:
1. The backup application or requestor sends a command to the VSS to make a shadow copy of the mailbox database.
2. The VSS sends a command to the Exchange writer to prepare for a snapshot backup.
3. The VSS sends a command to the appropriate provider to create a shadow copy of the mailbox database. (The storage provider can be a hardware storage provider provided by the hardware vendor or the default Windows storage provider.)
4. The Exchange writer temporarily stops or pauses the mailbox database and puts it into read-only mode; all data in the server memory is then flushed to the mailbox database. Also, a log file rollover is performed to make sure that all data will be in the backup set. This holds for a couple of seconds while the snapshot is taken. All write IOs are queued at this point.
5. The shadow copy is created.
6. The VSS releases the Exchange server to resume ordinary operations, and all queued write IOs are completed.
7. The VSS queries the Exchange writer to confirm that the write IOs were successfully held while the snapshot was taken. If the write operations were not successfully held, there could potentially be an inconsistent shadow copy. If this is the case, the shadow copy is deleted and the requestor is notified of the failed snapshot.
8. If the snapshot was successful, the requestor verifies the integrity of the backup set (the clone). If the clone integrity is good, the requestor informs Exchange Server that the snapshot was successful. The snapshot can now be transferred to a backup device.
9. When all data is successfully moved to the backup device, the requestor informs VSS that the backup was successful and that the log files can be purged.
In contrast to the streaming backup, where consistency is checked by ESE, the Exchange writer itself does not perform this consistency check.
Steps 1 through 6 usually take between 10 and 15 seconds. Note that this is the time it takes to take the actual snapshot; it does not include the time needed to write all the data to the backup device which is step 9 in the above list. Depending on the size of the mailbox database, this step can take up to several hours to complete.
If you’re already working with Exchange Server, you might be familiar with a VSS administrative tool called VSSADMIN. You can use VSSADMIN to quickly check the various components in the VSS infrastructure. For example, to list all the VSS writers on a server, simply use theVSSADMIN List Writers command in a command prompt window. For Exchange Server 2013, the output will be something like that shown in Figure 7-2.
Figure 7-2. A list of VSS writers on an Exchange 2013 server
Similarly, you can list the VSS providers, existing shadow copies, or volumes eligible for creating shadow copies. VSSADMIN, however, was replaced in Windows 2008 by a new administrative tool called DiskShadow, which is more powerful than VSSADMIN and as such will return more detailed information about the VSS infrastructure. It is even possible to create backups using DiskShadow (discussed later in this chapter).
Backing up a Mailbox Database
For backing up your mailbox databases, there are various applications available. Which one you use is a matter of personal experience (or maybe your company has a license for some applications), but the most important factor in making your choice is that the application be “Exchange aware.” Windows Server 2008 R2 and later versions were shipped with Windows Server Backup (WSB), which is also capable of backing up Exchange 2013 mailbox databases. It’s a pretty simple and limited backup application, but it is used quite often. DiskShadow is a low-level tool you can use for backing up your Exchange 2010 mailbox databases, but I don’t recommend it for daily use because it is quite complex. Let’s discuss both applications.
Using WSB in PowerShell
Exchange 2013 contains a VSS plug-in that can be used with WSB. Although WSB has limited functionality, it can create backups and restore them if needed. Another advantage of WSB is that it’s free and comes with Windows Server 2008 R2 and Windows Server 2012.
To install Windows Server Backup, you open PowerShell and enter the following command:
Add-WindowsFeature -Name Windows-Server-Backup
Note PowerShell 3.0, included in Windows 2012, has a new feature called module auto-loading. When you’re adding features that are dependent on other modules, these modules are automatically loaded. If you want to add the Windows Server Backup feature in Windows 2008 R2, you have to use the Import-Module ServerManager command before using the Add-WindowsFeature command.
When WSB is installed, you then load the WSB PowerShell snap-in each time you run Windows PowerShell. You can do this by entering the following command in PowerShell:
Import-Module WindowsServerBackup
Note To get an overview of all available WSB cmdlets, you can use the following URL: http://bit.ly/WSBCommands. You can also use the Get-Command –Module WindowsServerBackup command on the Exchange Server.
WSB uses a backup policy that needs to be created at the outset. You can use the New-WBPolicy command to create a new policy; to store it in a variable, you then use the following command: $WBPolicy = New-WBPolicy. This will create an empty policy that can be shown on the console by calling the variable, as displayed in Figure 7-3.
Figure 7-3. Newly created, and empty, WSB policy
This policy now needs to be populated with the following configuration options:
· Schedule: when the backup will be running.
· VSS options: whether a full backup or a copy backup will be created.
· Volumes to backups: the volumes containing the mailbox databases.
· Target disk: a disk where the backup will be stored.
To set the schedule and add it to the backup policy, you can use the following commands:
$Schedule = Set-Date "10/29/2014 01:30:00"
Set-WBSchedule -Policy $WBPolicy -Schedule $Schedule
This will set the next backup to be run at October 15, 2014, at 01:30 a.m., and it will run every 24 hours.
The VSS options are set to establish what kinds of backups will be created. For example, a full backup will create a complete backup of the mailbox database (or databases) on the stated volume and purge the accompanying transaction log files when the backup has been completed successfully. You can set the VSS full backup option by using the –VssFullBackup switch parameter. The other option is to create a VSS copy backup, which is a full backup, but the transaction log files aren’t purged when the backup is created successfully. To set this latter option, you can use the following command:
Set-WBVssBackupOptions -Policy $WBPolicy –VssFullBackup
In our example, there are three disks attached to the Mailbox server, and these disks are mounted in a directory called C:\ExchVols. You can use the following commands to create an array that contains all the volumes in the C:\ExchVols directory, and add it to the backup policy, as follows:
$Volumes = Get-WBVolume -AllVolumes | Where-Object {$_.MountPath -like "C:\ExchVols\*"}
Add-WBVolume -Policy $WBPolicy -Volume $Volumes
A target disk will then be used to store the backup set. In our example, there’s dedicated disk attached to the Mailbox server, and this is disk number 5. (Note that the first disk in your Mailbox server always start with disk 0.) The following three commands will retrieve all disks from the Mailbox server, create a target disk, and add this to the backup policy:
$Disks = Get-WBDisk
$Target = New-WBBackupTarget -Disk $Disks[5] -Label "Backup Disks"
Add-WBBackupTarget -Policy $WBPolicy -Target $Target
When everything in the backup policy is configured, you can use the following command to activate the backup policy:
Set-WBPolicy -Policy $WBPolicy –Force
Note The –Force option is used to bypass a confirmation when running the Set-WBPolicy command.
The backup will start automatically during the next cycle that’s been configured with the schedule option, but to start the backup before then, you can use the following command:
Get-WBPolicy | Start-WBBackup
When this command is executed, a full VSS backup of the volumes containing the mailbox databases will be created. It will start with a VSS snapshot of these volumes, followed by a consistency check of the mailbox databases to ensure the mailbox database’s integrity.
When the consistency check is finished successfully, the data is backed up to the backup location; the backup can take a couple of hours if you have a large mailbox database.
If you check the application log in the event viewer, you’ll see the following entries:
· Event ID 2021 (MSExchangeRepl) Successfully collected metadata document in preparation for backup.
· Event ID 2110 (MSExchangeRepl) Successfully prepared for a full or a copy backup of database MDB01.
· Event ID 2023 (MSExchangeRepl) VSS writer successfully prepared for backup.
· Event ID 2005 (ESE) Shadow copy instance started.
· Event ID 2025 (MSExchangeRepl) VSS successfully prepared for a snapshot.
· Event ID 2001 (ESE) MDB01 shadow copy freeze started.
· Event ID 2027 (MSExchangeRepl) VSS writer instance has successfully frozen the databases.
· Event ID 2003 (ESE) MDB01 shadow copy freeze ended.
· Event ID 2029 (MSExchangeRepl) VSS writer instance has successfully thawed the databases.
· Event ID 2035 (MSExchangeRepl) VSS writer has successfully processed the post-snapshot event.
· Event ID 2021 (MSExchangeRepl) VSS writer has successfully collected the metadata document in preparation for backup.
· Event ID 224 (ESE) MDB01 deleting log files C:\ExchVols\MDB01\Log Files\E0000000001.log to C:\ExchVols\MDB01\Log Files\E000000002B.log.
· Event ID 225 (ESE) MDB01—no log files can be truncated; will be logged instead of Event ID 224 when circular logging is used.
· Event ID 2046 (MSExchangeRepl) VSS writer has successfully completed the backup of database MDB01.
· Event ID 2006 (ESE) MDB01 shadow copy completed successfully.
· Event ID 2033 (MSExchangeRepl) VSS writer has successfully processed the backup completion event.
· Event ID 2037 (MSExchangeRepl) VSS writer backup has been successfully shut down.
When you check the location of the log files using Windows Explorer, you’ll notice that most of the log files have indeed been purged. The information in the mailbox database itself has also been updated with backup information. You can use the following command to retrieve backup information from the mailbox database in EMS, and see information similar to that shown in Figure 7-4:
Get-MailboxDatabase -Identity MDB01 -Status | Select Name,*backup*
Figure 7-4. Checking the backup status using the EMS
You can also use the ESEUTIL /MH MDB01.edb command to check the header for backup information; if so, you’ll see something like this:
Previous Full Backup:
Log Gen: 4636-4637 (0x121c-0x121d) - OSSnapshot
Mark: (0x121E,1,0)
Mark: 10/14/2014 17:10:54.015
Current Full Backup:
Log Gen: 0-0 (0x0-0x0)
Mark: (0x0,0,0)
Mark: 00/00/1900 00:00:00.000
Note It is only possible to check the header information of a mailbox database when the database is dismounted. If the database is mounted, an error saying “The file is in use” is shown.
You might ask yourself why this information is written under Previous Full Backup, while the entry under Current Full Backup is empty. This is because when you create a backup, the entry Current Full Backup is written with information regarding the backup as it runs. When the backup is finished, the entries are moved from the Current Full Backup section to the Previous Full Backup section, and the Current Full Backup is emptied. When a backup is restored to disk, and you check the header information after a restore, you can always identify the database as a backup that’s been restored instead of a “normal” mailbox database because it will show as Previous Full Backup.
Using the Windows Server Backup GUI
I can imagine that, for backup purposes, you don’t want to use PowerShell; although it’s interesting, it can also become complex; let’s look at the Graphical User Interface (GUI) as well.
To create a backup of your Exchange 2013 databases, you use the following steps:
1. In the administrative tools section, select Windows Server Backup. This is an MMC snap-in. In the Actions pane, select Backup Once.
2. In the Backup Options page, select Different Options and click Next to continue.
3. In this example we want only to back up the mailbox database, so select the Custom option and click Next to continue.
4. The mailbox database is located on disk C:\ExchVols\MDB01; use the Add Items button to select this disk.
5. To change the type of backup that’s being created, click the Advanced Settings button, select the VSS Settings tab, and select the VSS Full Backup radio button. Click OK to return to the previous page and click Next to continue.
6. WSB has the option of backing up to a remote share or to a local disk, whichever you prefer. On our server there’s an additional backup disk (disk X), so select Local Drives and click Next to continue.
7. In the Backup Destination drop-down box, select the disk where the backup needs to be stored; on our server this would be disk X. Click Next to continue.
8. The selection is now complete. Click Backup to start the actual backup process.
Note By default, WSB will create a copy backup instead of a full backup. This is understandable, as it will not interfere with a normal backup cycle when you’re using another backup solution and you want to test using WSB. If you want to make a regular backup using WSB, make sure you change this setting. This is a common pitfall with WSB.
The backup will now start with the creation of the VSS snapshot, and it will perform a consistency check of the mailbox data, as shown in Figure 7-5. Be aware that this information can be visible for a small amount of time, especially when the mailbox database is not that large and the consistency check takes only a few seconds. This backup status check is the only visual indication you have that an Exchange-aware backup is running.
Figure 7-5. Windows Server Backup automatically checks for database consistency
When the consistency check is finished successfully, the data is backed up to the backup location, and after a while (which can take a couple of hours if you have a large mailbox database), the backup is completed.
Using DiskShadow to Create Backups
As mentioned earlier, DiskShadow is a VSS management tool you can use to manage your VSS infrastructure at a low level. DiskShadow lets you check your writers, providers, shadow copies, and so on much as with VSSADMIN, but you’ll get more detail. You can also use DiskShadow to create VSS snapshots on a low level, which is a great way to demonstrate what exactly happens during VSS backup creation.
DiskShadow can be used in interactive mode, but it can also be used in scripting mode, where it accepts input from a text file. For example, in our earlier Contoso environment, there were two Exchange 2013 multi-role servers at the Amsterdam site, and these were configured in a DAG. Each server has one active mailbox database and one passive mailbox database. The mailbox databases are located on drive F: (DB01) and driver G: (DB02).
When employing DiskShadow to create a VSS backup in this scenario, the following input file can be used:
SET verbose on
SET context persistent
# Exclude other writers on Exchange Server
# Can be retrieved using VSSADMIN List Writers
Writer Exclude {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
Writer Exclude {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
Writer Exclude {0bada1de-01a9-4625-8278-69e735f39dd2}
Writer Exclude {e8132975-6f93-4464-a53e-1050253ae220}
Writer Exclude {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Exclude {1072ae1c-e5a7-4ea1-9e4a-6f7964656570}
Writer Exclude {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Exclude {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Exclude {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Exclude {4969d978-be47-48b0-b100-f328f07ac1e0}
Writer Exclude {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Exclude {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
Writer Exclude {7e47b561-971a-46e6-96b9-696eeaa53b2a}
Writer Exclude {41e12264-35d8-479b-8e5c-9b23d1dad37e}
Writer Exclude {59b1f0cf-90ef-465f-9609-6ca8b2938366}
# Exchange writer is required
Writer Verify {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
# Take the actual snapshot
begin backup
add volume F: alias VSS_Backup_F
add volume G: alias VSS_Backup_G
create
# Expose the snapshot as additional drive S:
expose %VSS_Backup_F% S:
expose %VSS_Backup_G% T:
End backup
The first two entries are to turn on verbose logging and set the context to persistent so the information won’t be lost. Then, all VSS writers on the Exchange servers are disabled except the VSS Exchange writer. The GUIDs of the VSS writers can be retrieved by using the VSSADMIN List Writers command.
Volumes F: and G: are used for creating the VSS snapshots, and an alias VSS_Backup_F and VSS_Backup_G is assigned. (You could use any readable name you like.)
At this point, the actual snapshot is created using the CREATE command. In Windows, this is a copy-on-write snapshot whereby the snapshot information is written on the same disk and on the mailbox database (i.e., drive F: and drive G:), but it’s not visible for a regular user or administrator. (All the steps as explained in the previous section are performed here, and all the events are recorded in the event log as well.) This step usually takes between 15 and 30 seconds, depending on the hardware that’s used.
When the snapshot is created, it is exposed using the EXPOSE command. This way it is visible as a regular disk to the operating system, and thus accessible, as shown in Figure 7-6.
Figure 7-6. The VSS snapshots are published as additional drives in Windows
When the backup in DiskShadow is ended using the End Backup command, a request is sent to ESE to purge any log files that are no longer needed for recovery purposes. If the VSS snapshot was created on a single Exchange server, or a DAG member with active copies of the mailbox database, then the request is processed locally. If the snapshot was created using a passive copy of a mailbox database, the request is automatically sent to the Mailbox server that is hosting the active copy of the mailbox database. The log files are purged on that particular Mailbox server, and the truncation itself is replicated to the Mailbox servers hosting the passive copies of the mailbox database. This means that you no longer need to know which mailbox database in a DAG you back up; the appropriate log files are automatically purged.
This is the pure VSS snapshot function: a snapshot of the mailbox database is created, and its log files are purged (when successful, of course). But remember that the mailbox database has not been checked for consistency, nor has it been backed up to a safe location. To check the consistency of a mailbox database, you use the ESEUTIL tool again, but now with the /k switch:
ESEUTIL /K db01.edb
This command will read all the pages in the database and check the consistency of all those pages, as shown in Figure 7-7.
Figure 7-7. Checking the database consistency using the ESEUTIL tool
The final step in a backup, then, is to copy the mailbox database snapshots from the published locations (drive S: and drive T:) to another safe location using, for example, Explorer. Once this is done, you have been successful in manually creating a full backup.
Note Don’t forget to remove the exposed VSS snapshots using the UNEXPOSE command in DiskShadow, and remove the actual VSS shadow copies using the DELETE SHADOWS command.
Backup of Other Configuration Information
The previous section discussed how to back up mailbox databases in Exchange 2013, since this is the most important aspect of an Exchange server’s role. There are a couple more things that need to be backed up, however, either in a regular backup sequence or maybe after there have been configuration changes. For example:
· Log files (not transaction log files), as Exchange 2013 logs quite a lot of information in log files—for instance, in IIS log files located C:\inetpub\logs\logfiles, which contain logging from all HTTPS-based clients like Outlook WebApp, Exchange Web Services, Autodiscover, Outlook Anywhere, and ActiveSync.
· SMTP Transport Protocol logs, located in C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\ Hub\ProtocolLog. These are disabled by default, but when enabled these might be included in a daily backup routine.
· Message tracking information, located in C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\ Logs\MessageTracking.
· Entire directory, depending on the (legal) backup requirements of your company; it might be necessary to back up the entire logging directory in C:\Program Files\Microsoft\Exchange Server\V15\Logging.
· CONFIG files, such as transport configuration files located in C:\Program Files\Microsoft\Exchange Server\V15\Bin. This file is used to relocate the SMTP message database and log files to another location. No need to include this in a daily backup sequence, but it is necessary to back up after configuration changes.
· SSL certificate, especially on the Exchange 2013 Client Access server. No need to back up this on a daily basis, but you should back up after the initial installation of the certificate. It is needed when rebuilding a Client Access server, or maybe when adding an additional Client Access server. Make sure that when you create a backup of your SSL certificate you also include the private key in the certificate backup. If not, it’s still useless when rebuilding your Client Access server.
· System State or entire server, depending on your disaster recovery plan. You might want to back up the server’s system state, or maybe the entire server, for rebuilding purposes. (We’ll get back to this later in this chapter.)
When using server virtualization, it is an option to back up the entire virtual machine using a backup solution. Veeam, for example, is a third-party vendor that offers backup solutions in a virtual environment. Veeam can back up the VMs, and using the Hyper-V Integration Components, the VSS backup information is sent to the operating system inside the VM. This way the virtual machine is also aware that a snapshot is being made.
Restoring Exchange Server 2013
Backing up your Exchange 2013 environment does make sense, but restoring the backup is even more important. In all my years as an Exchange consultant I have regularly met customers who thought that their backup solutions were fine, but they did not have any idea how to restore them if needed. The worst way to find out is during a disaster, when you must restore information rapidly, as there likely are hundreds of users and managers complaining. Most often, the restoration will fail at this moment if there’s lack of experience.
Note Restoring your Exchange backups is not a daily management routine. Therefore, it is not feasible to create an entire restore process in PowerShell. Doing so is a complex task and very labor-intensive. This section only covers the Windows Server Backup GUI.
Here, I focus on restoring the mailbox databases, since these are where all the data is. In the next part, I explain how to restore the Exchange servers as part of a disaster recovery operation.
There are two options for restoring mailbox databases:
· Restore the mailbox database to its original location In this scenario, the original mailbox database is taken offline and is overwritten by the mailbox database in the backup set. Since all the information is also stored in the log files, the information processed by the Mailbox server since the last backup was created will automatically be replayed by the Mailbox server. If all goes well, no information, or almost no information, will be lost and the mailbox database will be in the same state as before.
· Restore the mailbox database to another location In this scenario, the mailbox database from the backup set is restored to another location, most likely a dedicated restore disk on your Mailbox server. When the mailbox database is restored to another location, the original mailbox database can continue running and thus continue servicing client requests. A recovery mailbox database can be used as well. This is a special mailbox database, not visible for regular clients (only for the Exchange administrator), that can be used to restore a particular mailbox; you can move this mailbox into the production mailbox database or export it to a PST file.
Restoring a Mailbox Database to Its Original Location
A lot of interesting technologies are used to make the mailbox database backup process as smooth as possible without interrupting any users. Restoring a mailbox database to its original location is straightforward work.
The mailbox database has to be taken offline, and thus users will face some downtime because Exchange Server is not available anymore. The mailbox database is then restored from the backup set, additional log files are replayed automatically, and the mailbox database can then be mounted again. There’s nothing fancy to do when restoring a mailbox database.
In the previous section, WSB was used to back up the mailbox database, so you have to use WSB again to restore the mailbox database. For testing purposes, you can log on to the mailbox before the restore action and send some messages around to see if these are actually replayed after the backup is restored. To restore a previous backup using WSB, you can use these steps:
1. Log on to the Exchange 2013 Mailbox server and dismount the mailbox database.
2. Log in to the EAC as an administrator. In the Feature pane, select Servers and then select the Databases tab.
3. Select the Mailbox database you want to restore (and you dismounted in step 1), and open its properties. In the navigation menu, select Maintenance and check the “This database can be overwritten by a restore” checkbox, as shown in Figure 7-8.
Figure 7-8. Setting the option “This database can be overwritten by a restore”
Note This option can be set in PowerShell by using the command Set-MailboxDatabase –Identity MDB01 –AllowFileRestore $true.
4. Open WSB, and in the Actions pane, select Recover.
5. In the Recovery wizard that starts, select where the backup is located. This can be on a disk attached to the server itself or it can be a remote location. Select the appropriate option and click Next to continue.
6. In the Select Backup Date window, select the backup set you want to restore to this server. Once selected, click Next to continue.
7. Next is the Select Recovery Type window, which is very important. WSB is Exchange-aware and thus an Exchange backup is an application backup. This way, the backup is restored as the Exchange writer would like it to be. If you select Volumes, for example, the backup is restored as an ordinary file backup and this is useless from an Exchange perspective. Select Applications as shown in Figure 7-9 and click Next to continue.
Figure 7-9. Select the Applications radio button when restoring a mailbox database
8. In the Select Application window, make sure Exchange is selected and click Next to continue.
Note This option is only available when the mailbox database is not located on the system drive and boot drive, typically drive C:\. That is, it’s only available when the mailbox database is on a separate drive.
9. Since the backup will be restored to its original location, select the Recover to Original Location option and click Next to continue.
10.In the Confirmation window, WSB will show the backup set that was selected. If this is correct, click Recover to start the restore process.
11.The restore operation will run in the background so it is safe to close WSB at this point.
During the restore process, the mailbox database and the log files that are in the backup set are restored to the original location. After the initial restore of the individual files, the mailbox database is still in an inconsistent (i.e., dirty shutdown) state. Exchange will try to start a recovery process with the log files that are also restored from the backup set.
When finished, any additional log files that are written to disk after the previous backup was taken are replayed into the mailbox database as well, so no mail data will be lost when restoring a mailbox database from a backup.
Note Exchange Server has always been very sensitive when it comes to transaction log files and with replaying these into the mailbox database. In the “A Refresher on Mailbox Database Technologies” section earlier in this chapter, I explained the ESE internals and so it should be clear that Exchange Server is 100 percent dependent on all transaction log files. If only one log file is missing, the replay of the log files will fail. Therefore, you should never delete any of the log files manually; or do so only if you’re 150 percent sure of what you’re doing, or if Microsoft support instructs you to do so!
During the restore process, several events are written to the event log, indicating the progress of the restore operation or if any problems have arisen during the process. For example:
· Event ID 4347 (MSExchangeRepl) Exchange Replication Service VSS writer will restore a backup set to database MDB01\EXCH01, which is the same database from which the backup was originally taken.
· Event ID 4367 (MSExchangeRepl) Exchange Replication Service VSS writer successfully restored the backup set. In order to bring the restored databases to a clean shutdown state, database recovery will be performed using the information in the restore environment document MDB01\EXCH01.
· Event ID 4370 (MSExchangeRepl) Exchange Replication Service VSS writer will perform database recovery on database MDB01.edb as part of the restore process for MDB01\EXCH01, followed by a number of events from ESE, indicating the recovery steps for the restored Mailbox database.
· Event ID 40008 (MSExchangeIS) Mount completed successfully for database <<GUID>>.
· Event ID 3156 (MSExchangeRepl) Active Manager successfully mounted database MDB01 on server EXCH01.contoso.com.
· Event ID 737 (Backup) The operation to recover component(s) d7f01671-c9c7-46d6-abac-00fd27e1ebf2 has completed successfully at 2013-07-25T14:52:38.059000000Z.
If you log on to your mailbox and check any messages that were sent after the last backup was taken, you’ll see that these are still available in the mailbox and thus successfully recovered.
Restoring a Mailbox Database to Another Location
Restoring a mailbox database to its original location is only useful when the original mailbox database is lost, for whatever reason. Restoring to its original location means you have to dismount the mailbox database, resulting in an outage for users. So, in a normal production situation you don’t want to dismount your mailbox database for restoring purposes unless there’s no other option.
Restoring a backup to another location has the advantage of leaving the original mailbox mounted, allowing users to continue working. Also, when using this method, there’s no risk in accidentally overwriting the mailbox database with a database that’s restored, since the original database is still mounted and thus reports as “file in use.” The procedure for restoring the mailbox database to another location does not differ that much from restoring the mailbox to its original location.
The Restore Process
If you want to restore to another location, you follow these steps:
1. Log on to the Exchange 2013 Mailbox server, open WSB, and in the Actions pane, select Recover.
2. In the Recovery wizard that starts, select where the backup is located. This can be on a disk attached to the server itself or it can be a remote location. Select the appropriate option here and click Next to continue.
3. In the Select Backup Date window, select the backup set you want to restore to this server. Once selected, click Next to continue.
4. Next, the Select Recovery Type window is very important. WSB is Exchange-aware and thus an Exchange backup is an application backup. This way, the backup is restored as the Exchange writer would like it to be. If you select Volumes, for example, the backup will be restored as an ordinary file backup and this is useless from an Exchange perspective. Select Applications and click Next to continue.
5. In the Select Application window, make sure Exchange is selected and click Next to continue.
6. Since the backup will be restored to another location, select the Recover to Another Location option, and use the Browse button to select a disk and directory where you want to restore the mailbox database. Typically a dedicated restore LUN is used for this purpose. In our example, we use G:\RestoreDB to restore the mailbox database from backup. This is shown in Figure 7-10.
Figure 7-10. Restoring a mailbox database to an alternative location
7. The selection is shown in the confirmation window. If all is okay, then click the Restore button to start the recover process.
8. Since only the mailbox database and its log files are restored from backup, the process finishes much faster than when restoring to its original location. In this process, no additional recovery steps are performed so you have to do this manually.
The mailbox database and the accompanying log files are now restored from the backup set. The file location is also restored, so now there’s a mailbox database DB01 located in G:\RestoreDB\F_\MDB01 and the log files are stored in G:\RestoreDB\F_\MDB01\LogFiles. This file is taken from a running copy of the mailbox database and thus the mailbox database is in a dirty shutdown state. You can check this using the ESEUTIL /MH command, which would produce something like this:
State : Dirty Shutdown
Log Required: 82-84 (0x52-0x54)
To bring this database back into a consistent state, the mailbox database has to be recovered—something that can be achieved using the ESEUTIL tool with the /R option for recovery and the /l and /s options for the file locations of the log files and the system file (i.e., the checkpoint file). This is shown in Figure 7-11.
ESEUTIL /R E00 /lG:\RestoreDB\F_\MDB01\Logfiles /sG:\RestoreDB\F_\MDB01\Logfiles
Figure 7-11. Recovering the mailbox database after a restore to an alternative location
Now when you check the database again using the ESEUTIL /MH command, the database will be in a clean shutdown state and ready to use.
Recovery Database
A recovery database in Exchange 2013 is a special mailbox database, invisible to normal users, where you can mount a normal database restored from backup. This means that you will have one normal mailbox database MDB01 running in its original location and one recovery mailbox MDB01 running in recovery mode.
Creating a recovery mailbox database is not very different from creating a regular mailbox database, but it can be managed only by using EMS. When creating the recovery mailbox database, you have to use the -Recovery switch to tell Exchange that a recovery mailbox database is created.
To create a recovery mailbox database using the database we’ve restored in the previous section, you would use an EMS command like this:
New-MailboxDatabase -Name "MDB01 Recovery" -Recovery -Server EXCH01 -EdbFilePath G:\RestoreDB\F_\MDB01\MDB01.edb -LogFolderPath G:\RestoreDB\F_\MDB01\LogFiles
When the recovery mailbox database is created, it can be mounted, again only using the EMS since the recovery mailbox database is not visible in EAC.
Now that the recovery mailbox database is up and running, you can view what’s inside this database. An ordinary Get-Mailbox command is not going work, since this is targeted to the normal mailbox database, but the Get-MailboxStatistics command does work to get a recovery mailbox database. Just use the following command to retrieve the mailbox data from the recovery mailbox database. Figure 7-12 shows the output from this command.
Get-MailboxStatistics -Database "MDB01 Recovery" | Select-Object DisplayName,ItemCount | –Format-Table -AutoSize
Figure 7-12. Using the Get-MailboxStatistics command to view what's inside the recovery mailbox database
This information can be used to restore mailbox content from the recovery mailbox database into the normal production mailbox database by using the New-MailboxRestoreRequest command in the EMS.
To retrieve mailbox content from the administrator mailbox inside the recovery mailbox database into the normal administrator mailbox, you can use the following command:
New-MailboxRestoreRequest -SourceDatabase "MDB01 Recovery" -SourceStoreMailbox Administrator
-TargetMailbox administrator
The content will be imported into the normal mailbox, but content will not be overwritten. If items already exist in the target mailbox, an additional copy of the item is created so that no information gets lost.
Note If you’re uncertain what will happen during a command like the New-MailboxRestoreRequest, you can always use the -WhatIf switch parameter. The command will be the same, but it won’t be executed. The results, however, will be shown on the console. If you’re satisfied with the results, you can then rerun the command but without the -WhatIf parameter.
The example above will restore the entire mailbox from the recovery mailbox database, but it is possible to use a more granular approach. For instance, you can use the -IncludeFolders parameter to specify the folder in the mailbox that needs to be restored. To include the contents from the inbox, the option -IncludeFolders "Inbox/*" can be used; or in case of restoring only the Deleted Items folder, the option -IncludeFolders "DeletedItems/*" can be used. For example,
New-MailboxRestoreRequest -SourceDatabase "MDB01 Recovery" -SourceStoreMailbox Administrator -TargetMailbox administrator -IncludeFolders "DeletedItems/*"
Note There’s a dependency here on the regional settings of the mailbox. In English, you have an “inbox” while in Dutch the same folder is called “Postvak in”; in Spanish, it is “bandeja de entrada.” You have to be aware of the regional setting of the mailbox when performing this command.
For certain purposes, it is possible to restore mailbox content from the mailbox in the recovery mailbox database to another mailbox that’s not the original mailbox—for example, a legal mailbox. Suppose there’s a mailbox called “Legal” and this mailbox is used to gather information from a mailbox from a backup. A command similar to this can be used:
New-MailboxRestoreRequest -SourceDatabase "MDB01 Recovery" -SourceStoreMailbox Administrator -TargetMailbox legal -TargetRootFolder "Recovery Items"
Dial-tone Recovery
A recovery database is also used in a process called dial-tone recovery. In this recovery scenario, you get the users back online as quickly as possible after a mailbox database crash, and you work on mailbox database recovery in the background.
Suppose a mailbox database has crashed beyond repair, but you need to get users back online immediately. In this case you can remove all corrupted files from the disk and mount the mailbox database again. Since Exchange does not find any mailbox database files, it creates a new mailbox database and new log files (after showing a warning message).
Users who had their mailboxes in the crashed mailbox database can now start their mail client, and new mailboxes will automatically be created in the new mailbox database. Of course, that database is empty, but users are online again. They can send mail, but more important, they can receive mail again. The last thing you want to have happen is for external customers to send email to your organization and they receive error messages like “Mailbox info@contoso.com is not available.”
Since users are online, they can continue to function and you can work in the background on restoring the last mailbox database backup to a recovery mailbox database. When the mailbox database is restored and all remaining log files are replayed into the recovered mailbox database, you can swap the two mailbox databases. The recovered mailbox database is then moved to the production location while the newly created mailbox database, which now also has items in it, is moved to the recovery mailbox database location.
The trick you perform at this point is to mount the newly created mailbox database as a recovery mailbox database and move the new content, using the New-MailboxRestoreRequest command as explained in the previous section, into the restored mailbox database. Once finished, you will have the mailbox database up and running again, without having lost any data.
The good part here is that you can restore the mailbox database from a backup, but your users are able to log on again and continue sending and receiving email. Yes, at that moment they won’t have their “old” mailbox content available (not only email items but also the temporary mailbox don’t contain any information or additional permissions), but at least they are online during the restore procedures.
Note Some third-party backup applications use a granular restore technology, whereby it is possible to restore individual items directly from the VSS backup. What happens is that the VSS backup is completely indexed after the backup, indexing all individual email messages. These messages can then be restored directly from the backup, where the MAPI CDO is used for storing directly into the mailbox. While this works very well, it is not an official Microsoft-supported solution.
Disaster Recovery with Exchange Server 2013
The previous section discussed mailbox database technologies: how they work, how to back them up, and how to restore them. It showed how it is possible to restore a mailbox database to its original location or to an alternative location. In the last scenario, you created a recovery mailbox database, restored data from this recovery mailbox database, or used it in a dial-tone scenario.
But what happens if an entire server is lost and it is beyond repair? Then you need to rely on your disaster recovery skills.
Rebuilding the Exchange Server
When the Exchange 2013 Mailbox server is lost, it needs to be rebuilt, and the services, configuration, and data need to be restored. Earlier in this chapter I described where all the information is stored that’s needed to rebuild an Exchange 2013 server; for example,
· Mailbox data is stored in the mailbox database, which should be on additional disks. If you’re in luck, these are still safe after losing your Exchange server.
· Configuration data is sometimes stored in config files, located somewhere in the C:\Program Files\Microsoft\Exchange Server directory.
· All kinds of log files are also stored in the C:\Program Files\Microsoft\Exchange Server directory.
· SSL certificates are stored somewhere safe.
· Server configuration is stored in Active Directory.
Now, assuming that you’ve take care of the first four items listed above, the last one about Active Directory is interesting. All configuration data that’s not in the config files is stored in Active Directory, and that can be used when rebuilding the Exchange 2013 Mailbox server. But instead of entering all the details manually during setup, this information is retrieved from Active Directory during installation.
Rule number 1 in a crisis situation is: Don’t panic and don’t destroy any data—not from disks and not from Active Directory. Otherwise, this action will backfire on you at some point!
To successfully rebuild an Exchange 2013 server, you can use the following steps:
1. Reset the computer account. When the server has crashed beyond repair and you have to rebuild your server, do not remove the Computer object from Active Directory. Instead, reset the Computer object in Active Directory, as follows:
a. Log on to a domain controller, or any other member server that has the Active Directory tools installed, and open the Active Directory and Computers MMC snap-in.
b. Locate the Computer object, right-click it, and select Reset Account (see Figure 7-13). This will reset the Computer object so you can join a new Windows Server (using the same name) to Active Directory.
Figure 7-13. Resetting the Computer object in a disaster recovery scenario
2. Install a new Windows server with the same specifications as the “old” Mailbox server. Use the same operating system and bring it up to date with the same hot fixes and service packs as were applied to the old Exchange 2013 server. Very important: use the same server name as was for the old Exchange 2013 server. That is, when the old server name was EXCH01, the new server name needs to be EXCH01 as well.
3. Join the new server, using the original name, to the Active Directory domain. When joined, reboot the server and log on to the new server as a domain administrator.
4. When logged on as a domain administrator, install the prerequisite software. This is explained in Chapter 2, but as a friendly reminder this is the prerequisite software when using a Windows Server 2012 server:
Import-Module ServerManager
Install-WindowsFeature AS-HTTP-Activation, Desktop-Experience, NET-Framework-45-Features, RPC-over-HTTP-proxy, RSAT-Clustering, RSAT-Clustering-CmdInterface, Web-Mgmt-Console, WAS-Process-Model, Web-Asp-Net45, Web-Basic-Auth, Web-Client-Auth, Web-Digest-Auth, Web-Dir-Browsing, Web-Dyn-Compression, Web-Http-Errors, Web-Http-Logging, Web-Http-Redirect, Web-Http-Tracing, Web-ISAPI-Ext, Web-ISAPI-Filter, Web-Lgcy-Mgmt-Console, Web-Metabase, Web-Mgmt-Console, Web-Mgmt-Service, Web-Net-Ext45, Web-Request-Monitor, Web-Server, Web-Stat-Compression, Web-Static-Content, Web-Windows-Auth, Web-WMI, Windows-Identity-Foundation
The following prerequisite software need to be installed as well:
· Microsoft Unified Communications Managed API 4.0, Core Runtime 64-bit - http://go.microsoft.com/fwlink/p/?linkId=258269
· Microsoft Office 2010 Filter Pack 64 bit - http://go.microsoft.com/fwlink/p/?linkID=191548
· Microsoft Office 2010 Filter Pack SP1 64 bit - http://go.microsoft.com/fwlink/p/?LinkId=254043
Note It looks strange, but the filter pack components used here are really version 2010.
5. When the server is fully up to date, the (external) disks containing the “old” mailbox databases must be accessible to the new Exchange 2013 server. The setup application will look for these disks; this is one kind of information that is stored in Active Directory as part of the Exchange 2013 server object. If you omit this step, the setup application will halt and will generate error messages on the console.
6. When the disks are connected and configured with the previous drive letters, the Exchange 2013 server can be recovered. To do this, open a command prompt, navigate to the installation media, and enter the following command:
Setup.exe /mode:RecoverServer /IAcceptExchangeServerLicenseTerms
This will install Exchange 2013, and all the configuration information that’s normally entered in the setup application is now retrieved from Active Directory.
7. When the setup application is finished and the server is reinstalled, shown in Figure 7-14, reboot the server.
Figure 7-14. Recovering an Exchange 2013 server with information from Active Directory
After rebooting, you can check the mailbox databases that were on the (external) disk drives; they should be good, although dismounted. When you mount the mailbox databases on the recovered server, you’re good to log in to your mailbox using OWA. The last steps are to restore or reconfigure additional items like the SSL certificate, virtual directories, additional config files, or other log files, as explained earlier in this chapter (see “Backup of Other Configuration Information”).
When done, though your server has been unavailable for some time, it is now restored to its original location. If you’re unhappy with the downtime, you should refer to the “Database Availability Group” section in Chapter 5, which explains how to avoid or minimize downtime.
ESEUTIL and Corrupted Databases
Although rare these days, it can happen that you end up with a corrupted mailbox database and no backup of your mailbox database. In this case, you have to rely on tools that can repair your mailbox database. ESEUTIL is such a tool and it comes with Exchange Server; you can use ESEUTIL to repair a corrupted mailbox database.
When a mailbox database is corrupted, that means it has corrupted pages in it and most likely it will not mount. When you perform an integrity check on the mailbox database using ESEUTIL /G, it will report that the mailbox database is corrupted (see Figure 7-15).
Figure 7-15. ESEUTIL report of mailbox database as corrupted
ESEUTIL also has an option to repair a mailbox database, but it is a very destructive way of repairing. What it does is open the mailbox database and check all the pages in the database. When a page is found to contain corrupted pointers (i.e., pointers to other pages containing data), it will remove these pointers from the page. The result is that no pages contain invalid pointers anymore, but the data that was referenced using those pages is automatically lost. It is not possible to predict which pages and pointers are corrupted, and thus it’s not possible to anticipate what data you will be missing. Unfortunately, your users will find this out in the end.
You can start a repair with ESEUTIL /P MDB01.edb. A warning message is shown saying that you should only run Repair on damaged or corrupted databases. The caution is that Repair will not apply information in the transaction log files to the database and may cause information to be lost. And it will ask you if you want to continue.
If you do, click the OK button to continue. ESEUTIL will perform a consistency check first, then scan the database and repair any damaged information that it found (see Figure 7-16).
Figure 7-16. ESEUTIL /P does several checks before repairing damaged information
One question that always pops up is how long it actually takes for a mailbox database repair to be completed. As a rule of thumb, I use 10 GB per hour for processing. So, if you have a 250 GB mailbox database you have to repair, it will take approximately 25 hours to complete. There’s no need to panic when you don’t see any dots moving on the console; ESEUTIL just needs time to do its work.
This is the reason Microsoft recommends not using large mailbox databases (i.e., larger than 200 GB) when a DAG is not used. If you’re using a 500 GB mailbox database in a single Mailbox server environment, and you run into a situation like this, you will have a hard time ensuring service delivery (which should be documented in the SLA).
Note It is my personal experience that it’s best not to use a large mailbox database when using a single Mailbox server. Although Microsoft recommends a maximum size of 200 GB for a mailbox database in such a scenario, I think it’s still too large. When you run into problems and have to start repairing your mailbox database, it will take too much time to do so. Stay on the safe side and keep your mailbox database between 50 and 75 GB. When more space is needed, just create additional mailbox databases.
It is recommended that you create a new mailbox database after a corrupted mailbox database has been repaired. This can be done in two ways:
· Create a new mailbox database on the Exchange server and move all the mailboxes from the old and repaired mailbox database to the new mailbox database. Once they are moved, the old mailbox database can be removed. Do not forget to create a new backup of the new mailbox database when done.
· Use ESEUTIL /D to perform an offline defragmentation. The net effect of this is that a new mailbox database is created with the old name. When you use ESEUTIL /D to perform an offline defragmentation, it will create a new mailbox database file next to the old mailbox database file. Then it will read all information from the old file and merge it into the new file. This way not only is a new file created but also new indices, new tables, new pointers, and so on—so basically you end up with a new mailbox database. When the copy process is complete, the old (and previously corrupted) mailbox database will be deleted.
In the Exchange Server 5.5 days, it was a best practice to run ESEUTIL /D on a biweekly or monthly basis to maintain relatively fresh mailbox databases. Considering the quality and stability of ESE in those days, this made perfect sense. But Exchange Server and ESE have come a long way, and now it doesn’t make much sense to run this on a regular basis.
The only time you should run ESE is when you have moved a large number of mailboxes (and thus a lot of data) from one mailbox database to another mailbox database. This will result in large amounts of white space inside the mailbox database, which won’t be reclaimed by online defragmentation (OLD) as part of online maintenance. Yes, this will restructure the data inside the mailbox database, but it won’t shrink the mailbox database.
If you are in a situation like this and you do not have a DAG, you can use ESEUTIL /D to offline defragment the mailbox database and reclaim this white space.
Note Offline defragmentation is what its name implies—an offline process. During this, the mailbox database is not available for users. It is hard to predict how long it will take for an offline defrag to finish. A good rule of thumb is 10 GB per hour, depending on the amount of database inside the mailbox database that needs to be moved. If you have a 200 GB mailbox database with 150 GB of white space, you have to move 50 GB of data, which can take up to five hours to complete.
File Recovery Tools
This is not really a native Exchange Server topic, but it is interesting enough to mention when viewed from a disaster recovery perspective.
There are situations when you do have a mailbox database file and you are in urgent need of data inside this mailbox database file, but you don’t have an Exchange server available to perform these kinds of recovery services. In this case, a file recovery tool can be helpful. Several tools are available, but the most well known is called PowerControls, from Kroll Ontrack.
Tools such as these let you open the mailbox database file (the EDB file), regardless of whether they are in a clean shutdown state or are taken from a backup and thus in a dirty shutdown state. Either way, you don’t need an Exchange server for opening the EDB file.
You can connect to a source, which in Figure 7-17 is a mailbox database taken from a backup. This is shown in the top pane. You can also connect to a target, which can be a PST file or a live Exchange 2013 server. It is possible to connect to all the folders in a mailbox; even the Recoverable Items folder is available.
Figure 7-17. PowerControls can open an EDB file and connect to a Mailbox
Messages can be exported to MSG, TXT, or EML files, but can also be exported to a PST file. Besides being able to export to one of these files, you can move these items from the source mailbox database files to a live mailbox when the tool is connected to an Exchange 2013 server.
The tool uses MAPI under the hood, so Outlook needs to be installed on the server and there must be a working Outlook profile on the recover server to make sure that the tool can connect to the Exchange 2013 server.
A recovery tool like Kroll Ontrack PowerControl can be an interesting addition to your toolbox of restore or disaster recovery solutions.
Note Opening a mailbox database directly by a recovery tool is not supported by Microsoft, nor is a direct restore by this tool using the MAPI interfaces.
Exchange Native Data Protection
Exchange Native Data Protection is a solution that was introduced in Exchange Server 2010, sometimes also referred to as a “backupless environment.” This is an Exchange server environment where a traditional backup solution is not used, and where native Exchange functions are the replacement—when possible, of course.
Note In some market segments, such as legal, finance, or health care, there are (legal) requirements that dictate how to create regular backups, keep them for a certain amount of time, and/or store them on a separate location. For these, Exchange Native Data Protection is not a solution.
Exchange Native Data Protection should be able to help you in scenarios where:
· Users have unintentionally deleted items from their mailboxes and need them urgently.
· You need to restore items from a user’s mailbox for legal purposes.
· A mailbox was deleted unintentionally.
· Hardware failures caused loss of a mailbox database or maybe loss of an entire server.
· There has been failure or loss of an entire datacenter.
Exchange 2013 has a lot of built-in features that can take care of some of these items; for example, there are deleted items retention, in-place hold, single item recovery, archive mailboxes, retention policies, and a database availability group (DAG).
Using these features, it is possible to create an environment where a traditional backup is not used, but where the disasters mentioned earlier would be fully covered. The advantage of this arrangement is that the total cost of ownership of a full-fledged Exchange 2013 environment is most likely lower than that of a regular Exchange 2013 environment with a traditional backup solution.
Microsoft recommends evaluating your various requirements and the available options before deciding whether this is a viable solution for your organization. For more information, visit the Microsoft Exchange team site at http://bit.ly/BackupLess.
Deleted Items Retention
When users delete items from their mailboxes, the items are not visible to the users anymore, but they are still in the users’ mailboxes. They are stored in hidden folders in their mailboxes, which is part of the “dumpster.”
Deleted items are kept in the Recoverable Items folder for a time called the deleted item retention period. By default, the retention time for deleted items is 14 days, while for a deleted mailbox it is 30 days. This means that after these times, the items are deleted from the server. Deleted item retention is a property of the mailbox database and can be set using the Set-MailboxDatabase command with the –DeletedItemRetention parameter, and is specified as a time span: dd.hh:mm:ss. To set the deleted item retention time to 60 days, for example, you can use the following command:
Set-MailboxDatabase -Identity MDB01 -DeletedItemRetention 60.00:00:00
To set the deleted item retention time for mailboxes, the same command is used but with the MailboxRetention option. For example, to change the deleted mailbox retention time to seven days, you can use the following command:
Set-MailboxDatabase -Identity MDB01 -MailboxRetention 7.00:00:00
Within this deleted items retention time, users can recover deleted items themselves in Outlook. In Outlook 2013, you select the Folders tab and then click the Recover Deleted Items button. A new window will pop up showing all messages that are not in the Deleted Items folder but can still be recovered from the Recoverable Items folder (see Figure 7-18).
Figure 7-18. End users can recover deleted items
In Outlook WebApp, the deleted items can be recovered as well. In the OWA navigation pane, right-click the Deleted Items folder and select the Recover Deleted Items option.
When the user permanently deletes a message (i.e., purges the Deleted Items folder), the message can no longer be recovered—not even by administrators—unless the Single Item Recovery option is enabled. (This is covered later in this section.)
In-Place Hold
Although you now know it is possible to stretch the deleted item retention period, it is not really a good idea to stretch this time to a couple of years; doing so would have a disastrous effect on the size of the mailbox database. So Exchange 2010 came up with a new feature called litigation hold, which was then replaced by in-place hold in Exchange 2013. With in-place hold, you have the possibility of:
· Placing user mailboxes on hold and preserving mailbox content from any possible alteration.
· Preserving delete items; that is, items that can be deleted manually by the user or by an automated process like messaging records management (MRM).
· Searching for and retaining items matching specified criteria.
· Preserving items indefinitely or for a specific duration.
· Keeping in-place hold transparent from the user by not having to suspend MRM.
· Enabling in-place eDiscovery searches of items placed on hold.
In short, items can be on hold in a user’s mailbox without the user knowing it.
Litigation hold allowed putting all items on hold for an indefinite amount of time. In-place hold is more granular, with the following possible parameters:
· What to hold It is possible to specify which items to put on hold by using parameters such as keywords, recipients, senders, start and end dates, and type of items, such as messages, calendar items, and so forth.
· How long to hold It’s possible to specify how long these items should be on hold.
Note With in-place hold, the previous indefinite hold is still possible.
To place a mailbox on in-place hold, you need to have specific permissions. These permissions are granted to users who are members of the Discovery Management RBAC group or users who are assigned the Legal Hold and Mailbox Search management roles.
Note In-place hold is a premium feature that requires an Enterprise Client Access License (CAL). You only need the Enterprise CAL for mailboxes that use this feature.
To create a new in-place hold using PowerShell, you can use the following command:
New-MailboxSearch -Name "In-place hold Joe Jackson Mailbox" -SourceMailboxes "joe@contoso.com" -InPlaceHoldEnabled $true
When a new mailbox search is begun, an informational warning message is shown saying that the new Mailbox server won’t be effective immediately and that it can take up to 60 minutes before becoming active (see Figure 7-19).
Figure 7-19. Creating a new mailbox search using the EMS. Note the warning message
For example, all items in Joe’s mailbox will now be on hold for an indefinite period. When Joe deletes items and purges his deleted items manually, they will be stored. If Joe receives a message and deliberately makes changes to this message, the original and the changed messages will both be kept. It is good practice to monitor the deleted items quota in the mailboxes for which you have enabled in-place hold.
Note This will work only for mailboxes on Exchange 2013. For mailboxes stored on an Exchange 2010 Mailbox server, the corresponding cmdlet has to be used on an Exchange 2010 server.
Single Item Recovery
Single item recovery was introduced in Exchange 2010 and is still available in Exchange 2013.
When a message is deleted from a mailbox, it is stored in the Recoverable Items folder of that mailbox. Items stay in this folder until the deleted items retention period expires. Only then are they fully removed from the Exchange server by the server’s online maintenance. When a user permanently deletes a message, it is removed immediately from the Recoverable Items folder and the user is no longer able to recover that message—not even the administrator is able to recover it. One exception, however, is when the mailbox is on in-place hold. Then, deleted items are retained until the in-place hold is removed.
To be able to recover these deleted items when they have been permanently deleted by the user, you can enable single item recovery on the mailbox. When this is enabled, permanently deleted messages are retained in the Recoverable Items folder until the deleted item retention period is passed and then the message is fully removed from the Recoverable Items folder.
Single item recovery can be enabled only with PowerShell and by using the following command:
Set-Mailbox -Identity "April Summers" -SingleItemRecoveryEnabled $true
For mailboxes that have single item recovery enabled, it is also possible to set a different deleted items retention time using the -RetainDeletedItemsFor option, as follows:
Set-Mailbox -Identity "Kim Akers" -SingleItemRecoveryEnabled $true -RetainDeletedItemsFor 30
Recovering items when single item recovery is enabled consists of two steps:
1. Searching for the missing items and recovering them.
2. Restoring the recovered items.
Searching for deleted items in mailboxes that have single item recovery enabled or mailboxes that are on in-place hold is part of in-place eDiscovery. As such, you need special permissions to search for these deleted items or to search in these mailboxes. This makes sense, since you’re looking into somebody else’s potentially private information. With the permission stipulation in place, it is impossible to “accidentally” search others’ mailboxes. Permission is granted to members of the Discovery Management RBAC role group. To perform a search, you follow these steps:
1. Select Compliance Management in the Feature pane and select the In-Place eDiscovery and Hold tab. In the toolbar, click the New icon.
2. In the New In-Place eDiscovery and Hold wizard, enter a name and a description for the new hold you’re creating. Click Next to continue.
3. The next window lets you choose if you want to search all mailboxes or only specific mailboxes. Assuming the latter, select the Specify Mailboxes to Search radio button and use the Add button to select a mailbox that needs to be added to the new search. Select the mailbox, use the Add button, and click OK to return to the previous window. This mailbox should now be listed in the Results pane. Click Next to continue.
4. In the Search Query window, you can define the query that needs to be used. Enter the keywords, start and end dates, recipient and sender, and the item type, like email, meeting, notes, and so forth. Select the options you need and click Next to continue.
5. If needed, you can opt to place the content matching the search query on hold. It can be on hold indefinitely or for a certain number of days. Click Finish to continue.
The search is not executed immediately, but an estimate of the search results is shown in the Details pane, including the number of items returned by the search and its size (see Figure 7-20).
Figure 7-20. Estimate of results for an in-place eDiscovery
6. When the results are satisfactory, the search can be executed. Click the Down arrow next to the search icon and select the Copy Search Results option. You can select which options can be enabled, such as:
· Include unsearchable items
· Enable de-duplication
· Enable full logging
· Send me mail when the copy is completed
7. The last action is to select where the search will store its results; typically this is the Discovery Search mailbox. Click Copy to start copying the search results to the Discovery Search mailbox.
Members of the Discovery Search RBAC role group are automatically assigned full access permission to the Discovery Search mailbox. When you open the Discovery Search mailbox, either in Outlook or in OWA, you should see a folder with the name you specified in the eDiscovery wizard. This is where the search results are stored (see Figure 7-21).
Figure 7-21. Search results are stored in the Discovery Search mailbox
When the results are stored in the Discovery Search mailbox, it is possible to export those results to the user’s mailbox using the Search-Mailbox command:
Search-Mailbox "Discovery Search Mailbox" -SearchQuery "Welcome" -TargetMailbox "Kim Akers" -TargetFolder "Recovered Messages" -LogLevel Full -DeleteContent
For example, this command moves the items from the search results into Kim Akers’s mailbox in a folder called Recovered Messages. When this is done, the items in the Discovery Search mailbox are deleted.
Using New-MailboxExportRequest, it is also possible to export the search results to a PST file. The same prerequisites apply as when using a normal export request. The MRS processes the request, so the PST file needs to be stored to a fileshare where the Exchange Trusted Subsystem USG has permission for full access.
For example, the following command will start a new export request of the folder called Kim Akers in the Discovery Search mailbox, where the subject of the items is “Welcome,” and will store the PST file on a fileshare called HelpDeskPst on a server called AMS-FS01:
New-MailboxExportRequest -Mailbox "Discovery Search Mailbox" -SourceRootFolder "Kim Akers" -ContentFilter {Subject -eq "Welcome"} -FilePath \\AMS-FS01\HelpDeskPst\KimAkersRecovery.pst
Archive Mailboxes
Besides normal mailboxes, in Exchange 2013 there are archive mailboxes, a concept introduced in Exchange 2010.
An archive mailbox is just a normal mailbox, but it’s connected to a user as a secondary mailbox and is used for archiving purposes. The archive mailbox, however, is completely separate from the user’s primary mailbox and can be located on additional storage, on a separate server, or even in Exchange Online.
An archive mailbox is visible from Outlook 2007 Professional Plus and higher, and also from Outlook WebApp; it appears as an additional mailbox in the client.
To create an archive mailbox for a user call Joe, and locate this archive mailbox in a mailbox database called MDB01, you can use the following command (see Figure 7-22):
Enable-Mailbox -Identity Joe@contoso.com -Archive -ArchiveDatabase MDB01
Figure 7-22. Creating an archive Mailbox using EMS
Note In-place archiving is a premium feature and requires an Exchange Enterprise Client Access License (CAL).
By default, an archive mailbox has an archive quota of 100 GB and an archive quota warning at 90 GB. This means that a warning message will be sent to the user when the archive reaches 90 GB, but this is sufficient to store tons of information.
Note Archive mailboxes are a perfect “PST killer.” By nature, PST files are stored on the user’s workstation or laptop, where they are not backed up and are prone to getting lost when the laptop is stored. Storing PST files on a network share is not supported and can give erratic results in Outlook. Also, a backup application can skip these files when they are still open in Outlook. But when they are moved to archive mailboxes, they are stored safely, available for the user and backed up frequently. It’s a perfect solution for getting rid of your PST files.
The user can manually move data to the archive mailbox, but an administrator can also import data into that archive mailbox. The New-MailboxImportRequest command can be used to import PST files from a fileshare directly into an archive mailbox. To achieve this in EMS, use a command similar to this example:
New-MailboxImportRequest -Mailbox Joe@contoso.com -FilePath \\AMS-FS01\PSTFiles\Archive2011.pst -IsArchive
Another way to automatically move data from the user’s mailbox into the archive mailbox is by using retention policies.
Retention Policies
Over the years, the amount of information stored in mailboxes has kept growing and growing. To deal with this, Microsoft introduced messaging records management (MRM) in Exchange 2007. The implementation of MRM was by means of managed folders, which are folders created by an Exchange administrator. On top of these managed folders, the Exchange administrator had to create rules that govern certain actions performed on items in the folders, such as marking them as past retention or to permanently delete. Users were able to store information in these managed folders, and after a retention time set by the administrator, the information would be processed.
In addition to using MRM to manage amounts of information, and thus meet larger storage requirements, MRM can be used to:
· Meet business requirements Depending on your organization’s messaging policies, you may need to retain important email messages for a certain period. It is not uncommon that HR information with resumes of non-hired people are kept for only six months, or information regarding business strategies or transactions is kept for only a certain amount of time. When items are stored for a limited time in Exchange 2013, the items are not set to read-only, so that users are able to delete these items if they want to.
· Meet legal and regulatory requirements Many organizations have a legal or regulatory requirement to store messages for a designated period and to remove messages older than that time. Storing messages longer than necessary may increase your organization’s legal or financial risks.
· Increase user productivity The amount of information, or the ever-increasing amount of information, has a negative impact on productivity. For example, consider those newsletters that are kept for ages but never read. Using MRM, it is possible to remove these after a certain amount of time, thereby reducing the clutter in a user’s mailbox and increasing the productivity.
MRM version 2 was introduced in Exchange 2010 and included the personal archive and retention tags and retention policies, a concept further developed in Exchange 2013.
As an Exchange administrator, you can choose the tags that can be assigned to items in a mailbox. There are various types of tags available:
· A default policy tag (DPT)
· Personal tags applied to folders or individual items by the user
· Retention policy tags (RPT) assigned to specific folders to expire items in those folders
Retention policy tags can be created for the folders listed in Table 7-1.
Table 7-1. Folder in a Mailbox that Can Have RPTs Applied
|
Folder Name |
Details |
|
Calendar |
Default folder used to store meeting dates and appointments. |
|
Conversation History |
Created by Microsoft Lync (when implemented); although not treated as a default folder by Outlook, it’s regarded as a special folder by Exchange and can have RPTs applied. |
|
Delete Items |
Default folder used to store items deleted from other folders in the mailbox. Outlook and Outlook WebApp users can manually empty this folder. Users can also configure Outlook to empty the folder upon closing Outlook. |
|
Drafts |
Default folder used to store draft messages that haven’t been sent by the user. Outlook WebApp also uses this folder to save messages that were sent by the user but not submitted to the Hub Transport server. |
|
Inbox |
Default folder used to store messages delivered to a mailbox. |
|
Journal |
Default folder for actions selected by the user. These actions are automatically recorded by Outlook and placed in a timeline view. |
|
Junk Email |
Default folder used to save messages marked as junk email by the content filter on an Exchange server or by the anti-spam filter in Outlook. |
|
Notes |
Contains notes created by users in Outlook. |
|
Outbox |
Default folder used to temporarily store sent messages until processed by a Hub Transport server. Messages usually remain in this folder for a brief period so it isn’t necessary to create an RPT for this folder. |
|
RSS Feeds |
Default folder containing RSS feeds. |
|
Recoverable Items |
Hidden folder in the non-IPM sub-tree. It contains the Deletions, Versions, Purges, and Audits sub-folders. Retention tags for this folder move items from the Recoverable Items folder in the user’s primary mailbox to the Recoverable Items folder in the user’s archive mailbox. You can assign only the Move To Archive retention action to tags for this folder. |
|
Sent Items |
Default folder used to store messages that have been sent. |
|
Sync Issues |
Contains synchronization logs. |
|
Tasks |
Default folder used to store tasks. |
You can create a DPT that will move all items in a mailbox to the personal archive in 180 days, but create personal tags that will keep items in the mailbox for one or two years. These personal tags can be assigned by the user to items and will override the DPT.
Additionally, it is possible to create RPTs that will mark messages as past retention after a certain number of years or when legal or company requirements dictate, then permanently remove those items. Another RPT can be applied to the junk email folder in the mailbox that will permanently remove items after, for example, seven days.
When creating an RPT, you can specify the following actions to be taken when the retention time expires:
· Move to Archive Items are moved to the archive mailbox. Naming in the archive mailbox will be identical, so when a folder named “Finance” exists in the primary mailbox, it will automatically be created as such in the archive mailbox.
· Delete and Allow Recovery Items are deleted, but kept in the Recoverable Items folder.
· Permanently Delete Items are permanently deleted (and not stored in the Recoverable Items folder) and cannot be recovered. When the in-place hold is set on a mailbox, the item is kept in the Recoverable Items folder. If single item recovery is enabled for a mailbox, the item will be kept in the Recoverable Items folder until the deleted item retention time for this mailbox database or the single item recovery setting is reached.
· Mark as Past Retention Messages are marked as past retention. Outlook and OWA will show this message with a notification string “This item has expired.”
To apply the retention tags to a mailbox, you must add them to a retention policy, and this retention policy is applied to the mailbox. A mailbox can have only one retention policy applied, but a retention policy can have multiple retention tags.
By default, 11 retention policy tags are created and two retention policies are created. One retention policy is for the arbitration mailboxes; the other retention policy is applied to mailboxes that have an archive mailbox.
Note The EMS shows 11 retention policy tags (Get-RetentionPolicyTag) and two retention policies (Get-RetentionPolicy). For some reason, EAC only shows 10 retention policy tags and one retention policy.
Table 7-2 shows the default retention policy tags that are used in the default retention policy.
Table 7-2. The Default Retention Policy and its Retention Policy Tags
The managed folder assistant is a service running on the Mailbox server that’s responsible for processing the mailboxes with retention policies indicated. It applies the retention policy by inspecting items in the mailbox and determines whether they are subject to retention. It then stamps the items subject to retention with the appropriate retention tags and takes the specified retention action when items are past their retention age.
The managed folder assistant is a throttled service. This means that it is running 24 hours a day, but it does not have full control over system resources to prevent depletion of resources.
Note The managed folder assistant doesn’t take any action on messages that aren’t subject to retention.
When items are moved from one folder in the inbox to another folder in the inbox, the items inherit the retention policy tag that has been applied to the target folder.
The default 11 retention policy tags are most likely sufficient for 99 percent of all Exchange 2013 implementations, but if you want to create additional RPTs, it’s easily done. To create a new RPT that deletes an item after 14 days, but allows recovery for the item, for example, you can use the following PowerShell command:
New-RetentionPolicyTag -Name "Delete after 14 days" -Type Personal -RetentionAction DeleteAndAllowRecovery -AgeLimitForRetention 14 -Comment "Items with this tag will be deleted after 14 days"
RPTs are grouped together in retention policies. Suppose you want to create a new policy called “My Exchange 2013 Policy” and assign the default policy tag “Default 2 years move to archive” and the RPT you just created; you would use the following PowerShell command:
New-RetentionPolicy "My Exchange 2013 Policy" -RetentionPolicyTagLinks "Default 2 years move to archive","Delete after 14 days"
To assign this retention policy to a mailbox in PowerShell, you use the following command:
Set-Mailbox -Identity joe@contoso.com -RetentionPolicy "My Exchange 2013 Policy"
Note Personal tags are a premium feature. Mailboxes with retention policies require an Enterprise Client Access License.
Database Availability Groups
Chapter 5 talked about the database availability groups, or DAGs, and how a DAG can be used to improve the uptime of an Exchange 2013 environment. Here, we revisit the DAG, but now it’s to see what it can do for a backupless environment.
A DAG is a logical grouping of Exchange 2013 Mailbox servers that are used to host multiple copies of mailbox databases. One Mailbox server in a DAG holds an active copy of a mailbox database; this is the mailbox database that’s used by clients. One or more Mailbox servers in a DAG hold a passive copy of a mailbox database, and these passive copies are not used by any clients; they are only for redundancy purposes. One exception can be a backup application that uses a passive copy to create backups.
Figure 7-23 illustrates how redundancy works with a DAG. Clients like Outlook use a load balancer to connect to an Exchange 2013 Client Access server, where they are authenticated. After authentication, the request is proxied to the Mailbox server holding the active copy of the mailbox database. On this Mailbox server, the data is retrieved from the mailbox and returned to the client. The most important part of this process to remember is that all mail processing is taking place on the Mailbox server holding the active copy of the mailbox database.
Figure 7-23. The DAG makes sure your mailbox databases are redundant
As shown earlier in this chapter, mailbox data is in the Mailbox server’s memory, the transaction log file, or the mailbox database. During processing, new transaction files are created; once they are stored on the server’s hard disk, they are sent to the Mailbox server holding a passive copy of that mailbox database. At minimum, there’s one passive copy of the mailbox database, but more passive copies can be used as well.
In a DAG, there’s always one active copy of the mailbox database and replication always takes place from this active copy to all the passive copies. So how do the passive copies get on the other Mailbox servers? This is a process called seeding. It’s the actual copying of the mailbox database from the server holding the active copy to another Mailbox server. Seeding a mailbox database is not an NTFS copy; instead, a technique called streaming is used—very similar to the streaming backup technology explained earlier in this chapter. When streaming, the active mailbox copy is read (database) page by page. When the page is read, its checksum is checked; when it’s okay, the page is sent to the other Mailbox server, where it is stored. This way the new passive copy is always identical to the active copy of the mailbox database and it’s 100 percent consistent.
What happens when a Mailbox server fails? Clients are connected to the Client Access server. Referring back to Figure 7-21, you can see that when the mailbox database on Mailbox server MBX01 fails, the passive copy of the mailbox database on server MBX02 is automatically activated. The Client Access server connects to this Mailbox server within seconds. Clients do notice a small disturbance; in the taskbar, a balloon will pop up saying “Connection lost,” followed shortly by a balloon saying “Connection restored.”
So, if a mailbox database is lost, the service continues to run even when the mailbox database is beyond repair. When a disk containing one or more mailbox databases is lost, though, you have to replace the disk and manually perform a reseed of the mailbox database to the new disk. Depending on the size of the mailbox database, this can take considerable time, but since the users can continue to work with another copy of the mailbox database, this is no big deal.
Note Multiple mailbox databases in a DAG have another advantage. When a Mailbox server encounters a corrupted page, a process called page patching is started. A special marker is placed in the current transaction log file, which is then closed and replicated to another Mailbox server holding a passive copy of the mailbox database. When this Mailbox server encounters the marker, it tries to retrieve a copy of the patch (which should be fine on the passive copy) and returns the healthy page to the first Mailbox server. This places the page in its cache buffer, where it is then written into one of the log files and eventually written into the mailbox database.
When this happens, the header of the mailbox database file is updated as well. There’s a special entry called “patch count” for this with the number of patches applied. It is possible to check this using the ESEUTIL /MH command. When the patches are applied and the database header is updated, the mailbox database is repaired.
By using a DAG, you achieve higher availability because of the multiple mailbox databases and replication technology, but it doesn’t prevent human failures, such as the accidental deletion of mailbox content or even deletion of a mailbox itself. As far as Exchange is concerned, these kinds of deletions are legitimate actions and thus are replicated across all mailbox databases.
A DAG does, however, protect you against hardware failures, like server failure, controller failure, or a hard disk failure, resulting in physical database corruption. It does not protect you against logical corruption of a mailbox database. Though logical corruption of a mailbox database is rare, it can lead to data loss.
There are two types of logical corruption:
· Database logical corruption In this case, the checksum of a page in the mailbox database is correct but the data inside the page is wrong. This can happen when ESE tries to write a page to disk (and ESE gets a success message back from disk), but the page isn’t actually written to disk or is written to the wrong location. This is also referred to as a “lost flush.” A common cause is that an NFS volume is used instead of block-level storage.
· Store logical corruption Data inside a page is added, modified, deleted, and so on in an unexpected way, usually caused by a third-party application. From an Exchange Server point of view, this is not really corruption, as the database pages contain valid operations, but the user still experiences this as message corruption in his or her mailbox.
Among other solutions like in-place hold, which was explained earlier, a means to help guard against logical corruption is to use a lagged copy in the DAG.
Lagged Copies
A lagged copy is a passive mailbox database copy, but one that has a lag time in replaying the log files into the mailbox database. This lag time can be anywhere between one second and 14 days. The lagged copy of a mailbox database can be in the same (Active Directory) site as the active copy of the mailbox database, but it can also be located in another, remote (Active Directory) site for offsite storage.
Note Lagged copies are not a high-availability solution, as outlined in Chapter 5, but they are a disaster-recovery solution. Manual intervention is needed to activate a lagged copy, and when you’re using a 14-day lag time, that means you need a substantial amount of time to replay the log files and activate the lagged mailbox copy.
In a DAG, the log files are replicated from the Mailbox server holding the active copy of the mailbox database to the Mailbox servers holding the passive copies of the mailbox database. This is no different for lagged copies. Normal passive copies of the mailbox database inspect the log file that was received, and at some point the log file is read and the information is replayed into the passive copy of the mailbox database. Just like the active copy of the mailbox database, there can be some delay in replaying—in other words, there is a certain checkpoint depth.
For lagged copies, however, the log files are not replayed into the passive copy of the mailbox database during the time, as specified in the lag time. When the lag time is two hours, this is not a big deal; but when the lag time is seven days, it means that seven days of log files will be stored on the Mailbox server holding the passive copy. Consequently, you have to take this into account when it comes to your storage design, since seven days of log files can consume a considerable amount of storage space. The Exchange 2013 Server role requirements calculator can do this math for you.
As stated, a lagged copy is not part of a high-availability solution but is part of a disaster-recovery solution. The lagged copy is activated manually, not automatically. When a lagged copy is activated, the log files that are stored during the lag time need to be replayed into the passive copy first. When the lag time is set to seven days, this can take quite some time to finish.
When creating a lagged copy, there are two choices:
· There are already copies of a mailbox database running and the only thing you have to do is add the lag time.
· You want to create an additional copy of a mailbox database and include the lag time during creation of that additional copy.
To create an additional copy of a mailbox database and add the lag time during creation, you can use the following command to create an additional copy and set the replay lag time to four days, seven hours, and 36 minutes:
Add-MailboxDatabaseCopy -Identity MDB01 -MailboxServer AMS-EXCH02 -ReplayLagTime 00:10:00 -TruncationLagTime 4.7:36:00 -ActivationPreference 2
When you already have a copy of a mailbox database and want to add a replay lag time to this copy, you can use the following example:
Set-MailboxDatabaseCopy -Identity MDB01\AMS-EXCH01 -ReplayLagTime 7.0:0:0
Note For redundancy purposes in transport, the SafetyNet feature is used. To prevent unwanted data loss, the SafetyNet hold time should be higher than the replay lag time of the mailbox database copy. The Set-TransportConfig cmdlet can be used to set this using the -SafetyNetHoldTime option.
When you have a lagged copy of a mailbox database, the log files from the active copy of the mailbox database are continuously sent to the passive copies, including the lagged copy. On the Mailbox server holding the lagged copy of the mailbox database, the log files are kept in a queue until the lag time expires. If a lag time of seven days is used, this can become a tremendous amount of data. You can check the number of log files in the replay queue using the Get-MailboxDatabase | Get-MailboxDatabaseCopyStatus command in EMS. This will show the mailbox database, its status, the copy queue length, and the replay queue length. The copy queue length should be close to zero (although it can increase occasionally), but the replay queue will show the number of log files waiting for the lag time to expire. So don’t be alarmed when you see a high number of log files in this queue!
Another option that’s only settable using EMS is the TruncationLagTime option. This is the lag time that the Microsoft Exchange Replication service should wait after replaying the information into the passive copy of the mailbox database before the log file is actually deleted. The minimum value of the truncation lag time is 0 seconds, the maximum value is 14 days.
Having lagged copies is only one part of the story; recovering from a lagged copy is another part. This is covered in the “Point in Time (PIT) Recovery” section a bit later in this chapter.
Circular Logging
As explained in Chapter 5 about the Mailbox server, using circular logging on a production Mailbox server is, generally speaking, not a good idea since this will block log file replay during recovery. You can restore from a backup without the proper log files, but you cannot recover to a newer point in time, since this information is missing.
When a DAG is configured in your Exchange 2013 environment, things are a bit different. Since there are multiple copies of a mailbox database, there’s no need to restore from a backup. In case of a failure, another copy of the mailbox database takes over and continues servicing the user requests.
If there are no legal requirements to create backups, it is possible to use only the Mailbox servers running in a DAG. When one mailbox database fails, another one takes over without the need for an Exchange administrator to restore a mailbox database from backup.
But one of the functions of a backup is to purge log files that are not needed anymore after a backup was taken. To overcome this, circular logging can be enabled within the DAG. Circular logging in a DAG isn’t a function of ESE as in a normal scenario, but it is performed by the Microsoft Exchange eplication service.
Circular logging in a DAG is called continuous replication circular logging (CRCL). There’s a major difference with the default circular logging as used by ESE. In a normal circular logging scenario, log files are deleted once the data in the files is stored in the mailbox database. When using CRCL, however, the log files are deleted only when they have been replicated to the passive copies of the mailbox database. The Microsoft Exchange Replication service communicates using RPC with each other to determine the status of the replication and which log files can be safely deleted.
The following must be true for log files to be deleted:
· CRCL must be enabled.
· The log file is below the checkpoint.
· Other copies of the mailbox database agree with the deletion.
· The log file has been inspected by all lagged copies of the mailbox database.
Log files are also deleted on Mailbox servers holding a lagged copy of the mailbox database. For this to happen, the following must be true:
· The log file is below the checkpoint.
· The log file is older than the combined value of the replay lag time and the truncation lag time.
· The log file is already deleted on the active copy of the mailbox database.
So when using CRCL, you should always be safe, able to recover from a mailbox database failure with minimum loss of data.
Point-in-Time Recovery
When using a lagged copy of a mailbox database, it is possible to return to a specific point in time between the active copy of the mailbox database and the replay lag time of the lagged copy of a mailbox database. This means that it is possible to recover to an exact point in time between 0 seconds and a maximum of 14 days ago.
The technology behind point-in-time (PIT) recovery is that all information stored in log files is replayed into the lagged copy of the mailbox database—up to the exact moment you want to recover to.
Note Although a very rare occasion, this technique can be used to recover from a logical mailbox database corruption.
The steps to recover to a certain PIT are as follows:
1. Suspend the replication to the lagged copy of the mailbox database.
2. Determine the last log file you want to replay into the mailbox database. This is the PIT you want to recover to.
3. Use ESEUTIL to replay the log files into the lagged copy of the mailbox database.
4. Use this copy of the mailbox database for recovery purposes.
A PIT recovery cannot be performed by the EAC, so the EMS is the only option you have here. To suspend the replication of the lagged copy of the mailbox database, use an EMS command similar to this:
Suspend-MailboxDatabaseCopy DB01\AMS-EXCH02 -SuspendComment "Activate lagged copy of DB01 on Server AMS-EXCH02" -Confirm:$false
Determine which log file you want to use for recovery purposes. All log files that are more recent than the chosen one should be moved to a different location to prevent accidental replay of information.
Before replaying the log files, you need to remove the checkpoint file. Replaying the log files starts at the location of the checkpoint file; by removing the checkpoint file, replay starts automatically at the oldest available log file. If information found in these log files is already in the mailbox database, then that’s no problem; it will just be skipped. At one point, though, information inside the log files that’s not in the mailbox database will be replayed into the mailbox database until the last available log file is replayed. Then, the replaying stops and the mailbox database is closed, as in a clean shutdown state. The mailbox database is available for any recovery purpose you have in mind—for example, a recovery database.
When the checkpoint file is deleted, you open a command prompt, navigate to the location where the lagged copy is stored, and enter a command similar to this:
ESEUTIL /R E00 /LF:\DB01\Logfiles /D /I
The E00 prefix is the prefix used by this lagged copy, the /L option is used to point to the location of the log files, and the /S option is used to point to the location of the system file. The /D option is used to point to the location of the mailbox database, but in this example it is the current directory. The /I option is used to ignore any missing attachments into the log files that need to go into the mailbox database. All options should have values that correspond to your environment, of course.
Summary
This chapter explained about backup, restore, and disaster recovery, and especially how the three work together. The last part of the chapter described the Microsoft Exchange Native Data Protection, sometimes referred to as a “backupless environment.”
Exchange 2013 uses snapshot technology provided by the Virtual Shadowcopy Services (VSS). The out-of-the-box tool Windows Server Backup is able to create VSS backups of the mailbox database. New to Exchange 2013 is the fact that Windows Server Backup can back up passive copies of a mailbox database—something that was not possible to do before.
Mailbox databases can be restored from a backup on the original location, but a recovery mailbox database can be used as well, where the mailbox is restored to a different location, after which all kinds of recovery actions can be taken.
The ultimate disaster recovery is needed when an entire server is lost. It is possible to rebuild a new but identical Exchange 2013 server using the setup application with the /Mode:RecoverServer option. This will retrieve all information from Active Directory.
The Exchange Native Data Protection is a set of technologies that can be used to overcome all kinds of disasters, ranging from messages that are accidentally deleted to additional copies of a mailbox database stored in another location. When things are designed properly, there are situations where a traditional backup is not needed anymore but, of course, this fully depends on your organizational needs and requirements.
The next chapter is about monitoring Exchange 2013—something that’s needed as well, and quite a lot of times is forgotten because the typical administrator is too busy. But when monitoring is not performed properly, there’s always a risk encountering disaster and having to perform the subsequent recovery.