Chapter 5.Virtual Machine Storage - Automating Microsoft Azure Infrastructure Services (2015)

Automating Microsoft Azure Infrastructure Services

Chapter 5. Virtual Machine Storage

Storage Management

Microsoft Azure virtual machines are based on some of the same virtualization technologies in Windows Server Hyper-V. From a storage perspective, this means that the underlying disks and images are based on Microsoft Virtual Hard Disk (VHD) format. As you have seen in Chapter 3, virtual machine disks are stored in Microsoft Azure Storage, which itself is a highly scalable, durable, and available service. In this chapter we will dig deeper into how you can use PowerShell to manage these disks and configure storage for your virtual machines.

Uploading and Downloading VHDs

To start with, let’s explore how to upload and download VHD files with Microsoft Azure storage. As you have already seen, it is possible to create VHD files in the cloud without the need to create them locally and upload them. However, I feel that starting at this point makes some of the concepts later in the chapter easier to explain.

The Microsoft Azure PowerShell cmdlets provide two cmdlets for uploading and downloading VHD files to a storage account: Add-AzureVHD and Save-AzureVHD. Let’s explore each of these cmdlets using a simple walk-through.

Uploading a VHD

Using the Microsoft Azure PowerShell cmdlets over most generic storage tools has its advantages. In addition to easy scriptability, the cmdlets do not treat the VHD files as plain old blobs of data. Instead, the cmdlets have knowledge of the native VHD file format. This means that the cmdlets can optimize how they upload or download VHDs.

The Add-AzureVHD cmdlet supports an optimized upload. When you upload a VHD, the cmdlet evaluates which bytes of the file have data and transmits only the written bytes to storage while keeping the overall structure of the file intact.

Microsoft Azure currently supports only VHD files in fixed format. That does not mean that you have to convert your dynamic disks to fixed before uploading. The Add-AzureVHD cmdlet can dynamically convert from dynamic to fixed on upload. Unfortunately, at this time it does not support converting from VHDX to VHD, but that scenario can easily be accomplished with the Convert-VHD cmdlet that comes with Server 2012 R2 and Windows 8.1.

The Add-AzureVHD cmdlet supports three optional parameters shown in Table 5-1.

Table 5-1. Add-AzureVHD optional parameters

-OverWrite

If the VHD exists on the destination, you can overwrite it by passing this parameter.

-NumberOfThreads

The default value is 8. Change if you feel that more or less parallelization would improve your upload.

-BaseImageUriToPatch

When uploading a differencing disk, specify the image name to patch.

Authentication with the Add-AzureVHD and Save-AzureVHD cmdlets works the same way as the other Microsoft Azure cmdlets. You can call Select-AzureSubscription prior to their usage, and they will use the selected subscription and authentication method. As an alternative, shared access signatures are also fully supported. You can use a shared access signature by specifying it as part of the destination URL.

To try uploading a VHD on your own, create a new PowerShell script in the PowerShell ISE named chapter5upload.ps1. Add the code shown in Example 5-1 to the file. Ensure that you replace the placeholder values with real values. Do not execute the script yet because you still have to create the mydatadisk.vhd file using the disk management tool in Windows.

This example assumes that you have the virtual machine named ps-vm1 from Chapter 3 (feel free to substitute another virtual machine in its place).

Example 5-1. Using Add-AzureVHD to upload a VHD (Script pane)

$subscription = "[subscription name]"

$storageAccount = "[storage account name]"

$serviceName = "[cloud service name]"

$vmName = "ps-vm1"

Select-AzureSubscription $subscription

$source = "C:\VHDFiles\mydatadisk.vhd"

$destination = "https://$storageAccount.blob.core.windows.net/upload/mydatadisk.vhd"

Add-AzureVHD -LocalFilePath $source -Destination $destination

Add-AzureDisk -DiskName "mydatadisk" -MediaLocation $destination

Get-AzureVM -ServiceName $serviceName -Name $vmName |

Add-AzureDataDisk -Import -DiskName "mydatadisk" -LUN 1 |

Update-AzureVM

The call to the Add-AzureDisk cmdlet associates the VHD file with the mydatadisk disk name. Once the name is registered, you can easily attach or detach the disk by name using the Add-AzureDataDisk and Remove-AzureDataDisk cmdlets.

These cmdlets work by modifying the returned virtual machine configuration from Get-AzureVM by either adding or removing the referenced disk name. The modified configuration is then passed to Update-AzureVM, which in turn calls the appropriate API for passing the configuration to Azure to perform the update.

Creating a Local VHD with Windows

Before you can execute the script, you need to create the VHDFiles folder on the C: drive and the mydatadisk.vhd file. Using the built-in functionality in Windows 7 and above, you can create the VHD file locally, mount it, add data to it, and then upload it using the Azure cmdlets.

To do this on your own, open Control Panel→Administrative Tools, and then open Computer Management (see Figure 5-1).

Computer Management

Figure 5-1. Computer Management

When Computer Management is open, expand Storage, right-click Disk Configuration, and click Create VHD (see Figure 5-2).

Create VHD

Figure 5-2. Create VHD

When the dialog opens, specify C:\VHDFiles\mydatadisk.vhd in the path (make sure you have created the VHDFiles folder first!) and specify 50 MB to keep the disk small but still usable. You should also ensure that VHD is selected and not VHDX before creating the VHD (see Figure 5-3).

Specifying local VHD location

Figure 5-3. Specifying local VHD location

After the disk is created, you will then need to intialize and format it before adding data. To initialize the VHD, right-click on the left side (where it says Disk 1) and click Initialize Disk (see Figure 5-4). A new dialog will open; change the partition table format to MBR and click OK.

Initializing a disk

Figure 5-4. Initializing a disk

After the disk is initialized, right-click on the right side of the disk and select New Simple Volume. Accept the defaults for the New Simple Volume Wizard and let Windows format the drive (see Figure 5-5).

Creating a new simple volume

Figure 5-5. Creating a new simple volume

When the disk formatting has completed, open the new drive in File Explorer and put data of some kind on it (such as a simple text file). See Figure 5-6.

Adding data to the disk

Figure 5-6. Adding data to the disk

Finally, detach the disk from Windows by going back to Computer Management→Storage→Disk Configuration, and then right-clicking on the new disk and selecting Detach VHD (see Figure 5-7).

Detaching the VHD

Figure 5-7. Detaching the VHD

You are now ready to upload the VHD and attach it to the virtual machine!

Execute the script by pressing F5 or by highlighting the script and pressing F8 to start the upload (see Figure 5-8).

Completing the upload

Figure 5-8. Completing the upload

Validating the Disk

If you are following the exercise, the next step is to validate that the disk was actually uploaded and attached to the virtual machine (see Example 5-2). This step just requires you to log in using remote desktop via the Microsoft Azure management portal, or by using the Get-AzureRemoteDesktopFile cmdlet.

Example 5-2. Logging in to validate the attached disk (Console pane)

Get-AzureRemoteDesktopFile -ServiceName $serviceName -Name "ps-vm1" -Launch

When you are logged in, launch File Explorer and browse your disks. You should have a 49 to 50 MB disk attached with a simple file in it (see Figure 5-9).

Validating the uploaded disk

Figure 5-9. Validating the uploaded disk

At this point I like to modify the file with some changes. I do this because the next step is to download the VHD again, and I can validate the changes locally after the download.

Downloading a VHD

Downloading a VHD from Microsoft Azure with PowerShell is very similar to uploading, since the the parameters are reversed from Add-AzureVHD. The source parameter is now the location of the VHD in storage, and the destination is the location on the local file system. To try this out on your own, create a new PowerShell script in the PowerShell ISE named chapter5download.ps1 and add the code shown in Example 5-3.

Example 5-3. Downloading a VHD

$subscription = "[subscription name]"

$storageAccount = "[storage account name]"

Select-AzureSubscription $subscription

$destination = "C:\VHDFiles\mydatadisk_downloaded.vhd"

$source = "https://$storageAccount.blob.core.windows.net/upload/mydatadisk.vhd"

Save-AzureVhd -Source $source -LocalFilePath $destination

Press F5, or highlight the script and press F8 to execute the script (see Figure 5-10).

VHD download complete

Figure 5-10. VHD download complete

When the VHD file is downloaded, you can double-click the downloaded file to mount it in Windows and examine the file for your changes.

DETACHING A VHD

After you have mounted the VHD in Windows, you can’t delete it until you first detach it by using Disk Configuration.

Save-AzureVHD Tips

There are some things you should know about downloading using the Save-AzureVHD cmdlet. First, if you attempt to download a VHD that is actively being written to, chances are very high that the operation will fail due to conflict between the new I/O from the virtual machine and the cmdlet downloading the written bytes.

The second thing you should know is that download is also optimized like the upload. The cmdlet will download only the written bytes of the VHD. However, the cmdlet currently does not support converting to a dynamic VHD on save and saves only to fixed disks. If the disk you are downloading is 1 TB in size but has only 50 GB written to it, the cmdlet will download only 50 GB but will require the full 1 TB of local storage to hold the file.

Similar to Add-AzureVHD, the Save-AzureVHD cmdlet supports the -Overwrite and -NumberOfThreads parameters. This cmdlet also supports a -StorageKey parameter that allows you to specify the authentication key for the storage account (see Table 5-2). If the key is not specified, the currently selected subscription is used to attempt authentication.

Table 5-2. Save-AzureVHD optional parameters

-OverWrite

If the VHD exists on the destination, you can overwrite it by passing this parameter.

-NumberOfThreads

Default value is 8. Change if you feel that more or less parallelization would improve your download.

-StorageKey

The authentication key to the storage account.

Disks and Images

Virtual machine VHD files are referenced indirectly through disks and images in Microsoft Azure. Disks and images are named entities that contain metadata and a link to the underlying VHD.

What Is an Image?

In Microsoft Azure an image is a named entity that is mapped to an operating system disk and optionally a set of data disks. Images are used to instantiate virtual machines with a specific operating system with or without customizations applied. Images can reside in the Microsoft Azure image gallery if they are part of the platform, or they can reside in a storage account in your own subscription if they are your own custom images. Images are designed to be a customized version of what a virtual machine should look like.

There are two types of images that can be created. The original image type is called an OS image. This image type can reference only the operating system disk and works only with images that have been generalized with Sysprep or the Linux Waagent. The second image type is called avirtual machine (VM) image. This image type can reference data disks in addition to the operating system disk and can be generalized or specialized.

A VM image that has been generalized is used for creating multiple virtual machine instances with the same starting point. These virtual machines will be customized with their own computer names at provision time. A specialized VM image is not generalized. This means that it will retain its own identity such as a computer name and even domain-join information. This image type is especially useful for taking backups of a specific virtual machine for development and test, or as a general-purpose backup solution at the VHD level. It is very similar in concept to exporting a virtual machine in Hyper-V. A key difference is that you have to take special care to shut the virtual machine down, because currently the image creation process does not freeze the state of your applications before capture.

To register a VHD file as an image, the VHD file must first be in a Microsoft Azure storage account. Whether it was uploaded there or copied from another location (we’ll talk about this capability a bit later) doesn’t really matter. To register a VHD file located in a storage account, simply call the Add-AzureVMImage cmdlet.

For example, let’s assume that you have previously used the Add-AzureVHD cmdlet to upload the myWindowsImage.vhd file to your storage account in the upload container. The call to Add-AzureVMImage creates the image named MyWinImage that is backed by the file you uploaded, and it is marked as being a Windows-based virtual machine (see Example 5-4). Specifying the OS on the image is important because Linux and Windows have distinct provisioning processes for creating a new virtual machine instance from a generalized image.

Example 5-4. Registering an operating system image

$storageAccount = "[storage account name]"

$source = "https://$storageAccount.blob.core.windows.net/upload/myWindowsImage.vhd"

$imageName = "MyWinImage"

Add-AzureVMImage -ImageName $imageName -MediaLocation $source -OS Windows

What Is a Disk?

A Microsoft Azure disk is another entity mapped to a VHD file in Microsoft Azure. Unlike an image, it is never generalized and it is not required to have an operating system on it, as it could just hold data. A disk with an operating system is called an OS disk, and one is created each time you create a virtual machine from an image. A disk without an operating system in Microsoft Azure is a data disk. These are meant to hold exactly what you would expect to be on a data disk: data.

Example 5-5 is a simple example of how you could register two previously created or uploaded VHD files in your subscription as usable disks. The first call to Add-AzureDisk registers the myosdrive.vhd as an OS disk by specifying the -OS Windows parameter to the cmdlet.

The second call to Add-AzureDisk registers a simple data disk. When execution is complete, this disk could then be attached to a virtual machine at provision time or to a virtual machine that already exists.

Example 5-5. Registering an OS and data disk

$storageAccount = "[storage account name]"

$osdisk = "https://$storageAccount.blob.core.windows.net/upload/myosdrive.vhd"

$osdiskName = "os disk"

$datadisk = "https://$storageAccount.blob.core.windows.net/upload/mydatadrive.vhd"

$datadiskname = "data disk"

# Register the operating system disk

Add-AzureDisk -DiskName $osdiskName -MediaLocation $osdisk -OS Windows

# Register the data disk

Add-AzureDisk -DiskName $datadiskname -MediaLocation $datadisk

Creating a virtual machine from disks is similar to creating a virtual machine from an image, with just a few differences.

The first difference is you need to specify the OS disk name to the -DiskName parameter on New-AzureVMConfig instead of an ImageName. The second difference is when creating a virtual machine from a disk, you do not need to use the Add-AzureProvisioningConfig cmdlet to populate the provisioning configuration. The provisioning configuration data is used only with a generalized image. The third change is regarding attaching the data disk. Instead of using the -CreateNew parameter with Add-AzureDataDisk, you specify -Import and the name of the data disk with the -DiskName parameter (see Example 5-6).

Example 5-6. Creating a virtual machine from a disk

$serviceName = "[cloud service name]"

$location = "[region name]"

$vmName = "migratedVM"

$vmSize = "Small"

$osdiskName = "os disk"

$datadiskname = "data disk"

# Create the configuration specifying the disk name instead of an image

$vmConfig = New-AzureVMConfig -Name $vmName `

-InstanceSize $vmSize `

-DiskName $osdiskName

# Import the data disk to the first LUN

$vmConfig | Add-AzureDataDisk -Import -DiskName $datadiskname -LUN 0

# Create the virtual machine

$vmConfig | New-AzureVM -ServiceName $serviceName -Location $location

Managing Images

Now that you understand the basics of images and disks, I want to dig deeper into each of these entities, starting with images.

There are cmdlets to help you manage images. We have seen one of them in Chapter 3 (Get-AzureVMImage), and we have also just seen how to register an image based on a VHD using the Add-AzureVMImage cmdlet. Let’s investigate some of the additional options you have with managing virtual machine images.

Viewing Image Properties

Running Get-AzureVMImage on its own will give you a list of all of the platform images and user images available to your subscription.

Each image has a list of properties that can be filtered on using PowerShell (see Table 5-3). For custom images that you create, some of these properties can also be set with the Set-AzureVMImage cmdlet.

Table 5-3. Image properties

Category

Public (Microsoft Azure images) or User (your images).

Location

The location(s) where the image is usable.

LogicalSizeInGB

The size of the image (and the size your OS disk will be).

Label

Descriptive label.

MediaLink

Full URI to the underlying VHD. Ths is applicable only to User images.

ImageName

The name of the image. This is the value passed to create a virtual machine.

OS

Whether the OS is Windows or Linux.

OSDiskConfiguration

Configuration of the operating system disk.

DataDiskConfigurations

Configuration of the captured data disks (if any).

Eula

End-user license agreement for the image.

Description

Descriptive text.

ImageFamily

A logical grouping of images (SQL, Server 2012 R2, etc.)

PublishedDate

When the image was made available.

IsPremium

If True, there will be an additional usage charge on top of compute.

IconUri

URI to the icon for the image that will show up in the portal.

PrivacyUri

URI to the privacy policy for the image.

RecommendedVMSize

Recommended virtual machine size for the image.

PublisherName

Name of the group or company that published the image.

Filtering images by using these properties is simple with PowerShell. For instance, if you would like to see all of the images that charge a premium on top of the regular compute hours, run the command shown in Example 5-7.

Example 5-7. Filtering on only premium images

Get-AzureVMImage | where { $_.IsPremium -eq $true }

With PowerShell, you can combine these filters to make more-advanced queries. Example 5-8 shows filtering on only premium images that recommend the A6 instance size.

Example 5-8. Filtering on only premium images and recommended virtual machine size

Get-AzureVMImage |

where { $_.IsPremium -eq $true -and $_.RecommendedVMSize -eq "A6" }

The only value you will use from a virtual machine image to create a virtual machine is the ImageName property. It is useful to know how to select only that property when querying images, as shown in Example 5-9.

Example 5-9. Filtering on only premium images and selecting ImageName

Get-AzureVMImage | where { $_.IsPremium -eq $true } | select ImageName

In Chapter 3 I showed how you can use the filtering capabilities to return the image by publish date so you always get the latest version with the latest patches (see Example 5-10).

Example 5-10. Filtering by image publish date

$imageName = Get-AzureVMImage |

where { $_.ImageFamily -eq $imageFamily } |

sort PublishedDate -Descending |

select -ExpandProperty ImageName -First 1

If you would like to view images that are specialized, you can filter on the OSState property of the OSDiskConfiguration property. This value can be Specialized or Generalized (see Example 5-11).

Example 5-11. Filtering only specialized images

Get-AzureVMImage | where { $_.OSDiskConfiguration.OSState -eq "Specialized" }

Capturing a Generalized Image

Now that you know how to view and filter images, how do you create one of your own? To create a custom image, let’s walk through the steps from the very beginning by creating a new virtual machine, customizing it, and then capturing it as an image.

The image capture process works by starting with a base virtual machine. When the virtual machine is booted, you customize it (install software, make configuration changes, and so on) to make the virtual machine exactly how you want new virtual machines to start.

When the virtual machine is customized, you then run the tool to generalize the operating system. In Windows, you use the Sysprep tool. In Linux, you can use the Microsoft Azure agent (Waagent) to accomplish a similar task. This step prepares the operating system in the same way that OEMs do after installing software on a new computer. When the virtual machine boots up, it will go through the provisioning process just like a new install of Windows would, except your changes will already be on the virtual machine. As you can imagine, this is a fairly destructive process, so read the documentation for any applications you are considering imaging. SQL Server and SharePoint are just two examples of applications that support Sysprep but have very specific instructions on how to accomplish it in a supported manner.

Finally, after you have successfully run Sysprep and the virtual machine is shut down, you can then capture the virtual machine as an image. Capturing a generalized image also deletes the virtual machine instance that was captured, as it no longer has a machine name and requires going through the full provisioning process to be useful again. You now have the saved image stored in a storage account to allow you to easily provision one or more instances of the virtual machine that you captured.

Hopefully, the imaging process is a little more clear to you now if it was not already. With that in mind, let’s use PowerShell to create an image (we’ll cheat ever so slightly by using remote desktop to customize the virtual machine, but I’ll show how you can script that in Chapter 7).

Creating the virtual machine

Using the PowerShell ISE, create a new script named chapter5vmimage.ps1 and add the code shown in Example 5-12. Make sure to replace all of the placeholder values with correct values. For this example, ensure that your cloud service name is globally unique.

Example 5-12. Creating a virtual machine for imaging (Script pane)

$subscription = "[subscription name]"

Select-AzureSubscription $subscription

# Specify the admin credentials

$adminUser = "[admin username]"

$password = "[admin password]"

$location = "[region name]"

$serviceName = "[cloud service name]"

$vmName = "ps-vmImage"

$vmSize = "Small"

$imageFamily = "Windows Server 2012 R2 Datacenter"

$imageName = Get-AzureVMImage |

where { $_.ImageFamily -eq $imageFamily } |

sort PublishedDate -Descending |

select -ExpandProperty ImageName -First 1

New-AzureQuickVM -Windows `

-ServiceName $serviceName `

-Name $vmName `

-ImageName $imageName `

-Location $location `

-AdminUsername $adminUser `

-Password $password `

-InstanceSize $vmSize `

-WaitForBoot

In this example I have also introduced the -WaitForBoot flag. This flag ensures that New-AzureQuickVM (or New-AzureVM) does not return until the virtual machine is in the ReadyRole state. This makes it very convenient to know when you can log in to the virtual machine or execute a script remotely to customize the virtual machine.

Customizing the virtual machine

Next, run the code shown in Example 5-13 to launch a remote desktop session into the virtual machine.

Example 5-13. Launching a Remote Desktop into the virtual machine for customization (Console pane)

Get-AzureRemoteDesktopFile -ServiceName $serviceName -Name $vmName -Launch

When prompted, enter the username and password specified in your script to log in. When you are logged in to the newly created virtual machine, click the Windows PowerShell icon on the task bar.

The whole point of creating an image is that you would like the ability to create multiple instances of a virtual machine that has a preexisting configuration on it. In this case, we can install IIS so that we can easily deploy a web server using the custom image.

Launch PowerShell, and at the PowerShell command line on the virtual machine, execute the command shown in Example 5-14 to install IIS (see Figure 5-11).

Example 5-14. Installing IIS on the new virtual machine (Console pane)

Add-WindowsFeature -Name "Web-Server" -IncludeAllSubFeature -IncludeManagementTools

Installing IIS using PowerShell

Figure 5-11. Installing IIS using PowerShell

When all customizations for your image have been made (which in this case is just installing the Web-Server feature), the next step is to run Sysprep.

Running Sysprep on the virtual machine

Within the virtual machine, press Windows Key + R and enter C:\Windows\System32\Sysprep\sysprep.exe as the command to run, and press Enter. See Figure 5-12.

Starting Sysprep

Figure 5-12. Starting Sysprep

When the Sysprep wizard is displayed on the screen, as shown in Figure 5-13, check the Generalize checkbox and change Shutdown Options to Shutdown, and then click OK.

Configuring Sysprep

Figure 5-13. Configuring Sysprep

The system preparation process can take several minutes to complete. Eventually, your remote desktop session will be stopped (this is supposed to happen), and the next step is to wait until the virtual machine is in the StoppedVM state. You can check this by running the Get-AzureVMcmdlet to view the virtual machine’s status.

Capturing the virtual machine image

When the virtual machine is stopped, you can capture the image by using the Microsoft Azure management portal or with PowerShell.

Run the code shown in Example 5-15 in the console of the PowerShell ISE. I have split the call to Save-AzureVMImage across multiple lines for clarity, but it can be executed on one line by removing the line continuation character (`). This example creates a new virtual machine image and saves it in the storage account where the virtual machine disks are currently located.

Example 5-15. Capturing and setting properties of an image (Console pane)

$imageName = "WEBSERVERIMAGE"

$imageLabel = "Image with IIS Pre-Installed"

Save-AzureVMImage -ServiceName $serviceName `

-Name $vmName `

-NewImageName $imageName `

-NewImageLabel $imageLabel `

-OSState Generalized

Provisioning a virtual machine from your own image

To create a virtual machine using this new image, create a new script using the PowerShell ISE named chapter5createfromimage.ps1 and enter the code shown in Example 5-16. Ensure that you replace the placeholder values with real ones from your subscription.

Example 5-16. Creating a virtual machine from the image (Script pane)

$subscription = "[subscription name]"

$location = "[region name]"

$serviceName = "[cloud service name]"

# Specify the admin credentials

$adminUser = "[admin username]"

$password = "[admin password]"

Select-AzureSubscription $subscription

$vmSize = "Small"

$vmName = "ps-webserver1"

$imageName = "WEBSERVERIMAGE"

New-AzureQuickVM -Windows `

-ServiceName $serviceName `

-Name $vmName `

-ImageName $imageName `

-Location $location `

-AdminUsername $adminUser `

-Password $password `

-InstanceSize $vmSize

Updating a Virtual Machine Image

When the image is created, there are two pieces of the image that you may want to modify in the future.

The first is the properties of the virtual machine image entity in Microsoft Azure. To update the properties, use the Update-AzureVMImage cmdlet along with the image name.

To try this on your own, create a new script called chapter5updateimage.ps1 and add the code shown in Example 5-17.

Example 5-17. Updating a virtual machine image (Script pane)

$subscription = "[subscription name]"

Select-AzureSubscription $subscription

$imageName = "WEBSERVERIMAGE"

Update-AzureVMImage -ImageName $imageName `

-Description "A Pre-Built web server" `

-ImageFamily "My Company Custom Images - IIS" `

-Label "Image with IIS Pre-Installed"

Execute the script by pressing F5, or by highlighting the text and pressing F8 to update the image.

Execute the Get-AzureVMImage cmdlet to verify your changes (see Example 5-18).

Example 5-18. Verifying the properties (Console pane)

$imageName = "WEBSERVERIMAGE"

Get-AzureVMImage -ImageName $imageName

The second type of update is to update the underlying image bits themselves. This can be accomplished in one of two ways. The first and likely the simplest way is to create a new virtual machine instance from the image, and then update the virtual machine by applying patches and software updates. When the virtual machine instance is updated, run Sysprep again and capture the image with a new name.

The alternative method assumes that you have created the image offline and uploaded it using the Add-AzureVHD cmdlet. If you have, then you can create differencing disks offline with the updates and upload only the differencing disks and have them applied to the base image using theAdd-AzureVHD cmdlet and specifying the image URL with the -BaseImageUriToPatch parameter.

Deleting a Virtual Machine Image

Deleting a custom image is straightforward with the Remove-AzureVMImage cmdlet. Remove-AzureVMImage accepts two parameters: the -ImageName and, optionally, the -DeleteVHD parameter.

Example 5-19 will delete the custom virtual machine image just created and, with the -DeleteVHD parameter added, will also delete the underlying VHD in your storage account.

Example 5-19. Deleting a custom image (Console pane)

Remove-AzureVMImage -ImageName $imageName -DeleteVHD

OS Images and VM Images

The original imaging technology that was released when Microsoft shipped virtual machines is known as OS images. Since then, a new imaging architecture has been released and this new architecture is known as VM images. The biggest difference between OS images and VM images is that VM images can also capture data disks. The second difference is that VM images also support capturing specialized images (see Table 5-4). This support for specialized images means you can create a virtual machine and capture it without running Sysprep or Waagent (Linux), and the OS disk and data disks will be captured as is, including the unique identity of the VM.

Table 5-4. Image type comparison

OS image

VM image

OS disk only

OS and data disks

Generalized only

Generalized or specialized

The Save-AzureVMImage cmdlet can be used to create each image type.

To create an OS image, simply omit the -OSState parameter, and the image will be created as an OS image, as shown in Example 5-20. This is an important point to remember, because if you were anticipating capturing data disks, you will be disappointed if you forget to specify the -OSState parameter.

Example 5-20. Creating an OS image (no data disks included)

$serviceName = "[cloud service name]"

$vmName = "[virtual machine name]"

$imageName = "[image name]"

$imageLabel = "[image label]"

Save-AzureVMImage -ServiceName $serviceName `

-Name $vmName `

-NewImageName $imageName `

-NewImageLabel $imageLabel

ACCIDENTLY LOSING DATA DISKS

Calling Save-AzureVMImage without the -OSState parameter reverts to using the OS image type, which does not support data disks.

Specifying the -OSState Generalized parameter creates a VM image, and if there are data disks associated with the virtual machine, they will be captured as well (see Example 5-21). Using this parameter assumes that the virtual machine has been generalized with Sysprep on Windows or the Microsoft Azure Linux agent (Waagent).

Example 5-21. Creating a generalized virtual machine image

$serviceName = "[cloud service name]"

$vmName = "[virtual machine name]"

$imageName = "[image name]"

$imageLabel = "[image label]"

Save-AzureVMImage -ServiceName $serviceName `

-Name $vmName `

-NewImageName $imageName `

-NewImageLabel $imageLabel `

-OSState Generalized

Specifying the -OSState Specialized parameter creates a VM image, and if there are data disks associated with the virtual machine, they will be captured as well (see Example 5-22). Using this parameter assumes that the virtual machine has not been generalized with Sysprep on Windows or the Azure Linux agent (Waagent). This scenario is great for capturing a virtual machine exactly how it should be when deployed. An example is creating an image of the virtual machine before applying updates or an application deployment. If the update or deployment fails, you can delete the virtual machine and re-create it from the image, and the state will be exactly as it was before provisioning.

Example 5-22. Creating a specialized virtual machine image

$serviceName = "[cloud service name]"

$vmName = "[virtual machine name]"

$imageName = "[image name]"

$imageLabel = "[image label]"

Save-AzureVMImage -ServiceName $serviceName `

-Name $vmName `

-NewImageName $imageName `

-NewImageLabel $imageLabel `

-OSState Specialized

SHUTDOWN FOR A CLEAN IMAGE CAPTURE

Before creating a specialized image, it is a good idea to shut the virtual machine down first to ensure that all writes to the disks are flushed before capture. When creating a generalized image, the VM is already shut down due to the generalization process. With the specialization option, it is up to you whether you want to shut down the virtual machine first.

Managing Disks

As you now know, a disk is similar in concept to an image. It is an entity in Microsoft Azure that is backed by a VHD file in a storage account. The VHD file can be a bootable OS disk with a supported version of Windows or Linux on it or it can be a simple data disk that contains only data.

When you create a virtual machine from an image, a copy of that image is made to the target location you specified for your virtual machine, and this becomes the OS disk.

Just like images, there are additional cmdlets to help you manage your disks. You have already seen how to attach a new disk to a virtual machine by using Add-AzureDataDisk with the -CreateNew parameter, and you have seen how to register a VHD file to a Microsoft Azure disk by using the Add-AzureDisk cmdlet. In this section we will dig deeper and show more scenarios for how disks can be managed using PowerShell.

OS Disks

An OS disk is simply a disk that contains a bootable operating system on it, and the OS property of the disk entity is set to Windows or Linux. An OS disk is created each time you create a virtual machine from an image or when you add an existing VHD to storage and register it by calling the Add-AzureDisk cmdlet with the -OS parameter specified.

In this partial example, shown in Example 5-23, the $mediaLocation variable points to a previously uploaded VHD in a storage account. To make this VHD usable to a virtual machine, the Add-AzureDisk cmdlet is used to create the disk entity that is mapped to the underlying VHD in storage. The -OS Windows parameter is also specified to register the disk as a Windows OS disk.

Example 5-23. Adding an OS disk

$mediaLocation = "https://$storageAccount.blob.windows.net/upload/myosdisk.vhd"

Add-AzureDisk -DiskLabel "My OS disk" -MediaLocation $mediaLocation -OS Windows

Data Disks

A data disk is similar to an OS disk except that it does not have an OS property specified. Microsoft Azure uses this property to decide whether the disk can be booted or just attached to a virtual machine.

As we have seen in Chapter 3, using the PowerShell cmdlets makes it easy to create additional empty disks to a virtual machine with up to 1023 GB in storage.

In this partial example, shown in Example 5-24, the $vmConfig variable is a virtual machine configuration object. It is piped to the Add-AzureDataDisk cmdlet, which has the -CreateNew parameter specified. The -CreateNew parameter makes the Add-AzureDataDisk cmdlet populate the configuration object with properties that, when sent to the Microsoft Azure API, will create new empty disks on the virtual machine.

Example 5-24. Attaching new data disks to a virtual machine configuation

$vmConfig | Add-AzureDataDisk -CreateNew `

-DiskSizeInGB 50 `

-DiskLabel "data 1" `

-LUN 0

$vmConfig | Add-AzureDataDisk -CreateNew `

-DiskSizeInGB 50 `

-DiskLabel "data 2" `

-LUN 1

Besides creating a new disk with the -CreateNew parameter, you can also attach existing disks by using the -Import or -ImportFrom parameters. This can be a disk that you have uploaded previously or other data disks that were created in Azure on other virtual machines. One thing to remember is that a disk cannot be attached to multiple virtual machines at the same time.

In this partial example, shown in Example 5-25, a data disk is created with the Add-AzureDisk cmdlet using the MyDataDisk name and the URL to an existing VHD in an Azure storage account. The -Import parameter of Add-AzureDataDisk allows you to just specify the disk name and attach it to a virtual machine.

Example 5-25. Importing a data disk by name

$mediaLocation = "https://$storageAccount.blob.windows.net/upload/mydatadisk.vhd"

Add-AzureDisk -DiskName "MyDataDisk" -MediaLocation $mediaLocation

$vmConfig | Add-AzureDataDisk -Import `

-DiskName "MyDataDisk" `

-LUN 0

Example 5-26 shows that it is also possible to specify an existing VHD in a storage account that is in the same subscription and same region by using the Add-AzureDataDisk cmdlet with the -ImportFrom parameters parameter. What does -ImportFrom actually mean? When you specify the -ImportFrom parameter and a storage account URI to a VHD, the cmdlet will automatically register the VHD as a disk and add it to your virtual machine configuration. This is a shortcut that is the same as calling Add-AzureDisk to register the disk and using the -Importparameter to add it to the virtual machine configuration at the same time.

Example 5-26. Importing a VHD as a data disk

$mediaLocation = "https://$storageAccount.blob.windows.net/upload/mydatadisk.vhd"

$vmConfig | Add-AzureDataDisk -ImportFrom `

-DiskLabel "imported vhd" `

-LUN 0 `

-MediaLocation $mediaLocation

Viewing Disk Properties

To view disks in your Azure subsription or to view the properties of a specific disk (listed in Table 5-5), you can use the Get-AzureDisk cmdlet. Running Get-AzureDisk with no parameters will return all of the disks in your subscription (see Example 5-27).

Example 5-27. Viewing disk properties (Console pane)

Get-AzureDisk

Table 5-5. Disk properties

AffinityGroup

The affinity group (if any) of the parent storage account.

AttachedTo

The virtual machine that the disk is attached to (null if not attached).

IsCorrupted

Whether corruption has been detected on the VHD.

Label

Descriptive label.

Location

The Microsoft Azure region (if not affinity group) of the parent storage account.

DiskSizeInGB

The size, in gigabytes, of the disk.

MediaLink

Full URI to the underlying VHD.

DiskName

The name of the disk. This is the value passed to any disk-related API.

SourceImageName

Source image name if the disk was created by an image.

OS

Windows or Linux if an OS disk or null if a data disk.

Get-AzureDisk also supports specifying just the -DiskName parameter if you would like to return only properties of a specific disk.

As with images, you can filter on the properties of disks. Example 5-28 calls Get-AzureDisk to return all of the disks from your subscription. The output of Get-AzureDisk is piped to the where command, which filters the results further by including only disks where the AttachedToproperty equals null (not mounted to a virtual machine) and OS equals null (not an operating system disk).

Example 5-28. Viewing only unattached data disks (Console pane)

Get-AzureDisk | where { $_.AttachedTo -eq $null -and $_.OS -eq $null }

Example 5-29 is another example of using the PowerShell filtering abilities. The commands are almost identical, except for filtering by a different value of the operating system. These commands may or may not return any data, depending on whether you have unattached OS disks for Windows or Linux in your current subscription.

Example 5-29. Viewing by operating system (Console pane)

Get-AzureDisk | where { $_.AttachedTo -eq $null -and $_.OS -eq "Windows" }

Get-AzureDisk | where { $_.AttachedTo -eq $null -and $_.OS -eq "Linux" }

Specifying Disk Locations at VM Creation

Under the covers, the PowerShell cmdlets automatically set the location of the disk to be in the storage account you specified with the -CurrentStorageAccountName parameter of the Set-AzureSubscription cmdlet and the default container vhds. Using the PowerShell cmdlets, you have the power to override this default behavior and have full control over the location and filenames of the VHDs, including the OS disk, by specifying the -MediaLocation parameter.

Example 5-30 is a full example. I am using the -MediaLocation parameter to specify the location of the underlying disks of a virtual machine when it is created. The -MediaLocation parameter is specified in the call to New-AzureVMConfig to override the location of the operating system disk, and each call to Add-AzureDataDisk specifies the location of each data disk.

If you would like to try this example on your own, create a new PowerShell script using the PowerShell ISE named chapter5medialocation.ps1.

Example 5-30. Creating a virtual machine with data disks (Script pane)

$subscription = "[subscription name]"

Select-AzureSubscription $subscription

$location = "[region name]"

$serviceName = "[cloud service name]"

# Specify the admin credentials

$adminUser = "[admin username]"

$password = "[admin password]"

$storageAccount = (Get-AzureSubscription -Current).CurrentStorageAccountName

$vmName = "ps-vmDisks"

$vmSize = "Small"

$imageFamily = "Windows Server 2012 R2 Datacenter"

$imageName = Get-AzureVMImage |

where { $_.ImageFamily -eq $imageFamily } |

sort PublishedDate -Descending |

select -ExpandProperty ImageName -First 1

# Custom location and filenames

$oslocation = "https://$storageAccount.blob.core.windows.net/custom/osdisk.vhd"

$d1Location = "https://$storageAccount.blob.core.windows.net/custom/data1.vhd"

$d2Location = "https://$storageAccount.blob.core.windows.net/custom/data2.vhd"

$vmConfig = New-AzureVMConfig -Name $vmName `

-InstanceSize $vmSize `

-ImageName $imageName `

-MediaLocation $oslocation

$vmConfig | Add-AzureProvisioningConfig -Windows `

-AdminUsername $adminUser `

-Password $password

$vmConfig | Add-AzureDataDisk -CreateNew `

-DiskSizeInGB 50 `

-DiskLabel "data 1" `

-LUN 0 `

-MediaLocation $d1Location

$vmConfig | Add-AzureDataDisk -CreateNew `

-DiskSizeInGB 50 `

-DiskLabel "data 2" `

-LUN 1 `

-MediaLocation $d2Location

$vmConfig | New-AzureVM -ServiceName $serviceName -Location $location

Press F5, or highlight the script and press F8 to create the virtual machine. You can then use the Get-AzureVM cmdlet to return the virtual machine configuration and pipe it to Get-AzureOSDisk or Get-AzureDataDisk to view the custom media location (see Figures 5-14 and 5-15).

Custom OS disk media location

Figure 5-14. Custom OS disk media location

Custom data disk media location

Figure 5-15. Custom data disk media location

Specifying Cache

By default, ReadWrite caching is enabled on the OS disk of a virtual machine for performance. This means that most reads and writes to the C: drive are cached on the physical disk that the virtual machine is hosted on before being committed to the disk backed in Microsoft Azure storage. You can change the setting of the OS disk during disk creation by specifying the -HostCaching parameter to New-AzureVMConfig, or use the Set-AzureOSDisk cmdlet for changing the cache setting on an existing virtual machine’s OS disk.

OS DISK CACHE SETTINGS

Supported -HostCaching values for the OS disk are ReadOnly and ReadWrite.

Up to four data disks per virtual machine can be enabled for disk caching. The default cache setting of a data disk is None and is most likely the safest bet if you are actively writing data to the disk. However, some workloads can be optimized by enabling local read or write cache on the disk. During creation, you can specify the cache settings of a data disk with the -HostCaching flag of the Add-AzureDataDisk cmdlet, or use the Set-AzureDataDisk cmdlet to change the cache configuration on an existing virtual machine.

DATA DISK CACHE SETTINGS

Supported -HostCaching flag values for the OS disk are None, ReadOnly, and ReadWrite.

Example 5-31 is a partial example that shows how you can specify the -HostCaching parameter during creation to change the default cache behavior of OS and data disks.

Example 5-31. Specifying disk HostCaching during creation

# -HostCaching on New-AzureVMConfig refers to the OS disk setting

$vmConfig = New-AzureVMConfig -Name $vmName `

-InstanceSize $vmSize `

-ImageName $imageName `

-HostCaching ReadOnly

$vmConfig | Add-AzureProvisioningConfig -Windows `

-AdminUsername $adminUser `

-Password $password

$vmConfig | Add-AzureDataDisk -CreateNew `

-DiskSizeInGB 50 `

-DiskLabel "data 1" `

-LUN 0 `

-HostCaching ReadWrite

$vmConfig | Add-AzureDataDisk -CreateNew `

-DiskSizeInGB 50 `

-DiskLabel "data 2" `

-HostCaching ReadWrite

-LUN 1

Example 5-32 is a partial example that shows changing the -HostCaching property of an existing virtual machine’s disks using the Set-AzureOSDisk and Set-AzureDataDisk cmdlets.

Example 5-32. Specifying HostCaching properties on an existing virtual machine

$vmConfig = Get-AzureVM -ServiceName $serviceName -Name $vmName

$vmConfig | Set-AzureOSDisk -HostCaching ReadOnly

$vmConfig | Set-AzureDataDisk -LUN 0 -HostCaching ReadWrite

$vmConfig | Set-AzureDataDisk -LUN 1 -HostCaching ReadWrite

$vmConfig | Update-AzureVM

Custom Images, Disks, and Storage Accounts

Before completing this section of the chapter, I want to highlight a few common pitfalls that occur with cloud services, images, disks, and storage accounts.

When creating a virtual machine using your own custom OS image, it is a requirement that the CurrentStorageAccountName property of your subscription is set to the same storage account where the image is located. This applies even if they are in the same region. Remember, you set this property by using the Set-AzureSubscription cmdlet passing the -CurrentStorageAccountName property.

If you do not follow this rule, you will receive the error:

BadRequest: The disk’s VHD must be in the same account as the VHD of the source image (source account: storageaccount1.blob.core.windows.net, target account: storageaccount2.blob.core.windows.net).

If you are using a VM image as the source, the behavior is slightly different. Even though the CurrentStorageAccountName is not the same location as the VM image location, the cmdlet will succeed. However, the newly created virtual machine’s disks will be created in the storage account that the user image is in and not the storage account specified in CurrentStorageAccountName.

Another common error that you will receive is when provisioning virtual machines where the image or disk is in one region and the location of the cloud service is in another.

The error for this common problem is as follows:

BadRequest: The location or affinity group North Europe of the storage account where the source image CUSTOMIMAGE resides is not in the same location or affinity group as the specified cloud service. The source image must reside in a storage account that has the same affinity group or location as the cloud service West US.

The resolution to all of these problems is to ensure that all of your assets are in the same location. The cloud service container must always be in the same region as your storage. If you are provisioning from a custom image, the CurrentStorageAccountName should match the same region and storage account as your source image.

Managing Storage with PowerShell

You have already seen several of the cmdlets for managing blob storage in Microsoft Azure. While this book is mainly focused on managing infrastructure services, there are some key things to know about managing the underlying system where the virtual machine disks are actually stored.

In addition to the storage account, container, and blob cmdlets, there are also cmdlets that allow you to configure Microsoft Azure storage metrics to monitor storage usage. There are also cmdlets to directly manage tables, queues, and Azure File Services (which is in preview at the time of this writing).

If you would like to see all of the Microsoft Azure storage-related cmdlets, you can use the PowerShell Get-Command cmdlet and filter the Name property for AzureStorage, as shown in Example 5-33.

Example 5-33. Viewing all Microsoft Azure storage cmdlets

Get-Command | Where { $_.Name -like "*AzureStorage*" }

In the remainder of this chapter I want to focus on some of these cmdlets that specifically impact running and managing virtual machines.

Storage Account Geo-Replication

In Chapter 3 we discussed the basics of creating a Microsoft Azure storage account by using the New-AzureStorageAccount cmdlet. There are some items that merit further review.

Enabling geo-replication on a storage account tells Microsoft Azure to replicate all of the data in the storage account to a remote Microsoft Azure region. Each Microsoft Azure region has a failover region that is used for such purposes. For example, a storage account in the East US region with geo-replication enabled will asynchronously copy all of its data to the West US region.

This is meant to ensure a higher level of durability. Microsoft Azure storage already makes three copies of each blob within the same region. Enabling geo-replication enables an additional three copies of the blob to be replicated to the failover region for additional durability. A storage account with geo-replication enabled gives you six copies of every blob in the storage account. This means there is very little chance that you will have significant data loss due to a failure in the data center.

To enable or disable geo-replication from PowerShell, you can use the Set-AzureStorageAccount cmdlet (see Example 5-34).

Example 5-34. Modifying geo-replication for a storage account

Set-AzureStorageAccount -StorageAccountName $storageAccount `

-GeoReplicationEnabled $true

There are a few reasons why you would want to disable geo-replication on a storage account.

The first should be obvious, and that is cost. Having data stored in multiple regions is going to cost more than storing the data in just a single region. If you are storing non-critical data that does not need the assurance of being replicated to a remote region, then you should disable this option.

The second reason is that not all workloads support geo-replication. For instance, SQL Server does not support having data and transaction logs on separate disks on a geo-replicated storage account. Geo-replication is completely asynchronous. This means that the writes are not guaranteed to be written in order, which makes a transactionally consistent database difficult to restore in the event of a data center failover.

Authenticating Access to Storage

So far, from a security perspective, we have only discussed using the Select-AzureSubscription cmdlet and a brief mention of shared access signatures. The Select-AzureSubscription cmdlet tells the other cmdlets which subscription to use and, more important, which credentials to use to perform the operation. The current implementation of this approach implies that you are either an administrator or a co-administrator of the Microsoft Azure subscription.

However, there are times when you need to directly authenticate against storage instead of using authentication at the subscription level. A good example is when you need to access a storage account, container, or just a file in a container and you are not an administrator on the Microsoft Azure subscription itself.

There are two methods for authenticating access to storage.

The first method is to use a combination of the storage account name and one of the storage account authentication keys. The second method is generating a shared access signature and passing that to a lower privilege client to use for authentication.

The differences between the two methods is significant. With the storage account name and key, you have full access to everything in the storage account and can perform all operations on its contents. With a shared access signature, the level of access and even the duration of access is defined during the creation of the shared access signature itself. Using a shared access signature is a two-step process. The first step is to generate the SAS token, and this does require full rights on the storage account. The second step is to actually use the SAS token.

Storage context objects

Many of the operations regarding storage accept a storage context parameter. The storage context is simply a data type that stores the authentication method and the credentials used for whatever storage operation you wish to perform. This can either be the storage account name and key or a generated SAS token.

In Example 5-35 I’m using the Get-AzureStorageKey cmdlet to retrieve the primary authentication key value and store it in the $storageKey variable. I then pass it along with the name of the storage account itself to the New-AzureStorageContext cmdlet. This cmdlet creates the context object that can be used by other storage cmdlets to authenticate their operations.

Accessing the storage account key by using the Get-AzureStorageKey cmdlet requires Microsoft Azure administrative access. It is possible to retrieve this key as an administrator and pass it to less-elevated code or users. Just remember that whoever has access to the name and key can perform any operation on the contents of the storage account.

Example 5-35. Creating a storage context object using the storage account name and key

$storageAccount = "[storage account name]"

$storageKey = (Get-AzureStorageKey -StorageAccountName $storageAccount).Primary

$context = New-AzureStorageContext -StorageAccountName $storageAccount `

-StorageAccountKey $storageKey

Passing the $context object to the Get-AzureStorageContainer cmdlet tells the cmdlet to use the specified storage account and key, whether your local PowerShell configuration has access to the subscription or not (see Example 5-36). You can think of it as an authentication override for storage. If you do not specify a context object to the storage cmdlets, they will assume you want to work on the storage account specified in the CurrentStorageAccountName property of your subscription (remember, you can change this setting with the Set-AzureSubscriptioncmdlet).

Example 5-36. Using the context object with Get-AzureStorageBlob

$container = "vhds"

Get-AzureStorageBlob -Context $context -Container $container

The alternative to passing out the storage account name and key (and full permissions to your storage account) is to generate a shared access signature instead (see Example 5-37). This type of access allows for much finer-grained permissions, duration of permissions, and is also revokable.

To create the shared access signature in the first place, you do need full access to the storage account (see Table 5-6).

Example 5-37. Creating a storage context object using a shared access signature

$sas = New-AzureStorageContainerSASToken -Name $container `

-Permission rwdl `

-Context $context

Table 5-6. Container permissions for the -Permission parameter

Permission

Symbol

Description

Read

r

Read the content, properties, metadata, or block list of any blob in the container. Use any blob in the container as the source of a copy operation.

Write

w

For any blob in the container, create or write content, properties, metadata, or block list. Snapshot or lease the blob. Resize the blob (page blob only). Use the blob as the destination of a copy operation within the same account.

Delete

d

Delete any blob in the container.

List

l

List blobs in the container.

For more details on blob and container permissions, see the following article in MSDN: http://bit.ly/shared_access_sig_URI.

The New-AzureStorageContainerSASToken cmdlet creates the actual shared access signature token. You can specify a start time and an end time for how long the token is valid and, optionally, you can specify a previously created Shared Access Signature Policy from which to base the token settings.

SHARED ACCESS SIGNATURE POLICIES

At the time of this writing, you can only reference an existing shared access signature policy but you cannot create one using the PowerShell cmdlets. Of course, you can call .NET, which would allow you to call in the Storage Client Libraries where you have full access to create policies.

When the token has been created, it can be passed to a client and used based on the access that has been granted. To use this token from the Microsoft Azure PowerShell cmdlets, you need to create a storage context object. This time, instead of passing in the the storage account name and key, you will pass in the storage account name and SAS token (see Example 5-38).

When the context object is created, you pass it as a parameter or through the pipeline to any Azure storage cmdlet that accepts a Context object as a parameter.

Example 5-38. Using a SAS token from PowerShell

$sascontext = New-AzureStorageContext -StorageAccountName $storageAccount `

-SasToken $sas

Get-AzureStorageBlob -Context $sascontext -Container $container

Setting the Public Access Policy for a Container

The final security topic I want to mention is setting the public access policy for a storage container. Each container within a Microsoft Azure storage account can have an access policy for nonauthenticated (public) requests (see Table 5-7). By default this policy is Off on each container but can be changed using the Set-AzureStorageContainerAcl cmdlet (see Example 5-39).

Table 5-7. Container access policies

Off

No anonymous access.

Container

Allows read-only access to the blobs in the container and allows the blobs to be enumerated.

Blob

Allows read-only access to the blobs in the container. The requestor needs to know the full path to the blob in question.

Example 5-39. Changing the public access policy of a container

$storageAccount = "[storage account name]"

$storageKey = (Get-AzureStorageKey -StorageAccountName $storageAccount).Primary

$context = New-AzureStorageContext -StorageAccountName $storageAccount `

-StorageAccountKey $storageKey

# Create a new storage container with Blob public access

New-AzureStorageContainer -Name "newcontainer" -Permission Blob -Context $context

# Modify the container to have the Container public access instead

Set-AzureStorageContainerAcl -Name "newcontainer" `

-Permission Container `

-Context $context

Managing Blob Data

So far I have skirted around managing blob data directly. Let us tackle it head-on in this section.

You have already seen how to create a container by using the New-AzureStorageContainer cmdlet. Let’s work off of that knowledge and fill in some of the gaps.

For instance, how do you enumerate all of the existing containers in your storage account? Before creating a new container, how do you test whether the container exists already? How do you delete a container? These are all very good questions that we can quickly address with some examples.

Enumerating containers (see Example 5-40) will show you the current public access policy and the modification date of the container (see Figure 5-16). You can pipe the container output to the Get-AzureStorageBlob container and enumerate all of the files in the storage account as well (see Example 5-41 and Figure 5-17).

Example 5-40. Enumerating containers

Get-AzureStorageContainer

Enumerating containers in a storage account

Figure 5-16. Enumerating containers in a storage account

Example 5-41. Enumerating containers and blobs

Get-AzureStorageContainer | Get-AzureStorageBlob

Enumerating blobs in a storage account

Figure 5-17. Enumerating blobs in a storage account

One of the more useful storage-related tasks is scripting the upload or download of blob data to or from a Microsoft Azure storage account. The cmdlets make this a fairly painless process while at the same time providing you all of the power you would expect as a PowerShell user by supporting the PowerShell pipeline operator. This allows you to enumerate multiple files locally or remotely, and process them as a batch.

You can, of course, take the output of Get-AzureStorageBlob and process it via the pipeline too. Passing the output to Remove-AzureStorageBlob to delete files or to the Get-AzureStorageBlobContent file to download the files locally are two examples of processing you could do with the pipeline output of Get-AzureStorageBlob.

In Example 5-42 I am using the Get-AzureStorageContainer output and piping it to Get-AzureStorageBlob. From there, the output is passed to the PowerShell for-each command that allows me to execute a script block for each blob returned. The script block is simple; it creates a local path using the Join-Path cmdlet, the C:\Temp directory, and the name of the file (see Figure 5-18).

The returned blob information $_ is piped to the Get-AzureStorageBlobContent cmdlet, which does the heavy lifting of actually downloading the files from the videoexample container. Without specifying the videoexample container, this same code could download all of the files in the storage account. It would need a little help to account for creating local folders, but the code change would be minimal.

Example 5-42. Downloading files from a container

Get-AzureStorageContainer -Name "videoexample" | Get-AzureStorageBlob | foreach {

$localPath = Join-Path "C:\Temp" $_.Name

$_ | Get-AzureStorageBlobContent -Destination $localPath

}

Downloading multiple files from storage

Figure 5-18. Downloading multiple files from storage

Uploading files from the local hard drive to storage is just as simple. In Example 5-43 there are two things going on in order to upload files.

The call to Get-AzureStorageContainer is also passed the -ErrorAction SilentlyContinue in order to test whether the storage container exists or not. If the specified container does not exist, the cmdlet returns $null and the container is created in the New-AzureStorageContainer call.

When the container has been verified to exist, the script uses the Get-ChildItem cmdlet to enumerate all of the files in the C:\AdventureWorksDB folder on the C: drive. The path of each file and the name of the container is then passed to the Set-AzureStorageBlobContent cmdlet, which does the heavy lifting of uploading the files. Figure 5-19 shows the upload status using PowerShell ISE.

Example 5-43. Uploading files to a container

$localPath = "C:\AdventureWorksDB"

$container = "adventureworks"

$existingContainer = Get-AzureStorageContainer -Name $container `

-ErrorAction SilentlyContinue

If($existingContainer -eq $null)

{

New-AzureStorageContainer -Name $container

}

Get-ChildItem $localPath | foreach {

Set-AzureStorageBlobContent -File $_.FullName -Container $container

}

Uploading multiple files to storage

Figure 5-19. Uploading multiple files to storage

AUTHENTICATION WITHOUT PASSING CONTEXT

You have probably noticed in the previous examples that I have not passed a storage context to any of these calls to storage. When the storage cmdlets are called and the -Context parameter is not specified, the cmdlets will use the currently selected subscription and the CurrentStorageAccountName assocated with it. This property is set by using the Set-AzureSubscription cmdlet.

Asynchronous Blob Copy

The final subject regarding Microsoft Azure storage that I want to cover in this chapter is using the asynchronous blob copy cmdlets. From the perspective of an infrastructure services administrator, this is one of the more critical features of the Microsoft Azure PowerShell cmdlets.

The asynchronous blob copy service allows you to initiate a copy from a source location to a destination location. The beauty is that the source location does not have to be local. You can specify any URL that the copy service itself can access. This could be another Microsoft Azure storage account or just a file on a website. When the copy is initiated, the blob copy service will copy the file from the source to the destination without using your local machine as a middle man.

This opens several scenarios for infrastructure services. You can copy a virtual machine’s underlying disks between regions and even between subscriptions. For example, if you have a functioning environment running in the West US region and you would like to copy or move it to East US or even East Asia, you can, all through PowerShell!

Before I dive into the practical aspects of how to do these types of copies in PowerShell, there are some things that you should know about how Azure storage works. For the examples in the table below, I am using four storage accounts.

If I ping each storage account using the full Domain Name System (DNS) name, I can see that opsgilitywest1 and opsgilitywest2 share the same IP address, and that opsgilitywest0 and opsgilityeast1 do not match opsgilitywest1 or opsgilitywest2.

Storage account

FQDN

IP address

opsgilitywest0

opsgilitywest0.blob.core.windows.net

168.62.0.14

opsgilitywest1

opsgilitywest1.blob.core.windows.net

168.63.89.142

opsgilitywest2

opsgilitywest2.blob.core.windows.net

168.63.89.142

opsgilityeast1

opsgilityeast1.blob.core.windows.net

138.91.96.142

Within each Azure region, storage is further subdivided into storage stamps. When you create a storage account, Azure places that storage account in one of the stamps for that region.

How do you know if two storage accounts in the same region are actually in the same stamp? Simply ping the name, and if the IP address returned for each storage account is the same, then they are in the same stamp. In Figure 5-20, the storage accounts opsgilitywest1 and opsgilitywest2 are in the same stamp, but not in the same stamp as opsgilitywest0, even though they are all in the same region.

Storage stamps do not span regions, so opsgilityeast1 is in a completely separate region and stamp than the storage accounts in the West US region (see Figure 5-21).

Why does all of this matter? If you just want to copy files from PowerShell, it shouldn’t matter, right? It matters because copying blob data between storage accounts in the same stamp is ridiculously fast when copying files between storage accounts even in the same region, but copying between different stamps is not so fast. Of course, copying files between regions is going to be slowest of all and will vary based on the distance the regions are from each other.

Stamps within the same region

Figure 5-20. Stamps within the same region

Stamps in separate regions

Figure 5-21. Stamps in separate regions

Table 5-8 shows some example copies from testing the copy of a 127 GB OS disk between storage accounts in Microsoft Azure.

Table 5-8. Blob copy time examples

Source

Destination

Result

opsgilitywest1

opsgilitywest1

Shadow Copy—instantaneous

opsgilitywest1

opsgilitywest2

Shadow Copy—instantaneous

opsgilitywest0

opsgilitywest1

Cross-Stamp Copy (12 minutes and 3 seconds)

opsgilitywest1

opsgilityeast1

Cross-Region Copy (22 minutes and 54 seconds)

As you can see, the location of your storage accounts can have a dramatic impact on the performance of a copy operation using the asynchronous blob copy cmdlets. Unfortunately, now that you know this critical information, I do have bad news for you. The bad news is that you cannot directly control the placement of which stamp your storage account will be created in. When you create a new storage account, the stamp is automatically selected for you.

Now that you understand how storage is architected and how the asynchronous blob copy works behind the scenes, let’s look at triggering a copy operation between two storage accounts in the same subscription.

Example 5-44 starts by using the Select-AzureSubscription cmdlet to set the PowerShell context to the correct subscription. From there it creates a variable $vhdName that references the name of the VHD file to copy in the storage account, and the source and destination container names. The next step is to use the Get-AzureStorageKey cmdlet to retrieve the primary authentication key for the source and destination storage accounts.

The code then creates two storage context objects for both storage accounts using the New-AzureStorageContext cmdlet. This cmdlet is passed the storage account name and key and generates the context object that will be used by the asynchronous blob copy API to authenticate at the source and destination.

The New-AzureStorageContainer cmdlet is used with the destination context to create a container on the destination storage account that will hold the copied VHD file.

Finally, the code calls the Start-AzureStorageBlobCopy cmdlet, passing the appropriate information for the VHD name on the source and destination, along with the container names and the storage security context objects for both storage accounts.

This cmdlet executes a request to the async blob copy API, and the blob copy state is returned and stored in the $blobCopyState variable to use for polling the copy status.

Example 5-44. Copying a VHD in the same subscription between storage accounts

# Select the subscription

Select-AzureSubscription "[subscription name]"

$vhdName = "[filename.vhd]"

$srcContainer = "[source container]"

$destContainer = "[destination container]"

# Source storage account

$srcStorage = "[source storage]"

# Destination storage account

$destStorage = "[dest storage]"

$srcStorageKey = (Get-AzureStorageKey -StorageAccountName $srcStorage).Primary

# If the destination storage account is in a separate subscription, switch to the

# destination subscription first to retrieve the storage account key.

# Select-AzureSubscription "[destination subscription name]"

$destStorageKey = (Get-AzureStorageKey -StorageAccountName $destStorage).Primary

# If the destinaton is in a separate subscription,

# switch back to the source subscription here

# Select-AzureSubscription "[subscription name]"

# Create the source storage account context

$srcContext = New-AzureStorageContext –StorageAccountName $srcStorage `

-StorageAccountKey $srcStorageKey

# Create the destination storage account context

$destContext = New-AzureStorageContext –StorageAccountName $destStorage `

-StorageAccountKey $destStorageKey

# Create the container on the destination

New-AzureStorageContainer -Name $destContainer -Context $destContext

# Start the asynchronous copy - specify the source authentication with -Context

$blobCopyState = Start-AzureStorageBlobCopy -srcBlob $vhdName `

-srcContainer $srcContainer `

-Context $srcContext `

-DestContainer $destContainer `

-DestBlob $vhdName `

-DestContext $destContext

The Get-AzureStorageBlobCopyState cmdlet can then be used to validate whether the copy has completed or not. Simply pipe the returned value of a call to Start-AzureStorageBlobCopy to the cmdlet, and it will return the current status of the copy operation. You can use this to monitor the status of the async copy to know when it is complete (see Example 5-45).

Example 5-45. Monitoring the copy status

# Retrieve the current status of the copy operation

$status = $blobCopyState | Get-AzureStorageBlobCopyState

# Print out status

$status

# Loop until complete

While($status.Status -eq "Pending"){

$status = $blobCopyState | Get-AzureStorageBlobCopyState

Start-Sleep 10

# Print out status

$status

}

Example 5-46 is a working example with values from my subscription added to show a practical aspect to the sample.

Example 5-46. Full blob copy example

# Select the subscription

Select-AzureSubscription "opsgilitytraining"

# Retrieved the OS disk name using:

# Get-AzureVM -ServiceName "psdeploysvc" -Name "psdeploy" | Get-AzureOSDisk

$vhdName = "psdeploysvc-psdeploy-2014-03-17.vhd"

$srcContainer = "vhds"

$destContainer = "copiedvhds"

# Source storage account

$srcStorage = "opsgilitywest"

# Destination storage account

$destStorage = "opsgilityeast1"

$srcStorageKey = (Get-AzureStorageKey -StorageAccountName $srcStorage).Primary

$destStorageKey = (Get-AzureStorageKey -StorageAccountName $destStorage).Primary

# Create the source storage account context

$srcContext = New-AzureStorageContext –StorageAccountName $srcStorage `

-StorageAccountKey $srcStorageKey

# Create the destination storage account context

$destContext = New-AzureStorageContext –StorageAccountName $destStorage `

-StorageAccountKey $destStorageKey

# Create the container on the destination

New-AzureStorageContainer -Name $destContainer -Context $destContext

# Start the asynchronous copy - specify the source authentication with -Context

$blobCopyState = Start-AzureStorageBlobCopy -srcBlob $vhdName `

-srcContainer $srcContainer `

-Context $srcContext `

-DestContainer $destContainer `

-DestBlob $vhdName `

-DestContext $destContext

# Retrieve the current status of the copy operation

$status = $blobCopyState | Get-AzureStorageBlobCopyState

# Print out status

$status

# Loop until complete

While($status.Status -eq "Pending"){

$status = $blobCopyState | Get-AzureStorageBlobCopyState

Start-Sleep 10

# Print out status

$status

}

To retrieve the OS disk name, I piped the results of Get-AzureVM to the Get-AzureOSDisk cmdlet to extract the OS disk information (see Figure 5-22).

Viewing the OS disk name with Get-AzureOSDisk

Figure 5-22. Viewing the OS disk name with Get-AzureOSDisk

The storage accounts are in separate regions, so the copy operation takes anywhere from 7 to 12 minutes. The simple loop at the end of the script will print out the status of the copy operation every 10 seconds, updating the progress (see Figure 5-23).

Summary

In this chapter, we explored Microsoft Azure storage in the context of Microsoft Azure Virtual Machines. While this chapter is not comprehensive, I do hope it helps you understand the capabilities of the platform and hits on the key tasks that a Microsoft Azure administrator will likely be given with automating. In the next chapter we will discuss automation with Microsoft Azure Virtual Networks, including features such as the internal load balancer, and hybrid technologies such as site-to-site, point-to-site, and ExpressRoute.

Checking the blob copy state

Figure 5-23. Checking the blob copy state