Managing Microsoft Hybrid Clouds (2015)
Chapter 6. Managing the Microsoft Hybrid Cloud
In the previous chapter, you learned about the preparations required for the deployment of a virtual machine. You also learned about virtual networks and how to connect from on-premises infrastructures to Azure using a VPN connection.
In this chapter, we will go a bit deeper into managing Microsoft Azure. You will learn about making our virtual machines highly available and perform daily operations such as monitoring the Azure infrastructure. As automation using PowerShell has become a must for administrators, we will spend some time on the basics of using this powerful tool.
In this chapter, we will cover the following topics:
· Understanding Azure Active Directory
· Using Active Directory for authentication to Azure
· Importing and exporting data into Azure using offline media
· Managing disks
· Monitoring your Azure infrastructure
· Automation
· License mobility
Understanding Azure Active Directory
Microsoft offers many online services which run in the cloud. To sign into services such as Office 365 and Microsoft Azure, a multitenant identity management service is used.
This service is called Azure Active Directory (AD). It is a comprehensive identity and access management cloud solution. It combines core directory services, advanced identity governance, security, and application access management. Azure AD also offers developers an identity management platform to deliver access control to their applications, based on a centralized policy and rules.
Azure AD, which was known as Windows Azure Active Directory, was developed by Microsoft and is based on the Active Directory available in Windows Server. Each Azure AD directory is distinct and separate from other Azure AD directories. Just as a corporate office building is a secure asset specific to only your organization, an Azure AD was also designed to be a secure asset for use by your organization only. The Azure AD architecture isolates customer data and identity information from commingling. This means that users and administrators of one Azure AD cannot accidentally or maliciously access data in another directory.
Just as in Software as a Service (SaaS), customers are restricted in adjusting Azure AD to their needs. For example, customers cannot change the schema, groups are limited, additional third-party tooling is nonexistent, and so on.
User accounts in Azure AD are organized in containers that are called directories and are similar to Active Directory domains.
A typical use case for multiple directories is for management reasons. Cloud services such as Office 365 can be used by a single organization that has multiple IT departments or multiple business units; another example is using one directory for production and one for test.
When an Azure subscription is created, a directory is created automatically. This directory is named default directory. The directory is used to store user accounts and groups. It is also used to connect Internet-based SaaS applications to user accounts in a particular directory.
Each directory comes with a domain name that is constructed like this:
<customer supplied domain name>. onmicrosoft.com
You can add a so-called custom domain. This should be a domain your organization owns. Adding a custom domain allows users to authenticate using their local Active Directory account and password.
To be able to use the local user account and password, we need to synchronize those accounts and passwords.
Each directory has a domain that is labeled primary domain. This is the domain name that is displayed by default when you add a new user.
In Azure Management Portal, there are no features for changing or disabling user passwords, or deleting user accounts. All this can be done using the on-premises AD when it is synchronized with a directory hosted by Azure AD.
Azure AD is available in three offerings:
· Azure Active Directory Free: This has options for the management of user accounts and directory synchronization. It is limited to 500,000 objects.
· Azure Active Directory Premium: This has all the features of Active Directory Free, along with an unrestricted number of objects, free Multi-Factor Authentication, usage of groups for application access, synchronization with on-premises directories, machine-learning-based security, and usage reports. Azure Active Directory Premium also empowers end users with, delegated group management, customizable environment for launching enterprise and consumer applications, Microsoft Identity Manager (MIM) server licenses, and many more features.
· Azure Active Directory Basic: There are no limits in the number of objects. Microsoft offers a SLA. There is a self-service password reset for cloud users
Azure Active Directory Premium will be available for purchase through Microsoft's Enterprise Agreement volume licensing program.
Now that we understand what the role of Azure AD is, we are ready for a more detailed look into Azure AD.
Authentication models in Azure AD
For authentication on Microsoft Azure and Office 365, three authentication models are available:
· Cloud Identity: This is also known as the standard authentication and uses a cloud-based username and password stored in Microsoft Azure Active Directory. A typical example of a username is useraccount@<organization>.onmicrosoft.com.
· Synchronized Identity: This is also known as managed authentication, and this allows synced usernames and passwords with on-premises AD as the source.
· Federated identity: This is also known as federated authentication. This allows single sign-on to Microsoft Office 365 and Azure because of a federation with an on-premise Microsoft Active Directory. Azure kind of trusts the Active Directory a user was authenticated on and does not require additional authentication, making live for users easier.
Cloud Identity is the simplest way to authenticate users. Administrators manage user accounts in Azure AD. There is no connection between Azure and any other external directory (like Active Directory). Accounts are managed using Azure Management Portal.
While this works for smaller organizations, it is not preferred for larger organizations. It is not easy to efficiently manage both an on-premises directory as well as an Azure AD. Also, users cannot use their on-premises Active Directory credentials for access to Azure and Office 365.
Synchronized Identity allows administrator access to Azure Management Portal using their on-premises user credentials stored in AD. In the next sections, you will learn how to synchronize user accounts and passwords using AD as a source and Azure AD as a target.
Federated Identity allows authentication on Microsoft cloud services using on-premises AD as source for authentication. This allows for single sign-on so users already signed( )in to AD are not presented with another request for credentials while signing in to Office 365, Azure, and so on.
To enable Federated Identity, several servers are required on-premises. Also, public IP addresses and certificates and required. One of the services to be installed is Active Directory Federation Services (ADFS). Availability of ADFS is essential for users to be able to sign in to Office 365 and Azure. So, in many cases, ADFS needs to be made redundant using load balancers and multiple installations of ADFS. This makes it a complex configuration.
It is recommended that you use the simplest identity model that meets your needs. If your needs change, you can switch between these models easily.
Connecting an on-premises Active Directory to Microsoft Azure
When using a hybrid cloud, you obviously want a single directory service. Most organizations use Microsoft AD for identity management.
It is very easy to extend your on-premises AD to Microsoft Azure. This extension has many advantages:
· Single identity source for both on-premises and Azure
· Users can authenticate to Microsoft Azure using their corporate credentials
· No dependency on availability of on-premises AD and the VPN or ExpressRoute connection with Azure.
· Ability to authenticate to third-party SaaS solutions using AD
· Allow single sign-on access to the Azure Management Portal using on-premises AD credentials
The following are possible ways to extend your on-premises AD:
· Synchronize AD objects using DirSync to Microsoft Azure AD
· Replicate Active Directory to a virtual machine running in Azure
The first option is best suited to organizations that want users to be authenticated on Microsoft and third-party hosted SaaS applications using their corporate credentials.
The advantages of synchronizing AD objects using DirSync to Microsoft Azure AD are:
· No costs for running virtual machines and storage. Azure AD is free to use for up to 500,000 objects. Note that Microsoft has a default quota of 150,000 objects. You can contact Microsoft support to increase this quota.
· The ability of Azure AD to provide single sign-on to Microsoft-hosted applications plus to a growing number of third-party SaaS services such as Salesforce.com.
· No need to set( )up a VPN connection. Syncing with Azure Active Directory uses port 443.
· Azure AD is very scalable.
· Azure AD is very robust and highly available.
· The simplicity of setting up Azure AD. It requires just a few mouse clicks.
· Outsourcing a part of the management AD. You only manage objects, not servers and replication.
· Azure AD is also used by Office 365. If your organization is using Office 365, there is only one identity management solution.
If your organization is already using Office 365, it is quite simple to activate Azure AD for Azure authentication.
In scenario 1, a one-way synchronization is used. Accounts and groups in on-premises AD are synchronized to Azure AD. Synchronization from Azure AD towards on-premises AD (so in reverse) is not possible.
When using Azure AD, you are still in control over policies such as password length and complexity. There will always be a requirement for an on-premises AD server, which is a master for both AD and Azure AD.
The second option is best suited to running a hybrid cloud with your own managed application, which needs AD authentication. This scenario is also suited to securing your AD.
If you are looking for a replica of your AD for disaster recovery purposes, then this scenario is the best option.
The advantage of the second option is two-way replication. Even if your on-premise site is lost, the AD running in Azure is still operational.
The replication of on-premises AD to Azure AD is performed by a tool named Microsoft Azure Active Directory Sync Tool or DirSync for short. DirSync is a free version of the Microsoft Forefront Identity Management (FIM) server.
DirSync is installed on a Windows server on-premises. The supported versions are as follows:
· Windows Server 2008 SP1 or higher
· Windows Server 2008 R2 SP1 or higher
· Windows Server 2012 or higher
DirSync can be installed on a domain controller or on a member server.
Note
Microsoft uses cookies for authentication to Microsoft Azure and other online services. You might encounter issues with signing in to Azure. If you are sure you used the correct username and password but are still denied access, it is advised to delete cookies or use another browser. In many cases, you will be able to sign in successfully afterwards.
Synchronizing an on-premises AD with Azure Active Directory
In this section, we are going to synchronize our on-premises AD user accounts and passwords to Azure AD. To do so, perform the following steps:
1. In Azure Management Portal, select Active Directory in the left pane. You will see a directory named Default Directory.
2. Click on the Add button in the lower menu bar.
Note
Since May 2014, it is possible to delete directories that you do not need anymore.
3. Fill in a descriptive name for the field titled Name.
4. Fill in a domain name. The default AD will be named <organization name>.onmicrosoft.com. This can be adjusted later. Do not worry if the domain is not unique. Just type something that is descriptive.
5. Next, select the country. This choice determines the Azure datacenter that your Azure AD will be active in.
6. Click on the tick mark.
7. Next, we are going to verify our domain name. Select the AD we just created and click on the name.
8. Select the Domains menu item at the top.
9. We are ready to add a custom domain. Select Add in the lower menu.
10. Type in the domain name you are an owner of. I used azurebook.nl for this example. Do not select I plan to configure this domain for single sign-on with my Local Active Directory.
11. Click on add.
12. Click on the right arrow.
13. Now we need to verify that the domain name typed in is actually owned by us. We do this by adding either a TXT or MX record in our DNS settings. As this procedure depends on the domain registrar, we are not going to explain this step.
14. After adding the record in your DNS, click on the Verify button. It may take a while (around 15 minutes), but when Azure has verified that you actually own the domain, a status bar should appear to show that the domain has been verified. The status of the verification process is also shown in the main window. For some unknown reason, sometimes it can be hard to get your domain verified by Azure. You can check the domain record by using a domain query tool such as http://dnsquery.org/.
15. Select the newly created domain, and select Domains from the top menu.
16. Select the Change primary button on the lower menu bar.
17. Make sure your domain is listed under NEW PRIMARY DOMAIN and click on the tick mark.
We now need to create a global administrator in our Microsoft Azure AD:
1. Select the AD we made primary in the previous step. Then, click on the Users menu time in the top menu.
2. Click on the Add User button in the lower menu bar.
3. Fill in a user name. Admin, and click on the right arrow.
4. Fill in first name, for example last name, and display name.
5. Make sure to select global administrator as the role.
6. Fill in the alternate e-mail address.
7. Click on the right arrow.
8. Click on the Create button. A temporary password will be shown.
9. Click on the check mark.
10. Now, log out of Azure Management Portal by clicking on the account displayed in the upper-right corner. Select Sign out.
11. Next, sign in to Azure using the organizational account you just created. In the example, this is admin@azurebook.nl.
12. After a successful login, you are requested to change the password.
13. You will see a message showing We were unable to find any subscriptions associated with your account. This is correct as the Azure subscription we used was originally initiated by a Microsoft account we used.
Next, we are going to set up the directory synchronization:
1. Sign in to Azure Management Portal with the account that is associated with a subscription containing the directory we created.
2. Select Active Directory in the left pane, then select the directory we created.
3. Select Directory Integration in the top menu.
4. Set Directory Sync to Activated.
5. At the bottom of the screen, click on Save.
6. Click on Yes for the question Are you sure you want to activate directory sync.
Next, we are going to prepare the on-premises AD controller for the directory synchronization. The first step is to install .NET 3.5 on the Domain Controller:
1. Select the Add roles and features on the DC you like to use for DirSync.
2. Check the .NET 3.5 Framework 3.5 Features checkbox.
3. Click on Next.
4. On the confirmation screen, click on Specify an alternate source path.
5. Set the path to d:\sources\SxS. This is the media on which the Windows Server 2012 R2 install files are located.
6. Click on Install and then click on Close.
Next, we are going to install DirSync. It uses TCP port 443 for communication with Microsoft Azure. Make sure this port is open at the server DirSync is installed on as well as on your firewall.
1. Download DirSync from the Azure Management Portal.
2. Log in to a server on-premises with an administrator account. This server can be a member server or a domain controller.
3. Run the DirSync installation. This will take around 10 minutes.
4. Click on Next.
5. Make sure that you remove the check mark on Start configuration wizard.
6. Click on Finish.
7. If you installed DirSync, you need to log off from the domain controller and log in again using the account you used to install DirSync. The logoff is required as DirSync added the installation account to a newly created security group.
8. If you installed DirSync on a member server, sign out, and then the login is not required.
The next step is the configuration of the DirSync tool.
1. You will find a shortcut on the desktop named Directory Sync configuration. Double-click on this shortcut.
2. Click on Next.
3. Type in the credentials of the global administrator account of the Azure AD domain and click on Next.
4. Fill in the credentials of an account with administrator rights of the on-premises AD.
5. Click on Next.
6. Click on the checkbox for Enable Hybrid Deployment and click on Next.
7. Click on the checkbox for Enable Password Sync and click on Next.
8. After the configuration has finished, click on Next.
9. Click on the checkbox for Synchronize your directories now and click on Finish.
Within a few minutes, you should be able to notice user accounts and groups from your AD are available in Microsoft Azure AD.
To check, select Active Directory in the left pane of Azure Management Portal. Then, select the directory and select the users or groups.
The status of synchronization can be checked in the Windows Event Viewer. The default synchronization schedule is to synchronize user accounts and passwords once every 3 hours.
To force a synchronization, perform the following steps:
1. Double-click on C:\Program Files\Microsoft Azure Active Directory Sync\DirSyncConfigShell.psc1, and it will launch the DirSync PowerShell.
2. In the PowerShell, run Start-OnlineCoexistenceSync.
Synchronizing partially
It is possible that you do not want to synchronize all of your local AD accounts to Microsoft Azure.
One of the reasons to synchronize is that you want to have administrators sign in to Azure using their local AD credentials.
If your applications do not support authentication to Azure AD and you are interested in single sign-on with SaaS applications, there is no direct need to synchronize.
It is possible to configure DirSync to synchronize just a single or multiple organizational units instead of the complete directory. The following steps show how to configure a partial synchronization:
1. To select which OUs to synchronize, start miisclient.exe. This is located in the folder C:\Program Files\Microsoft Azure Active Directory Sync\SYNCBUS\Synchronization Service\UIShell.
2. Select the Management agents tab and double-click on Active Directory Connector.
3. Select Configure directory partitions.
4. Click on the Container button.
5. Reset the password of the account name listed and fill it in.
6. Select the OUs (containers) you would like to synchronize.
7. Click on OK.
Deleting a domain
If for some reason you wish to delete a domain, you need to follow the steps described here:
1. Disable the directory synchronization.
2. Make sure the domain you want to delete is not the primary domain.
3. Delete all user accounts member of the domain. This can be done manually or by using a PowerShell script. Use the PowerShell module for Microsoft Azure AD.
4. Finally, delete the domain itself.
The domain being deleted must be empty; that is, there cannot be any users or groups with e-mail addresses in this domain.
Adding an AD account as a co-administrator
It is advised not to use Microsoft accounts for management tasks in Azure Management Portal. There is no control over user accounts by IT.
What you want is to use accounts from your on-premises AD and assign them the co-administrator role. In this section, we are going to discuss how to do this. If you are signed in to the Azure Management Portal using a Microsoft account you will not be able to add an organizational account as a co-administrator or service administrator for security reasons. Someone could be trying to find out if a corporate account exists, for example, supervisor@contoso.com.
If you are signed in with an organizational account, you will be able to add both organizational accounts as well as Microsoft accounts.
The co-administrator must be either a Microsoft account or a user account homed in the directory named Default Directory.
The procedure to add a co-administrator from an AD not being the default AD is as follows:
1. Log in using the Microsoft account.
2. Navigate to Settings | Administrator and add the administrative account from the default directory.
3. Assign it the co-administrator role.
4. Log in using this account to the Azure Management Portal.
5. Add the on-premises directory.
6. Add an administrator from the on-premises replicated directory.
Importing and exporting data
If we are going to move our on-premises virtual machine to Azure, we might run into capacity issues when transferring especially when we are connected to Azure using a VPN. If the VPN has a low bandwidth, it might take an unacceptable amount of time to transfer the data. Azure ExpressRoute could be used but might not be an option in your scenario because of availability or costs.
To overcome this, Microsoft offers an Import/Export service that allows us to transfer large amounts of data to or from an Azure datacenter using external hard drives. For security reasons, all data needs to be encrypted using BitLocker. The service is currently only available when the hard disks sent to Azure originate from a US location. Exported data is limited to shipments to US locations only. An import job is free of charge. When data is exported from an Azure location, Microsoft will charge for data leaving the datacenter.
Import and export jobs can be created using the Azure Management Portal. Microsoft charges a fee for the Import/Export service itself.
Managing Azure disks and images
In the previous chapter, you learned how to use the Microsoft supplied operating system images. You learned how to upload application software so we can customize servers.
The next step is creating our own images. An image is a reference guest operating system that can be used to deploy operating systems. By using images, we will be sure that each server is configured and installed according to our standards.
Creating an image from a virtual machine
It is very likely the images supplied by Microsoft as listed in Azure Management Portal are not sufficient for your goal. You might want to include your own software to the image or change the time zone and so on.
This can be accomplished by creating your own image. Configure the guest operating system and applications the way you want, capture the image, and use it for deployment.
In this section, you are going to learn how to create an image for a Windows Server virtual machine. During the image capture, the virtual machine including the virtual hard disks will be deleted. If you want to be able to reuse the virtual machine if something goes wrong, make sure you make a backup of the disks (by exporting data) as described in the previous section.
Note that a virtual machine deployed from an image use the same storage account as the virtual machine that was used as a baseline. During deployment of an image there is no option to select a storage account!
The following steps describe how to create an image from a virtual machine:
1. Start an RDP session to the Windows server that is going to be used as a baseline for other servers.
2. Open a Command Prompt as an administrator.
3. Start %windir%\system32\sysprep\sysprep.exe.
4. Select Enter System Out-of-Box Experience (OOBE).
5. Make sure that you select the Generalize checkbox.
6. Select Shutdown.
7. Click on OK.
Sysprep will now generalize Windows by removing the computer name, IP settings, and so on. When the process is ready, the virtual machine will shut down. This will take about 5 minutes.
Make sure that the virtual machine you want to create an image from is shown as Stopped in the Management Portal. You might have to refresh the browser window to get an update on the status.
8. Select the virtual machine and click on the Capture button in the lower menu.
9. Type in a name for the image. Make sure you enable I have run Sysprep on the virtual machine.
10. Click on the check mark in the bottom-right corner.
11. When the capture has completed, you can deploy a new virtual machine. Click on the NEW button and the virtual machine from the gallery. Select My Images and select the image you just created.
Converting dynamically expanding disks to a fixed size VHD
Microsoft Azure does not support dynamically expanding disks. A dynamically expanding disk consumes as much storage as the amount of data stored in the disk file (.vhd). However, the guest operating system to which a dynamic disk is attached sees the maximum size of the disk. Using a dynamic disk is a way to overprovision your storage to keep storage costs low. However, when using dynamic disks, you should constantly monitor the disk usage to prevent a situation when the underlying storage runs out of disk space. This will result in stopped or crashed virtual machines.
If you want to upload a dynamic disk from on-premises to Microsoft Azure, you first need to convert that disk to a fixed size. Actually, there is no conversion. The process creates a new fixed disk and copies the data of the dynamically expanding disk on to it.
Note that you need the amount of free disk space available equal to the maximum size of the dynamic disk. If your dynamic disk has 3 GB of data but has a maximum disk size of 50 GB, the conversion requires 50 GB of free disk space.
Note
When converting a disk, make sure the file size of the virtual disk is a whole number in MB. So, 500 MB is accepted by Azure. If the VHD file is 500.5 MB in size, Azure will present an error when the VHD is registered as a disk in Azure Management Portal:
The VHD http://<path>/<filename>.VHD has an unsupported virtual size of <number> bytes. The size must be a whole number (in MBs).
Several methods are available for conversion to a fixed-sized disk:
· PowerShell
· Third-party tools
· Hyper-V manager
Perform the following steps to convert a dynamically expanding disk to a fixed-size disk:
1. In Hyper-V Manager, select the virtual machine to which the to-be converted disk is attached.
2. Select the virtual disk and click on the Edit button.
3. In the Choose action part, select Convert.
4. Make sure that the disk format is VHD.
5. Set Fixed size for the Disk type.
Managing disks and VHD files
When a virtual machine is created using Azure Management Portal, the disks and VHD files are named according to a naming convention designed by Microsoft. You cannot change the names of disks and VHD files.
VHD files in Azure have a fixed format, so they do not grow dynamically. Azure does not support the VHDX format that is supported by Windows Server 2012 Hyper-V.
To see the names of the disks and VHD files that are attached to a virtual machine, select that virtual machine in the management portal and click on the name. Then select the Dashboard menu item and scroll down.
An alternative method to find out the naming of VHD files is by selecting Storage in the left pane, selecting the storage account, selecting containers, and double-clicking on the folder named VHD. You will not be able to see the D: disk as this disk is stored locally on the host.
The naming convention for the OS disk is as follows:
<name of cloud service>-<name of virtual machine>-0-<year><month><day><time-of-creation><sequence number>
If you add an additional disk to the virtual machine, the naming convention by default is as follows:
<name of cloud service>-<name of virtual machine>-<month><day>-sequence number starting with 1>
You are able to change the naming of additional disks. It is advised to use your own naming convention. You could, for example, add the drive letter of the Windows server guest in the filename. This makes troubleshooting and maintenance a lot easier.
The naming convention for the VHD files is as follows:
<name of cloud service>-<name of virtual machine>--<year>-<month>-<day>.VHD
If you add an additional disk to the virtual machine, the naming convention by default for the VHD is the same as for the disk:
<name of cloud service>-<name of virtual machine>-<month><day>-sequence number
Disks and VHDs
Microsoft Azure uses two terms for the same object. Microsoft Azure stores virtual hard disk files or .vhd on disks. These .vhd files are registered as an object, which is called a disk.
A virtual machine object is linked to a disk object. A disk object is linked to a VHD. An image can be linked to a VHD as well. This means you can delete a disk object while the linked VHD is not physically removed from the blob storage.
Azure TRIM support
Costs are clearly visible when using cloud. The Azure portal allows a download of a CSV file that clearly shows costs of consumption of, for example, disk storage.
So, you might think about how large a virtual disk should be. It should not be too large, because then you will pay for using disk capacity which is not used by data. Also, it cannot be too small or you might run out of data space.
You shouldn't worry when using Microsoft Azure. Microsoft actually charges for the data written to Microsoft Azure Storage. Microsoft does not charge for reserved capacity.
So, when you store a 127 GB disk on Azure and only 50 GB is used, you will pay for 50 GB of data.
In 2013, Microsoft introduced TRIM support. Without TRIM support, data that has been written on a Blob but has been deleted later is still being accounted for in the blob. This is because while in the metadata (the file allocation table of the blob) the block is free, but in the blob it is not. Microsoft Azure Storage is a sparse file system, which means the data is not charged; the allocated data in the blob means that over time the blob will expand. This is because the guest operating system believes there is no more free space.
With TRIM support, the guest operating system instructs the storage stack that a block of data should be deleted. Azure Storage will not allocate that block of data and you are not charged for it.
TRIM support is available on Windows Server 2012 R2 virtual machines for both the OS and data disks. For Windows Server 2008 R2 virtual machines, TRIM support is only available for the OS disk. For Linux servers, TRIM support is not available yet.
Basically, it comes down to TRIM enabling you to only pay for the amount of disk space you see being consumed in the virtual machine.
Monitoring Microsoft Azure
Monitoring your infrastructure running on Microsoft Azure is just as important as monitoring your on-premises infrastructure. In this section, we will get a high-level overview of available ways to get an insight in health, and performance and capacity of infrastructure components in Azure.
Microsoft Health Dashboard
We want to know the status of Microsoft Azure services like Storage, Network, Active Directory, and Compute at the highest possible level.
Microsoft provides the so-called Service Dashboard or Health Dashboard. This is a web-based dashboard providing the actual health status of all Azure services. For each service, you can further zoom in to Azure datacenters. The dashboard shows four statuses:Normal Service Availability, Performance Degradation, Service Interruption, and Information.
The dashboard URL is http://www.windowsazure.com/en-us/support/service-dashboard/.
The status of each service and Azure region can be monitored using a RSS feed. If you add the RSS feed to an application supporting RSS feeds, you will be able to have an automated push of status information.
Besides proving up-to-date status info, it is also possible to see historical data. This makes it very easy to search and see whether a particular Azure service in a particular period was affected. Microsoft informs customers about the progress of bringing the state of a service back to normal. However, the cause of the problem is not stated in the service dashboard.
Issues that had a serious impact on availability or performance for multiple customers are often documented in a Post Mortem blog. These blogs are published at MSDN.com and often titled Summary of Microsoft Azure Service Disruption. They provide a detailed description of what went wrong, why it went wrong, and what Microsoft has done to prevent similar issues in the future. These post mortems are very interesting to read as they often provide a unique insight into the inner workings of Microsoft Azure.
The Azure Health Dashboard will not warn you when a virtual machine fails, when an application fails, or any other failure in the infrastructure you are responsible for. We have several options available to have a more detailed insight.
First, the Azure Management Portal is able to provide information about service types and their performance, capacity, and health. Rules can be created which send an e-mail alert when a user-defined metric crosses a threshold value.
For the virtual machine service metrics, CPU percentage, disk read bytes/s, disk write/s, network in and network out can be monitored.
The storage service offers many alerts. Examples are alerts on failed authentication, inbound and outbound data transfers, and capacity and availability.
Rules that are used to monitor services are created in the Azure Management Portal by selecting Management Services in the left pane. Then, select Alert in the top menu. Next, click on Add rule in the bottom menu.
Another option for monitoring PaaS-related applications is using third-party monitoring tools. Many of them are offered as a SaaS service themselves. The setup is very easy. I used AzureWatch myself for a while. AzureWatch allows you to monitor cloud services and virtual machines, Azure Websites, SQL Azure databases, Azure storage accounts, and web HTTP/HTTPS URLs (endpoints).
AzureWatch is especially useful to scale down your instances when demand is low and scale up when demand increases. Automated scale up and scale down can also be done using Azure features.
In November 2014 Microsoft made available the Preview of Azure Operational Insights. Microsoft Azure Operational Insights is a Software as a Service offering which allows customers to monitor both Azure and on-premises components. A similar Microsoft service called System Center Advisor has been integrated in Operational Insights.
Monitoring using System Center Operations Manager
Microsoft Azure does not provide an API like Amazon CloudWatch does. Such an API allows third-party software or your in-house written scripts to monitor availability and performance of Azure services from a remote.
Remote monitoring of Azure virtual machines can, for example, be done using PowerShell. PowerShell remoting needs to be enabled in a virtual machine running Windows Server. Port 5986 must be open in the firewall of the guest. If you're not using a site-to-site VPN, you will also need to create an endpoint allowing traffic to flow to port 5986 from the Internet.
So, we have to use the tooling which is used in on-premises infrastructures.
System Center Operations Manager (SCOM), part of Microsoft System Center, allows monitoring of availability and performance of many components in an infrastructure. It is by far the most comprehensive monitoring tool for Microsoft solutions. SCOM is able to monitor non-Microsoft software and devices as well, for example, routers, Linux servers, or VMware vCenter. SCOM uses so-called management packs to get knowledge about the items it monitors.
One of the management packs (MP) available is the System Center Monitoring Pack for Microsoft Azure Application. This MP monitors applications that are deployed in the Azure platform as a service offering.
When you have deployed your own virtual machines in Azure, you can use any available MP to monitor availability and performance, just like doing the same thing on-premises. You will need a site-to-site VPN to prevent having to open all kinds of ports while at the same time weakening security.
Monitoring via open source tools
Other monitoring tools such as OpsView, Nagios, and Zabbix can be used for the purpose of monitoring as well. Microsoft Open Tech released a couple of plugins for both Nagios and Zabbix. These plugins are for:
· Microsoft Azure Storage (ingress, egress, requests, success ratio, availability, latency, and more)
· Microsoft Azure SQL databases( )(ingress, egress, requests, success ratio, availability, latency, and so on)
· Microsoft Azure Active Directory to monitor changes in user and group information (userdelta and groupdelta)
· Microsoft Azure Platform as a Service (PaaS) worker roles
Shutting down virtual machines
As you know, Microsoft charges compute per minute usage. If a virtual machine is not running, you are not charged for compute costs. However, to benefit from that you must shut down using the right method.
Virtual machines running in Microsoft Azure can be shut down using two methods:
· From inside the guest operating system
· From the Azure Management console
When a virtual machine is shut down from the guest operating system, it will keep its IP address. However, Microsoft will still charge you for compute costs even when the virtual machine is not running. This is because the virtual machine still has reservations open of compute resources, although it is not running. This can be compared to leaving your hotel room earlier but not checking out.
To prevent being charged for compute costs, a virtual machine needs to be shut down using the Azure Management console. What will happen is that the virtual machine status is shown as Stopped deallocated. The disadvantage of this method is that you might lose the IP address assigned to the virtual machine. This happens when there are no running virtual machines in a single cloud service.
Monitoring Azure and on-premises infrastructures
The software runs in the Azure North America datacenter.
It has four main features:
· Log Management
· Capacity Planning
· Change Tracking
· Update Assessment
Log Management analyzes logfiles for errors. It will collect events from eventlogs like the System log of a Windows server. Administrators can run all kinds of queries to identify potential issues in their infrastructure.
Capacity planning provides insight into the capacity of the datacenter. It enables you to perform what-if scenarios. It will predict when a datacenter will have unsufficient resources.
Change Tracking keeps an eye on changes on systems.
Update Assessment will scan the server for missing updates and provide reporting.
Operational Insights uses agents that are deployed on servers and virtual machines. The agents use so-called Intelligence Packs to collect information. At the moment (January 2015) there are 8 Intelligence Packs available.
The service is offered in three tiers: Free, Standard, and Premium. The free tier has a limit on the amount of data collected daily and date will be deleted after 7 days. The Standard and Premium tiers do not have a limit on the amount of data collected daily and have greater data retention periods.
Mobile management of Microsoft Azure
When you are not sitting at your desk, it is possible to manage Microsoft Azure services using mobile devices. A short overview of useful apps is as follows:
· As the Azure Management Portal is written in HTML5, it will work perfectly on any mobile device installed with a HTML5-compatible browser. As long as the screen of the device is big enough, management will work perfectly. I used an Apple iPad and had no issues at all.
· Microsoft offers a free Remote Desktop app available for iOS, Android, and Windows Phone. This app allows connections to Windows Server running on Azure.
· Quite a few SSH apps are available to manage Linux machine from mobile devices.
· CloudTools for Microsoft Azure is an iOS app that allows you to manage Azure services. It shows the health status of services, deployments, services, and much more. It is targeted at the PaaS service. The cost for this app is $4.99.
Tips and tricks
This section has some useful tips to make the management of Microsoft Azure a bit easier.
How to reset the IP configuration of a virtual machine
Virtual machines running on Microsoft Azure do not have a possibility for console access. Console access can become handy when the guest operating system does not have network connectivity. Without network connectivity, an Azure virtual machine cannot be accessed over RDP so cannot be managed either.
Loss of network connectivity can occur when an administrator makes a mistake in the networking configuration. Someone might have filled in the wrong IP address, for example.
So how do we regain access to this virtual machine if there is no network connectivity? There are two options. The most efficient one is to adjust the size of the virtual machine.
The procedure is as follows:
1. Stop the virtual machine using the Azure Management Portal.
2. Select the configure menu item.
3. Change the virtual machine size and click on Save.
4. Restart the virtual machine.
5. Connect using RDP or SSH.
The other option is to download the VHD file that contains the guest operating system. Set the IP address to dynamic (DHCP) and upload the VHD file again. Recreate the virtual machine using the VHD and start the virtual machine. This is obviously more complex and time consuming.
However, this trick will not be able to restore RDP connectivity in all situations.
Checking the usage of Azure resources
Azure has soft quotas on a number of resources:
· Cores
· Cloud services
· Storage accounts
To check the number of cores in use, the number of cloud services and the number of storage accounts in use, go the Settings in Azure Management Portal. Then, select the Usage menu item.
The following screenshot shows an example of Usage. If the maximum of 20 cores has been reached, Microsoft support can be contacted to upgrade the number of cores. There is no cost associated with this upgrade.
Automation
Automation is one of the most important features of cloud computing. Provisioning speed and agility can be reached by running scripts instead of doing things manually.
Several methods are available to automate certain tasks in Microsoft Azure. We will discuss all methods in later sections of the chapter:
· Microsoft Azure Automation
· REST API calling the service management API
· Windows PowerShell
· System Center Orchestrator
· Microsoft Azure Command-line Interface
Performing manual administrative tasks is done in Azure Management Portal. However, for certain repetitive tasks, you want to use automation. I advise you to start using PowerShell first. Most common tasks can be performed using this. For some other tasks, REST can be used.
Managing Azure using PowerShell
Knowledge of how to use PowerShell will save you a lot of time. PowerShell is being pushed by Microsoft as the tool for configuration and deployment of workloads in Azure. In some solutions, there is no graphical user interface feature available to perform a task and only a PowerShell script can do the job.
Some of us might be a bit too lazy to study PowerShell and believe it is hard to learn. I will try to prove it is actually very easy to learn.
To be able to use PowerShell for management of Microsoft Azure, we need to install the Microsoft Azure PowerShell module first. Besides the Microsoft PowerShell commandlets, there are also third-party commandlets (cmdlets) available. The Cerebrata Azure Management cmdlets are very useful.
Installation of the PowerShell module
Use the Microsoft Web Platform installer to install the PowerShell module. It can be found at http://go.microsoft.com/fwlink/p/?linkid=320376&clcid=0x409.
Install the web platform and accept the defaults. Installing the module also installs a customized console for Azure PowerShell. You can run the cmdlets from either the standard Windows PowerShell console or the Azure PowerShell console.
After the installation of PowerShell for Azure has completed, we are ready for authentication to our Azure subscription.
This can be done in two ways:
· Run PowerShell and authenticate using an organizational account / Microsoft account and the associated password
· Use a certificate
Running PowerShell and authenticating there can make it easier to manage access to a subscription, but may disrupt automation since it only retains the credentials for 12 hours so is not suitable for long-running tasks. We are going to use the certificate. To do so, perform the following steps:
1. Start the PowerShell for Azure console using Run As Administrator. How to do this depends on your system. For Windows 8 or Windows Server 2012 or later, type power on the Start screen. This command will search for all programs whose name starts with "power". For other systems, go to Programs | Microsoft Azure | Microsoft Azure PowerShell.
2. Type the command: Get-AzurePublishSettingsFile.
3. This will open a browser window which connects to Azure Management Portal. Type in the credentials of your subscription.
4. Next, you will be asked to select a folder for a file named <subscription>-downloaddate-credentials.publishsettings. Save this file and remember the location.
5. In PowerShell, type Import-AzurePublishSettingsFile <path to the.publishsettings file>.
6. You might have to change the execution policy. By default, PowerShell will restrict running scripts. You can change the policy.
7. In PowerShell, type Set-ExecutionPolicy –ExecutionPolicy unrestricted. This will run any script. If you want a restriction for running scripts, see the other options.
We are now ready to use PowerShell commands for the management of Azure.
Note
While using the certificate method makes it easier to use automation for long-running tasks, this method makes it harder to manage access to a shared subscription, such as when more than one person is authorized to access the account.
Managing Azure using command-line tools
The command-line tools are a very easy way to start using automation to manage Azure. Unlike PowerShell, which is very powerful but has a steep learning curve, the command line tools are quite easy to use. In this chapter, we will learn how to use the tools. Also, the command-line tools are available not only for Windows, but also for Mac and Linux. So, the command-line tools are suitable to get a consistent tool for multiple platforms.
First, we need to download the free Microsoft command-line tool. The software can be downloaded from http://www.windowsazure.com/en-us/develop/mobile/tutorials/command-line-administration/.
We select the Windows installer. Install the software and choice the defaults. After the installation has finished, start a Command Prompt.
Type in Azure. If the response is Azure is not recognized as an internal or external command, make sure C:\Program Files (x86)\Microsoft SDKs\Microsoft Azure\CLI\wbin is part of the path setting of the Command Prompt.
The first step in enabling command line management is to import the publish settings or subscription settings. These contain secure credentials and additional information about subscriptions.
At the command prompt type in azure account download.
Save the .publishsettings file to your workstation and remember the location. Make sure there is no space in the filename.
Then import the .publishsettings file using the following command. Replace the path to the settings file with the location of the file storage in the previous step.
azure account import <path-to-settings-file>
After importing your publish settings, you should delete the .publishsettings file for security reasons.
The azure vm command will show the possible commands. Many actions on virtual machines are possible.
The command-line syntax has three components: topic, verb, and options.
Microsoft Azure Automation
Microsoft Azure Automation uses PowerShell workflows that provide the ability to automate administrative processes to manage and deploy cloud servers or any other function that a Windows PowerShell script can perform. It is based on Windows PowerShell 4.0.
PowerShell scripts are called runbooks. While Azure tasks can be performed using PowerShell scripts initiated from on-premises clients, Azure Automation offers a workflow execution engine. Runbooks can be scheduled and are executed in Azure. Two versions of runbooks exist. The draft version is used for editing and the published version is used for execution.
Customers do not have to rely on their on-premises hardware and Internet connection to be able to execute scripts. Azure Automation offers a highly scalable and available script execution engine.
To import PowerShell cmdlets into Azure Automation's so-called Integration Modules are used. This is a package that contains a Windows PowerShell Module.
The pricing of Azure Automation is based on:
· Runtime of the job
· Number of runbooks
· Size of the integration module
Azure Automation has the ability to run scripts inside Azure virtual machines without the use of the PowerShell endpoint or PowerShell Remoting. Company policies might prevent opening an endpoint for PowerShell access.
The VM Agent that can be installed at the deployment of a new virtual machine provides a way to interact with the VM. It enables the Azure Fabric Controller to perform tasks inside a virtual machine. One method of access the VM Agent provides is the ability to use the Azure VM Custom Script Extension.
Custom Script Extension scripts can be used for example to automatically configure a virtual machine once the guest operating system is running. Custom Script Extension will download scripts and files from Azure storage and then run these inside the VM using PowerShell.
A good explanation of the use of Custom Script Extension can be found here:
http://fabriccontroller.net/blog/posts/customizing-your-microsoft-azure-virtual-machines-with-the-new-customscript-extension/
Azure Automation was in Preview and free at the time of writing this book.
Managing Azure using REST API
Another way to automate is using REST API. This is a standard method of enabling automation using a standard set of commands over HTTPS. REST API is not exclusively available in Microsoft Azure. Many cloud platforms such as Amazon and VMware vCloud Air offer REST API. It is also available in storage solutions (Nutanix) or backup (Veeam Backup & Replication).
REST API does not use an user account and password for authentication. To authenticate using REST, you will need a X509 certificate. These certificates can be self-signed, since they are used only for private communication between an application using the Service Management API and Azure Management Service. They can be or through using the makecert utility. Each subscription can have a maximum of 400 certificates.
While this book is not on automation, it is important to understand the basics of using REST API.
Microsoft Azure has several REST APIs available. The most frequently used one is the one connected to the Service Management API. There is also a Storage Services API. If you search for Microsoft Azure Storage Metrics, you will find a lot of information on how to query storage related data.
The purpose of the service management API is to allow third parties to programmatically access Microsoft Azure. The API is used by software vendors for tooling such as monitoring Azure or deploying services.
The API allows us to perform almost all tasks available in the Management Portal but now using scripts. You can create, stop, start, and delete virtual machines.
It supports the entire Azure Service lifecycle of deployment, configuration, staging, suspension, and deletion of Azure Services.
The API is available in two versions. Each version has a version number that looks like this: YYYY-MM-DD.
Each version of the API offers new commands and depreciated commands. So when calling the API, you must send the API version number using x-ms-version.
Authentication is done using a X509 v3 certificate. Each subscription can have a maximum of 400 certificates. You can either use IIS, the makecert tool, or the Visual Studio publish wizard to create self-signed certificates.
This Microsoft site has all the information you need to learn more about the Service Management REST API: http://msdn.microsoft.com/en-us/library/windowsazure/ee460799.aspx.
Exploring the service management API
To start exploring the Service Management API, we need to perform a couple of tasks:
1. Create a self-signed certificate.
2. Upload the certificate to Azure.
3. Explore the API.
To explore the API, a REST client is very handy. Several clients are available. Some good ones are RESTClient, which is a plugin for Firefox, and Fiddler for Chrome.
Introducing System Center Orchestrator
System Center Orchestrator is a software component of the System Center suite that allows us to automate tasks, for example, deploying an application on a server or provisioning a virtual machine on Microsoft Azure.
Instead of writing a PowerShell script or directly addressing the Service Management API, Orchestrator allows a relatively simple drag and drop of out-of-the-box tasks.
Orchestrator is an orchestration engine. Its knowledge of functions is received by importing so-called Integration Packs. Many Integration Packs are available. The Microsoft Azure Integration Pack contains a lot of activities that can be used to automate tasks. Available activities include creation, modification, and deletion of virtual machines as well as management of virtual machine images.
It is important to understand that the Azure Integration Pack uses a specific version of the service management API. So, when a new API is released, you will not be able to benefit from new functions in that release until Microsoft releases a new integration pack.
The procedure to use the Azure Integration Pack is as follows:
1. Download the Azure Integration Pack.
2. Register and deploy. Register it with the Orchestrator Management Server and then deploy it to runbook servers and Runbook Designers.
3. Configure the Azure connection.
As with all connections to the API, you will need a certificate. A connection establishes a reusable link between Orchestrator and Microsoft Azure. You can specify as many connections as you require to create links to multiple Microsoft Azure subscriptions.
Licensing and license mobility
Correct licensing of applications and operating systems is not an easy task. Many vendors have special rules when transferring licenses to a multi-tenant cloud infrastructure. Make sure you understand the licensing rules of the vendor. If uncertain, discuss this with your vendor. It may prevent some nasty fines from the vendor for not respecting the licensing end user agreement.
Basically, two scenarios are possible for licensing:
· You use licenses offered by the cloud vendor. This is the easy option. You pay the service provider a monthly fee for using the license and you are done. However, the availability of licenses is restricted. Windows Server will always be an option, along with SQL Server and SharePoint. Your service provider will not offer all the application licenses you are currently using.
· Use your own licenses in Microsoft Azure. We will discuss this in depth.
When using your owned licenses in a public cloud, there are a couple of possible restriction you should be aware of:
· License mobility of licenses in a shared hardware environment
· Licenses tied to physical sockets
The big difference in licensing depends on whether you are running your software on non-shared hardware or on shared hardware.
If your organization is running their licensed software on a non-shared hardware environment and your organization is the only user of that hardware, the same licensing rules applies as if the software is running on-premises.
If your software is running in a shared server environment, some vendors have restrictions for moving licenses from on-premises to a public cloud.
License mobility for Microsoft software
License mobility enables the owner of a software license to run that software either on-premises or on a hosted environment. Each vendor has its own license rules, so I will concentrate on the license mobility of Microsoft software.
Some Microsoft software is licensed by the Microsoft License Mobility through Software Assurance. This allows you to run selected software in a shared hardware environment offered by a service provider.
Windows server is not part of this License Mobility through SA, which means you cannot transfer Windows Server licenses from on-premises workloads to cloud workloads. For the licensing of Windows Server in the cloud, you will have to use the cloud provider Services Provider License Agreement licenses.
The following three requirements must be met by organizations wanting to move Microsoft licenses to the cloud:
· You will need to have a Volume Licensing program with SA to be able to move licenses to a cloud platform
· The service provider you are using must be an Authorized License Mobility Partner
· Fill in and submit a License Mobility Validation form
Products whose license is covered by the License Mobility through Software Assurance are listed in the Product Use Rights document, for example, Microsoft Exchange Server, Microsoft SharePoint Server, Microsoft Lync Server, and Microsoft Dynamics CRM business software.
In addition, the following products are also eligible for License Mobility through SA:
· SQL Server Standard: Per Processor and Server/CAL (processor and server licenses only) with Software Assurance
· System Center: All Server Management Licenses (MLs), including SMSE and SMSD with Software Assurance, System Center 2012 Standard, and Datacenter with Software Assurance
Microsoft restricts customers to deploy software in an environment in where hardware is shared by multiple organizations. To overcome this issue, some cloud providers offer a dedicated cloud service. Here, the customer is able to provision their own virtual machines on dedicated servers. So, the compute capacity is not shared with other tenants while physical network and storage is.
Licenses tied to physical sockets
Some vendors count the maximum number of servers an application can run on by the number of licenses for CPU sockets. In a cloud environment, this makes it impossible to use that software. In Microsoft Azure, there is no way customers can pin applications to specific physical host servers. However, vendors are changing their licensing. For example, Oracle now allows you to run Oracle software on Microsoft Azure.
Billing and cost management
Azure is very useful for its scale. Customers can create an unlimited number of servers and consume unlimited amounts of storage. However, this can be costly if there is no management of the consumption.
In server virtualization, server sprawl is used for a symptom where many virtual machines are running, while nobody really knows what the purpose is of some of those servers. It is so easy to create a virtual machine while the perception is that there are no associated costs.
Azure has two ways of charging:
· Pay upfront using an Enterprise Agreement. The customer has a commitment with Microsoft to pay in advance for a certain amount of resources.
· Pay by credit card or invoice. This is the pay-as-you-go model.
Microsoft makes an invoice available once per month. This is a PDF file that provides an overview of the costs per Azure service (virtual machines, data management, identity, recovery services, and so on).
The only way to get a more detailed insight into costs is by downloading a comma-separated variable file CSV file from Azure. The CSV file provides an overview of costs for all the services you consumed. The CSV file shows which services are consumed and for how long for each day.
Currently, cost management on Microsoft Azure is quite limited. Azure lacks an application programming interface (API) like Amazon provides that can be used to query actual costs by third-party software. However, Microsoft Azure Pack, which is used in private clouds as well as in clouds managed by service providers, does provide such an API.
To get insight into costs and produce reports, Cloud Cruiser can be used. This is a third-party tool that is able to display the costs made by customers (showback) or charge customers for consumption of cloud resources (chargeback) in real time.
Part of Microsoft Azure Pack is a free edition of Cloud Cruiser. This edition is limited in the number of objects it can handle. The commercially available version does not have such a limit.
Understanding the CSV
The CSV which contains detailed consumption information is a bit difficult to understand. Consumption is shown per day. You might notice a virtual machine that has been running the whole day but has been charged for 16 hours; or a single virtual machine running for 24 hours was billed for 48 hours that particular day.
The explanation for this billing is that compute hours for all instances are converted into small A1 instance hours. For example, one clock hour of a medium (A2) cloud services instance will be presented on your bill as two small (A1) cloud services instance hours.
The following figure shows the conversion:
Summary
In this chapter, we took another step towards learning about Microsoft Azure infrastructure services. To allow user accounts in on-premises AD access to Azure, you learned how to set( )up directory synchronization. You now understand how to handle VHD files and know the restrictions on moving licenses to the cloud.
Also, you learned about monitoring Microsoft Azure services such as storage and virtual machines, and their billing and cost management.
You also learned how to configure Azure. In Chapter 8, Migrating to Microsoft Azure, you will learn about the available options, including the tools used for this move.