C# 5.0 in a Nutshell (2012)

Chapter 16. Networking

The Framework offers a variety of classes in the System.Net.* namespaces for communicating via standard network protocols, such as HTTP, TCP/IP, and FTP. Here’s a summary of the key components:

§ A WebClient façade class for simple download/upload operations via HTTP or FTP

§ WebRequest and WebResponse classes for low-level control over client-side HTTP or FTP operations

§ HttpClient for consuming HTTP web APIs and RESTful services

§ HttpListener for writing an HTTP server

§ SmtpClient for constructing and sending mail messages via SMTP

§ Dns for converting between domain names and addresses

§ TcpClient, UdpClient, TcpListener, and Socket classes for direct access to the transport and network layers

Metro applications can access only a subset of these types, namely WebRequest/WebResponse, and HttpClient. However, they can also use WinRT types for TCP and UDP communication in Windows.Networking.Sockets, which we demonstrate in the final section in this chapter.

The .NET types in this chapter are in the System.Net.* and System.IO namespaces.

Network Architecture

Figure 16-1 illustrates the .NET networking types and the communication layers in which they reside. Most types reside in the transport layer or application layer. The transport layer defines basic protocols for sending and receiving bytes (TCP and UDP); the application layer defines higher-level protocols designed for specific applications such as retrieving web pages (HTTP), transferring files (FTP), sending mail (SMTP), and converting between domain names and IP addresses (DNS).

Figure 16-1. Network architecture

It’s usually most convenient to program at the application layer; however, there are a couple of reasons you might want to work directly at the transport layer. One is if you need an application protocol not provided in the Framework, such as POP3 for retrieving mail. Another is if you want to invent a custom protocol for a special application such as a peer-to-peer client.

Of the application protocols, HTTP is special in that its use has extended to general-purpose communication. Its basic mode of operation—“give me the web page with this URL”—adapts nicely to “get me the result of calling this endpoint with these arguments.” (In addition to the “get” verb, there is “put,” “post,” and “delete,” allowing for REST-based services.)

HTTP also has a rich set of features that are useful in multitier business applications and service-oriented architectures, such as protocols for authentication and encryption, message chunking, extensible headers and cookies, and the ability to have many server applications share a single port and IP address. For these reasons, HTTP is well supported in the Framework—both directly, as described in this chapter, and at a higher level, through such technologies as WCF, Web Services, and ASP.NET.

The Framework provides client-side support for FTP, the popular Internet protocol for sending and receiving files. Server-side support comes in the form of IIS or Unix-based server software.

As the preceding discussion makes clear, networking is a field that is awash in acronyms. Table 16-1 is a handy Network TLA (three-letter and more acronym buster).

Table 16-1. Network TLA (three-letter acronym) buster

Addresses and Ports

For communication to work, a computer or device requires an address. The Internet uses two addressing systems:

IPv4

Currently the dominant addressing system; IPv4 addresses are 32 bits wide. When string-formatted, IPv4 addresses are written as four dot-separated decimals (e.g., 101.102.103.104). An address can be unique in the world—or unique within a particular subnet (such as on a corporate network).

IPv6

The newer 128-bit addressing system. Addresses are string-formatted in hexadecimal with a colon separator (e.g., [3EA0:FFFF:198A:E4A3:4FF2:54fA:41BC:8D31]). The .NET Framework requires that you add square brackets around the address.

The IPAddress class in the System.Net namespace represents an address in either protocol. It has a constructor accepting a byte array, and a static Parse method accepting a correctly formatted string:

IPAddress a1 = new IPAddress (new byte[] { 101, 102, 103, 104 });

IPAddress a2 = IPAddress.Parse ("101.102.103.104");

Console.WriteLine (a1.Equals (a2)); // True

Console.WriteLine (a1.AddressFamily); // InterNetwork

IPAddress a3 = IPAddress.Parse

("[3EA0:FFFF:198A:E4A3:4FF2:54fA:41BC:8D31]");

Console.WriteLine (a3.AddressFamily); // InterNetworkV6

The TCP and UDP protocols break out each IP address into 65,535 ports, allowing a computer on a single address to run multiple applications, each on its own port. Many applications have standard port assignments; for instance, HTTP uses port 80; SMTP uses port 25.

NOTE

The TCP and UDP ports from 49152 to 65535 are officially unassigned, so they are good for testing and small-scale deployments.

An IP address and port combination is represented in the .NET Framework by the IPEndPoint class:

IPAddress a = IPAddress.Parse ("101.102.103.104");

IPEndPoint ep = new IPEndPoint (a, 222); // Port 222

Console.WriteLine (ep.ToString()); // 101.102.103.104:222

NOTE

Firewalls block ports. In many corporate environments, only a few ports are in fact open—typically, port 80 (for unencrypted HTTP) and port 443 (for secure HTTP).

URIs

A URI is a specially formatted string that describes a resource on the Internet or a LAN, such as a web page, file, or email address. Examples include http://www.ietf.org, ftp://myisp/doc.txt, and mailto:joe@bloggs.com. The exact formatting is defined by the Internet Engineering Task Force (http://www.ietf.org/).

A URI can be broken up into a series of elements—typically, scheme, authority, and path. The Uri class in the System namespace performs just this division, exposing a property for each element. This is illustrated in Figure 16-2.

Figure 16-2. URI properties

NOTE

The Uri class is useful when you need to validate the format of a URI string or to split a URI into its component parts. Otherwise, you can treat a URI simply as a string—most networking methods are overloaded to accept either a Uri object or a string.

You can construct a Uri object by passing any of the following strings into its constructor:

§ A URI string, such as http://www.ebay.com or file://janespc/sharedpics/dolphin.jpg

§ An absolute path to a file on your hard disk, such as c:\myfiles\data.xls

§ A UNC path to a file on the LAN, such as \\janespc\sharedpics\dolphin.jpg

File and UNC paths are automatically converted to URIs: the “file:” protocol is added, and backslashes are converted to forward slashes. The Uri constructors also perform some basic cleanup on your string before creating the Uri, including converting the scheme and hostname to lowercase and removing default and blank port numbers. If you supply a URI string without the scheme, such as “www.test.com”, a UriFormatException is thrown.

Uri has an IsLoopback property, which indicates whether the Uri references the local host (IP address 127.0.0.1), and an IsFile property, which indicates whether the Uri references a local or UNC (IsUnc) path. If IsFile returns true, the LocalPath property returns a version ofAbsolutePath that is friendly to the local operating system (with backslashes), on which you can call File.Open.

Instances of Uri have read-only properties. To modify an existing Uri, instantiate a UriBuilder object—this has writable properties and can be converted back via its Uri property.

Uri also provides methods for comparing and subtracting paths:

Uri info = new Uri ("http://www.domain.com:80/info/");

Uri page = new Uri ("http://www.domain.com/info/page.html");

Console.WriteLine (info.Host); // www.domain.com

Console.WriteLine (info.Port); // 80

Console.WriteLine (page.Port); // 80 (Uri knows the default HTTP port)

Console.WriteLine (info.IsBaseOf (page)); // True

Uri relative = info.MakeRelativeUri (page);

Console.WriteLine (relative.IsAbsoluteUri); // False

Console.WriteLine (relative.ToString()); // page.html

A relative Uri, such as page.html in this example, will throw an exception if you call almost any property or method other than IsAbsoluteUri and ToString(). You can instantiate a relative Uri directly as follows:

Uri u = new Uri ("page.html", UriKind.Relative);

WARNING

A trailing slash is significant in a URI and makes a difference as to how a server processes a request if a path component is present.

For instance, given the URI http://www.albahari.com/nutshell/, you can expect an HTTP web server to look in the nutshell subdirectory in the site’s web folder and return the default document (usually index.html).

Without the trailing slash, the web server will instead look for a file called nutshell (without an extension) directly in the site’s root folder—which is usually not what you want. If no such file exists, most web servers will assume the user mistyped and will return a 301 Permanent Redirect error, suggesting the client retries with the trailing slash. A .NET HTTP client, by default, will respond transparently to a 301 in the same way as a web browser—by retrying with the suggested URI. This means that if you omit a trailing slash when it should have been included, your request will still work—but will suffer an unnecessary extra round trip.

The Uri class also provides static helper methods such as EscapeUriString(), which converts a string to a valid URL by converting all characters with an ASCII value greater than 127 to hexadecimal representation. The CheckHostName() and CheckSchemeName() methods accept a string and check whether it is syntactically valid for the given property (although they do not attempt to determine whether a host or URI exists).

Client-Side Classes

WebRequest and WebResponse are the common base classes for managing both HTTP and FTP client-side activity, as well as the “file:” protocol. They encapsulate the “request/response” model that these protocols all share: the client makes a request, and then awaits a response from a server.

WebClient is a convenient façade class that does the work of calling WebRequest and WebResponse, saving you some coding. WebClient gives you a choice of dealing in strings, byte arrays, files, or streams; WebRequest and WebResponse support just streams. Unfortunately, you cannot rely entirely on WebClient because it doesn’t support some features (such as cookies).

HttpClient is another class that builds on WebRequest and WebResponse (or more specifically, HttpWebRequest and HttpWebResponse) and is new to Framework 4.5. Whereas WebClient acts mostly as a thin layer over the request/response classes, HttpClient adds functionality to help you work with HTTP-based web APIs, REST-based services, and custom authentication schemes.

For simply downloading/uploading a file, string or byte array, both WebClient and HttpClient are suitable. Both have asynchronous methods, although only WebClient offers progress reporting.

NOTE

WinRT applications can’t use WebClient at all and must use either WebRequest/WebResponse or HttpClient (for HTTP).

WebClient

Here are the steps for using WebClient:

1. Instantiate a WebClient object.

2. Assign the Proxy property.

3. Assign the Credentials property if authentication is required.

4. Call a DownloadXXX or UploadXXX method with the desired URI.

Its download methods are as follows:

public void DownloadFile (string address, string fileName);

public string DownloadString (string address);

public byte[] DownloadData (string address);

public Stream OpenRead (string address);

Each is overloaded to accept a Uri object instead of a string address. The upload methods are similar; their return values contain the response (if any) from the server:

public byte[] UploadFile (string address, string fileName);

public byte[] UploadFile (string address, string method, string fileName);

public string UploadString(string address, string data);

public string UploadString(string address, string method, string data);

public byte[] UploadData (string address, byte[] data);

public byte[] UploadData (string address, string method, byte[] data);

public byte[] UploadValues(string address, NameValueCollection data);

public byte[] UploadValues(string address, string method,

NameValueCollection data);

public Stream OpenWrite (string address);

public Stream OpenWrite (string address, string method);

The UploadValues methods can be used to post values to an HTTP form, with a method argument of “POST”. WebClient also has a BaseAddress property; this allows you to specify a string to be prefixed to all addresses, such as http://www.mysite.com/data/.

Here’s how to download the code samples page for this book to a file in the current folder, and then display it in the default web browser:

WebClient wc = new WebClient();

wc.Proxy = null;

wc.DownloadFile ("http://www.albahari.com/nutshell/code.aspx", "code.htm");

System.Diagnostics.Process.Start ("code.htm");

NOTE

WebClient implements IDisposable under duress—by virtue of deriving from Component (this allows it to be sited in the Visual Studio’s Designer’s component tray). Its Dispose method does nothing useful at runtime, however, so you don’t need to dispose WebClient instances.

From Framework 4.5, WebClient provides asynchronous versions of its long-running methods (Chapter 14) that return tasks that you can await:

await wc.DownloadFileTaskAsync ("http://oreilly.com", "webpage.htm");

(The “TaskAsync” suffix disambiguates these methods from the old EAP-based asynchronous methods which use the “Async” suffix). Unfortunately, the new methods don’t support the standard “TAP” pattern for cancellation and progress reporting. Instead, for cancellation you must call theCancelAsync method on the WebClient object, and for progress reporting, handle the DownloadProgressChanged/UploadProgressChanged event. The following downloads a web page with progress reporting, canceling the download if it takes longer than 5 seconds:

var wc = new WebClient();

wc.DownloadProgressChanged += (sender, args) =>

Console.WriteLine (args.ProgressPercentage + "% complete");

Task.Delay (5000).ContinueWith (ant => wc.CancelAsync());

await wc.DownloadFileTaskAsync ("http://oreilly.com", "webpage.htm");

NOTE

When a request is canceled, a WebException is thrown whose Status property is WebExceptionStatus.RequestCanceled. (For historical reasons, an OperationCanceledException is not thrown.)

The progress-related events capture and post to the active synchronization context, so their handlers can update UI controls without needing Dispatcher.BeginInvoke.

WARNING

Using the same WebClient object to perform more than one operation in sequence should be avoided if you’re relying on cancellation or progress reporting, as it can result in race conditions.

WebRequest and WebResponse

WebRequest and WebResponse are more complex to use than WebClient, but also more flexible. Here’s how to get started:

1. Call WebRequest.Create with a URI to instantiate a web request.

2. Assign the Proxy property.

3. Assign the Credentials property if authentication is required.

To upload data:

1. Call GetRequestStream on the request object, and then write to the stream. Go to step 5 if a response is expected.

To download data:

1. Call GetResponse on the request object to instantiate a web response.

2. Call GetResponseStream on the response object, and then read the stream (a StreamReader can help!).

The following downloads and displays the code samples web page (a rewrite of the preceding example):

WebRequest req = WebRequest.Create

("http://www.albahari.com/nutshell/code.html");

req.Proxy = null;

using (WebResponse res = req.GetResponse())

using (Stream rs = res.GetResponseStream())

using (FileStream fs = File.Create ("code.html"))

rs.CopyTo (fs);

Here’s the asynchronous equivalent:

WebRequest req = WebRequest.Create

("http://www.albahari.com/nutshell/code.html");

req.Proxy = null;

using (WebResponse res = await req.GetResponseAsync())

using (Stream rs = res.GetResponseStream())

using (FileStream fs = File.Create ("code.html"))

await rs.CopyToAsync (fs);

WARNING

The web response object has a ContentLength property, indicating the length of the response stream in bytes, as reported by the server. This value comes from the response headers and may be missing or incorrect. In particular, if an HTTP server chooses the “chunked” mode to break up a large response, the ContentLength value is usually −1. The same can apply with dynamically generated pages.

The static Create method instantiates a subclass of the WebRequest type, such as HttpWebRequest or FtpWebRequest. Its choice of subclass depends on the URI’s prefix, and is shown in Table 16-2.

Table 16-2. URI prefixes and web request types

NOTE

Casting a web request object to its concrete type (HttpWebRequest or FtpWebRequest) allows you to access its protocol-specific features.

You can also register your own prefixes by calling WebRequest.RegisterPrefix. This requires a prefix along with a factory object with a Create method that instantiates an appropriate web request object.

The “https:” protocol is for secure (encrypted) HTTP, via Secure Sockets Layer or SSL. Both WebClient and WebRequest activate SSL transparently upon seeing this prefix (see SSL under Working with HTTP later in this chapter). The “file:” protocol simply forwards requests to aFileStream object. Its purpose is in meeting a consistent protocol for reading a URI, whether it be a web page, FTP site, or file path.

WebRequest has a Timeout property, in milliseconds. If a timeout occurs, a WebException is thrown with a Status property of WebExceptionStatus.Timeout. The default timeout is 100 seconds for HTTP and infinite for FTP.

You cannot recycle a WebRequest object for multiple requests—each instance is good for one job only.

HttpClient

HttpClient is new to Framework 4.5 and provides another layer on top of HttpWebRequest and HttpWebResponse. It was written in response to the growth of HTTP-based web APIs and REST services, to provide a better experience than WebClient when dealing with protocols more elaborate than simply fetching a web page. Specifically:

§ A single HttpClient instance supports concurrent requests. To get concurrency with WebClient, you need to create a fresh instance per concurrent request, which can get awkward when you introduce custom headers, cookies, and authentication schemes.

§ HttpClient lets you write and plug in custom message handlers. This enables mocking in unit tests, and the creation of custom pipelines (for logging, compression, encryption, and so on). Unit-testing code that calls WebClient is a pain.

§ HttpClient has a rich and extensible type system for headers and content.

NOTE

HttpClient is not a complete replacement for WebClient, because it doesn’t support progress reporting. WebClient also has the advantage of supporting FTP, file:// and custom URI schemes. It’s also available in all Framework versions.

The simplest way to use HttpClient is to instantiate it and then call one its Get* methods, passing in a URI:

string html = await new HttpClient().GetStringAsync ("http://linqpad.net");

(There’s also GetByteArrayAsync and GetStreamAsync.) All I/O-bound methods in HttpClient are asynchronous (there are no synchronous equivalents).

Unlike with WebClient, to get the best performance with HttpClient, you must re-use same instance (otherwise things such as DNS resolution may be unnecessarily repeated.) HttpClient permits concurrent operations, so the following is legal and downloads two web pages at once:

var client = new HttpClient();

var task1 = client.GetStringAsync ("http://www.linqpad.net");

var task2 = client.GetStringAsync ("http://www.albahari.com");

Console.WriteLine (await task1);

Console.WriteLine (await task2);

HttpClient has a Timeout property and a BaseAddress property which prefixes a URI to every request. HttpClient is somewhat of a thin shell: most of the other properties that you might expect to find here are defined in another classed called HttpClientHandler. To access this class, you instantiate it and then pass the instance into HttpClient’s constructor:

var handler = new HttpClientHandler { UseProxy = false };

var client = new HttpClient (handler);

...

In this example, we told the handler to disable proxy support. There are also properties to control cookies, automatic redirection, authentication, and so on (we’ll describe these in the following sections, and in Working with HTTP).

GetAsync and response messages

The GetStringAsync, GetByteArrayAsync, and GetStreamAsync methods are convenient shortcuts for calling the more general GetAsync method, which returns a response message:

var client = new HttpClient();

// The GetAsync method also accepts a CancellationToken.

HttpResponseMessage response = await client.GetAsync ("http://...");

response.EnsureSuccessStatusCode();

string html = await response.Content.ReadAsStringAsync();

HttpResponseMessage exposes properties for accessing the headers (see Working with HTTP) and the HTTP StatusCode. Unlike with WebClient, an unsuccessful status code such as 404 (not found) doesn’t cause an exception to be thrown unless you explicitly callEnsureSuccessStatusCode. Communication or DNS errors, however, do throw exceptions (see Exception Handling).

HttpResponseMessage has a CopyToAsync method for writing to another stream, which is useful in writing the output to a file:

using (var fileStream = File.Create ("linqpad.html"))

await response.Content.CopyToAsync (fileStream);

GetAsync is one of four methods corresponding to HTTP’s four verbs (the others are PostAsync, PutAsync and DeleteAsync). We demonstrate PostAsync later in Uploading Form Data.

SendAsync and request messages

The four methods just described are all shortcuts for calling SendAsync, the single low-level method into which everything else feeds. To use this, you first construct an HttpRequestMessage:

var client = new HttpClient();

var request = new HttpRequestMessage (HttpMethod.Get, "http://...");

HttpResponseMessage response = await client.SendAsync (request);

response.EnsureSuccessStatusCode();

...

Instantiating an HttpRequestMessage object means you can customize properties of the request, such as the headers (see Headers) and the content itself, allowing you to upload data.

Uploading data and HttpContent

After instantiating an HttpRequestMessage object, you can upload content by assigning its Content property. The type for this property is an abstract class called HttpContent. The Framework includes the following concrete subclasses for different kinds of content (you can also write your own):

§ ByteArrayContent

§ StreamContent

§ FormUrlEncodedContent (see Uploading Form Data)

§ StreamContent

For example:

var client = new HttpClient (new HttpClientHandler { UseProxy = false });

var request = new HttpRequestMessage (

HttpMethod.Post, "http://www.albahari.com/EchoPost.aspx");

request.Content = new StringContent ("This is a test");

HttpResponseMessage response = await client.SendAsync (request);

response.EnsureSuccessStatusCode();

Console.WriteLine (await response.Content.ReadAsStringAsync());

HttpMessageHandler

We said previously that most of the properties for customizing requests are defined not in HttpClient but in HttpClientHandler. The latter is actually a subclass of the abstract HttpMessageHandler class, defined as follows:

public abstract class HttpMessageHandler : IDisposable

{

protected internal abstract Task<HttpResponseMessage> SendAsync

(HttpRequestMessage request, CancellationToken cancellationToken);

public void Dispose();

protected virtual void Dispose (bool disposing);

}

The SendAsync method is called from HttpClient’s SendAsync method.

HttpMessageHandler is simple enough to subclass easily and offers an extensibility point into HttpClient.

Unit testing and mocking

We can subclass HttpMessageHandler to create a mocking handler to assist with unit testing:

class MockHandler : HttpMessageHandler

{

Func <HttpRequestMessage, HttpResponseMessage> _responseGenerator;

public MockHandler

(Func <HttpRequestMessage, HttpResponseMessage> responseGenerator)

{

_responseGenerator = responseGenerator;

}

protected override Task <HttpResponseMessage> SendAsync

(HttpRequestMessage request, CancellationToken cancellationToken)

{

cancellationToken.ThrowIfCancellationRequested();

var response = _responseGenerator (request);

response.RequestMessage = request;

return Task.FromResult (response);

}

}

Its constructor accepts a function that tells the mocker how to generate a response from a request. This is the most versatile approach, as the same handler can test multiple requests.

We’ve thunked down to being synchronous by virtue of using Task.FromResult. We could have maintained asynchrony by having our response generator return a Task<HttpResponseMessage>, but this is pointless given that we can expect a mocking function to be short-running. Here’s how to use our mocking handler:

var mocker = new MockHandler (request =>

new HttpResponseMessage (HttpStatusCode.OK)

{

Content = new StringContent ("You asked for " + request.RequestUri)

});

var client = new HttpClient (mocker);

var response = await client.GetAsync ("http://www.linqpad.net");

string result = await response.Content.ReadAsStringAsync();

Assert.AreEqual ("You asked for http://www.linqpad.net/", result);

(Assert.AreEqual is a method you’d expect to find in a unit-testing framework such as NUnit.)

Chaining handlers with DelegatingHandler

You can create a message handler that calls another (resulting in a chain of handlers) by subclassing DelegatingHandler. This can be used to implement custom authentication, compression, and encryption protocols. The following demonstrates a simple logging handler:

class LoggingHandler : DelegatingHandler

{

public LoggingHandler (HttpMessageHandler nextHandler)

{

InnerHandler = nextHandler;

}

protected async override Task <HttpResponseMessage> SendAsync

(HttpRequestMessage request, CancellationToken cancellationToken)

{

Console.WriteLine ("Requesting: " + request.RequestUri);

var response = await base.SendAsync (request, cancellationToken);

Console.WriteLine ("Got response: " + response.StatusCode);

return response;

}

}

Notice that we’ve maintained asynchrony in overriding SendAsync. Introducing the async modifier when overriding a task-returning method is perfectly legal—and desirable in this case.

A better solution than writing to the Console would be to have the constructor accept some kind of logging object. Better still would be to accept a couple of Action<T> delegates which tell it how to log the request and response objects.

Proxies

A proxy server is an intermediary through which HTTP and FTP requests can be routed. Organizations sometimes set up a proxy server as the only means by which employees can access the Internet—primarily because it simplifies security. A proxy has an address of its own and can demand authentication so that only selected users on the local area network can access the Internet.

You can instruct a WebClient or WebRequest object to route requests through a proxy server with a WebProxy object:

// Create a WebProxy with the proxy's IP address and port. You can

// optionally set Credentials if the proxy needs a username/password.

WebProxy p = new WebProxy ("192.178.10.49", 808);

p.Credentials = new NetworkCredential ("username", "password");

// or:

p.Credentials = new NetworkCredential ("username", "password", "domain");

WebClient wc = new WebClient();

wc.Proxy = p;

...

// Same procedure with a WebRequest object:

WebRequest req = WebRequest.Create ("...");

req.Proxy = p;

To use a proxy with HttpClient, first create an HttpClientHandler, assign its Proxy property and then feed that into HttpClient’s constructor:

WebProxy p = new WebProxy ("192.178.10.49", 808);

p.Credentials = new NetworkCredential ("username", "password", "domain");

var handler = new HttpClientHandler { Proxy = p };

var client = new HttpClient (handler);

...

WARNING

If you know there’s no proxy, it’s worth setting the Proxy property to null on WebClient and WebRequest objects. Otherwise, the Framework may attempt to “auto-detect” your proxy settings, adding up to 30 seconds to your request. If you’re wondering why your web requests execute slowly, this is probably it!

HttpClientHandler also has a UseProxy property which you can assign to false instead of nulling out the Proxy property to defeat auto-detection.

If you supply a domain when constructing the NetworkCredential, Windows-based authentication protocols are used. To use the currently authenticated Windows user, assign the static CredentialCache.DefaultNetworkCredentials value to the proxy’s Credentials property.

As an alternative to repeatedly setting the Proxy, you can set the global default as follows:

WebRequest.DefaultWebProxy = myWebProxy;

or:

WebRequest.DefaultWebProxy = null;

Whatever you set applies for the life of the application domain (unless some other code changes it!).

Authentication

You can supply a username and password to an HTTP or FTP site by creating a NetworkCredential object and assigning it to the Credentials property of WebClient or WebRequest:

WebClient wc = new WebClient();

wc.Proxy = null;

wc.BaseAddress = "ftp://ftp.albahari.com";

// Authenticate, then upload and download a file to the FTP server.

// The same approach also works for HTTP and HTTPS.

string username = "nutshell";

string password = "oreilly";

wc.Credentials = new NetworkCredential (username, password);

wc.DownloadFile ("guestbook.txt", "guestbook.txt");

string data = "Hello from " + Environment.UserName + "!\r\n";

File.AppendAllText ("guestbook.txt", data);

wc.UploadFile ("guestbook.txt", "guestbook.txt");

HttpClient exposes the same Credentials property through HttpClientHandler:

var handler = new HttpClientHandler();

handler.Credentials = new NetworkCredential (username, password);

var client = new HttpClient (handler);

...

This works with dialog-based authentication protocols, such as Basic and Digest, and is extensible through the AuthenticationManager class. It also supports Windows NTLM and Kerberos (if you include a domain name when constructing the NetworkCredential object). If you want to use the currently authenticated Windows user, you can leave the Credentials property null and instead set UseDefaultCredentials true.

NOTE

Assigning Credentials is useless for getting through forms-based authentication. We discuss forms-based authentication separately (see Forms Authentication).

The authentication is ultimately handled by a WebRequest subtype (in this case, FtpWebRequest), which automatically negotiates a compatible protocol. In the case of HTTP, there can be a choice: if you examine the initial response from a Microsoft Exchange server web mail page, for instance, it might contain the following headers:

HTTP/1.1 401 Unauthorized

Content-Length: 83

Content-Type: text/html

Server: Microsoft-IIS/6.0

WWW-Authenticate: Negotiate

WWW-Authenticate: NTLM

WWW-Authenticate: Basic realm="exchange.somedomain.com"

X-Powered-By: ASP.NET

Date: Sat, 05 Aug 2006 12:37:23 GMT

The 401 code signals that authorization is required; the “WWW-Authenticate” headers indicate what authentication protocols are understood. If you configure a WebClient or WebRequest object with the correct username and password, however, this message will be hidden from you because the Framework responds automatically by choosing a compatible authentication protocol, and then resubmitting the original request with an extra header. For example:

Authorization: Negotiate TlRMTVNTUAAABAAAt5II2gjACDArAAACAwACACgAAAAQ

ATmKAAAAD0lVDRdPUksHUq9VUA==

This mechanism provides transparency, but generates an extra round trip with each request. You can avoid the extra round trips on subsequent requests to the same URI by setting the PreAuthenticate property to true. This property is defined on the WebRequest class (and works only in the case of HttpWebRequest). WebClient doesn’t support this feature at all.

CredentialCache

You can force a particular authentication protocol with a CredentialCache object. A credential cache contains one or more NetworkCredential objects, each keyed to a particular protocol and URI prefix. For example, you might want to avoid the Basic protocol when logging into an Exchange Server, as it transmits passwords in plain text:

CredentialCache cache = new CredentialCache();

Uri prefix = new Uri ("http://exchange.somedomain.com");

cache.Add (prefix, "Digest", new NetworkCredential ("joe", "passwd"));

cache.Add (prefix, "Negotiate", new NetworkCredential ("joe", "passwd"));

WebClient wc = new WebClient();

wc.Credentials = cache;

...

An authentication protocol is specified as a string. The valid values are as follows:

Basic, Digest, NTLM, Kerberos, Negotiate

In this particular example, WebClient will choose Negotiate, because the server didn’t indicate that it supported Digest in its authentication headers. Negotiate is a Windows protocol that boils down to either Kerberos or NTLM, depending on the capabilities of the server.

The static CredentialCache.DefaultNetworkCredentials property allows you to add the currently authenticated Windows user to the credential cache without having to specify a password:

cache.Add (prefix, "Negotiate", CredentialCache.DefaultNetworkCredentials);

Authenticating via headers with HttpClient

If you’re using HttpClient, another way to authenticate is to set the authentication header directly:

var client = new HttpClient();

client.DefaultRequestHeaders.Authorization =

new AuthenticationHeaderValue ("Basic",

Convert.ToBase64String (Encoding.UTF8.GetBytes ("username:password")));

...

This strategy also works with custom authentication systems such as OAuth. We discuss headers in more detail soon.

Exception Handling

WebRequest, WebResponse, WebClient, and their streams all throw a WebException in the case of a network or protocol error. HttpClient does the same but then wraps the WebException in an HttpRequestException. You can determine the specific error via theWebException’s Status property; this returns a WebExceptionStatus enum that has the following members:

An invalid domain name causes a NameResolutionFailure; a dead network causes a ConnectFailure; a request exceeding WebRequest.Timeout milliseconds causes a Timeout.

Errors such as “Page not found”, “Moved Permanently”, and “Not Logged In” are specific to the HTTP or FTP protocols, and so are all lumped together under the ProtocolError status. With HttpClient, these errors are not thrown unless you call EnsureSuccessStatusCode on the response object. Prior to doing so, you can get the specific status code by querying the StatusCode property:

var client = new HttpClient();

var response = await client.GetAsync ("http://linqpad.net/foo");

HttpStatusCode responseStatus = response.StatusCode;

With WebClient and WebRequest/WebResponse, you must actually catch the WebException and then:

1. Cast the WebException’s Response property to HttpWebResponse or FtpWebResponse.

2. Examine the response object’s Status property (an HttpStatusCode or FtpStatusCode enum) and/or its StatusDescription property (string).

For example:

WebClient wc = new WebClient();

try

{

wc.Proxy = null;

string s = wc.DownloadString ("http://www.albahari.com/notthere");

}

catch (WebException ex)

{

if (ex.Status == WebExceptionStatus.NameResolutionFailure)

Console.WriteLine ("Bad domain name");

else if (ex.Status == WebExceptionStatus.ProtocolError)

{

HttpWebResponse response = (HttpWebResponse) ex.Response;

Console.WriteLine (response.StatusDescription); // "Not Found"

if (response.StatusCode == HttpStatusCode.NotFound)

Console.WriteLine ("Not there!"); // "Not there!"

}

else throw;

}

NOTE

If you want the three-digit status code, such as 401 or 404, simply cast the HttpStatusCode or FtpStatusCode enum to an integer.

By default, you’ll never get a redirection error because WebClient and WebRequest automatically follow redirection responses. You can switch off this behavior in a WebRequest object by setting AllowAutoRedirect to false.

The redirection errors are 301 (Moved Permanently), 302 (Found/Redirect), and 307 (Temporary Redirect).

If an exception is thrown because you’ve incorrectly used the WebClient or WebRequest classes, it will more likely be an InvalidOperationException or ProtocolViolationException than a WebException.

Working with HTTP

This section describes HTTP-specific request and response features of WebClient, HttpWebRequest/HttpWebResponse, and the HttpClient class.

Headers

WebClient, WebRequest, and HttpClient all let you add custom HTTP headers, as well as enumerate the headers in a response. A header is simply a key/value pair containing metadata, such as the message content type or server software. Here’s how to add a custom header to a request, then list all headers in a response message in a WebClient:

WebClient wc = new WebClient();

wc.Proxy = null;

wc.Headers.Add ("CustomHeader", "JustPlaying/1.0");

wc.DownloadString ("http://www.oreilly.com");

foreach (string name in wc.ResponseHeaders.Keys)

Console.WriteLine (name + "=" + wc.ResponseHeaders [name]);

Age=51

X-Cache=HIT from oregano.bp

X-Cache-Lookup=HIT from oregano.bp:3128

Connection=keep-alive

Accept-Ranges=bytes

Content-Length=95433

Content-Type=text/html

...

HttpClient instead exposes strongly typed collections with properties for standard HTTP headers. The DefaultRequestHeaders property is for headers which apply to every request:

var client = new HttpClient (handler);

client.DefaultRequestHeaders.UserAgent.Add (

new ProductInfoHeaderValue ("VisualStudio", "2012"));

client.DefaultRequestHeaders.Add ("CustomHeader", "VisualStudio/2012");

whereas the Headers property on the HttpRequestMessage class is for headers specific to a request.

Query Strings

A query string is simply a string appended to a URI with a question mark, used to send simple data to the server. You can specify multiple key/value pairs in a query string with the following syntax:

?key1=value1&key2=value2&key3=value3...

WebClient provides an easy way to add query strings through a dictionary-style property. The following searches Google for the word “WebClient”, displaying the result page in French:

WebClient wc = new WebClient();

wc.Proxy = null;

wc.QueryString.Add ("q", "WebClient"); // Search for "WebClient"

wc.QueryString.Add ("hl", "fr"); // Display page in French

wc.DownloadFile ("http://www.google.com/search", "results.html");

System.Diagnostics.Process.Start ("results.html");

To achieve the same result with WebRequest or with HttpClient, you must manually append a correctly formatted string to the request URI:

string requestURI = "http://www.google.com/search?q=WebClient&hl=fr";

If there’s a possibility of your query including symbols or spaces, you can leverage Uri’s EscapeDataString method to create a legal URI:

string search = Uri.EscapeDataString ("(WebClient OR HttpClient)");

string language = Uri.EscapeDataString ("fr");

string requestURI = "http://www.google.com/search?q=" + search +

"&hl=" + language;

This resultant URI is:

http://www.google.com/search?q=(WebClient%20OR%20HttpClient)&hl=fr

(EscapeDataString is similar to EscapeUriString except that it also encodes characters such as & and = which would otherwise mess up the query string.)

NOTE

Microsoft’s Web Protection library (http://wpl.codeplex.com) offers another encoding/decoding solution which takes into account cross-site scripting vulnerabilities.

Uploading Form Data

WebClient provides UploadValues methods for posting data to an HTML form:

WebClient wc = new WebClient();

wc.Proxy = null;

var data = new System.Collections.Specialized.NameValueCollection();

data.Add ("Name", "Joe Albahari");

data.Add ("Company", "O'Reilly");

byte[] result = wc.UploadValues ("http://www.albahari.com/EchoPost.aspx",

"POST", data);

Console.WriteLine (Encoding.UTF8.GetString (result));

The keys in the NameValueCollection, such as searchtextbox and searchMode, correspond to the names of input boxes on the HTML form.

Uploading form data is more work via WebRequest. (You’ll need to take this route if you need to use features such as cookies.) Here’s the procedure:

1. Set the request’s ContentType to “application/x-www-form-urlencoded” and its Method to “POST”.

2. Build a string containing the data to upload, encoded as follows:

name1=value1&name2=value2&name3=value3...

3. Convert the string to a byte array, with Encoding.UTF8.GetBytes.

4. Set the web request’s ContentLength property to the byte array length.

5. Call GetRequestStream on the web request and write the data array.

6. Call GetResponse to read the server’s response.

Here’s the previous example written with WebRequest:

var req = WebRequest.Create ("http://www.albahari.com/EchoPost.aspx");

req.Proxy = null;

req.Method = "POST";

req.ContentType = "application/x-www-form-urlencoded";

string reqString = "Name=Joe+Albahari&Company=O'Reilly";

byte[] reqData = Encoding.UTF8.GetBytes (reqString);

req.ContentLength = reqData.Length;

using (Stream reqStream = req.GetRequestStream())

reqStream.Write (reqData, 0, reqData.Length);

using (WebResponse res = req.GetResponse())

using (Stream resSteam = res.GetResponseStream())

using (StreamReader sr = new StreamReader (resSteam))

Console.WriteLine (sr.ReadToEnd());

With HttpClient, you instead create and populate FormUrlEncodedContent object, which you can then either pass into the PostAsync method, or assign to a request’s Content property:

string uri = "http://www.albahari.com/EchoPost.aspx";

var client = new HttpClient();

var dict = new Dictionary<string,string>

{

{ "Name", "Joe Albahari" },

{ "Company", "O'Reilly" }

};

var values = new FormUrlEncodedContent (dict);

var response = await client.PostAsync (uri, values);

response.EnsureSuccessStatusCode();

Console.WriteLine (await response.Content.ReadAsStringAsync());

Cookies

A cookie is a name/value string pair that an HTTP server sends to a client in a response header. A web browser client typically remembers cookies, and replays them to the server in each subsequent request (to the same address) until their expiry. A cookie allows a server to know whether it’s talking to the same client it was a minute ago—or yesterday—without needing a messy query string in the URI.

By default, HttpWebRequest ignores any cookies received from the server. To accept cookies, create a CookieContainer object and assign it to the WebRequest. The cookies received in a response can then be enumerated:

var cc = new CookieContainer();

var request = (HttpWebRequest) WebRequest.Create ("http://www.google.com");

request.Proxy = null;

request.CookieContainer = cc;

using (var response = (HttpWebResponse) request.GetResponse())

{

foreach (Cookie c in response.Cookies)

{

Console.WriteLine (" Name: " + c.Name);

Console.WriteLine (" Value: " + c.Value);

Console.WriteLine (" Path: " + c.Path);

Console.WriteLine (" Domain: " + c.Domain);

}

// Read response stream...

}

Name: PREF

Value: ID=6b10df1da493a9c4:TM=1179025486:LM=1179025486:S=EJCZri0aWEHlk4tt

Path: /

Domain: .google.com

To do the same with HttpClient, first instantiate an HttpClientHandler:

var cc = new CookieContainer();

var handler = new HttpClientHandler();

handler.CookieContainer = cc;

var client = new HttpClient (handler);

...

The WebClient façade class does not support cookies.

To replay the received cookies in future requests, simply assign the same CookieContainer object to each new WebRequest object, or with HttpClient, keep using the same object to make requests. CookieContainer is serializable, so it can be written to disk—see Chapter 17. Alternatively, you can start with a fresh CookieContainer, and then add cookies manually as follows:

Cookie c = new Cookie ("PREF",

"ID=6b10df1da493a9c4:TM=1179...",

"/",

".google.com");

freshCookieContainer.Add (c);

The third and fourth arguments indicate the path and domain of the originator. A CookieContainer on the client can house cookies from many different places; WebRequest sends only those cookies whose path and domain match those of the server.

Forms Authentication

We saw in the previous section how a NetworkCredentials object can satisfy authentication systems such as Basic or NTLM (that pop up a dialog box in a web browser). Most websites requiring authentication, however, use some type of forms-based approach. Enter your username and password into text boxes that are part of an HTML form decorated in appropriate corporate graphics, press a button to post the data, and then receive a cookie upon successful authentication. The cookie allows you greater privileges in browsing pages in the website. With WebRequest orHttpClient, you can do all this with the features discussed in the preceding two sections.

A typical website that implements forms authentication will contain HTML like this:

<form action="http://www.somesite.com/login" method="post">

<input type="text" id="user" name="username">

<input type="password" id="pass" name="password">

<button type="submit" id="login-btn">Log In</button>

</form>

Here’s how to log into such a site with WebRequest/WebResponse:

string loginUri = "http://www.somesite.com/login";

string username = "username"; // (Your username)

string password = "password"; // (Your password)

string reqString = "username=" + username + "&password=" + password;

byte[] requestData = Encoding.UTF8.GetBytes (reqString);

CookieContainer cc = new CookieContainer();

var request = (HttpWebRequest)WebRequest.Create (loginUri);

request.Proxy = null;

request.CookieContainer = cc;

request.Method = "POST";

request.ContentType = "application/x-www-form-urlencoded";

request.ContentLength = requestData.Length;

using (Stream s = request.GetRequestStream())

s.Write (requestData, 0, requestData.Length);

using (var response = (HttpWebResponse) request.GetResponse())

foreach (Cookie c in response.Cookies)

Console.WriteLine (c.Name + " = " + c.Value);

// We're now logged in. As long as we assign cc to subsequent WebRequest

// objects, we'll be treated as an authenticated user.

And with HttpClient:

string loginUri = "http://www.somesite.com/login";

string username = "username";

string password = "password";

CookieContainer cc = new CookieContainer();

var handler = new HttpClientHandler { CookieContainer = cc };

var request = new HttpRequestMessage (HttpMethod.Post, loginUri);

request.Content = new FormUrlEncodedContent (new Dictionary<string,string>

{

{ "username", username },

{ "password", password }

});

var client = new HttpClient (handler);

var response = await client.SendAsync (request);

response.EnsureSuccessStatusCode();

...

SSL

WebClient, HttpClient, and WebRequest all use SSL automatically when you specify an “https:” prefix. The only complication that can arise relates to bad X.509 certificates. If the server’s site certificate is invalid in any way (for instance, if it’s a test certificate), an exception is thrown when you attempt to communicate. To work around this, you can attach a custom certificate validator to the static ServicePointManager class:

using System.Net;

using System.Net.Security;

using System.Security.Cryptography.X509Certificates;

...

static void ConfigureSSL()

{

ServicePointManager.ServerCertificateValidationCallback = CertChecker;

}

ServerCertificateValidationCallback is a delegate. If it returns true, the certificate is accepted:

static bool CertChecker (object sender, X509Certificate certificate,

X509Chain chain, SslPolicyErrors errors)

{

// Return true if you're happy with the certificate

...

}

Writing an HTTP Server

You can write your own .NET HTTP server with the HttpListener class. The following is a simple server that listens on port 51111, waits for a single client request, and then returns a one-line reply.

static void Main()

{

ListenAsync(); // Start server

WebClient wc = new WebClient(); // Make a client request.

Console.WriteLine (wc.DownloadString

("http://localhost:51111/MyApp/Request.txt"));

}

async static void ListenAsync()

{

HttpListener listener = new HttpListener();

listener.Prefixes.Add ("http://localhost:51111/MyApp/"); // Listen on

listener.Start(); // port 51111.

// Await a client request:

HttpListenerContext context = await listener.GetContextAsync();

// Respond to the request:

string msg = "You asked for: " + context.Request.RawUrl;

context.Response.ContentLength64 = Encoding.UTF8.GetByteCount (msg);

context.Response.StatusCode = (int) HttpStatusCode.OK;

using (Stream s = context.Response.OutputStream)

using (StreamWriter writer = new StreamWriter (s))

await writer.WriteAsync (msg);

listener.Stop();

}

OUTPUT: You asked for: /MyApp/Request.txt

HttpListener does not internally use .NET Socket objects; it instead calls the Windows HTTP Server API. This allows many applications on a computer to listen on the same IP address and port—as long as each registers different address prefixes. In our example, we registered the prefixhttp://localhost/myapp, so another application would be free to listen on the same IP and port on another prefix such as http://localhost/anotherapp. This is of value because opening new ports on corporate firewalls can be politically arduous.

HttpListener waits for the next client request when you call GetContext, returning an object with Request and Response properties. Each is analogous to a WebRequest and WebResponse object, but from the server’s perspective. You can read and write headers and cookies, for instance, to the request and response objects, much as you would at the client end.

You can choose how fully to support features of the HTTP protocol, based on your anticipated client audience. At a bare minimum, you should set the content length and status code on each request.

Here’s a very simple web page server, written asynchronously:

using System;

using System.IO;

using System.Net;

using System.Threading.Tasks;

class WebServer

{

HttpListener _listener;

string _baseFolder; // Your web page folder.

public WebServer (string uriPrefix, string baseFolder)

{

_listener = new HttpListener();

_listener.Prefixes.Add (uriPrefix);

_baseFolder = baseFolder;

}

public async void Start()

{

_listener.Start();

while (true)

try

{

var context = await _listener.GetContextAsync();

Task.Run (() => ProcessRequestAsync (context));

}

catch (HttpListenerException) { break; } // Listener stopped.

catch (InvalidOperationException) { break; } // Listener stopped.

}

public void Stop() { _listener.Stop(); }

async void ProcessRequestAsync (HttpListenerContext context)

{

try

{

string filename = Path.GetFileName (context.Request.RawUrl);

string path = Path.Combine (_baseFolder, filename);

byte[] msg;

if (!File.Exists (path))

{

Console.WriteLine ("Resource not found: " + path);

context.Response.StatusCode = (int) HttpStatusCode.NotFound;

msg = Encoding.UTF8.GetBytes ("Sorry, that page does not exist");

}

else

{

context.Response.StatusCode = (int) HttpStatusCode.OK;

msg = File.ReadAllBytes (path);

}

context.Response.ContentLength64 = msg.Length;

using (Stream s = context.Response.OutputStream)

await s.WriteAsync (msg, 0, msg.Length);

}

catch (Exception ex) { Console.WriteLine ("Request error: " + ex); }

}

}

Here’s a main method to set things in motion:

static void Main()

{

// Listen on port 51111, serving files in d:\webroot:

var server = new WebServer ("http://localhost:51111/", @"d:\webroot");

try

{

server.Start();

Console.WriteLine ("Server running... press Enter to stop");

Console.ReadLine();

}

finally { server.Stop(); }

}

You can test this at the client end with any web browser; the URI in this case will be http://localhost:51111/ plus the name of the web page.

WARNING

HttpListener will not start if other software is competing for the same port (unless that software also uses the Windows HTTP Server API). Examples of applications that might listen on the default port 80 include a web server or a peer-to-peer program such as Skype.

Our use of asynchronous functions makes this server scalable and efficient. Starting this from a UI thread, however, would hinder scalability because for each request, execution would bounce back to the UI thread after each await. Incurring such overhead is particularly pointless given that we don’t have shared state, so in a UI scenario we’d get off the UI thread, either like this:

Task.Run (Start);

or by calling ConfigureAwait(false) after calling GetContextAsync.

Note that we used Task.Run to call ProcessRequestAsync, even though the method was already asynchronous. This allows the caller to process another request immediately, rather than having to first wait out the synchronous phase of the method (up until the first await).

Using FTP

For simple FTP upload and download operations, you can use WebClient as we did previously:

WebClient wc = new WebClient();

wc.Proxy = null;

wc.Credentials = new NetworkCredential ("nutshell", "oreilly");

wc.BaseAddress = "ftp://ftp.albahari.com";

wc.UploadString ("tempfile.txt", "hello!");

Console.WriteLine (wc.DownloadString ("tempfile.txt")); // hello!

There’s more to FTP, however, than just uploading and downloading files. The protocol also lists a set of commands or “methods,” defined as string constants in WebRequestMethods.Ftp:

To run one of these commands, you assign its string constant to the web request’s Method property, and then call GetResponse(). Here’s how to get a directory listing:

var req = (FtpWebRequest) WebRequest.Create ("ftp://ftp.albahari.com");

req.Proxy = null;

req.Credentials = new NetworkCredential ("nutshell", "oreilly");

req.Method = WebRequestMethods.Ftp.ListDirectory;

using (WebResponse resp = req.GetResponse())

using (StreamReader reader = new StreamReader (resp.GetResponseStream()))

Console.WriteLine (reader.ReadToEnd());

RESULT:

.

..

guestbook.txt

tempfile.txt

test.doc

In the case of getting a directory listing, we needed to read the response stream to get the result. Most other commands, however, don’t require this step. For instance, to get the result of the GetFileSize command, just query the response’s ContentLength property:

var req = (FtpWebRequest) WebRequest.Create (

"ftp://ftp.albahari.com/tempfile.txt");

req.Proxy = null;

req.Credentials = new NetworkCredential ("nutshell", "oreilly");

req.Method = WebRequestMethods.Ftp.GetFileSize;

using (WebResponse resp = req.GetResponse())

Console.WriteLine (resp.ContentLength); // 6

The GetDateTimestamp command works in a similar way, except that you query the response’s LastModified property. This requires that you cast to FtpWebResponse:

...

req.Method = WebRequestMethods.Ftp.GetDateTimestamp;

using (var resp = (FtpWebResponse) req.GetResponse() )

Console.WriteLine (resp.LastModified);

To use the Rename command, you must populate the request’s RenameTo property with the new filename (without a directory prefix). For example, to rename a file in the incoming directory from tempfile.txt to deleteme.txt:

var req = (FtpWebRequest) WebRequest.Create (

"ftp://ftp.albahari.com/tempfile.txt");

req.Proxy = null;

req.Credentials = new NetworkCredential ("nutshell", "oreilly");

req.Method = WebRequestMethods.Ftp.Rename;

req.RenameTo = "deleteme.txt";

req.GetResponse().Close(); // Perform the rename

Here’s how to delete a file:

var req = (FtpWebRequest) WebRequest.Create (

"ftp://ftp.albahari.com/deleteme.txt");

req.Proxy = null;

req.Credentials = new NetworkCredential ("nutshell", "oreilly");

req.Method = WebRequestMethods.Ftp.DeleteFile;

req.GetResponse().Close(); // Perform the deletion

NOTE

In all these examples, you would typically use an exception handling block to catch network and protocol errors. A typical catch block looks like this:

catch (WebException ex)

{

if (ex.Status == WebExceptionStatus.ProtocolError)

{

// Obtain more detail on error:

var response = (FtpWebResponse) ex.Response;

FtpStatusCode errorCode = response.StatusCode;

string errorMessage = response.StatusDescription;

...

}

...

}

Using DNS

The static Dns class encapsulates the Domain Name Service, which converts between a raw IP address, such as 66.135.192.87, and a human-friendly domain name, such as ebay.com.

The GetHostAddresses method converts from domain name to IP address (or addresses):

foreach (IPAddress a in Dns.GetHostAddresses ("albahari.com"))

Console.WriteLine (a.ToString()); // 205.210.42.167

The GetHostEntry method goes the other way around, converting from address to domain name:

IPHostEntry entry = Dns.GetHostEntry ("205.210.42.167");

Console.WriteLine (entry.HostName); // albahari.com

GetHostEntry also accepts an IPAddress object, so you can specify an IP address as a byte array:

IPAddress address = new IPAddress (new byte[] { 205, 210, 42, 167 });

IPHostEntry entry = Dns.GetHostEntry (address);

Console.WriteLine (entry.HostName); // albahari.com

Domain names are automatically resolved to IP addresses when you use a class such as WebRequest or TcpClient. If you plan to make many network requests to the same address over the life of an application, however, you can sometimes improve performance by first using Dns to explicitly convert the domain name into an IP address, and then communicating directly with the IP address from that point on. This avoids repeated round-tripping to resolve the same domain name, and it can be of benefit when dealing at the transport layer (via TcpClient, UdpClient, orSocket).

The DNS class also provides awaitable task-based asynchronous methods:

foreach (IPAddress a in await Dns.GetHostAddressesAsync ("albahari.com"))

Console.WriteLine (a.ToString());

Sending Mail with SmtpClient

The SmtpClient class in the System.Net.Mail namespace allows you to send mail messages through the ubiquitous Simple Mail Transfer Protocol. To send a simple text message, instantiate SmtpClient, set its Host property to your SMTP server address, and then call Send:

SmtpClient client = new SmtpClient();

client.Host = "mail.myisp.net";

client.Send ("from@adomain.com", "to@adomain.com", "subject", "body");

To frustrate spammers, most SMTP servers on the Internet will accept connections only from the ISP’s subscribers, so you need the SMTP address appropriate to the current connection for this to work.

Constructing a MailMessage object exposes further options, including the ability to add attachments:

SmtpClient client = new SmtpClient();

client.Host = "mail.myisp.net";

MailMessage mm = new MailMessage();

mm.Sender = new MailAddress ("kay@domain.com", "Kay");

mm.From = new MailAddress ("kay@domain.com", "Kay");

mm.To.Add (new MailAddress ("bob@domain.com", "Bob"));

mm.CC.Add (new MailAddress ("dan@domain.com", "Dan"));

mm.Subject = "Hello!";

mm.Body = "Hi there. Here's the photo!";

mm.IsBodyHtml = false;

mm.Priority = MailPriority.High;

Attachment a = new Attachment ("photo.jpg",

System.Net.Mime.MediaTypeNames.Image.Jpeg);

mm.Attachments.Add (a);

client.Send (mm);

SmtpClient allows you to specify Credentials for servers requiring authentication, EnableSsl if supported, and change the TCP Port to a nondefault value. By changing the DeliveryMethod property, you can instruct the SmtpClient to instead use IIS to send mail messages or simply to write each message to an .eml file in a specified directory:

SmtpClient client = new SmtpClient();

client.DeliveryMethod = SmtpDeliveryMethod.SpecifiedPickupDirectory;

client.PickupDirectoryLocation = @"c:\mail";

Using TCP

TCP and UDP constitute the transport layer protocols on top of which most Internet—and local area network—services are built. HTTP, FTP, and SMTP use TCP; DNS uses UDP. TCP is connection-oriented and includes reliability mechanisms; UDP is connectionless, has a lower overhead, and supports broadcasting. BitTorrent uses UDP, as does Voice over IP.

The transport layer offers greater flexibility—and potentially improved performance—over the higher layers, but it requires that you handle such tasks as authentication and encryption yourself.

With TCP in .NET, you have a choice of either the easier-to-use TcpClient and TcpListener façade classes, or the feature-rich Socket class. (In fact, you can mix and match, because TcpClient exposes the underlying Socket object through the Client property.) The Socket class exposes more configuration options and allows direct access to the network layer (IP) and non-Internet-based protocols such as Novell’s SPX/IPX.

(TCP and UDP communication is also possible in WinRT: see TCP in Windows Runtime.)

As with other protocols, TCP differentiates a client and server: the client initiates a request, while the server waits for a request. Here’s the basic structure for a synchronous TCP client request:

using (TcpClient client = new TcpClient())

{

client.Connect ("address", port);

using (NetworkStream n = client.GetStream())

{

// Read and write to the network stream...

}

}

TcpClient’s Connect method blocks until a connection is established (ConnectAsync is the asynchronous equivalent). The NetworkStream then provides a means of two-way communication, for both transmitting and receiving bytes of data from a server.

A simple TCP server looks like this:

TcpListener listener = new TcpListener (<ip address>, port);

listener.Start();

while (keepProcessingRequests)

using (TcpClient c = listener.AcceptTcpClient())

using (NetworkStream n = c.GetStream())

{

// Read and write to the network stream...

}

listener.Stop();

TcpListener requires the local IP address on which to listen (a computer with two network cards, for instance, may have two addresses). You can use IPAddress.Any to tell it to listen on all (or the only) local IP addresses. AcceptTcpClient blocks until a client request is received (again, there’s also an asynchronous version), at which point we call GetStream, just as on the client side.

When working at the transport layer, you need to decide on a protocol for who talks when, and for how long—rather like with a walkie-talkie. If both parties talk or listen at the same time, communication breaks down!

Let’s invent a protocol where the client speaks first, saying “Hello”, and then the server responds by saying “Hello right back!” Here’s the code:

using System;

using System.IO;

using System.Net;

using System.Net.Sockets;

using System.Threading;

class TcpDemo

{

static void Main()

{

new Thread (Server).Start(); // Run server method concurrently.

Thread.Sleep (500); // Give server time to start.

Client();

}

static void Client()

{

using (TcpClient client = new TcpClient ("localhost", 51111))

using (NetworkStream n = client.GetStream())

{

BinaryWriter w = new BinaryWriter (n);

w.Write ("Hello");

w.Flush();

Console.WriteLine (new BinaryReader (n).ReadString());

}

}

static void Server() // Handles a single client request, then exits.

{

TcpListener listener = new TcpListener (IPAddress.Any, 51111);

listener.Start();

using (TcpClient c = listener.AcceptTcpClient())

using (NetworkStream n = c.GetStream())

{

string msg = new BinaryReader (n).ReadString();

BinaryWriter w = new BinaryWriter (n);

w.Write (msg + " right back!");

w.Flush(); // Must call Flush because we're not

} // disposing the writer.

listener.Stop();

}

}

Hello

Hello right back!

In this example, we’re using the localhost loopback to run the client and server on the same machine. We’ve arbitrarily chosen a port in the unallocated range (above 49152) and used a BinaryWriter and BinaryReader to encode the text messages. We’ve avoided closing or disposing the readers and writers in order to keep the underlying NetworkStream open until our conversation completes.

BinaryReader and BinaryWriter might seem like odd choices for reading and writing strings. However, they have a major advantage over StreamReader and StreamWriter: they prefix strings with an integer indicating the length, so a BinaryReader always knows exactly how many bytes to read. If you call StreamReader.ReadToEnd you might block indefinitely—because a NetworkStream doesn’t have an end! As long as the connection is open, the network stream can never be sure that the client isn’t going to send more data.

NOTE

StreamReader is in fact completely out of bounds with NetworkStream, even if you plan only to call ReadLine. This is because StreamReader has a read-ahead buffer, which can result in it reading more bytes than are currently available, blocking indefinitely (or until the socket times out). Other streams such as FileStream don’t suffer this incompatibility with StreamReaderbecause they have a definite end—at which point Read returns immediately with a value of 0.

Concurrency with TCP

TcpClient and TcpListener offer task-based asynchronous methods for scalable concurrency. Using these is simply a question of replacing blocking method calls with their *Async versions, and awaiting the task that’s returned.

In the following example, we write an asynchronous TCP server that accepts requests of 5000 bytes in length, reverses the bytes, and then sends them back to the client:

async void RunServerAsync ()

{

var listener = new TcpListener (IPAddress.Any, 51111);

listener.Start ();

try

{

while (true)

Accept (await listener.AcceptTcpClientAsync ());

}

finally { listener.Stop(); }

}

async Task Accept (TcpClient client)

{

await Task.Yield ();

try

{

using (client)

using (NetworkStream n = client.GetStream ())

{

byte[] data = new byte [5000];

int bytesRead = 0; int chunkSize = 1;

while (bytesRead < data.Length && chunkSize > 0)

bytesRead += chunkSize =

await n.ReadAsync (data, bytesRead, data.Length - bytesRead);

Array.Reverse (data); // Reverse the byte sequence

await n.WriteAsync (data, 0, data.Length);

}

}

catch (Exception ex) { Console.WriteLine (ex.Message); }

}

Such a program is scalable in that it does not block a thread for the duration of a request. So, if a thousand clients were to connect at once over a slow network connection (so that each request took several seconds from start to finish, for example), this program would not require 1000 threads for that time (unlike with a synchronous solution). Instead, it leases threads only for the small periods of time required to execute code before and after the await expressions.

Receiving POP3 Mail with TCP

The .NET Framework provides no application-layer support for POP3, so you have to write at the TCP layer in order to receive mail from a POP3 server. Fortunately, this is a simple protocol; a POP3 conversation goes like this:

Each command and response is terminated by a new line (CR + LF) except for the multiline LIST and RETR commands, which are terminated by a single dot on a separate line. Because we can’t use StreamReader with NetworkStream, we can start by writing a helper method to read a line of text in a nonbuffered fashion:

static string ReadLine (Stream s)

{

List<byte> lineBuffer = new List<byte>();

while (true)

{

int b = s.ReadByte();

if (b == 10 || b < 0) break;

if (b != 13) lineBuffer.Add ((byte)b);

}

return Encoding.UTF8.GetString (lineBuffer.ToArray());

}

We also need a helper method to send a command. Because we always expect to receive a response starting with “+OK,” we can read and validate the response at the same time:

static void SendCommand (Stream stream, string line)

{

byte[] data = Encoding.UTF8.GetBytes (line + "\r\n");

stream.Write (data, 0, data.Length);

string response = ReadLine (stream);

if (!response.StartsWith ("+OK"))

throw new Exception ("POP Error: " + response);

}

With these methods written, the job of retrieving mail is easy. We establish a TCP connection on port 110 (the default POP3 port), and then start talking to the server. In this example, we write each mail message to a randomly named file with an .eml extension, before deleting the message off the server:

using (TcpClient client = new TcpClient ("mail.isp.com", 110))

using (NetworkStream n = client.GetStream())

{

ReadLine (n); // Read the welcome message.

SendCommand (n, "USER username");

SendCommand (n, "PASS password");

SendCommand (n, "LIST"); // Retrieve message IDs.

List<int> messageIDs = new List<int>();

while (true)

{

string line = ReadLine (n); // e.g., "1 1876"

if (line == ".") break;

messageIDs.Add (int.Parse (line.Split (' ')[0] )); // Message ID

}

foreach (int id in messageIDs) // Retrieve each message.

{

SendCommand (n, "RETR " + id);

string randomFile = Guid.NewGuid().ToString() + ".eml";

using (StreamWriter writer = File.CreateText (randomFile))

while (true)

{

string line = ReadLine (n); // Read next line of message.

if (line == ".") break; // Single dot = end of message.

if (line == "..") line = "."; // "Escape out" double dot.

writer.WriteLine (line); // Write to output file.

}

SendCommand (n, "DELE " + id); // Delete message off server.

}

SendCommand (n, "QUIT");

}

TCP in Windows Runtime

Windows Runtime exposes TCP functionality through the Windows.Networking.Sockets namespace. As with the .NET implementation, there are two primary classes to handle server and client roles. In WinRT, these are StreamSocketListener and StreamSocket.

The following method starts a server on port 51111, and waits for a client to connect. It then reads a single message comprising a length-prefixed string:

async void Server()

{

var listener = new StreamSocketListener();

listener.ConnectionReceived += async (sender, args) =>

{

using (StreamSocket socket = args.Socket)

{

var reader = new DataReader (socket.InputStream);

await reader.LoadAsync (4);

uint length = reader.ReadUInt32();

await reader.LoadAsync (length);

Debug.WriteLine (reader.ReadString (length));

}

listener.Dispose(); // Close listener after one message.

};

await listener.BindServiceNameAsync ("51111");

}

In this example, we used a WinRT type called DataReader (in Windows.Networking) to read from the input stream, rather than converting to a .NET Stream object and using a BinaryReader. DataReader is rather like BinaryReader except that it supports asynchrony. TheLoadAsync method asynchronously reads a specified number of bytes into an internal buffer, which then allows you to call methods such as ReadUInt32 or ReadString. The idea is that if you wanted to, say, read 1000 integers in a row, you’d first call LoadAsync with a value of 4000, and then ReadInt32 1000 times in a loop. This avoids the overhead of calling asynchronous operations in a loop (as each asynchronous operation incurs a small overhead).

NOTE

DataReader/DataWriter have a ByteOrder property to control whether numbers are encoding in big- or little-endian format. Big-endian is the default.

The StreamSocket object that we obtained from awaiting AcceptAsync has separate input and output streams. So, to write a message back, we’d use the socket’s OutputStream.

We can illustrate the use of OutputStream and DataWriter with the corresponding client code:

async void Client()

{

using (var socket = new StreamSocket())

{

await socket.ConnectAsync (new HostName ("localhost"), "51111",

SocketProtectionLevel.PlainSocket);

var writer = new DataWriter (socket.OutputStream);

string message = "Hello!";

uint length = (uint) Encoding.UTF8.GetByteCount (message);

writer.WriteUInt32 (length);

writer.WriteString (message);

await writer.StoreAsync();

}

}

We start by instantiating a StreamSocket directly, then call ConnectAsync with the host name and port. (You can pass either a DNS name or an IP address string into HostName’s constructor.) By specifying SocketProtectionLevel.Ssl, you can request SSL encryption (if configured on the server).

Again, we used a WinRT DataWriter rather than a .NET BinaryWriter, and wrote the length of the string (measured in bytes rather than characters), followed by the string itself which is UTF-8 encoded. Finally, we called StoreAsync which writes the buffer to the backing stream, and closed the socket.