Take Control of Apple Mail (1.0) (2014)

Learn about Email Protocols

The word “protocol” may sound complicated, but it’s just a way of describing how your email program (in this case, Mail) talks to a mail server. If you know a few basics about email protocols, you’ll have an easier time understanding Mail’s interface and solving problems.

Although you may not be aware of it, most email accounts involve two separate systems—one for receiving and another for sending—and these often use entirely different servers. You probably use the same username and password for each, but behind the scenes, each account may function as two separate accounts:

· Your incoming account fetches email from your mail server and delivers it to you using a mail delivery protocol—such as POP (Post Office Protocol) or IMAP (Internet Message Access Protocol). Just ahead, I explain more about POP and IMAP, I discuss commonIMAP and POP Misconceptions, and I offer guidance if you’re still using POP and want to Switch from POP to IMAP (you probably do). I also mention a few important points about iCloud and Gmail accounts, both of which also use IMAP.

· Your outgoing account uses a mail transfer protocol called SMTP (Simple Mail Transfer Protocol) to send email from your machine to your mail server, and then (usually through a number of intermediate steps) to the recipient’s mail server.

Microsoft Exchange accounts are a bit different. They act like IMAP accounts in many ways, but use a single—and an entirely different—protocol for receiving and sending mail, as well as calendar items, contacts, and other information.

Note: In this book I refer to IMAP (including iCloud and Gmail) accounts and Exchange accounts, which manage and work with messages on the mail server, as server-based accounts to distinguish them from POP accounts, which store all retrieved messages locally.

In this chapter I also discuss the concepts Fetch, Push, and IMAP IDLE, which apply to multiple account types and affect how rapidly your email client can find out about new messages.

POP

A POP account works like this:

· First, your client (that is, your email program) asks the server for the list of messages in your Inbox.

· Then, depending on your client’s capabilities and settings, it downloads either all the messages, just the messages you select, or all messages under a specified size, to your local device.

· Finally (and optionally), your client instructs the server to delete the server copy of some or all of the messages.

The server doesn’t keep track of whether a message has been downloaded, read, or deleted; only your client knows this, and the assumption underlying the protocol is that you want to read, organize, and store your mail on a single device, using the server only as a conduit to receive your mail. If you want to store any messages in mailboxes other than your Inbox, when you create those mailboxes in your email client, they’re stored locally (not on the server); the messages you move to these mailboxes are thus also stored locally.

You can usually set your client to leave messages on the server (rather than delete them immediately after retrieving them) and then check your email with a different client (or on a different device), but all those messages will appear to be “new” and unread on every other client or device. This, among other reasons, makes POP a poor option if you work with email on multiple computers and devices.

In addition, although iOS supports POP, iOS Mail doesn’t let you create new mailboxes to file messages locally. That means an iOS device doesn’t work well as the sole device used with a POP account, and because POP works best when used with a single device, this mismatch of capabilities makes POP less than ideal for an iOS device.

IMAP

From a user’s point of view, the main distinction between POP and IMAP is that by default in an IMAP account, messages stay on the mail server even after you’ve downloaded and read them. You can create mailboxes on the server for filing messages, and those mailboxes are mirrored in all your client(s), on all your devices. (Nothing prevents you from moving messages off the server and into mailboxes that are stored only locally, if that’s your preference—but by doing so you lose most of the IMAP advantages for those messages since they’ll no longer sync to your other clients and devices.)

In addition, the server—not your local client—keeps track of which messages you’ve read, forwarded, or replied to, so you see the correct status indicators even if you check your email from another client or device. You can, of course, delete messages you no longer want in order to free up space on the server; doing so deletes the message from all devices. Although each client and provider handles deletions somewhat differently, deleting a message typically means moving it to a Trash or Deleted Messages mailbox, which you can then empty manually or have your email client empty on a preset schedule.

The biggest advantage of IMAP is that you can view your email using any client, on any device, and you will always see exactly the same thing. So, you can access a single email account on your iPad, your iPhone, your Mac (using, say, Apple Mail) and your PC (using, say, Microsoft Outlook), and all your mailboxes, sent and saved messages, read/unread status, and so on will look exactly the same, all the time, in all those places. (Many IMAP accounts also provide a Web-based client.) In other words, unlike POP, IMAP imposes no penalty for checking your mail from more than one device.

Mail and IMAP Subscriptions

Most IMAP clients let you select which of the stored mailboxes on the server you want to see in your local mailbox list, a process known as subscribing. But Mail in Mavericks does not, for the most part, understand the concept of subscriptions—with one exception, it simply shows you all the mailboxes that are on the server, all the time.

If you choose Get Account Info from the pop-up Action menu at the bottom of Mail’s sidebar, and then click Subscription List in the Account Info window, you’ll see a list of mailboxes to which you can subscribe or unsubscribe—use the pop-up Accounts menu at the top to switch between accounts. (Don’t see the sidebar? Choose View > Show Mailbox List.) But usually this list is blank; it shows only mailboxes in “Public” or “Shared” folders on the server, if such folders exist (which is rarely the case).

iCloud, AOL, and Yahoo (among other providers) use IMAP by default for incoming email, although they may do some additional sneaky things in the background that aren’t entirely standard. (I say a bit more about this in Fetch, Push, and IMAP IDLE, later in this chapter.) Gmail (see Gmail, ahead) uses a specially wacky version of IMAP.

Microsoft Exchange servers can be configured to support IMAP, though for various reasons, many Exchange-server administrators choose not to enable IMAP support. The much better, and more common, way to access an Exchange server on a Mac or iOS device is to use Exchange Web Services (in OS X) or Exchange ActiveSync (in iOS), both of which I describe next.

Note: iCloud and Exchange accounts handle not only email but also other data, such as contacts and calendars (and, in the case of iCloud, bookmarks, documents, and several other types of data). However, you’re not required to use all the services together. If you want to use an Exchange account only for calendars, an iCloud account only for email, or whatever, you can turn off the features you don’t want to use.

SMTP

For each POP or IMAP account you set up in Mail, an associated SMTP (Simple Mail Transfer Protocol) server, for sending outgoing mail, must be specified. (Exchange, by contrast, uses the same server for sending and receiving email.) In most cases, you can simply enter the server information provided by your email provider and not have to think about it again. However, if you encounter errors when sending email, you may have an SMTP-related problem; see Fix Outgoing Mail (SMTP) Problems.

IMAP and POP Misconceptions

Over the years that I’ve been writing about and evangelizing IMAP, I’ve had countless people tell me that IMAP is a nonstarter for them or that POP was the only protocol they’d ever use, based on what turned out to be complete misunderstandings.

In an attempt to clear up some of this confusion, I wrote an article for TidBITS called FlippedBITS: IMAP Misconceptions, and I think you’ll find it interesting reading. Let me briefly summarize a few of the points I make there:

· IMAP isn’t a new protocol by any means—it’s been around since 1986 (making it just two years younger than POP).

· The fact that IMAP stores messages on the server doesn’t mean you can read or process your email only when you’re online. It’s always an option to keep local copies of all your messages—and, in fact, most modern email clients keep those offline copies.

· “Stored on the server” doesn’t mean “impossible to delete.” You can delete messages from both your client and the server if you like.

· Just as with POP, you can store any or all of your messages locally and not on the IMAP server if you prefer.

· POP permits only one connection at a time per account, while IMAP lets more than one device connect simultaneously, making IMAP much better if you frequently use multiple devices—say, your Mac, an iPhone, and an iPad—to work with your email.

· Even if your ISP or other email provider doesn’t offer IMAP, you can almost certainly set up an IMAP account with another provider and then forward email from your old address to the IMAP account. The fact that an ISP doesn’t “support” IMAP doesn’t prevent you from using IMAP from another provider.

I find it a bit bewildering that anyone still uses POP these days, but lots of people do, largely out of habit. IMAP is a bit “chattier” and uses slightly more bandwidth, but in my own experience IMAP has performed well even on slow dial-up and cellular connections. Some IMAP servers limit the amount of data you can store on the server, but if you run into such a limit, it’s usually possible to have it raised; to switch to another provider; or to archive some of your messages locally by creating a local (“On My Mac”) mailbox, dragging messages or entire mailboxes into that local mailbox in Mail’s sidebar, and then deleting the server-based copies.

The only argument for POP that I find vaguely persuasive is that it may reduce your susceptibility to hacking or eavesdropping, because incoming messages stay on the server only until you retrieve them. Of course, your email provider may keep indefinite backups of all incoming messages, and a government agency could monitor and capture all messages as they come in, so I think this supposed advantage is largely illusory.

I can’t think of any compelling reason to keep using POP in this day and age, especially with so many robust IMAP servers and clients to choose from. IMAP gives you much more convenience and flexibility, while still enabling you to have a complete local copy of all your messages for offline access.

In short: If you’re still using POP, I suggest switching to IMAP (or Exchange)—especially if you use multiple devices to check your email.

Switch from POP to IMAP

Many mail servers allow both POP and IMAP to be used for retrieving email from the same account, which can make it easy to switch from POP to IMAP. If you currently use POP, try adding a second account to Mail with identical settings, except for an account type of IMAP (see Account Setup for a tip on doing this):

· If it works: Look for any messages in your POP Inbox that aren’t also in your IMAP Inbox, and copy them there; also, if you have mailboxes that you created to file your POP mail locally, copy those mailboxes to the IMAP server. Finally, if you wish, delete the POP account from Mail.

Tip: I go into more detail about why and how to switch from POP to IMAP in my Macworld article The IMAP advantage.

· If it doesn’t work: Contact your email provider to see if it can enable IMAP for you. In some cases, you may need to set up an entirely new email account. If your provider doesn’t offer IMAP and you’re interested in switching to one that does, consult this (somewhat dated, but still useful) list of providers.

Tip: My current IMAP provider is easyMail from easyDNS. To learn more about why I made that choice, see my Macworld article Why (and how) I’m saying goodbye to Gmail.

iCloud

For the most part, everything I said earlier in this chapter about IMAP accounts is also true of iCloud accounts. In fact, you can use any IMAP client—not just Mail—to connect to your iCloud account. But unlike other clients, Mail can use any iCloud email aliases you’ve set up (for details, read Use iCloud Aliases)—they appear automatically as options in the Account pop-up menu in the New Message window. In addition, as long as you’re signed in to your iCloud account with Documents & Data enabled on each of your Macs, the settings for all your email accounts—not just iCloud—will sync between Macs automatically, so once you set up an email account on one Mac, you don’t need to set it up again on any other Macs using the same iCloud account. (Unfortunately, this doesn’t apply to iOS devices.)

iCloud offers many other benefits, too, such as push updates of contacts, calendars, reminders, bookmarks, notes, and Keychain items; Photo Stream to ensure that your latest photos appear automatically on all your devices; and Documents in the Cloud to keep documents in sync in supported apps. And it’s free! Even if you use another provider for your primary email account (as I do), I suggest setting up an iCloud account as well.

Tip: I cover iCloud mail in detail in my book Take Control of iCloud.

Exchange

Microsoft Exchange servers handle email, contacts, shared calendars, and a variety of other network services for many organizations. Under Windows, Outlook is the application of choice for working with Exchange-based data. You can now buy a version of Outlook for OS X, too; it’s part of Microsoft Office for Mac Home & Business 2011. However, Mail—on both OS X and iOS—can also connect directly to an Exchange server (for email only, naturally, though Apple’s Contacts and Calendar apps can connect to Exchange servers to handle those types of data).

Exchange servers can communicate with email clients with a number of different protocols (depending on the version of the server software and which features the server’s administrator has enabled). If the Exchange server to which you’re connecting is running a version of Exchange Server older than 2007, IMAP and SMTP must be enabled by the administrator in order for Mail to communicate with it. But if the server is running Exchange Server 2007 or later, Mail in Mavericks can use a more sophisticated protocol called Exchange Web Services (EWS), which is based on SOAP.

Among other benefits, Exchange Web Services supports a feature called Autodiscover, which simplifies account setup—when adding an Exchange account to Mail, you just enter your email address and password, and in most cases Mail can figure out all the other details by itself. (You can enable or disable Autodiscover by going to Mail > Preferences > Accounts > Your Exchange Account > Account Information and selected or deselecting Use Autodiscover Service.)

EWS also enables Mail to use the contact information stored in the Global Address List on your Exchange server to automatically complete addresses as you type them. And, because the Contacts and Calendar apps on your Mac can also use Exchange Web Services, you can access contact and calendar information on Exchange servers with those applications (which, however, I don’t cover in this book).

On an iOS device, the experience of using Exchange is essentially the same as on a Mac, except that behind the scenes, yet another protocol is used—Exchange ActiveSync (EAS). EAS is a collection of capabilities built into Microsoft Exchange servers as well as a number of third-party servers that have licensed the technology. (For example, you can connect to a Kerio or Zimbra server using EAS.)

EAS can be set up to check the server for new messages manually or on a schedule, but one of the protocol’s major attractions is its support for push email (see Fetch, Push, and IMAP IDLE), along with push contacts and calendars. Like Exchange Web Services, EAS supports Autodiscover for easy account setup.

For all practical purposes, the experience of using Mail with an Exchange server (whether via Exchange Web Services or EAS) is almost identical to using Mail with an IMAP server, with the possible exception of push message delivery. Messages are stored on the server and mirrored locally, and you can create as many server-based mailboxes as you need.

Gmail

Coming up a bit later in this book is an entire chapter called Use Gmail with Mail, which discusses in detail how Gmail works (and doesn’t) with Mail in Mavericks (and, to a lesser extent, iOS 7). For now, since we’re discussing protocols, I want to point out that Mail can talk to Gmail using either POP or IMAP, but:

· IMAP is the default.

· Gmail’s implementation of IMAP is highly nonstandard.

· Mail in Mavericks handles Gmail’s “flavor” of IMAP differently than earlier versions of Mail did—and that’s not always a good thing.

· On an iOS device, there are two ways to set up Gmail as an IMAP account—the default method (selecting Google as the account type) and manually setting up the account (selecting the Other option). The two options result in somewhat different behaviors (detailed in Use Gmail with Mail).

No matter how you slice it, the experience of using Gmail in Mail won’t be exactly like using a conventional IMAP server, nor will it reflect what you might expect if you’re used to using Gmail in a Web browser. It’s a weird and potentially frustrating hybrid of the two, and while some people don’t mind it at all, others find it so annoying that they stop using either Mail or Gmail.

More details to come, but…consider yourself forewarned.

AOL, Yahoo, and Outlook.com

When you set up a new email account on your Mac or iOS device, you’ll notice several predefined options beyond those mentioned in this chapter—including AOL, Yahoo, and (on iOS only) Outlook.com. These account types don’t represent different protocols, as all these services support IMAP (although Outlook.com uses Exchange ActiveSync by default under iOS). Similarly, iCloud uses IMAP; and while Gmail supports both POP and IMAP, if you choose the Google option, Mail uses IMAP by default.

So why does Apple bother listing all these providers separately instead of simply showing POP, IMAP, and Exchange as options? It’s all for ease of configuration. Mail already knows these providers’ IMAP and SMTP server addresses, as well as several other configuration details, and this information enables you to set up an account with one of these services by providing only your email address and password (and, in some cases, your name).

Fetch, Push, and IMAP IDLE

Back in what I like to think of as the olden days (let’s say, the 20th century, give or take), email clients always bore the responsibility for checking to see if there were any new messages on the server waiting to be downloaded. You might click a button to do this manually, or set your client to check on a recurring schedule (every minute, every half hour, or whatever). Indeed, this is the way most desktop email programs still work. This process—asking whether there are new messages, and downloading any that are there—is called fetch or pull, emphasizing that the activity is always initiated from the receiving end.

Fetching works just fine, but it involves two kinds of inefficiency:

· If there’s no mail waiting, the transaction is wasted. Sure, it’s just a handful of bytes being sent back and forth, but given a server with many users, you’re talking about thousands of such wasted checks each day—this can add up to lots of bandwidth and processor cycles.

· What if an important message arrives in between scheduled checks? There’d be no way to know unless you kept checking your mail manually, or increased the checking frequency to, say, every minute (which takes us back to the first inefficiency).

So a different way of delivering email emerged: push, in which the server sends incoming messages (or, in some implementations, a command telling the client to fetch the messages) to the client as soon as they arrive. When no messages are waiting, no transactions occur.

Now, I’m oversimplifying in that push isn’t a specific technology, but rather a generic term, and there are several ways to accomplish it. iOS 7 supports several mobile-oriented push technologies (iCloud uses one method, Yahoo uses another, and Exchange ActiveSync uses a third). On a Mac, Mail uses a different technique, called IMAP IDLE, to accomplish essentially the same thing—but it works only if it is both enabled in Mail (Mail > Preferences > Accounts > Account Name > Advanced and check Use IDLE Command If the Server Supports It) and if the IMAP server supports it (iCloud and Gmail do; with other providers, it’s hit-and-miss). Curiously, although Exchange servers can push email to an iOS device or to Outlook on a Mac, Mail for Mavericks doesn’t support push email from Exchange servers.

Tip: To learn more about push and IMAP IDLE, read Michael E. Cohen’s TidBITS article How Apple Mail May Be Anything but IDLE when Pushing Email.

Although push email (of whatever sort) is nice, I don’t find it to be a huge advantage—especially on Mavericks, where Mail can be set to check all your accounts as frequently as once a minute.