THE RUBY WAY, Third Edition (2015)

---

18.2.2 Contacting an Official Timeserver

As we promised, here’s a bit of code to contact a Network Time Protocol (NTP) server on the Net, using TCPSocket to connect to a public time server and read the time:

require "socket"

resp = TCPSocket.new("time.nist.gov", 13).read
time = ts.split(" ")[2] + " UTC"
remote = Time.parse(time)

puts "Local : #{Time.now.utc.strftime("%H:%M:%S")}"
puts "Remote: #{remote.strftime("%H:%M:%S")}"

We establish a connection and read from the socket. The server response includes the Julian date, the date in YY-MM-DD format, the time in HH:MM:SS format, days left until a change to or from Daylight Saving Time, and some other informational flags. We read the entire response, parse only the time, and then print the time on the local machine for comparison.

Note that this example does not use the NTP protocol, which is a more complex format that contains additional data (such as the latency of responses with the time). Instead, we used the much simpler DAYTIME protocol. A full implementation of NTP is too long to reproduce here (although it is only about three pages of code). To see a full NTP implementation, read the source of the net-ntp gem online.

18.2.3 Interacting with a POP Server

The Post Office Protocol (POP) is commonly used by mail servers. Ruby’s POP3 class enables you to examine the headers and bodies of all messages waiting on a server and process them as you see fit. After processing, you can easily delete one or all of them.

The Net::POP3 class must be instantiated with the name or IP address of the server; the port number defaults to 110. No connection is established until the method start is invoked (with the appropriate username and password).

Invoking the method mails on this object will return an array of objects of class POPMail. (There is also an iterator each that will run through these one at a time.)

A POPMail object corresponds to a single email message. The header method will retrieve the message’s headers; the method all will retrieve the header and the body. (There are also other usages of all, as you’ll see shortly.)

A code fragment is worth a thousand words. Here’s a little example that will log on to the server and print the subject line for each email:

require "net/pop"

pop = Net::POP3.new("pop.fakedomain.org")
pop.start("gandalf", "mellon") # user, password
pop.mails.each do |msg|
puts msg.header.lines.grep /^Subject: /
end

The delete method will delete a message from the server. (Some servers require that finish be called to close the POP connection before such an operation becomes final.) Here is the world’s most trivial spam filter:

require "net/pop"

pop = Net::POP3.new("pop.fakedomain.org")
pop.start("gandalf", "mellon") # user, password
pop.mails.each do |msg|
if msg.all =~ /.*make money fast.*/
msg.delete
end
end
pop.finish

We’ll mention that start can be called with a block. By analogy with File.open, it opens the connection, executes the block, and closes the connection.

The all method can also be called with a block. This will simply iterate over the lines in the email message; it is equivalent to calling each_line on the string resulting from all:

# Print each line backwards... how useful!
msg.all { |line| print line.reverse }
# Same thing...
msg.all.each_line { |line| print line.reverse }

We can also pass an object into the all method. In this case, it will call the append operator (<<) repeatedly for each line in the string. Because this operator is defined differently for different objects, the behavior may be radically different, as shown here:

arr = [] # Empty array
str = "Mail: " # String
out = $stdout # IO object

msg.all(arr) # Build an array of lines
msg.all(str) # Concatenate onto str
msg.all(out) # Write to standard output

Finally, we’ll give you a way to return only the body of the message, ignoring all headers:

module Net
class POPMail
def body
# Skip header bytes
self.all[self.header.size..-1]
end
end
end

This doesn’t have all the properties that all has, but it could be extended. We’ll leave that to you.

For those who prefer IMAP to POP3, see Section 18.2.5, “Interacting with an IMAP Server.”

18.2.4 Sending Mail with SMTP

A child of five could understand this. Fetch me a child of five.

—Groucho Marx

The Simple Mail Transfer Protocol (SMTP) may seem like a misnomer. If it is “simple,” it is only by comparison with more complex protocols.

Of course, the smtp library shields the programmer from most of the details of the protocol. However, we have found that the design of this library is not entirely intuitive and perhaps overly complex (and we hope it will change in the future). In this section, we try to present a few examples to you in easily digested pieces.

The Net::SMTP class has two class methods, new and start. The new method takes two parameters—the name of the server (defaulting to localhost) and the port number (defaulting to the well-known port 25).

The start method takes these parameters:

• server is the IP name of the SMTP server, defaulting to "localhost".

• port is the port number, defaulting to 25.

• domain is the domain of the mail sender, defaulting to ENV["HOSTNAME"].

• account is the username; the default is nil.

• password is the user password, defaulting to nil.

• authtype is the authorization type, defaulting to :cram_md5.

Many or most of these parameters may be omitted under normal circumstances.

If start is called “normally” (without a block), it returns an object of class SMTP. If it is called with a block, that object is passed into the block as a convenience.

An SMTP object has an instance method called sendmail, which will typically be used to do the work of mailing a message. It takes three parameters:

• source is a string or array (or anything with an each iterator returning one string at a time).

• sender is a string that will appear in the “from” field of the email.

• recipients is a string or an array of strings representing the addressee(s).

Here is an example of using the class methods to send an email:

require 'net/smtp'

msg = <<EOF
Subject: Many things
"The time has come," the Walrus said,
"To talk of many things —
Of shoes, and ships, and sealing wax,
Of cabbages and kings;
And why the sea is boiling hot,
And whether pigs have wings."
EOF

Net::SMTP.start("smtp-server.fake.com") do |smtp|
smtp.sendmail msg, 'walrus@fake1.com', 'alice@fake2.com'
end

Because the string Subject: was specified at the beginning of the string, Many things will appear as the subject line when the message is received.

There is also an instance method named start, which behaves much the same as the class method. Because new specifies the server, start doesn’t have to specify it. This parameter is omitted, and the others are the same as for the class method. This gives us a similar example using anSMTP object:

require 'net/smtp'

msg = <<EOF
Subject: Logic
"Contrariwise," continued Tweedledee,
"if it was so, it might be, and if it
were so, it would be; but as it isn't,
it ain't. That's logic."
EOF

smtp = Net::SMTP.new("smtp-server.fake.com")
smtp.start
smtp.sendmail msg, 'tweedledee@fake1.com', 'alice@fake2.com'

In case you are not confused yet, the instance method can also take a block:

require 'net/smtp'

msg = <<EOF
Subject: Moby-Dick
Call me Ishmael.
EOF

addressees = ['reader1@fake2.com', 'reader2@fake3.com']

smtp = Net::SMTP.new("smtp-server.fake.com")
smtp.start do |obj|
obj.sendmail msg, 'narrator@fake1.com', addressees
end

As the example shows, the object passed into the block (obj) certainly need not be named the same as the receiver (smtp). I’ll also take this opportunity to emphasize that the recipient can be an array of strings.

There is also an oddly named instance method called ready. This is much the same as sendmail, with some crucial differences. Only the sender and recipients are specified; the body of the message is constructed using an adapter—an object of classNet::NetPrivate::WriteAdapter—which has a write method as well as an append method. This adapter is passed into the block and can be written to in an arbitrary way:

require "net/smtp"

smtp = Net::SMTP.new("smtp-server.fake1.com")

smtp.start

smtp.ready("t.s.eliot@fake1.com", "reader@fake2.com") do |obj|
obj.write "Let us go then, you and I,\r\n"
obj.write "When the evening is spread out against the sky\r\n"
obj.write "Like a patient etherised upon a table...\r\n"
end

Note here that the carriage-return linefeed pairs are necessary (if we actually want line breaks in the message). Those who are familiar with the actual details of the protocol should note that the message is “finalized” (with “dot” and “QUIT”) without any action on our part.

We can append instead of calling write if we want:

smtp.ready("t.s.eliot@fake1.com", "reader@fake2.com") do |obj|
obj << "In the room the women come and go\r\n"
obj << "Talking of Michelangelo.\r\n"
end

Finally, we offer a minor improvement. We add a puts method that will tack on the newline for us:

class Net::NetPrivate::WriteAdapter
def puts(args)
args << "\r\n"
self.write(*args)
end
end

This new method enables us to write this way:

smtp.ready("t.s.eliot@fake1.com", "reader@fake2.com") do |obj|
obj.puts "We have lingered in the chambers of the sea"
obj.puts "By sea-girls wreathed with seaweed red and brown"
obj.puts "Till human voices wake us, and we drown."
end

If your needs are more specific than what we’ve detailed here, we suggest you do your own experimentation. And if you decide to write a new interface for SMTP, please feel free.

18.2.5 Interacting with an IMAP Server

The IMAP protocol is not the prettiest in the world, but it is superior to POP3 in many ways. Messages may be stored on the server indefinitely (individually marked as read or unread). Messages may be stored in hierarchical folders. These two facts alone are enough to establish IMAP as more powerful than POP3.

The standard library net/imap enables us to interact with an IMAP server. As you would expect, you connect to the server and then log in to an account with a username and password, as shown in the following code:

require 'net/imap'

host = "imap.hogwarts.edu"
user, pass = "lupin", "riddikulus"

imap = Net::IMAP.new(host)
begin
imap.login(user, pass)
# Or alternately:
# imap.authenticate("LOGIN", user, pass)
rescue Net::IMAP::NoResponseError
abort "Could not login as #{user}"
end

# Process as needed...
imap.logout # break the connection

After you have a connection, you can do an examine on a mailbox; the default mailbox in IMAP is called INBOX. The responses method retrieves information about the mailbox, returning a hash of arrays (with the interesting data in the last element of each array). The following code finds the total number of messages in the mailbox ("EXISTS") and the number of unread messages ("RECENT"):

imap.examine("INBOX")
total = imap.responses["EXISTS"].last # total messages
recent = imap.responses["RECENT"].last # unread messages
imap.close # close the mailbox

Note that examine gives you read-only access to the mailbox. If, for example, you want to delete messages or make other changes, you should use select instead.

IMAP mailboxes are hierarchical and look similar to UNIX pathnames. You can use the create, delete, and rename methods to manipulate mailboxes:

imap.create("lists")
imap.create("lists/ruby")
imap.create("lists/rails")
imap.create("lists/foobar")

# Oops, kill that last one:
imap.delete("lists/foobar")

There are also methods named list (to list all the available mailboxes) and lsub (to list all the “active” or “subscribed” mailboxes). The status method will return information about the mailbox.

The search method will find messages according to specified criteria, and fetch will fetch a given message. Here is an example:

msgs = imap.search("TO","lupin")
msgs.each do |mid|
env = imap.fetch(mid, "ENVELOPE")[0].attr["ENVELOPE"]
puts "From #{env.from[0].name} #{env.subject}"
end

The fetch command in the preceding code appears convoluted because it returns an array of hashes. The envelope itself is similarly complex; some of its accessors are arrays of complex objects, and some are simply strings.

IMAP has the concept of unique IDs (UIDs) and sequence numbers for messages. Typically, methods such as fetch deal with sequence numbers and have counterparts such as uid_fetch that deal with UIDs. There is no room here to explain why both numbering systems are appropriate; if you are doing any significant programming with IMAP, however, you will need to know the difference (and never get them mixed up).

The net/imap library has extensive support for handling mailboxes, messages, attachments, and so on. For more details, refer to the API reference documentation.

18.2.6 Encoding/Decoding Attachments

Files are usually attached to email or news messages in a special encoded form. More often than not, the encoding is base64, which can be encoded or decoded with the Base64 class:

bin = File.read("new.gif")
str = Base64.encode64(bin) # str is now encoded
orig = Base64.decode64(str) # orig == bin

Older mail clients may prefer to work with uuencode and uudecode; in a case like this, an attachment is more a state of mind than anything else. The attachment is simply appended to the end of the email text, bracketed inside begin and end lines, with the begin line also specifying file permissions (which may be ignored) and filename. The pack directive u serves to encode a uuencoded string. The following code shows an example:

# Assume mailtext holds the text of the email

filename = "new.gif"
bin = File.read(filename)
encoded = [bin].pack("u")

mailtext << "begin 644 #{filename}"
mailtext << encoded
mailtext << "end"
# ...

On the receiving end, we would extract the encoded information and use unpack to decode it:

# ...
# Assume 'attached' has the encoded data (including the
# begin and end lines)

lines = attached.lines
filename = /begin \d\d\d (.*)/.scan(lines[0]).first.first
encoded = lines[1..-2].join("\n")
decoded = encoded.unpack("u") # Ready to write to filename

More modern mail readers usually use MIME format for email; even the text part of the email is wrapped (although the client strips all the header information before the user sees it).

A complete treatment of MIME would be lengthy and off topic here. However, the following code shows a simple example of encoding and sending an email with a text portion and a binary attachment. The encoding for binaries is usually base64, as shown here:

require 'net/smtp'

def text_plus_attachment(subject,body,filename)
marker = "MIME_boundary"
middle = "—#{marker}\n"
ending = "—#{middle}—\n"
content = "Content-Type: Multipart/Related; " +
"boundary=#{marker}; " +
"typw=text/plain"
head1 = <<-EOF
MIME-Version: 1.0
#{content}
Subject: #{subject}
EOF
binary = File.read(filename)
encoded = [binary].pack("m") # base64
head2 = <<EOF
Content-Description: "#{filename}"
Content-Type: image/gif; name="#{filename}"
Content-Transfer-Encoding: Base64
Content-Disposition: attachment; filename="#{filename}"

EOF

# Return...
head1 + middle + body + middle + head2 + encoded + ending
end

domain = "someserver.com"
smtp = "smtp.#{domain}"
user, pass = "elgar","enigma"

body = <<EOF
This is my email. There isn't
much to say. I attached a
very small GIF file here.

— Bob
EOF

mailtext = text_plus_attachment("Hi there...",body,"new.gif")

Net::SMTP.start(smtp, 25, domain, user, pass, :plain) do |mailer|
mailer.sendmail(mailtext, 'fromthisguy@wherever.com',
['destination@elsewhere.com'])
end

18.2.7 Case Study: A Mail-News Gateway

Online communities keep in touch with each other in many ways. Two of the most traditional of these are mailing lists and newsgroups.

Not everyone wants to be on a mailing list that may generate dozens of messages per day; some would rather read a newsgroup and pick through the information at random intervals. On the other hand, some people are impatient with Usenet and want to get the messages before the electrons have time to cool off.

Therefore, we get situations in which a fairly small, private mailing list deals with the same subject matter as an unmoderated newsgroup open to the whole world. Eventually someone gets the idea for a mirror—a gateway between the two.

Such a gateway isn’t appropriate in every situation, but in the case of the Ruby mailing list, there was such a gateway from 2001 to 2011. The newsgroup messages were copied to the list, and the list emails were posted on the newsgroup.

This task was accomplished by Dave Thomas (in Ruby, of course), and we present the code with his kind permission in Listings 18.6 and 18.7.

But let’s look at a little background first. We’ve taken a quick look at how email is sent and received, but how do we handle Usenet? As it turns out, we can access the newsgroups via a protocol called the Network News Transfer Protocol (NNTP). This creation, incidentally, was the work of Larry Wall, who later on gave us Perl.

Ruby doesn’t have a “standard” library to handle NNTP, but there are several gems available that provide NNTP functionality. These examples depend on the NNTP library written by a Japanese developer (known to us only as “greentea”).

The nntp.rb library defines a module NNTP containing a class called NNTPIO; it has instance methods connect, get_head, get_body, and post (among others). To retrieve messages, you connect to the server and call get_head and get_body, repetitively. (We’re oversimplifying this.) Likewise, to post a message, you basically construct the headers, connect to the server, and call the post method.

These programs use the smtp library, which we’ve looked at previously. The original code also does some logging to track progress and record errors; we’ve removed this logging for greater simplicity.

The file params.rb is used by both programs. This file contains the parameters that drive the whole mirroring process—the names of the servers, account names, and so on. The following is a sample file that you will need to reconfigure for your own purposes. (The domain names used in the code, which all contain the word fake, are obviously intended to be fictitious.)

# These are various parameters used by the mail-news gateway
module Params
NEWS_SERVER = "usenet.fake1.org" # name of the news server
NEWSGROUP = "comp.lang.ruby" # mirrored newsgroup
LOOP_FLAG = "X-rubymirror: yes" # avoid loops
LAST_NEWS_FILE = "/tmp/m2n/last_news" # last msg num read
SMTP_SERVER = "localhost" # host for outgoing mail
MAIL_SENDER = "myself@fake2.org" # Name used to send mail
# (On a subscription-based list, this
# name must be a list member.)

MAILING_LIST = "list@fake3.org" # Mailing list address
end

The module Params merely contains constants that are accessed by the two programs. Most are self-explanatory; we’ll only point out a couple of items here. First, the LAST_NEWS_FILE constant identifies a file where the most recent newsgroup message ID is stored; this is “state information,” so that work is not duplicated or lost.

Perhaps even more important, the LOOP_FLAG constant defines a string that marks a message as having already passed through the gateway. This avoids infinite recursion and prevents the programmer from being mobbed by hordes of angry netizens who have received thousands of copies of the same message.

You might be wondering, how do we actually get the mail into the mail2news program? After all, it appears to read standard input. Well, the author recommends a setup like this: The sendmail program’s .forward file first forwards all incoming mail to procmail. The.procmail file is set up to scan for messages from the mailing list and pipe them into the mail2news program. For the exact details of this, see the documentation associated with RubyMirror (found in the Ruby Application Archive). Of course, if you are on a non-UNIX system, you will likely have to come up with your own scheme for handling this situation.

Aside from what we’ve already said, we’ll let the code stand on its own, as shown in Listings 18.6 and 18.7.

Listing 18.6 Mail-to-News