Ruby and Data Formats - THE RUBY WAY, Third Edition (2015)

THE RUBY WAY, Third Edition (2015)

Chapter 15. Ruby and Data Formats

“Your information, sir,” the Librarian says. “Are you smart enough to tie that information into YOU ARE HERE?” Hiro says.“I’ll see what I can do, sir. The formats appear to be reconcilable.”

—Snow Crash, by Neal Stephenson

In computing, it is a fact of life that when information becomes complex enough, it evolves its own “mini-language” in which that information is best expressed. More often, it evolves multiple such mini-languages. We call these file formats or data formats.

Anyone who has used a computer can name many file formats. There are image formats such as GIF, JPG, and PNG; document formats such as DOC and PDF; “universal” formats such as CSV, JSON, and XML; and countless thousands of proprietary data formats, many of which are variations on the fixed-column data storage so common in the ancient times of the 1960s.

The simplest and most common data format is plain text. Files full of bytes encoded as ASCII (or its supersets, ISO 8859-1 and UTF-8) have proven themselves to be the most enduring and accessible form of data over the decades.

Even files containing only simple text, however, may have a structure (hence the popularity of JSON and XML). Other formats may be pure binary (such as Protocol Buffers) or some mixture of text and binary (such as RTF). The primary tradeoff between text and binary formats is readability versus speed; therefore, one or the other will be better depending on the circumstances.

No matter what format data is stored in, sooner or later we want to read it, parse it, and write it again. This chapter covers a few of the most common file formats, but there is no way to cover even all the common ones in a single book. If you want to parse such a format as vCard, iCal, or one of the hundreds of others, you will have to do a search for the appropriate libraries, or you may have to write your own.

15.1 Parsing JSON

JSON, short for JavaScript Object Notation, was designed as a human-readable format allowing arrays, hashes, numbers, and strings to be serialized as plain text to communicate data between programs, no matter what language they are written in. Designed partially as a reaction to the verboseness of XML, JSON is sparse both in the number of characters required and in the number of data types it supports.

We briefly looked at using JSON as a way to store Ruby data in Section 10.2.5, “Persisting Data with JSON” of Chapter 10, “I/O and Data Storage.” In this section, we’ll look at how to parse and manipulate the type of JSON data that might be provided by a website API. Before we do that, though, let’s review how to parse a string or file containing JSON data into a Ruby hash:

require 'json'

data = JSON.parse('{"students":[
{"name":"Alice","grade":4},
{"name":"Bob","grade":3}
]}')
p data["students"].first.values_at("name", "grade")
# ["Alice", 4]

What JSON calls an “object” corresponds almost perfectly to what we in Ruby-land know as a hash. Meanwhile, JSON contains several other easily recognizable types: arrays, strings, and numbers. JSON (including the strings contained inside it) is always encoded as UTF-8, and the only other types it is able to encode are true, false, and null (or nil in Ruby).

15.1.1 Navigating JSON Data

Once the JSON has been parsed, the nested hashes and arrays that are returned can be navigated to extract the specific data required. To illustrate this, we’ll use some JSON data provided by the GitHub public API:

require 'json'
require 'open-uri'
require 'pp'

json = open("https://api.github.com/repos/ruby/ruby/contributors")
users = JSON.parse(json)

pp users.first
# {"login"=>"nobu",
# "id"=>16700,
# "url"=>"https://api.github.com/users/nobu",
# "html_url"=>"https://github.com/nobu",
# [...other attributes...]
# "type"=>"User",
# "site_admin"=>false,
# "contributions"=>9850}

users.sort_by!{|u| -u["contributions"] }
puts users[0...10].map{|u| u["login"] }.join(", ")
# nobu, akr, nurse, unak, eban, ko1, drbrain, knu, kosaki, mame

In the preceding code, we are using the open-uri library for convenience. This is explained in greater detail in Chapter 18, “Network Programming”; for now, just be aware that it enables us to use the open method on a URI much as if it were a simple file.

Using open-uri, we download a string that contains a JSON array of the GitHub accounts of contributors to the main Ruby interpreter. (Many contributors who lack GitHub accounts are not included in this list.) Parsing the JSON gives us an array of hashes, with each hash containing information about one particular contributor, with keys including login, id, url, and contributions, among others.

Then, we use pp, the “prettyprint” library, to print the hash of attributes for the first contributor. (Every attribute is printed by pp, but we’ve elided some of them here to save space.) Next, we reverse-sort the list by the number of contributions so that the hashes are ordered from most commits to least commits. Finally, we map the sorted hashes of contributor information into just their login (or username) and print them as a comma-separated list.

Most real-world uses of JSON data are similar to this one: Write some code to fetch the data, parse the data into a native Ruby hash or array, and then find the array items or hash keys that contain the data you are interested in.

15.1.2 Handling Non-JSON Data Types

As you saw in Chapter 10, JSON cannot hold Ruby objects. Most of the time, using JSON will mean writing code to convert your objects into a hash of strings and numbers and then back again. Some JSON libraries, such as Oj, provide an automatic form of this object conversion.

Although most objects can be converted into a hash of string or number attributes, some data is not so easily converted. Symbols, instances of Time and Date, and currency amounts with exact cents are just a few examples of Ruby objects that cannot be directly represented in JSON.

For symbols, that means straightforward conversion into a JSON string. For Time objects, it means using the iso8601 or to_i method and then the parse or at method to reverse the process. For currencies, it means a decimal number stored as a JSON string. (The JSON spec allows any number to be floating point. As a result, numbers cannot be trusted to accurately contain hundredths without adding or subtracting small fractions.)

Ultimately, the way to use JSON to store such data always boils down to this: Convert it into a string or floating point number and then later parse that string or number back into the object you actually want.

15.1.3 Other JSON Libraries

In addition to the json standard library, a number of additional libraries provide specialized JSON functionality.

Sometimes JSON documents are so large that they will not fit entirely into memory, and need to be streamed from a file or over a network. In those cases, it is possible to use the json-stream gem to parse the JSON document as a stream. With stream parsing, it is possible to run a specific block of code any time a document, array, or object starts or ends, or whenever a key or value is parsed. Refer to the json-stream gem’s documentation for further information.

Another common use of JSON on the Web is to use a long-running HTTP connection to send a stream of events, with each event encoded as a separate JSON document. The Twitter API is perhaps the most common example of this use case, but many other activity streams offer similar APIs. The yajl-ruby gem is specifically optimized to handle this case. It can stream-parse JSON objects from an IO object as well as make the entire HTTP request itself, calling the provided block each time a JSON document is finished parsing.

Finally, there is the Oj gem, which is written in C and is highly optimized for high-speed encoding and parsing. It has sometimes been subtly incompatible with other JSON encoders, so be sure to test the results when using it. That said, its performance is highly impressive, and can be up to two to five times faster than the other JSON gems.

15.2 Parsing XML (and HTML)

XML, the eXtensible Markup Language, is a tag-based markup language. HTML, the HyperText Markup Language, is very similar (and, in fact, both XML and HTML are based on an earlier tag-based system called SGML). XML and HTML both rose to massive popularity in the 1990s, and are still used heavily today in development tools, program data storage and transfer, and all over the Web.

In XML and HTML, every element is named, and the entire document is hierarchically structured. Although it is highly verbose, everything is written in plain text and can be read by a human being directly, if needed. Another advantage over formats from the 70s and 80s is that XML allows variable-length data, rather than requiring each data field to fit into a specific number of bytes.

Three or four decades ago, memory constraints would have rendered XML largely impractical. However, if it had been used back then, issues such as the infamous “Y2K” problem would never have occurred (although even Y2K turned out to be more of a nuisance than a problem). There was a Y2K issue solely because most of our legacy data was stored and manipulated in fixed-length formats. So, although it has shortcomings, XML also has its uses.

In Ruby, the most common way to read, manipulate, and write XML or HTML is with the Nokogiri gem. The gem provides a Ruby interface (also called binding) for the libXML2 library, which is written in C. Nokogiri has two primary APIs, which could be called document based andstream based. We’ll look at both approaches.

15.2.1 Document Parsing

In a document-based approach, the entire file is parsed into a hierarchical, tree-like structure. This structure can be navigated, similar to the way we navigated hashes parsed from JSON earlier. Unlike hashes, Nokogiri documents also provide a query language that can be used to select only a particular subset of the elements in the document. Nokogiri supports two query languages: CSS selectors and XPaths.

Parsing HTML using the Nokogiri library can be done using the Nokogiri::HTML class method parse. The resulting document can be navigated and manipulated the same way as an XML document can, so we will only give XML examples here.

For our XML code examples, we’ll use the same simple XML file (shown in Listing 15.1). It represents part of a private library of books.

Listing 15.1 The books.xml File

<library shelf="Recent Acquisitions">
<section name="Ruby">
<book isbn="0321714636">
<title>The Ruby Way</title>
<author>Hal Fulton</author>
<author>André Arko</author>
<description>Third edition. The book you are now reading.
Ain't recursion grand?
</description>
</book>
</section>
<section name="Space">
<book isbn="0684835509">
<title>The Case for Mars</title>
<author>Robert Zubrin</author>
<description>Pushing toward a second home for the human
race.
</description>
</book>
<book isbn="074325631X">
<title>First Man: The Life of Neil A. Armstrong</title>
<author>James R. Hansen</author>
<description>Definitive biography of the first man on
the moon.
</description>
</book>
</section>
</library>

Let’s first parse our XML data as a document. We begin by requiring the nokogiri library and then parsing the XML:

require 'nokogiri'
doc = Nokogiri::XML.parse File.read("books.xml")

doc.root.name # library
doc.root["shelf"] # Recent Acquisitions

The root method returns an instance of Nokogiri::XML::Element that represents the “root” XML tag that contains the rest of the document. Any attributes on a tag (set inside the angle brackets) can be read with the [] method, similar to accessing a hash. Next, we navigate the document and select some other elements:

books = doc.css("section book") # a NodeSet, much like an array
books = doc.xpath("//section/book") # the same NodeSet

books.each do |book|
title = book.elements.first.text
authors = book.css("author").map(&:text)
puts "#{title} (ISBN #{book["isbn"]})"
puts " by #{authors.join(' and ')}"
end
# Output:
# The Ruby Way (ISBN 0672328844)
# by Hal Fulton and André Arko
# The Case for Mars (ISBN 0684835509)
# by Robert Zubrin
# First Man: The Life of Neil A. Armstrong (ISBN 074325631X)
# by James R. Hansen

Using the css and xpath methods, we are able to select only book tags that are also inside section tags. For detailed information about CSS selectors or XPath syntax, refer to a documentation website such as developer.mozilla.org. The methods both return a NodeSet, which acts much like an array. It provides methods such as each and size, as well as allowing indexed access with [].

Once we have selected the book tags, we use them to print the text from the first element (which is the title tag), and we then use the css method to get the author tags inside each individual book tag. Finally, we read the "isbn" attribute from each book using the [] method, and print information about each book. Now let’s look at the section tags:

doc.root.elements.map{|e| e["name"] } # ["Ruby", "Space"]

space = doc.at_css("section[name=Space]") # an Element
space = doc.at_xpath("//section[@name='Space']") # the same Element

books.include?(space.elements.first) # true

Each element has its own elements method, returning the list of elements contained inside that element. By reading the "name" attribute, we can see the section tag names. Next, we use the at_css and at_xpath methods to select the section tag whose "name" attribute is set to "Space". The at_ methods only return the first element selected, if more than one element matches the given selector. Finally, we show that the elements returned via different methods are all the same Ruby objects, because the first book inside the section named "Space" is also inside the books we selected earlier.

There are many different ways to select the same elements, including the elements array and the css and xpath methods. It is also possible to use any of the Enumerable methods such as find or select to select elements using a block.

You can also manipulate elements. Each method we’ve used here, such as text or [], has a corresponding setter that can be used to add, delete, or modify attributes and element contents. Afterward, the new XML or HTML can be written to a string using the to_xml or to_html, respectively.

For more information on using Nokogiri to parse XML and HTML documents, see the full API documentation at nokogiri.org.

15.2.2 Stream Parsing

The stream-style approach is a “parse as you go” technique, useful when your documents are large or you have memory limitations; it parses the file as it reads it from disk, and the entire file is never stored in memory. Instead, as each element is read, user-supplied methods or blocks are run as callbacks. This allows the entire file to be processed incrementally.

Let’s process the same XML data file in a stream-oriented way. (We probably wouldn’t do that in reality because this file is small.) There are variations on this concept, but Listing 15.2 shows one way. The trick is to define a listener class whose methods will be the target of callbacks from the parser. When you’re parsing XML, make your class a subclass of Nokogiri::XML::SAX::Document. When you’re parsing HTML, subclass Nokogiri::HTML::SAX::Document instead.

Listing 15.2 Stream Parsing

require 'nokogiri'
class Listener < Nokogiri::XML::SAX::Document
def start_element(name, attrs = [])
case name
when "book"
isbn = attrs.find{|k,v| k == "isbn" }.last
puts "New book with ISBN number #{isbn}"
when "title", "author", "description"
print "#{name.capitalize}: "
else
end
end

def characters(string)
return unless string =~ /\w/ # ignore all-whitespace
print string.tr("\n", " ").squeeze(" ")
end

def end_element(name)
print "\n" if %w[book title author description].include?(name)
end

def end_document
puts "The document has ended."
end
end

xml = File.read("books.xml")
Nokogiri::XML::SAX::Parser.new(Listener.new).parse(xml)

The Nokogiri::XML::SAX::Document class provides empty callback methods, which we override in our subclass Listener. For example, when the parser encounters an opening tag, it calls the start_element method with the name of the tag and any attributes set on that tag. Inside that method, we use the name of the tag to decide what to do: If it is a book tag, we announce that a new book has been found, and we print its ISBN number. If it is a title, author, or description, we print that name, and for other tags, we do nothing.

In the characters method, the guard clause at the beginning of the method returns without printing unless the string contains at least one “word” character. Then, we simply print the text inside any tag, after replacing newlines with spaces and replacing any repeated spaces with a single space.

The end_element method prints a newline when each book information tag ends, after we have printed out the text that was contained inside that tag. We check the list of tags because we do not want to print a newline at the end of section and library tags. The end_documentmethod works similarly, but it’s only called once.

Although Listing 15.2 is somewhat contrived, using it to process books.xml produces the organized (and hopefully useful) output seen in Listing 15.3.

Listing 15.3 Output from the Stream Parsing Example

New book with ISBN number 0321714636
Title: The Ruby Way
Author: Hal Fulton
Author: André Arko
Description: Third edition. The book you are now reading.
Ain't recursion grand?

New book with ISBN number 0684835509
Title: The Case for Mars
Author: Robert Zubrin
Description: Pushing toward a second home for the human race.

New book with ISBN number 074325631X
Title: First Man: The Life of Neil A. Armstrong
Author: James R. Hansen
Description: Definitive biography of the first man on the moon.

The document has ended.

For more information on using Nokogiri to parse XML and HTML documents, see the full API documentation at nokogiri.org.

15.3 Working with RSS and Atom

As the Web grew, developers noticed a need to announce events, changes, or just content that had been newly added to a particular site. The solution that eventually emerged is typically referred to as a site’s feed, served as XML structured according to the RSS or Atom feed standard.

RSS (Really Simple Syndication) was standardized early on in the development of the Internet, and different developers extended it with additional features in interesting (and sometimes contradictory) ways. A new standardization effort eventually produced Atom, which is the most capable and widely used feed format today.

RSS is XML based, so you could simply parse it as XML. However, the fact that it is slightly higher level makes it appropriate to have a dedicated parser for the format. Furthermore, the messiness of the RSS standard is legendary, and it is not unusual at all for broken software to produce RSS that a parser may have great difficulty parsing.

This inconvenience is even more true because there are multiple incompatible versions of the RSS standard; the most common versions are 0.9, 1.0, and 2.0. The RSS versions, like the manufacturing of hotdogs, are something whose details you don’t want to know unless you must.

15.3.1 Parsing Feeds

Let’s look briefly at processing both RSS and Atom using the Ruby standard library rss, which can seamlessly handle not only all three versions of RSS, but Atom as well. Here, we take the feed from NASA’s Astronomy Picture of the Day service and then print the title of each item in the feed:

require 'rss'
require 'open-uri'

xml = open("http://apod.nasa.gov/apod.rss").read
feed = RSS::Parser.parse(xml, false)

puts feed.channel.description
feed.items.each_with_index do |item, i|
puts "#{i + 1}. #{item.title.strip}"
end

Note how the RSS parser retrieves the channel for the RSS feed; our code then prints the title associated with that channel. There is also a list of items (retrieved by the items accessor), which can be thought of as a list of articles. Our code retrieves the entire list and prints the title of each one.

Of course, the output from this is highly time sensitive, but here are the results when I ran that code:

Astronomy Picture of the Day
1. Star Trails Over Indonesia
2. Jupiter and Venus from Earth
3. No X rays from SN 2014J
4. Perseid in Moonlight
5. Surreal Moon
6. Rings Around the Ring Nebula
7. Collapse in Hebes Chasma on Mars

Before going any further, let me talk about courtesy to feed providers. A program like the preceding one should be run with caution because it uses the provider’s bandwidth. In any real application, such as an actual feed aggregator, caching should always be done. Here is a naive (but functional) cache:

unless File.exist?("apod.rss")
File.write("apod.rss", open("http://apod.nasa.gov/apod.rss"))
end

xml = File.read("apod.rss")

This simple cache just reads the feed from a file and then fetches the feed into that file if it does not exist. More useful caching would check the age of the cached file and refetch the feed if the file is older than some threshold. It would also use the HTTP header If-Modified-Since orIf-None-Match. However, such a system is beyond the scope of this simple example.

Atom feeds can be parsed by the same RSS::Parser.parse method. The rss library will read the feed contents to determine which parser to use. The only real difference to note is that Atom lacks a channel attribute. Instead, look for the title and author attributes directly on the feed itself.

15.3.2 Generating Feeds

In addition to parsing feeds, it’s also possible to generate RSS or Atom using the RSS standard library’s RSS::Maker class. Here, we create a small Atom feed for a hypothetical website:

require 'rss'

feed = RSS::Maker.make("atom") do |f|
f.channel.title = "Feed Your Head"
f.channel.id = "http://nosuchplace.org/home/"
f.channel.author = "Y.T."
f.channel.logo = "http://nosuchplace.org/images/headshot.jpg"
f.channel.updated = Time.now

f.items.new_item do |i|
i.title = "Once again, here we are"
i.link = "http://nosuchplace.org/articles/once_again/"
i.description = "Don't you feel more like you do now than usual?"
i.updated = Time.parse("2014-08-17 10:23AM")
end

f.items.new_item do |i|
i.title = "So long, and thanks for all the fiche"
i.link = "http://nosuchplace.org/articles/so_long_and_thanks/"
i.description = "I really miss the days of microfilm..."
i.updated = Time.parse("2014-08-12 3:52PM")
end
end

puts feed.to_xml

The Maker is created with a block initializer similar to the one we discussed in Section 11.1.3, “Using More Elaborate Constructors,” of Chapter 11, “OOP and Dynamic Features in Ruby.” We use the block to set the feed’s title, id, author, and updated timestamp, because they are all required for the Atom feed to be valid.

Then, we use the new_item method, which creates a new feed entry and adds it to the feed’s list of entries, again using a block. Each item has a title, description (or summary), link to the URL with the full content, and an updated timestamp. The optional content attribute provides a place to include the full content in the feed, if desired.

The RSS standard library is fairly well developed, but keep in mind that other libraries exist to both parse and generate RSS and Atom feeds. If your needs aren’t well served by the standard library, check ruby-toolbox.com and search around the Web to see if there’s something that suits you better.

15.4 Manipulating Image Data with RMagick

As computers, tablets, and phones become the dominant way to consume media, working with images and graphics is becoming a more and more important part of creating programs. Programmers need ways to create and manipulate images across different platforms in many different, complicated formats. In Ruby, the easiest way to do this is with RMagick, a gem created by Tim Hunter.

The RMagick gem is a Ruby binding for the ImageMagick library. It can be installed by running gem install rmagick, but you must you have the ImageMagick library installed first. If you are on Linux, you probably already have it; if you are on Mac OS X, you can install it using Homebrew. If you need help installing it, the ImageMagick website at http://imagemagick.org might be a good place to start.

Because RMagick is just a binding, it is able to support all the image formats supported by the underlying library. Those include all the common ones, such as JPG, GIF, PNG, and TIFF, but also dozens of others. The same is true for the operations RMagick can perform. The gem implements the full ImageMagick API, adapted to Ruby via the use of symbols, blocks, and other features.

The ImageMagick API is really huge, by the way. This chapter would not be enough to cover it in detail, nor would this book. The upcoming sections will give you a good background in RMagick, however, and you can find out anything else you may need from the project website athttp://www.imagemagick.org/RMagick/doc/.

15.4.1 Common Graphics Tasks

One of the easiest and most common tasks you might want to perform on an image file is simply to determine its characteristics (width and height in pixels, and so on). Let’s look at retrieving a few of these pieces of metadata.

Figure 15.1 shows a pair of simple images that we’ll use for this code example (and later examples in the next section). The first one (smallpic.jpg) is a simple abstract picture created with a drawing program; it features a few different shades of gray, a few straight lines, and a few curves. The second is a photograph I took in 2002 of a battered automobile in rural Mexico. Both images were converted to grayscale for printing purposes. Listing 15.4 shows how to read these images and extract a few pieces of information.

Image

Figure 15.1 Two sample image files

Listing 15.4 Retrieving Information from an Image

require 'rmagick'

def show_info(fname)
img = Magick::Image::read(fname).first
fmt = img.format
w,h = img.columns, img.rows
dep = img.depth
nc = img.number_colors
nb = img.filesize
xr = img.x_resolution
yr = img.y_resolution
res = Magick::PixelsPerInchResolution ? "inch" : "cm"

puts <<-EOF.gsub(/^\s+/, '')
File: #{fname}
Format: #{fmt}
Dimensions: #{w}x#{h} pixels
Colors: #{nc}
Image size: #{nb} bytes
Resolution: #{xr}/#{yr} pixels per #{res}
EOF
puts
end

show_info("smallpic.jpg")
show_info("vw.jpg")

Here is the output of the Listing 15.4 code:

File: smallpic.jpg
Format: JPEG
Dimensions: 257x264 pixels
Colors: 248
Image size: 19116 bytes
Resolution: 72.0/72.0 pixels per inch

File: vw.jpg
Format: JPEG
Dimensions: 640x480 pixels
Colors: 256
Image size: 55892 bytes
Resolution: 72.0/72.0 pixels per inch

Now let’s examine the details of how the code in Listing 15.4 gave us that output. Notice how we retrieve all the contents of a file with Magick::Image::read. Because a file (such as an animated GIF) can contain more than one image, this operation actually returns an array of images (and we look at the first one by calling first). We can also use Magick::ImageList.new to read an image file.

The image object has a number of readers such as format (the name of the image format), filesize, depth, and others that are intuitive. It may be less intuitive that the width and height of the object are retrieved by columns and rows, respectively. (This is because we are supposed to think of an image as a grid of pixels.) It also may not be intuitive that the resolution is stored as two numbers. Images can have rectangular pixels, which produces different horizontal and vertical resolutions.

There are many other properties and pieces of metadata you can retrieve from an image. Refer to the online documentation for RMagick for more details.

One common task we often perform is to convert an image from one format to another. The easy way to do this in RMagick is to read an image in any supported format and then write it to another file. The file extension is used to determine the new format. Needless to say, it does a lot of conversion behind the scenes. Here is a simple example:

img = Magick::Image.read("smallpic.jpg")
img.first.write("smallpic.gif") # Convert to a GIF

Frequently, we want to change the size of an image (smaller or larger). The most common methods for this are thumbnail, resize, and scale. These can all accept either a floating point number (representing a scaling factor) or a pair of numbers (representing the actual new dimensions in pixels). Other differences are summarized in Listing 15.5 and its comments.

Listing 15.5 Four Ways to Resize an Image

require 'rmagick'

img = Magick::ImageList.new("vw.jpg")

# Thumbnail is designed to shrink a large image to a small
# preview. It is the fastest, especially with small sizes.

pic1 = img.thumbnail(0.2) # Reduce to 20%
pic2 = img.thumbnail(64,48) # Reduce to 64x48 pixels

# Resize is medium speed, and makes an image fit inside the
# given dimensions without changing aspect ratio. The
# optional 3rd and 4th parameters are the filter and blur,
# defaulting to LanczosFilter and 1.0, respectively.

pic3 = img.resize(0.40) # Reduce to 40%
pic4 = img.resize(320, 240) # Fit inside 320x240
pic5 = img.resize(300, 200, Magick::LanczosFilter, 0.92)

# Scale is the slowest, as it scales each dimension of the
# image independently (distorting it if necessary).

pic8 = img.scale(0.60) # Reduce to 60%
pic9 = img.scale(400, 300) # Reduce to 400x300

Many other transformations can be performed on an image. Some of these are simple and easy to understand, whereas others are complex. We’ll explore a few interesting transformations and special effects in the next section.

15.4.2 Special Effects and Transformations

Some operations we might want to do on an image are to flip it, reverse it, rotate it, distort it, alter its colors, and so on. RMagick provides literally dozens of methods to perform such operations, and many of these are highly “tunable” by their parameters.

Listing 15.6 demonstrates 12 different effects. To make the code a little more concise, the method example simply takes a filename, a symbol corresponding to a method, and a new filename; it basically does a read, a method call, and a write. The individual methods (such asdo_rotate) are simple for the most part; these are where the image passed in gets an actual instance method called (and then the resulting image is the return value).

Listing 15.6 Twelve Special Effects and Transformations

require 'RMagick'

def do_flip(img)
img.flip
end

def do_rotate(img)
img.rotate(45)
end

def do_implode(img)
img = img.implode(0.65)
end

def do_resize(img)
img.resize(120,240)
end

def do_text(img)
text = Magick::Draw.new
text.annotate(img, 0, 0, 0, 100, "HELLO") do
self.gravity = Magick::SouthGravity
self.pointsize = 72
self.stroke = 'black'
self.fill = '#FAFAFA'
self.font_weight = Magick::BoldWeight
self.font_stretch = Magick::UltraCondensedStretch
end
img
end

def do_emboss(img)
img.emboss
end


def do_spread(img)
img.spread(10)
end

def do_motion(img)
img.motion_blur(0,30,170)
end

def do_oil(img)
img.oil_paint(10)
end

def do_charcoal(img)
img.charcoal
end

def do_vignette(img)
img.vignette
end

def do_affine(img)
spin_xform = Magick::AffineMatrix.new(
1, Math::PI/6, Math::PI/6, 1, 0, 0)
img.affine_transform(spin_xform) # Apply the transform
end



def example(old_file, meth, new_file)
img = Magick::ImageList.new(old_file)
new_img = send(meth,img)
new_img.write(new_file)
end

example("smallpic.jpg", :do_flip, "flipped.jpg")
example("smallpic.jpg", :do_rotate, "rotated.jpg")
example("smallpic.jpg", :do_resize, "resized.jpg")
example("smallpic.jpg", :do_implode, "imploded.jpg")
example("smallpic.jpg", :do_text, "withtext.jpg")
example("smallpic.jpg", :do_emboss, "embossed.jpg")

example("vw.jpg", :do_spread, "vw_spread.jpg")
example("vw.jpg", :do_motion, "vw_motion.jpg")
example("vw.jpg", :do_oil, "vw_oil.jpg")
example("vw.jpg", :do_charcoal, "vw_char.jpg")
example("vw.jpg", :do_vignette, "vw_vig.jpg")
example("vw.jpg", :do_affine, "vw_spin.jpg")

The methods used here are flip, rotate, implode, resize, annotate, and others. The results are shown in Figure 15.2 in a montage.

Image

Figure 15.2 Twelve special effects and transformations

Many other transformations can be performed on an image. Consult the online documentation at http://www.imagemagick.org/RMagick/doc/.

15.4.3 The Drawing API

RMagick has an extensive drawing API for drawing lines, polygons, and curves of various kinds. It deals with filling, opacity, colors, text fonts, rotating/skewing, and other issues.

A full treatment of the API is beyond the scope of this book. Let’s look at a simple example, however, to understand a few concepts.

Listing 15.7 shows a program that draws a simple grid on the background and draws a few filled shapes on that grid. The image is converted to grayscale, resulting in the image shown in Figure 15.3.

Listing 15.7 A Simple Drawing

require 'rmagick'

img = Magick::ImageList.new
img.new_image(500, 500)

purplish = "#ff55ff"
yuck = "#5fff62"
bleah = "#3333ff"

line = Magick::Draw.new
50.step(450, 50) do |n|
line.line(n, 50, n, 450) # vert line
line.draw(img)
line.line(50, n, 450, n) # horiz line
line.draw(img)
end

# Draw a circle
cir = Magick::Draw.new
cir.fill(purplish)
cir.stroke('black').stroke_width(1)
cir.circle(250, 200, 250, 310)
cir.draw(img)

rect = Magick::Draw.new
rect.stroke('black').stroke_width(1)
rect.fill(yuck)
rect.rectangle(340, 380, 237, 110)
rect.draw(img)

tri = Magick::Draw.new
tri.stroke('black').stroke_width(1)
tri.fill(bleah)
tri.polygon(90, 320, 160, 370, 390, 120)
tri.draw(img)

img = img.quantize(256, Magick::GRAYColorspace)
img.write("drawing.gif")

Image

Figure 15.3 A simple drawing

Let’s examine Listing 15.7 in detail. We start by creating an “empty” image with ImageList.new and then calling new_image on the result. Think of this as giving us a “blank canvas” of the specified size (500 by 500 pixels).

For convenience, let’s define a few colors (with creative names such as purplish and yuck). These are strings that specify colors just as we would in HTML. The underlying ImageMagick library is also capable of understanding many color names such as "red" and "black"; when in doubt, experiment or specify the color in hex.

We then create a drawing object called line; this is the Ruby object corresponding to the graphical object we will see on the screen. The variable is sometimes named gc or something similar (probably standing for graphics context), but a more descriptive name seems natural here.

We then call the method line on our drawing object (which admittedly gets a little confusing). In fact, we call it repeatedly, twice in each iteration of a loop. If you spend a moment studying the coordinates, you’ll see that each iteration of the loop draws a horizontal line and a vertical one.

After each line call, we call draw on the same drawing object and pass in the image reference. This is an essential step because it is when the graphical object actually gets added to the canvas.

If you are like me, a call such as shape.draw(image) may be a little confusing. In general, my method calls look like this:

big_thing.operation(little_thing)
# For example: dog.wag(tail)

However, the call in question feels to me more like this:

little_thing.operation(big_thing)
# Continuing the analogy: tail.wag(dog)

But this idiom is actually common, especially in the realm of drawing programs and GUI frameworks. And it makes perfect sense in a classic OOP way: A shape should know how to draw itself, implying it should have a draw method. It needs to know where to draw itself, so it needs the canvas (or whatever) passed in.

If you’re not like me, though, you were never bothered by the question of which object should be the receiver. That puts you at a tiny advantage.

So after we draw the grid of lines, we then draw a few shapes. The circle method takes the center of the circle and a point on the circle as parameters. (Notice we don’t draw by specifying the radius!) The rectangle method is even simpler; we draw it by specifying the upper-left corner (lower-numbered coordinates) and the lower-right corner (higher-numbered coordinates). Finally, we draw a triangle, which is just a special case of a polygon; we specify each point in order, and the final line (from end point to start point) is added automatically.

Each of these graphical objects has a few methods called that we haven’t looked at yet. Look at this “chained” call:

shape.stroke('black').stroke_width(1)

This gives us a “pen” of sorts; it draws in black ink with a width of 1 pixel. The color of the stroke actually does matter in many cases, especially when we are trying to fill a shape with a color.

That, of course, is the other method we call on our three shapes. We call fill to specify what color it should have. (There are other more complex kinds of filling involving hatching, shading, and so on.) The fill method replaces the interior color of the shape with the specified color, knowing that the stroke color serves as a boundary between “inside” and “outside” the shape.

Numerous other methods in the drawing API deal with opacity, spatial transformations, and many more things. There are methods that analyze, draw, and manipulate graphical text strings. There is even a special RVG API (Ruby Vector Graphics) that is conformant to the W3C recommendation on SVG (Scalable Vector Graphics).

There is no room here to document these and many other features. For more information, go to http://www.imagemagick.org/RMagick/doc/, as usual.

15.5 Creating PDF Documents with Prawn

The Portable Document Format (PDF), originally popularized by Adobe’s Acrobat Reader, has been popular for many years. It has proven useful in distributing “printer-ready” documents in a rich format that is independent of any particular word processing software or operating system.

The most widely adopted library for creating PDFs from Ruby is the Prawn gem, created by Gregory Brown and many other contributors. We’ll look at how Prawn works and then use it to create a PDF document.

15.5.1 Basic Concepts and Techniques

The class that “drives” Prawn is Prawn::Document. This class enables the creation of PDF documents using two or three coding styles. Here is one example:

require "prawn"

doc = Prawn::Document.new # Start a new document
doc.text "Lorem ipsum dolor..." # Add some text
doc.render_file "my_first.pdf" # Write to a file

Alternatively, you may call the class method generate and pass it the output file name and a block:

Prawn::Document.generate("portrait.pdf") do
text "Once upon a time and a very good time it was "
text "there was a moocow coming down along the road..."
end

If you specify a parameter for the block, the generator object is passed in explicitly:

Prawn::Document.generate("ericblair.pdf") do |doc|
doc.text "It was a bright cold day in April, "
doc.text "and the clocks were striking thirteen."
end

These forms are all basically equivalent. Choosing one is mostly a matter of style and convenience.

Now let’s talk about page coordinates. The origin of a PDF page is [0,0] (which is at the bottom left of the page, as we’re used to in mathematics). A bounding box is an imaginary rectangle within the space of a page. There is one default bounding box called the margin box, which serves as a container or boundary for all content on the page.

The cursor in Prawn naturally starts at the top, even though the coordinate system has its origin at the bottom. Adding text to the document moves the cursor (or the cursor can be moved manually via such methods as move_cursor_to, move_up, and move_down). The methodcursor returns the current cursor position.

Prawn uses the “standard” measurement of a point (which is 1/72 inch). If you require 'prawn/measurement_extensions' you can use other units as needed.

Prawn also has a large set of text-handling routines as well as graphics primitives for drawing lines, curves, and shapes.

15.5.2 An Example Document

Let’s look at a fairly contrived example. This program (see Listing 15.8) divides the page into four rectangles and places different items in each of them. Figure 15.4 shows the result.

Listing 15.8 pdf-demo.rb

require 'prawn'

# Adapted from code contributed by Brad Ediger

class DemoDocument
def initialize
@pdf = Prawn::Document.new
end

def render_file(file)
render
@pdf.render_file(file)
end

def render
side = @pdf.bounds.width / 2.0
box(0, 0, side, side) { star }
box(side, 0, side, side) { heart }
box(0, side, side, side) { ruby }
box(side, side, side, side) { misc_text }
end

private

# Run the given block in a bounding box inset from the parent by
# 'padding' PDF points.
def inset(padding)
left = @pdf.bounds.left + padding
top = @pdf.bounds.top - padding
@pdf.bounding_box([left, top],
width: @pdf.bounds.width - 2*padding,
height: @pdf.bounds.height - 2*padding) { yield }
end

# Draw a width-by-heigt box at (x, y), yielding inside a bounding
# box so content may be drawn inside.
def box(x, y, w, h)
@pdf.bounding_box([x, @pdf.bounds.top - y], width: w, height: h) do
@pdf.stroke_bounds
inset(10) { yield }
end
end

def star
reps = 15
size = 0.24 * @pdf.bounds.width
radius = 0.26 * @pdf.bounds.width
center_x = @pdf.bounds.width / 2.0
center_y = @pdf.bounds.height / 2.0
reps.times do |i|
@pdf.rotate i * 360.0 / reps, origin: [center_x, center_y] do
edge = center_y + radius
@pdf.draw_text ")", size: size, at: [center_x, edge]
end
end
end

def ruby
@pdf.image "ruby.png",
at: [0, @pdf.cursor],
width: @pdf.bounds.width,
height: @pdf.bounds.height
end

def heart
10.times do |i|
inset(i*10) do
box = @pdf.bounds
center = box.width / 2.0
cusp_y = 0.6 * box.top

k = center * Prawn::Graphics::KAPPA
@pdf.stroke_color(0, 0, 0, 100-(i*10))
@pdf.stroke do
# Draw a heart using a Bezier curve with two paths
paths = [[0, 0.9*center], [box.right, 1.1*center]]
paths.each do |side, midside|
@pdf.move_to [center, cusp_y]
@pdf.curve_to [side, cusp_y],
bounds: [[center, cusp_y + k], [side, cusp_y + k]]
@pdf.curve_to [center, box.bottom],
bounds: [[side, 0.6 * cusp_y], [midside, box.bottom]]
end
end
end
end

# reset stroke color
@pdf.stroke_color 0, 0, 0, 100
end

def misc_text
first_lines = <<-EOF
Call me Ishmael. Somewhere in la Mancha, in a place whose
name I do not care to remember, a gentleman lived not long
ago, one of those who has a lance and ancient shield on a
shelf and keeps a skinny nag and a greyhound for racing.
The sky above the port was the color of television, tuned to
a dead channel. It was a pleasure to burn. Granted: I am an
inmate of a mental hospital; my keeper is watching me, he
never lets me out of his sight; there's a peephole in the
door, and my keeper's eye is the shade of brown that can
never see through a blue-eyed type like me. Whether I shall
turn out to be the hero of my own life, or whether that
station will be held by anybody else, these pages must show.
I have never begun a novel with more misgiving.
EOF
first_lines.gsub!(/\n/, " ")
first_lines.gsub!(/ +/, " ")
@pdf.text first_lines
end
end

DemoDocument.new.render_file("demo.pdf")

Image

Figure 15.4 Output of pdf-demo.rb

The top two “boxes” illustrate the use of the drawing API. In the upper left, we see an abstract “star” pattern created by drawing a sequence of arcs with the same radius but different centers. In the upper right, a more advanced use of the drawing API uses Bezier curves to produce a heart shape.

In the lower left, we embed a PNG image using the image method. Finally, in the lower right, we see simple text wrapping around its bounding box.

This example, however, illustrates only a tiny portion of the Prawn API. Consult the reference documentation at prawnpdf.org for complete information on the rich API.

15.6 Conclusion

In this chapter, we have looked at how to parse and manipulate JSON, and how to use Nokogiri to parse XML in both stream-oriented and document-oriented styles. We’ve looked at parsing feeds in XML-based formats, and have seen how the rss library handles reading and writing both RSS and Atom.

We’ve looked at reading and manipulating graphic images in many formats with RMagick; we’ve also seen its drawing API, which enables us to add arbitrary text and shapes to an image. Finally, we’ve seen how Prawn can produce complex, high-quality PDF documents in a programmatic fashion.

In the next chapter, we will look at a different topic entirely. The next chapter deals with effective testing and debugging in Ruby.