The Rails 4 Way (2014)

Chapter 20. Asset Pipeline

It’s not enough to solve the problem, we have to have the pleasure.

—DHH, RailsConf 2011 keynote

The asset pipeline is one of those Rails magic features that makes developer’s life so easy that once you master it you will never want to go back. It also significantly improves perceived performance of your application and reduces burdens on your application server. It’s a huge win for Rails overall that nonetheless might make you want to tear your hair out and switch to (shudder) Django until you understand how it works. Persevere! We promise it’s worth the learning curve. According to David, the asset pipeline was by far his favorite element of the Rails 3.1 release.

“Wait,” you might ask, “what is an asset”?

It’s simple - by “assets” we mean images, Javscript, CSS, and other static files that we need in order to properly render our pages.

Web applications built with early versions of Rails shared common problems with managing static assets. Before the asset pipeline, you just dumped all your JavaScript files into the public/javascripts directory, all your CSS files into public/stylesheets and your image files intopublic/images without any structure. Afterwards, you could load all your Javscript files within your templates using the helper <%= javascript_include_tag :all %>. It completely ignored files in subdirectories of public/javascripts, so that if you wanted to organize your assets into subdirectories you had to manually load them into your layout. What a mess!

There were other inconveniences as well. For instance, if you wanted to load the files in a certain order, you had to replace the :all directive with a manually maintained list of “includes” in the exact order that you needed. When you wanted to use a library that came with JavaScript and CSS files (e.g. twitter bootstrap) you had to copy those files into your public directory and keep it under source control so that they will be available for the running application. Worse, you had to read the README files carefully to figure out just what files exactly do you needed to copy and in which exact order you had to load them. Not fun.

20.1 Asset Pipeline

The major goal of the asset pipeline is to make management of static assets easy, even trivial. In this chapter, we discuss organization of assets, how can they be packaged into neat external gem dependencies, available asset pre-processors and compressors, helpers that assist us with the Asset Pipeline and more.

Incidentally, automated asset management is not a new concept. It has existed since before the Rails era, and plugins to add this critical functionality to Rails began appearing many years ago. The most successful one is Sprockets, written primarily by Sam Stephenson of 37signals and Rails core team fame. Sprockets was eventually incorporated into Rails itself and is at the core of the Rails asset pipeline implementation.

20.2 Wish List

Which features of asset management solutions would be most useful to us in building a Rails application?

For starters, we could organize the asset files into a sensible directory tree instead of ‘junk drawer’ directories filled haphazardly.

We might also want to compress all our assets, so that they can be served faster to web browsers and eat up less bandwidth.

We could also consolidate multiple source files of the same kind (JavaScript or CSS) into single files, reducing the number of HTTP requests made by the browser and significantly improving page load times.

On the other hand, compressing and consolidating all those source files could make debugging during development a nightmare, so our wish list would also include the ability to turn those features off except for production environments.

What else? To speed up page loading times even more, we might “pre-shrink” our asset files with the maximum compression level, so that our web server doesn’t waste CPU cycles zipping up the same files over and over again on each request.

We would also want to include cache-busting features, giving us the ability to force expiration of stale assets from all cache layers (HTTP proxies, browsers, etc.) when their content changes.

Furthermore, we might want the ability to transparently compile languages such as CoffeeScript for Javscript assets and Sass and Less for CSS stylesheets.

All the highlighted features in our wish list and more are part of the Asset Pipeline, making this aspect of Rails programming a lot more enjoyable than in earlier versions.

20.3 The Big Picture

We could try to describe how the entire Asset Pipeline works at the high level now, but it would require too many forward references to stuff we haven’t yet explained. Therefore we are going to build out our understanding from the bottom up. Keep in mind the overall goal: concatenating and serving asset files and “bundles” composed of multiple files, which can possibly be pre- and post-processed or compiled from different formats.

Now let’s dive in.

20.4 Organization. Where does everything go?

Asset Pipeline continues with the Rails tradition of separate directories for images, stylesheets and scripts, but adds an additional dimension of organization. There are now three locations where you store assets in your project directory. Those are app/assets, lib/assets, and vendor/assets.

This small change already gives us a much better way to organize the project files. Files specific to the current project go into app/assets, external libraries go into vendor/assets, and assets for your own libraries can go into lib/assets.

You no longer need to copy the static assets bundled with your gems into your project directory. The asset pipeline will find them automatically and make them available for your application (more on this later).

20.5 Manifest files

The organizational structure doesn’t just involve new directories. If you look into the app/assets directory of a freshly generated Rails 4 application you’ll notice a couple of files with include directives in them: app/assets/javascripts/application.js andapp/assets/stylesheets/application.css. Those are called asset manifest files, and they specify instructions on where the pipeline processor can find other assets and in which order to load them. The loaded files are concatenated into a single “bundle” file named after the manifest.

Lets take a look at application.js:

1 // This is a manifest file that'll be compiled into application.js,

2 // which will include all the files listed below.

3 //

4 // Any Javscript/Coffee file within this directory, lib/assets/javascripts,

5 // vendor/assets/javascripts, or vendor/assets/javascripts of plugins,

6 // if any, can be referenced here using a relative path.

7 //

8 // It's not advisable to add code directly here, but if you do, it'll

9 // appear at the bottom of the compiled file.

10 //

11 // WARNING: THE FIRST BLANK LINE MARKS THE END OF WHAT'S TO BE PROCESSED,

12 // ANY BLANK LINE SHOULD GO AFTER THE REQUIRES BELOW.

13 //

14 //= require jquery

15 //= require jquery_ujs

16 //= require turbolinks

17 //= require_tree .

And application.css:

1 /*

2 * This is a manifest file that'll be compiled into application.css,

3 * which will include all the files listed below.

4 *

5 * Any CSS and SCSS file within this directory, lib/assets/stylesheets,

6 * vendor/assets/stylesheets, or vendor/assets/stylesheets of plugins,

7 * if any, can be referenced here using a relative path.

8 *

9 * You're free to add application-wide styles to this file and they'll

10 * appear at the top of the compiled file, but it's generally better

11 * to create a new file per style scope.

12 *

13 *= require_self

14 *= require_tree .

15 */

A manifest is just a JavaScript or CSS file with a commented block at the beginning of the file that includes special directives in it that specify other files of the same format to concatenate in the exact order. Several comment formats are supported:

1 // This is a single line comment (JavaScript, SCSS)

2 //= require foo

1 /* This is a multi-line comment (CSS, SCSS, JavaScript)

2 *= require foo

3 */

1 # This is a single line comment too (CoffeeScript)

2 #= require foo

Make as many manifest files as you need. For example, the admin.css and admin.js manifest could contain the JS and CSS files that are used for the admin section of an application.

20.5.1 Manifest directives

There are several manifest directives available:

require

is the most basic one. It concatenates the content of the referenced file you specify into the final packaged asset “bundle”. It will only do it once, even if the same filename appears multiple times in the manifest, either directly or as a part of require_tree (see below).

include

is just like require, but will insert the file again if it appears in the manifest more then once.

require_self

inserts the content of the file itself (after the directives). This is often useful when you want to make sure that JavaScript code from a manifest comes before any other code that is loaded with require_tree. We see an example of that in the default application.css.

require_directory

will load all the files of the same format in the specified directory in an alphabetical order. It will skip files that were already loaded.

require_tree

is just like require_directory, but it will also recursively load all the files in subdirectories as well. It will skip files that were already loaded as well.

depend_on

declares a dependency on a file without actually loading it into the “bundle”. It can be useful to force Rails to recompile cached asset bundle in response to the change in this file, even if it is not concatenated into the bundle directly.

Directives are processed in the order they are read in the file, but when you use require_tree there is no guarantee of the order files in which will be included. If for dependency reasons you need to make sure of a certain order, just require those files explicitly.

20.5.2 Search path

When you require an asset from a manifest file Rails searches for it in all directories in its search path. You do not need to specify file extensions. The processor assumes you are looking for files that match the type of the manifest file itself.

The search path includes all the directories that are directly under the default assets locations app/assets, lib/assets, and vendor/assets by default, meaning you can easily add other directories for new asset types to the list by creating them under any of the standard asset locations, e.g.app/assets/fonts

Files in subdirectories can be accessed by using a relative path:

1 // this will load the app/assets/javascripts/library/foo.js

2 //= require 'library/foo'

The directories are traversed in the order that they appear in the search path. The first file with the required name “wins”.

Note that all the directories in the search path are “equal”, and can store files of any format. It means you can put your JavaScript files in app/assets/stylesheets, and CSS files in app/assets/javascripts and Rails will work just the same. But your fellow developers will probably stop talking to you.

20.5.3 Gemified assets

As mentioned before, gems can contain assets, and there are gems that exist with the sole purpose of packaging asset files for Asset Pipeline.

To make gem assets available to an application the gem has to define an “engine” i.e. a class that inherits from Rails::Engine. Once “required” it will add app/assets, lib/assets, and vendor/assets directories from the gem to the search path.

Lets see the example from jquery-rails. You can find its engine in the lib/jquery/ui/rails/engine.rb file of the gem’s source code:

1 moduleJquery

2 moduleUi

3 moduleRails

4 classEngine < ::Rails::Engine

5 end

6 end

7 end

8 end

This Ruby file is loaded when you include this Gem into your application and as a result all the subdirectories of gem’s vendor/assets directory are added to the search path.

20.5.4 Index files

Index files makes inclusion of ‘bundles’ of files easy. If, for example, your Foobar library has a directory lib/assets/foobar with index.js file inside, Rails will recognize this files as a manifest and let you include the whole “bundle” with a single directive:

//= require 'foobar'

As with your Rails project, manifest files can encapsulate all the gem asset files and ensure proper load order without any additional effort on your part.

20.5.5 Format handlers

Asset Pipeline is not called a pipeline for nothing. Source files go into one end, get processed and compiled (if necessary), concatenated and compressed, then come out of the other end of the pipeline as bundles. There are multiple stages that the source files go through while traversing the pipeline.

There are many format handlers available with Rails, with more available as third party gems. Some of them are compilers, like CoffeeScript, that compile one format into another. Others are more simple pre-processors like “Interpolated Strings” engine that performs ruby substitution, e.g. #{...} regardless of the underlying format of the file, so that it can process a CoffeeScript file before it will be compiled into JavaScript.

Before we continue with individual handlers, we should discuss the file naming scheme, because the file extensions used on an asset determine which handlers are invoked. Asset files that are intended for compilation/pre-processing can have more then one extension, concatenated one after the other.

When asked to serve products in a manifest, either explicitly or as part of a compound require directive, the asset pipeline constructs the output by iteratively processing the file from one format into the next. It starts with the processing corresponding to the right-most file extension and continues until the requested leftmost extension format is obtained.

For example, let’s dissect the processing of an asset source file named products.css.sass.erb.str.

The pipeline will first pass this file through an Interpolated Strings engine, then the ERB template engine, after which the result is treated as a Sass file. Sass files get compiled into normal CSS, which is in turn served to the browser as the final result.¹¹¹

In case it wasn’t obvious, the order in which you specify the file extensions is important. If you were to name a file foo.css.erb.sass, the first processor to get the file would be the Sass compiler, and it would blow up when it encountered ERB tags.

Naturally, for this entire scheme to work, pre-processors and/or compilers should be available for all the relevant formats. A wide swath of pre-processing power is provided to Rails by a gem named Tilt, a generic interface to multiple Ruby template engines.¹¹²

1 ENGINE FILE EXTENSIONS REQUIRED LIBRARIES

2 ------------------------ ----------------------- ----------------------------

3 Asciidoctor .ad, .adoc, .asciidoc asciidoctor (>= 0.1.0)

4 ERB .erb, .rhtml none (included ruby stdlib)

5 Interpolated String .str none (included ruby core)

6 Erubis .erb, .rhtml, .erubis erubis

7 Haml .haml haml

8 Sass .sass haml (< 3.1) or sass (>= 3.1)

9 Scss .scss haml (< 3.1) or sass (>= 3.1)

10 Less CSS .less less

11 Builder .builder builder

12 Liquid .liquid liquid

13 RDiscount .markdown, .mkd, .md rdiscount

14 Redcarpet .markdown, .mkd, .md redcarpet

15 BlueCloth .markdown, .mkd, .md bluecloth

16 Kramdown .markdown, .mkd, .md kramdown

17 Maruku .markdown, .mkd, .md maruku

18 RedCloth .textile redcloth

19 RDoc .rdoc rdoc

20 Radius .radius radius

21 Markaby .mab markaby

22 Nokogiri .nokogiri nokogiri

23 CoffeeScript .coffee coffee-script (+ javascript)

24 Creole (Wiki markup) .wiki, .creole creole

25 WikiCloth (Wiki markup) .wiki, .mediawiki, .mw wikicloth

26 Yajl .yajl yajl-ruby

27 CSV .rcsv none (Ruby >= 1.9),

28 fastercsv (Ruby < 1.9)

20.6 Custom format handlers

Even though Tilt provides quite a few formats, you might need to implement your own. Template handler classes have a simple interface. They define a class attribute named default_handler containing the desired Mime-type of the content, and a class method with the signaturecall(template) that receives the template content and returns the processed result.

For example, here is the handler class from the Rabl gem, used to generate JSON using templates.

1 moduleActionView

2 moduleTemplate::Handlers

3 classRabl

4 class_attribute :default_format

5 self.default_format = Mime::JSON

6

7 defself.call(template)

8 # ommitted for clarity...

9 end

10 end

11 end

12 end

Note that by convention, template handlers are defined in the ActionView::Template::Handlers module. Once your custom code is available to your application in the lib folder or as a gem, register it using the register_template_handler method, providing the extension to match, and the handler class:

1 ActionView::Template.register_template_handler :rabl, ActionView::Template::Handlers::Rabl

20.7 Post-Processing

In addition to pre-processing various formats into JavaScripts and stylesheets, the asset pipeline can also post-process the results. By default post-processing compressors are available for both stylesheets and JavaScripts.

20.7.1 Stylesheets

By default stylesheets are compressed using the YUI Compressor, which is the only stylesheets compressor available out of the box with Rails.

You can control it by changing the config.assets.css_compressor configuration option, that is set to yui by default.

When using Sass in a Rails project, one could set the CSS compressor to use Sass’ standard compressor with the config.assets.css_compressor = :sass option.

20.7.2 JavaScripts

There are several JavaScript compression options available: :closure, :uglifier, and :yui, provided by closure-compiler, uglifier or yui-compressor gems respectively.

The :uglifier option is the default, but you can control it by changing the config.assets.js_compressor configuration option.

20.7.3 Custom Compressor

You can use a custom post-processor by defining a class with a compress method that accepts a string and assigning an instance of it to one of the configuration options above, like this:

1 classMyProcessor

2 def compress(string)

3 # do something

4 end

5 end

6

7 config.assets.css_compressor = MyProcessor.new

20.8 Helpers

To link assets into your Rails templates, you use the same old helpers as always, javascript_include_tag and stylesheet_link_tag. Call these helpers in the <head> of your layout template, passing them the name of your manifest files.

1 <%= stylesheet_link_tag "application" %>

2 <%= javascript_include_tag "application" %>

One of the common frustrations of the Asset Pipeline learning curve is figuring out that you don’t need to explicitly link to every asset file in your layout template anymore. Unless you break off large portions of assets for different parts of your app (most commonly, for admin sections) you’ll just need one each for the application.js and application.css files. If you try to explicitly include or link to assets that are bundled up, your app will work in development mode where it’s possible to serve up assets dynamically. However, it will break in production where assets must be precompiled. The bundled-up assets will simply not exist.

You’ll know that you’re running into this problem when you get the following error:

ActionView::Template::Error (foo.js isn't precompiled)

To fix this problem, make sure that foo.js is required in one of your manifest files, and get rid of the call to `javascript_include_tag “foo”.

By default, Rails only seeks to precompile assets named “application.” If you have a good reason to break off additional bundles of assets, like for the admin section of your app, tell the pipeline to precompile those bundles by adding the names of the manifest files to theconfig.assets.precompile array in config/application.rb

config.assets.precompile += %w(admin.js optional.js}

Note that the most maddening incarnation of this problem happens with older Rails plugin gems that were written before the advent of the asset pipeline and contain explicit requires to their asset dependencies. Luckily, as gems are updated this problem is becoming less common than it was in the days of Rails 3.x.

20.8.1 Images

The venerable image_tag helper has been updated so that it knows to search asset/images and not just the public folder. It will also search through the paths specified in the config.assets.paths setting and any additional paths added by gems. If you’re passing user-supplied data to theimage_tag helper, note that a blank or non-existant path will raise a server exception during processing of the template.

20.8.2 Getting the URL of an asset file

The asset_path and asset_url helpers can be used if you need to generate the URL of an asset. But you’d need to make sure to include the .erb file extension at the right-most position. For example, consider the following snippet of JavaScript taken from a file named transitions.js.erbcontains the line:

this.loadImage('<%= asset_path "noise.jpg" %>');

The asset pipeline runs the source through ERB processing first, and interpolates in the correct path to the desired JPG file.

20.8.3 Built-in SASS asset path helpers

Similarly, in a SASS stylesheet named layout.css.scss.erb you might have the following code, but you wouldn’t for reasons that we’ll explain momentarily:

1 header {

2 background-image: url("<%= asset_path "header-photo-vert.jpg" %>");

3 }

Because this is such a common construct, Rails’ SASS processing has built-in helpers, useful for referencing image, font, video, audio, and other stylesheet assets.

1 header {

2 background-image: image-url("header-photo-vert.jpg");

3 }

Reusing a familiar pattern, image-url("rails.png") becomes url(/assets/rails.png) and image-path("rails.png") becomes "/assets/rails.png". The more generic form can also be used but the asset path and class must both be specified: asset-url("rails.png", image) becomesurl(/assets/rails.png) and asset-path("rails.png", image) becomes "/assets/rails.png".

20.8.4 Data URIs

You can easily embed the source of an image directly into a CSS file using the Data URL scheme¹¹³ with the asset_data_uri method like this:

1 icon {

2 background: url(<%= asset_data_uri 'icon.png' %>)

3 }

Many different kinds of content can be inlined using data urls, although a full explanation of each is outside the scope of this book. Generally speaking, you want to keep the size of inlined data small to avoid blowing up the size of your CSS file.

20.9 Fingerprinting

In the past, Rails encoded and appended an asset’s file timestamp to all asset paths like this:

<link href="/assets/foo.css?1385926153" media="screen" rel="stylesheet" />

This simple scheme allowed you to set a cache-expiration date for the asset far into the future, but still instantly invalidate it by updating the file. The updated timestamp changed the resulting URL, which busted the cache.

Note that in order for this scheme to work correctly, all your application servers had to return the same timestamps. In other words, they needed to have their clocks synchronized. If one of them drifted out of sync, you would see different timestamps at random and the caching wouldn’t work properly.

Another problem with the old approach was that it appended the timestamps as a query parameter. Not all cache implementations treat query parameters as parts of their cache key, leading to stale cache hits or no caching at all.

Yet another problem was that with many deployment methods, file timestamps would change on each deployment. This led to unnecessary cache invalidations after each production deploy.

The new asset pipeline drops the timestamping scheme and uses content fingerprinting instead. Fingerprinting makes the file name dependent on the files’ content, so that the filename only ever changes when the actual file content is changed.

It’s worth knowing that these two lines

1 <%= javascript_include_tag "application" %>

2 <%= stylesheet_link_tag "application" %>

will look like this in production:

1 <script src="/assets/application-908e25f4bf641868d8683022a5b62f54.js">

2 </script>

3 <link

4 href="/assets/application-4dd5b109ee3439da54f5bdfd78a80473.css"

5 media="screen" rel="stylesheet"></link>

20.10 Serving the files

To take full advantage of asset fingerprinting provided by the asset pipeline, you should configure your web server to set headers on your precompiled assets to a far-future expiration date. With cache headers in place, a client will only request an asset once until either the filename changes, or the cache has expired.

Here’s an example for Apache:

1 # The Expires* directives requires the Apache module `mod_expires` to be enabled.

2 <Location /assets/>

3 # Use of ETag is discouraged when Last-Modified is present

4 Header unset ETag

5 FileETag None

6 # RFC says only cache for 1 year

7 ExpiresActive On

8 ExpiresDefault "access plus 1 year"

9 </Location>

And here’s one for Nginx:

1 location ~ ^/assets/ {

2 expires 1y;

3 add_header Cache-Control public;

4 add_header Last-Modified "";

5 add_header ETag "";

6 break;

7 }

The fingerprinting feature is controller by the config.assets.digest Rails setting. By default it is only set in “production” environment.

Note that the asset pipeline always makes copies of non-fingerprinted asset files available in the same /assets directory.

While you’re thinking about how your asset files are being served, it’s worth investigating the possibility of seriously improving app performance by having your web server serve asset files directly instead of involving the Rails stack. Apache and nginx support this option out of the box, and you enable it by turning on right option in production.rb:

1 # config.action_dispatch.x_sendfile_header = "X-Sendfile" # for apache

2 # config.action_dispatch.x_sendfile_header = 'X-Accel-Redirect' # for nginx

20.10.1 GZip Compression

When the asset pipeline precompiles a file, it generates a full-compression gzipped version as well. So that alongside /assets/application.css, there is also /assets/application.css.gz. The benefit of doing it during the precompilation process and not on-the-fly (which is supported by default my most sane server configurations) is that it only happens once, allowing the use of the maximum compression level to minimize file size.

Some configuration is needed on the web server side to serve those pre-compressed files.

For Nginx you only need to add gzip_static on; to the configuration:

1 location ~ ^/assets/ {

2 expires 1y;

3 add_header Cache-Control public;

4 add_header Last-Modified "";

5 add_header ETag "";

6 gzip_static on;

7 break;

8 }

20.11 Rake Tasks

When in production mode, Rails expects all manifests and asset files to be pre-compiled on disk and available to be served up by your web server out of the location specified in config.assets.prefix setting, which defaults to public/assets. Compiled asset files should not be versioned in source control, and the default .gitignore file for Rails includes a line for public/assets/*.

As part of deploying your application to production, you’ll call the following rake task to create compiled versions of your assets directly on the server:

$ RAILS_ENV=production bundle exec rake assets:precompile

Note that Heroku automatically does this step for you in such a way that is compatible with its otherwise read-only filesystem. However, Heroku also prevents your Rails application from being initialized as part of asset pre-compilation, and certain references to objects or methods will not be available, causing the compile process to fail. To catch these errors, pre-compile assets on your development machine, noting any errors that arise.

Also note that local pre-compilation will result in a bunch of unwanted files in your /public/assets directory that will be served up instead of the originals. You’ll be scratching your head wondering why changes to your JS and CSS files are not being reflected in your browser. If that happens, you need to delete the compiled assets. Use the rake assets:clobber task to get rid of them.

The official Asset Pipeline guide¹¹⁴ goes into great detail about using pre-compiled assets with development mode, or even setting up Rails to compile assets on the fly. It’s rare that you would want or need to do either.

20.12 Conclusion

The asset pipeline was one of the great additions to Rails 3 and a big part of what makes Ruby on Rails a cutting-edge and industry leading framework. In this chapter we’ve covered the major aspects of working with the asset pipeline.