Developing Web Apps with Haskell and Yesod, Second Edition (2015)
Part I. Basics
Chapter 11. Deploying Your Web App
I can’t speak for others, but I personally prefer programming to system administration. But the fact is that eventually you’ll need to serve your app somehow, and odds are that you’ll need to be the one to set it up.
There are some promising initiatives in the Haskell web community toward making deployment easier. In the future, we may even have a service that allows you to deploy your app with a single command.
But we’re not there yet. And even if we were, such a solution will never work for everyone. This chapter covers the different deployment options, and gives some general recommendations on what you should choose in different situations.
Keter
The Yesod scaffolding comes with some built-in support for the Keter deployment engine, which is also written in Haskell and uses many of the same underlying technologies, like WAI and http-client. Keter works as a reverse proxy to your applications, as well as a system for starting, monitoring, and redeploying running apps. If you’d like to deploy with Keter, follow these steps:
1. Edit the config/keter.yaml file in your scaffolded application as necessary.
2. Set up some kind of server for hosting your apps. I recommend trying Ubuntu on Amazon EC2.
3. Install Keter on that machine (follow the instructions on the Keter website, as they will be the most up to date).
4. Run yesod keter to generate a Keter bundle (e.g., myapp.keter).
5. Copy myapp.keter to the /opt/keter/incoming directory on your server.
If you’ve gotten things configured correctly, you should now be able to view your website, running in a production environment! In the future, upgrades can be handled by simply rerunning yesod keter and recopying the myapp.keter bundle to the server. Note that Keter will automatically detect the presence of the new file and reload your application.
The rest of this chapter will provide some more details about various steps, and present some alternatives in case you’d prefer not to use the scaffolding or Keter.
Compiling
The biggest advice I can give is this: don’t compile on your server. It’s tempting to do so, as you just have to transfer source code around and you avoid confusing dependency issues. However, compiling a Yesod application takes significiant memory and CPU resources, which means:
§ While you’re recompiling your app, your existing applications will suffer performance-wise.
§ You will need to get a much larger machine to handle compilation, and that capacity will likely sit idle most of the time, as Yesod applications tend to require far less CPU and memory than GHC itself.
Once you’re ready to compile, you should always make sure to run cabal clean before a new production build, to make sure no old files are lying around. Then, you can run cabal configure && cabal build to get an executable, which will be located at dist/build/myapp/myapp. (The yesod keter command does cabal clean for you automatically.)
Files to Deploy
With a Yesod scaffolded application, there are essentially three sets of files that need to be deployed:
§ Your executable
§ The config/ folder
§ The static/ folder
Everything else (e.g., Shakespearean templates), gets compiled into the executable itself.
There is one caveat, however, regarding the config/client_session_key.aes file. This file controls the server-side encryption used for securing client-side session cookies. Yesod will automatically generate a new one of these keys if none is present. In practice, this means that, if you do not include this file in your deployment, all of your users will have to log in again when you redeploy. If you follow this advice and include the config/ folder, this issue will be partially resolved. Another approach is to put your session key in an environment variable.
The other half of the resolution is to ensure that once you generate a config/client_session_key.aes file, you keep the same one for all future deployments. The simplest way to ensure this is to keep that file in your version control. However, if your version control is open source, this will be dangerous: anyone with access to your repository will be able to spoof login credentials!
The problem described here is essentially one of system administration, not programming. Yesod does not provide any built-in approach for securely storing client session keys. If you have an open source repository, or do not trust everyone who has access to your source code repository, it’s vital to figure out a safe storage solution for the client session key.
SSL and Static Files
There are two commonly used features in the Yesod world: serving your site over HTTPS, and placing your static files in a separate domain name. Both of these are good practices, but used together they can lead to problems if you’re not careful. In particular, most web browsers will not load up JavaScript files from a non-HTTPS domain name if your HTML is served from an HTTPS domain name. In this situation, you’ll need to do one of two things:
§ Serve your static files over HTTPS as well.
§ Serve your static files from the same domain name as your main site.
Note that if you go for the first option (which is the better one), you’ll need either two separate SSL certificates or a wildcard certificate.
Warp
As we have mentioned before, Yesod is built on the Web Application Interface (WAI), allowing it to run on any WAI backend. At the time of writing, the following backends are available:
§ Warp
§ FastCGI
§ SCGI
§ CGI
§ WebKit
§ Development server
The last two are not intended for production deployments. Of the remaining four, all can be used for production deployment in theory. In practice, a CGI backend will likely be horribly inefficient, because a new process must be spawned for each connection. And SCGI is not nearly as well supported by frontend web servers as Warp (via reverse proxying) or FastCGI.
Between the two remaining choices, Warp gets a very strong recommendation, for the following reasons:
§ It is significantly faster.
§ Like FastCGI, it can run behind a frontend server like Nginx, using a reverse HTTP proxy.
§ It is a fully capable server of its own accord, and can therefore be used without any frontend server.
So that leaves one last question: should Warp run on its own, or via a reverse proxy behind a frontend server? For most use cases I recommend the latter, because:
§ Having a reverse proxy in front of your app makes it easier to deploy new versions.
§ If you have a bug in your application, a reverse proxy can give slightly nicer error messages to users.
§ You can host multiple applications from a single host via virtual hosting.
§ Your reverse proxy can function as a load balancer or SSL proxy as well, simplifying your application.
As already discussed, Keter is a great way to get started. If you have an existing web server running, like Nginx, Yesod will work just fine sitting behind it instead.
Nginx Configuration
Keter configuration is trivial, as it is designed to work with Yesod applications. But if you want to instead use Nginx, how do you set it up?
In general, Nginx will listen on port 80 and your Yesod/Warp app will listen on some unprivileged port (let’s say 4321). You will then need to provide an nginx.conf file, such as the following:
daemon off; # Don't run nginx in the background, good for monitoring apps
events {
worker_connections 4096;
}
http {
server {
listen 80; # Incoming port for Nginx
server_name www.myserver.com;
location / {
proxy_pass http://127.0.0.1:4321; # Reverse proxy to your Yesod app
}
}
}
You can add as many server blocks as you like. A common addition is to ensure users always access your pages with the www prefix on the domain name, guaranteeing the RESTful principle of canonical URLs. (You could just as easily do the opposite and always strip the www; just make sure that your choice is reflected in both the Nginx config and the approot of your site.) In this case, we would add the block:
server {
listen 80;
server_name myserver.com;
rewrite ^/(.*) http://www.myserver.com/$1 permanent;
}
A highly recommended optimization is to serve static files from a separate domain name, thereby bypassing the cookie transfer overhead. Assuming that our static files are stored in the static/ folder within our site folder, and the site folder is located at /home/michael/sites/mysite, this would look like:
server {
listen 80;
server_name static.myserver.com;
root /home/michael/sites/mysite/static;
# Because yesod-static appends a content hash in the query string,
# we are free to set expiration dates far in the future without
# concerns of stale content.
expires max;
}
In order for this to work, your site must properly rewrite static URLs to this alternative domain name. The scaffolded site is set up to make this fairly simple via the Settings.staticRoot function and the definition of urlRenderOverride. However, if you just want to get the benefit of Nginx’s faster static file serving without dealing with separate domain names, you can instead modify your original server block like so:
server {
listen 80; # Incoming port for Nginx
server_name www.myserver.com;
location / {
proxy_pass http://127.0.0.1:4321; # Reverse proxy to your Yesod app
}
location /static {
root /home/michael/sites/mysite; # Notice we do *not* include /static
expires max;
}
}
Server Process
Many people are familiar with an Apache/mod_php or Lighttpd/FastCGI kind of setup, where the web server automatically spawns the web application. With Nginx, either for reverse proxying or with FastCGI, this is not the case: you are responsible for running your own process. I strongly recommend using a monitoring utility that will automatically restart your application in case it crashes. There are many great options out there, such as angel or daemontools.
To give a concrete example, here is an Upstart config file. The file must be placed in /etc/init/mysite.conf:
description "My awesome Yesod application"
start on runlevel [2345];
stop on runlevel [!2345];
respawn
chdir /home/michael/sites/mysite
exec /home/michael/sites/mysite/dist/build/mysite/mysite
Once this is in place, bringing up your application is as simple as sudo start mysite.
Nginx + FastCGI
Some people may prefer using FastCGI for deployment. In this case, you’ll need to add an extra tool to the mix. FastCGI works by receiving new connections from a file descriptor. The C library assumes that this file descriptor will be 0 (standard input), so you need to use the spawn-fcgiprogram to bind your application’s standard input to the correct socket.
It can be very convenient to use Unix named sockets for this instead of binding to a port, especially when hosting multiple applications on a single host. A possible script to load up your app could be:
spawn-fcgi \
-d /home/michael/sites/mysite \
-s /tmp/mysite.socket \
-n \
-M 511 \
-u michael \
-- /home/michael/sites/mysite/dist/build/mysite-fastcgi/mysite-fastcgi
You will also need to configure your frontend server to speak to your app over FastCGI. This is relatively painless in Nginx:
server {
listen 80;
server_name www.myserver.com;
location / {
fastcgi_pass unix:/tmp/mysite.socket;
}
}
That should look pretty familiar. The only last trick is that, with Nginx, you need to manually specify all of the FastCGI variables. It is recommended to store these in a separate file (say, fastcgi.conf) and then add include fastcgi.conf; to the end of your http block. The contents of the file, to work with WAI, should be:
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param PATH_INFO $fastcgi_script_name;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx/$nginx_version;
fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;
Desktop
Another nifty backend is wai-handler-webkit. This backend combines Warp and QtWebKit to create an executable that a user simply double-clicks. This can be a convenient way to provide an offline version of your application.
One of the very nice conveniences of Yesod for this is that your templates are all compiled into the executable, and thus do not need to be distributed with your application. Static files do, however.
NOTE
There’s actually support for embedding your static files directly in the executable as well. See the yesod-static docs for more details.
A similar approach, without requiring the QtWebKit library, is using wai-handler-launch, which launches a Warp server and then opens up the user’s default web browser. There’s a little trickery involved here: in order to know that the user is still using the site, wai-handler-launchinserts a “ping” JavaScript snippet to every HTML page it serves. If wai-handler-launch doesn’t receive a ping for two minutes, it shuts down.
CGI on Apache
CGI and FastCGI work almost identically on Apache, so it should be fairly straight-forward to port this configuration. You essentially need to accomplish two goals:
1. Get the server to serve your file as (Fast)CGI.
2. Rewrite all requests to your site to go through the (Fast)CGI executable.
Here is a configuration file for serving a blog application, with an executable named bloggy.cgi, living in a subfolder named blog/ of the document root. This example was taken from an application living in the path /f5/snoyman/public/blog:
Options +ExecCGI
AddHandler cgi-script .cgi
Options +FollowSymlinks
RewriteEngine On
RewriteRule ^/f5/snoyman/public/blog$ /blog/ [R=301,S=1]
RewriteCond $1 !^bloggy.cgi
RewriteCond $1 !^static/
RewriteRule ^(.*) bloggy.cgi/$1 [L]
The first RewriteRule is to deal with subfolders. In particular, it redirects a request for /blog to /blog/. The first RewriteCond prevents directly requesting the executable, the second allows Apache to serve the static files, and the last line does the actual rewriting.
FastCGI on lighttpd
For this example, I’ve left off some of the basic FastCGI settings like MIME types. I also have a more complex file in production that prepends www when absent and serves static files from a separate domain. However, this should serve to show the basics.
Here, /home/michael/fastcgi is the FastCGI application. The idea is to rewrite all requests to start with /app, and then serve everything beginning with /app via the FastCGI executable:
server.port = 3000
server.document-root = "/home/michael"
server.modules = ("mod_fastcgi", "mod_rewrite")
url.rewrite-once = (
"(.*)" => "/app/$1"
)
fastcgi.server = (
"/app" => ((
"socket" => "/tmp/test.fastcgi.socket",
"check-local" => "disable",
"bin-path" => "/home/michael/fastcgi", # full path to executable
"min-procs" => 1,
"max-procs" => 30,
"idle-timeout" => 30
))
)
CGI on lighttpd
This is basically the same as the FastCGI version, but tells lighttpd to run a file ending in .cgi as a CGI executable. In this case, the file lives at /home/michael/myapp.cgi:
server.port = 3000
server.document-root = "/home/michael"
server.modules = ("mod_cgi", "mod_rewrite")
url.rewrite-once = (
"(.*)" => "/myapp.cgi/$1"
)
cgi.assign = (".cgi" => "")