Understanding a Request - Advanced - Developing Web Apps with Haskell and Yesod, Second Edition (2015)

Developing Web Apps with Haskell and Yesod, Second Edition (2015)

Part II. Advanced

Chapter 18. Understanding a Request

You can oftentimes get away with using Yesod for quite a while without needing to understand its internal workings. However, developing an understanding of its ins and outs is advantageous. This chapter will walk you through the request handling process for a fairly typical Yesod application. Note that a fair amount of this discussion involves code changes in Yesod 1.2. Most of the concepts are the same in previous versions, though the data types involved were a bit messier.

Yesod’s usage of Template Haskell to bypass boilerplate code can make it a bit difficult to understand this process sometimes. If you wish to go beyond the information in this chapter, it can be useful to view GHC’s generated code using -ddump-splices.

NOTE

A lot of this information was originally published as a blog series on the 1.2 release. You can see the blog posts at:

§ Yesod 1.2’s cleaner internals

§ Big Subsite Rewrite

§ Yesod dispatch, version 1.2

Handlers

When trying to understand Yesod request handling, we need to look at two components: how a request is dispatched to the appropriate handler code, and how handler functions are processed. We’ll start off with the latter, and then circle back to understanding the dispatch process itself.

Layers

Yesod builds itself on top of WAI, which provides a protocol for web servers (or, more generally, handlers) and applications to communicate with each other. This is expressed through two data types: Request and Response. Then, an Application is defined as:

typeApplication=Request

-> (Response->IOResponseReceived)

->IOResponseReceived

A WAI handler will take an application and run it.

NOTE

The structure of Application looks a bit complicated. It uses continuation passing style to allow an application to safely acquire resources, similar to the bracket function. See the WAI API documentation for more details.

Request and Response are both very low level, trying to represent the HTTP protocol without too much embellishment. This keeps WAI as a generic tool, but also leaves out a lot of the information we need in order to implement a web framework. For example, WAI will provide us with the raw data for all request headers. But Yesod needs to parse that to get cookie information, and then parse the cookies in order to extract session information.

To deal with this dichotomy, Yesod introduces two new data types: YesodRequest and YesodResponse. YesodRequest contains a WAI Request, and also adds in such request information as cookies and session variables. On the response side can either be a standard WAI Response or a higher-level representation of such a response including such things as updated session information and extra response headers. To parallel WAI’s Application, we have:

typeYesodApp=YesodRequest->ResourceTIOYesodResponse

NOTE

Yesod uses ResourceT for exception safety, instead of continuation passing style. This makes it much easier to write exception-safe code in Yesod.

But as a Yesod user, you never really see YesodApp. There’s another layer on top of that, which you are used to dealing with: HandlerT. When you write handler functions, you need to have access to three different things:

§ The YesodRequest value for the current request.

§ Some basic environment information, like how to log messages or handle error conditions. This is provided by the data type RunHandlerEnv.

§ A mutable variable to keep track of updateable information, such as the headers to be returned and the user session state. This is called GHState. (I know that’s not a great name, but it’s there for historical reasons.)

So when you’re writing a handler function, you’re essentially just writing a ReaderT transformer that has access to all of this information. The runHandler function will turn a HandlerT into a YesodApp. yesodRunner takes this a step further and converts it to a WAI Application.

Content

The preceding example, and many others you’ve already seen, gives a handler with a type of Handler Html. We’ve just described what the Handler means, but how does Yesod know how to deal with Html? The answer lies in the ToTypedContent typeclass. The relevant bit of code are:

dataContent=ContentBuilder !BBuilder.Builder !(MaybeInt)

-- ^ The content and optional content length.

| ContentSource !(Source (ResourceTIO) (FlushBBuilder.Builder))

| ContentFile !FilePath !(MaybeFilePart)

| ContentDontEvaluate !Content

dataTypedContent=TypedContent !ContentType !Content

classToContent a where

toContent ::a ->Content

classToContent a =>ToTypedContent a where

toTypedContent ::a ->TypedContent

The Content data type represents the different ways you can provide a response body. The first three mirror WAI’s representation directly. The fourth option (ContentDontEvaluate) is used to indicate to Yesod whether response bodies should be fully evaluated before being returned to users. The advantage to fully evaluating is that we can provide meaningful error messages if an exception is thrown from pure code. The downside is possibly increased time and memory usage.

In any event, Yesod knows how to turn a Content into a response body. The ToContent typeclass provides a way to allow many different data types to be converted into response bodies. Many commonly used types are already instances of ToContent, including strict and lazyByteString and Text, and of course Html.

TypedContent adds an extra piece of information: the content type of the value. As you might expect, there are ToTypedContent instances for a number of common data types, including Html, the aeson library’s Value (for JSON), and Text (treated as plain text):

instanceToTypedContentJ.Valuewhere

toTypedContent v =TypedContent typeJson (toContent v)

instanceToTypedContentHtmlwhere

toTypedContent h =TypedContent typeHtml (toContent h)

instanceToTypedContentT.Textwhere

toTypedContent t =TypedContent typePlain (toContent t)

Putting this all together, a Handler is able to return any value that is an instance of ToTypedContent, and Yesod will handle turning it into an appropriate representation and setting the Content-Type response header.

Short-Circuit Responses

One other oddity is how short-circuiting works. For example, you can call redirect in the middle of a handler function, and the rest of the function will not be called. The mechanism we use is standard Haskell exceptions. Calling redirect just throws an exception of typeHandlerContents. The runHandler function will catch any exceptions thrown and produce an appropriate response. For HandlerContents, each constructor gives a clear action to perform, be it redirecting or sending a file. For all other exception types, an error message is displayed to the user.

Dispatch

Dispatch is the act of taking an incoming request and generating an appropriate response. We have a few different constraints, depending on how we want to handle dispatch:

§ Dispatch based on path segments (or pieces).

§ Optionally dispatch on request method.

§ Support subsites: packaged collections of functionality providing multiple routes under a specific URL prefix.

§ Support using WAI Applications as subsites, while introducing as little runtime overhead to the process as possible. In particular, we want to avoid performing any unnecessary parsing to generate a YesodRequest if it won’t be used.

The lowest common denominator for this is to simply use a WAI Application. However, this doesn’t provide quite enough information: we need access to the foundation data type, and the logger, and for subsites, we need to know how a subsite route is converted to a parent site route. To address this, we have two helper data types—YesodRunnerEnv and YesodSubRunnerEnv—providing this extra information for normal sites and subsites.

With those types, dispatch now becomes a relatively simple matter: give me an environment and a request, and I’ll give you a response. This is represented by the typeclasses YesodDispatch and YesodSubDispatch:

classYesod site =>YesodDispatch site where

yesodDispatch ::YesodRunnerEnv site ->W.Application

classYesodSubDispatch sub m where

yesodSubDispatch ::YesodSubRunnerEnv sub (HandlerSite m) m

->W.Application

We’ll see a bit later how YesodSubDispatch is used. Let’s first understand how YesodDispatch comes into play.

toWaiApp, toWaiAppPlain, and warp

Let’s assume for the moment that you have a data type that is an instance of YesodDispatch. You’ll want to now actually run this thing somehow. To do this, you need to convert it into a WAI Application and pass it to some kind of WAI handler/server. To start this journey, we usetoWaiAppPlain. It performs any app-wide initialization necessary. At the time of writing, this means allocating a logger and setting up the session backend, but more functionality may be added in the future. Using this data, we can create a YesodRunnerEnv. And when that value is passed to yesodDispatch, we get a WAI Application.

We’re almost done. The final remaining modification is path segment cleanup. The Yesod typeclass includes a member function named cleanPath that can be used to create canonical URLs. For example, the default implementation would remove double slashes and redirect a user from/foo//bar to /foo/bar. toWaiAppPlain adds in some preprocessing to the normal WAI request by analyzing the requested path and performing cleanup/redirects as necessary.

At this point, we have a fully functional WAI Application. There are two other helper functions included. toWaiApp wraps toWaiAppPlain and additionally includes some commonly used WAI middlewares, including request logging and gzip compression (see the Haddocks for an up-to-date list). Finally, we have the warp function, which as you might guess, runs your application with Warp.

NOTE

There’s also the warpEnv function, which reads the port number information from the PORT environment variable. This is used for interacting with certain tools, including the Keter deployment manager and FP Haskell Center.

Generated Code

The last remaining black box is the Template Haskell generated code. This generated code is responsible for handling some of the tedious, error-prone pieces of your site. If you want to, you can write these all by hand instead. We’ll demonstrate what that translation would look like, and in the process elucidate how YesodDispatch and YesodSubDispatch work. Let’s start with a fairly typical Yesod application:

{-# LANGUAGE OverloadedStrings #-}

{-# LANGUAGE QuasiQuotes #-}

{-# LANGUAGE TemplateHaskell #-}

{-# LANGUAGE TypeFamilies #-}

{-# LANGUAGE ViewPatterns #-}

importqualifiedData.ByteString.Lazy.Char8as L8

import Network.HTTP.Types (status200)

import Network.Wai (pathInfo, rawPathInfo,

requestMethod, responseLBS)

import Yesod

dataApp=App

mkYesod "App" [parseRoutes|

/only-get OnlyGetR GET

/any-method AnyMethodR

/has-param/#IntHasParamR GET

/my-subsite MySubsiteRWaiSubsite getMySubsite

|]

instanceYesodApp

getOnlyGetR ::HandlerHtml

getOnlyGetR =defaultLayout

[whamlet|

<p>Accessed via GET method

<form method=post action=@{AnyMethodR}>

<button>POST to /any-method

|]

handleAnyMethodR ::HandlerHtml

handleAnyMethodR =do

req <-waiRequest

defaultLayout

[whamlet|

<p>In any-method, method == #{show $ requestMethod req}

|]

getHasParamR ::Int->HandlerString

getHasParamR i =return $ show i

getMySubsite ::App->WaiSubsite

getMySubsite _=

WaiSubsite app

where

app req sendResponse =sendResponse $ responseLBS

status200

[("Content-Type", "text/plain")]

$ L8.pack $ concat

[ "pathInfo == "

, show $ pathInfo req

, ", rawPathInfo == "

, show $ rawPathInfo req

]

main ::IO ()

main =warp 3000 App

For completeness, we’ve provided a full listing, but let’s focus on just the Template Haskell portion:

mkYesod "App" [parseRoutes|

/only-get OnlyGetR GET

/any-method AnyMethodR

/has-param/#IntHasParamR GET

/my-subsite MySubsiteRWaiSubsite getMySubsite

|]

Although this generates a few pieces of code, we only need to replicate three components to make our site work. Let’s start with the simplest—the Handler type synonym:

typeHandler=HandlerTAppIO

Next is the type-safe URL and its rendering function. The rendering function is allowed to generate both path segments and query string parameters. Standard Yesod sites never generate query string parameters, but it is technically possible. And in the case of subsites, this often does happen. Notice how we handle the qs parameter for the MySubsiteR case:

instanceRenderRouteAppwhere

dataRouteApp=OnlyGetR

| AnyMethodR

| HasParamRInt

| MySubsiteR (RouteWaiSubsite)

deriving (Show, Read, Eq)

renderRoute OnlyGetR= (["only-get"], [])

renderRoute AnyMethodR= (["any-method"], [])

renderRoute (HasParamR i) = (["has-param", toPathPiece i], [])

renderRoute (MySubsiteR subRoute) =

let (ps, qs) =renderRoute subRoute

in ("my-subsite" : ps, qs)

You can see that there’s a fairly simple mapping from the higher-level route syntax and the RenderRoute instance. Each route becomes a constructor, each URL parameter becomes an argument to its constructor, we embed a route for the subsite, and we use toPathPiece to renderparameters to text.

The final component is the YesodDispatch instance. Let’s look at this in a few pieces:

instanceYesodDispatchAppwhere

yesodDispatch env req =

case pathInfo req of

["only-get"] ->

case requestMethod req of

"GET" ->yesodRunner

getOnlyGetR

env

(JustOnlyGetR)

req

_->yesodRunner

(badMethod >> return ())

env

(JustOnlyGetR)

req

As just described, yesodDispatch is handed both an environment and a WAI Request value. We can now perform dispatch based on the requested path, or, in WAI terms, the pathInfo. Referring back to our original high-level route syntax, we can see that our first route is going to be the single piece only-get, which we pattern match for.

Once that match has succeeded, we additionally pattern match on the request method. If it’s GET, we use the handler function getOnlyGetR. Otherwise, we want to return a 405 Bad Method response, and therefore use the badMethod handler. At this point, we’ve come full circle to our original handler discussion. You can see that we’re using yesodRunner to execute our handler function. As a reminder, this will take our environment and WAI Request, convert it to a YesodRequest, construct a RunHandlerEnv, hand that to the handler function, and then convert the resulting YesodResponse into a WAI Response.

Wonderful; one down, three to go. The next one is even easier:

["any-method"] ->

yesodRunner handleAnyMethodR env (JustAnyMethodR) req

Unlike OnlyGetR, AnyMethodR will work for any request method, so we don’t need to perform any further pattern matching:

["has-param", t] | Just i <-fromPathPiece t ->

case requestMethod req of

"GET" ->yesodRunner

(getHasParamR i)

env

(Just $ HasParamR i)

req

_->yesodRunner

(badMethod >> return ())

env

(Just $ HasParamR i)

req

We add in one extra complication here: a dynamic parameter. While we used toPathPiece to render to a textual value earlier, we now use fromPathPiece to perform the parsing. Assuming the parse succeeds, we then follow a very similar dispatch system as was used for OnlyGetR. The prime difference is that our parameter needs to be passed to both the handler function and the route data constructor.

Next, we’ll look at the subsite, which is quite different:

("my-subsite":rest) ->yesodSubDispatch

YesodSubRunnerEnv

{ ysreGetSub =getMySubsite

, ysreParentRunner =yesodRunner

, ysreToParentRoute =MySubsiteR

, ysreParentEnv =env

}

req { pathInfo =rest }

Unlike the other pattern matches, here we just look to see if our pattern prefix matches. Any route beginning with /my-subsite should be passed off to the subsite for processing. This is where we finally get to use yesodSubDispatch. This function closely mirrors yesodDispatch. We need to construct a new environment to be passed to it. Let’s discuss the four fields:

§ ysreGetSub demonstrates how to get the subsite foundation type from the master site. We provide getMySubsite, which is the function we provided in the high-level route syntax.

§ ysreParentRunner provides a means of running a handler function. It may seem a bit boring to just provide yesodRunner, but by having a separate parameter we allow the construction of deeply nested subsites, which will wrap and unwrap many layers of interleaving subsites. (This is a more advanced concept, and we won’t be covering it in this chapter.)

§ ysreToParentRoute will convert a route for the subsite into a route for the parent site. This is the purpose of the MySubsiteR constructor. This allows subsites to use functions such as getRouteToParent.

§ ysreParentEnv simply passes on the initial environment, which contains a number of things the subsite may need (such as the logger).

The other interesting thing is how we modify the pathInfo. This allows subsites to continue dispatching from where the parent site left off. Figure 18-1 shows screenshots of a few requests.

wahy 1801

Figure 18-1. Path info in subsite

And finally, not all requests will be valid routes. For those cases, we just want to respond with a 404 Not Found:

_ -> yesodRunner (notFound >> return ()) env Nothing req

Complete Code

Here is the full code for the non-Template Haskell approach:

{-# LANGUAGE OverloadedStrings #-}

{-# LANGUAGE QuasiQuotes #-}

{-# LANGUAGE TemplateHaskell #-}

{-# LANGUAGE TypeFamilies #-}

{-# LANGUAGE ViewPatterns #-}

importqualifiedData.ByteString.Lazy.Char8as L8

import Network.HTTP.Types (status200)

import Network.Wai (pathInfo, rawPathInfo,

requestMethod, responseLBS)

import Yesod

import Yesod.Core.Types (YesodSubRunnerEnv (..))

dataApp=App

instanceRenderRouteAppwhere

dataRouteApp=OnlyGetR

| AnyMethodR

| HasParamRInt

| MySubsiteR (RouteWaiSubsite)

deriving (Show, Read, Eq)

renderRoute OnlyGetR= (["only-get"], [])

renderRoute AnyMethodR= (["any-method"], [])

renderRoute (HasParamR i) = (["has-param", toPathPiece i], [])

renderRoute (MySubsiteR subRoute) =

let (ps, qs) =renderRoute subRoute

in ("my-subsite" : ps, qs)

typeHandler=HandlerTAppIO

instanceYesodApp

instanceYesodDispatchAppwhere

yesodDispatch env req =

case pathInfo req of

["only-get"] ->

case requestMethod req of

"GET" ->yesodRunner

getOnlyGetR

env

(JustOnlyGetR)

req

_->yesodRunner

(badMethod >> return ())

env

(JustOnlyGetR)

req

["any-method"] ->

yesodRunner handleAnyMethodR env (JustAnyMethodR) req

["has-param", t] | Just i <-fromPathPiece t ->

case requestMethod req of

"GET" ->yesodRunner

(getHasParamR i)

env

(Just $ HasParamR i)

req

_->yesodRunner

(badMethod >> return ())

env

(Just $ HasParamR i)

req

("my-subsite":rest) ->yesodSubDispatch

YesodSubRunnerEnv

{ ysreGetSub =getMySubsite

, ysreParentRunner =yesodRunner

, ysreToParentRoute =MySubsiteR

, ysreParentEnv =env

}

req { pathInfo =rest }

_->yesodRunner (notFound >> return ()) env Nothing req

getOnlyGetR ::HandlerHtml

getOnlyGetR =defaultLayout

[whamlet|

<p>Accessed via GET method

<form method=post action=@{AnyMethodR}>

<button>POST to /any-method

|]

handleAnyMethodR ::HandlerHtml

handleAnyMethodR =do

req <-waiRequest

defaultLayout

[whamlet|

<p>In any-method, method == #{show $ requestMethod req}

|]

getHasParamR ::Int->HandlerString

getHasParamR i =return $ show i

getMySubsite ::App->WaiSubsite

getMySubsite _=

WaiSubsite app

where

app req sendResponse =sendResponse $ responseLBS

status200

[("Content-Type", "text/plain")]

$ L8.pack $ concat

[ "pathInfo == "

, show $ pathInfo req

, ", rawPathInfo == "

, show $ rawPathInfo req

]

main ::IO ()

main =warp 3000 App

Summary

Yesod abstracts away quite a bit of the plumbing from you as a developer. Most of this is boilerplate code that you’ll be happy to ignore. But it can be empowering to understand exactly what’s going on under the surface. At this point, you should hopefully be able—with help from the Haddocks—to write a site without any of the autogenerated Template Haskell code. Not that I’d recommend it; I think using the generated code is easier and safer.

One particular advantage of understanding this material is seeing where Yesod sits in the world of WAI. This makes it easier to see how Yesod will interact with WAI middleware, or how to include code from other WAI frameworks in a Yesod application (or vice versa!).