Introducing Elixir: Getting Started in Functional Programming (2014)

Chapter 12. Getting Started with OTP

At this point, it might seem like you have all you need to create process-oriented projects with Elixir. You know how to create useful functions, can work with recursion, know the data structures Elixir offers, and probably most important, know how to create and manage processes. What more could you need?

Process-oriented programming is great, but the details matter. The basic Elixir tools are powerful but can also bring you to frustrating mazes debugging race conditions that happen only once in a while. Mixing different programming styles can lead to incompatible expectations, and code that worked well in one environment may prove harder to integrate in another.

Ericsson encountered these problems early when developing Erlang (remember, Elixir runs on Erlang’s virtual machine), and created a set of libraries that eases them. OTP, the Open Telecom Platform, is useful for pretty much any large-scale project you want to do with Elixir and Erlang, not just telecom work. It’s included with Erlang, and though it isn’t precisely part of the language, it is definitely part of Erlang culture. The boundaries of where Elixir and Erlang end and OTP begins aren’t always clear, but the entrypoint is definitely behaviors. You’ll combine processes built with behaviors and managed by supervisors into an OTP application.

So far, the lifecycle of the processes shown in the previous chapters has been pretty simple. If needed, they set up other resources or processes to get started. Once running, they listen for messages and process them, collapsing if they fail. Some of them might restart a failed process if needed.

OTP formalizes those activities, and a few more, into a set of behaviors (or behaviours — this was originally created with British spelling). The most common behaviors are GenServer (generic server) and Supervisor. Through Erlang, you can use the gen_fsm (finite state machine) andgen_event behaviors. Elixir provides the Mix build tool for creating applications so that you can package your OTP code into a single runnable (and updatable) system.

The behaviors pre-define the mechanisms you’ll use to create and interact with processes, and the compiler will warn you if you’re missing some of them. Your code will handle the callbacks, specifying how to respond to particular kinds of events, and you will need to decide upon a structure for your application.

NOTE

If you’d like a free one-hour video introduction to OTP, though it is Erlang-centric, see Steve Vinoski’s “Erlang’s Open Telecom Platform (OTP) Framework” at http://www.infoq.com/presentations/Erlang-OTP-Behaviors. You probably already know the first half hour or so of it, but the review is excellent. In a very different style, if you’d like an explanation of why it’s worth learning OTP and process-oriented development in general, Francesco Cesarini’s slides at https://www.erlang-factory.com/upload/presentations/719/francesco-otp.pdf work even without narration, especially the second half.

Creating Services with gen_server

Much of the work you think of as the core of a program — calculating results, storing information, and preparing replies — will fit neatly into the GenServer behavior. It provides a core set of methods that let you set up a process, respond to requests, end the process gracefully, and even pass state to a new process if this one needs to be upgraded in place.

Table 12-1 shows the methods you need to implement in a service that uses GenServer. For a simple service, the first two or three are the most important, and you may just use placeholder code for the rest.

Table 12-1. What calls and gets called in GenServer

Example 12-1, which you can find in ch12/ex1-drop, shows an example that you can use to get started. It mixes a simple calculation from way back in Example 2-1 with a counter like that in Example 9-4.

Example 12-1. A simple gen_server example

defmodule DropServer do

use GenServer

defModule State do

defstruct count: 0

end

# This is a convenience method for startup

def start_link do

GenServer.start_link(__MODULE__, [], [{:name, __MODULE__}])

end

# These are the callbacks that GenServer.Behaviour will use

def init([]) do

{:ok, %State{}}

end

def handle_call(request, _from, state) do

distance = request

reply = {:ok, fall_velocity(distance)}

new_state = %State{count: state.count + 1}

{:reply, reply, new_state}

end

def handle_cast(_msg, state) do

IO.puts("So far, calculated #{state.count} velocities.")

{:noreply, state}

end

def handle_info(_info, state) do

{:noreply, state}

end

def terminate(_reason, _state) do

{:ok}

end

def code_change(_old_version, state, _extra) do

{:ok, state}

end

# internal function

def fall_velocity(distance) do

:math.sqrt(2 * 9.8 * distance)

end

end

The module name (DropServer) should be familiar from past examples. The second line specifies that the module is going to be using the GenServer module.

The nested defModule declaration should be familiar; it creates a structure that contains only one field, to keep a count of the number of calls made. Many services will have more fields, including things like database connections, references to other processes, perhaps network information, and metadata specific to this particular service. It is also possible to have services with no state, which would be represented by an empty tuple here. As you’ll see further down, every single GenServer function will reference the state.

NOTE

The State structure declaration is a good example of a declaration you should make inside of a module and not declare in a separate file. It is possible that you’ll want to share state models across different processes that use GenServer, but it’s easier to see what State should contain if the information is right there.

The first function in the sample, start_link/0, is not one of the required GenServer functions. Instead, it calls Elixir’s GenServer.start_link function to start up the process. When you’re just getting started, this is useful for testing. As you move toward production code, you may find it easier to leave these out and use other mechanisms.

The start_link/0 function uses the built-in MODULE declaration, which returns the name of the current module..

# This is a convenience method for startup

def start_link do

GenServer.start_link(__MODULE__, [], [{:name, __MODULE__}])

end

The first argument is an atom (MODULE) that will be expanded to the name of the current module, and that will be used as the name for this process. This is followed by a list of arguments to be passed to the module’s initialization procedure and a list of options. Options can specify things like debugging, timeouts, and options for spawning the process. By default, the name of the process is registered with just the local Elixir instance. Because we want it registered with all associated nodes, we have put the tuple {:name, MODULE} in the options list.

NOTE

You may also see a form of GenServer.start_link with :via as an atom in an option tuple. This lets you set up custom process registries, of which gproc is the best known. For more on that, see https://github.com/uwiger/gproc.

All of the remaining functions are part of GenServer’s behavior. init/1 creates a new state structure instance whose count field is zero — no velocities have yet been calculated. The two functions that do most of the work here are handle_call/3 and handle_cast/2. For this demonstration, handle_call/3 expects to receive a distance in meters and returns a velocity for a fall from that height on earth, while handle_cast/2 is a trigger to report the number of velocities calculated.

handle_call/3 makes synchronous communications between Erlang processes simple.

def handle_call(request, _from, state) do

distance = request

reply = {:ok, fall_velocity(distance)}

new_state = %State{count: state.count + 1}

{:reply, reply, new_state}

end

This extracts the distance from the request, which isn’t necessary except that I wanted to leave the variable names for the function almost the same as they were in the template. (handle_call(distance, _from, state) would have been fine.) Your request is more likely to be a tuple or a list rather than a bare value, but this works for simple calls.

The function then creates a reply based on sending that distance to the simple fall_velocity/1 function at the end of the module. It then creates a new_state containing an incremented count. Then the atom :reply, the reply tuple containing the velocity, and the new_statecontaining the updated count get passed back.

Because the calculation is really simple, treating the drop as a simple synchronous call is perfectly acceptable. For more complex situations where you can’t predict how long a response might take, you may want to considering responding with a :noreply response and using the _fromargument to send a response later. (There is also a :stop response available that will trigger the :terminate/2 method and halt the process.)

NOTE

By default, OTP will time out any synchronous calls that take longer than five seconds to calculate. You can override this by making your call using GenServer.call/3 to specify a timeout (in milliseconds) explicitly, or by using the atom :infinity.

The handle_cast/2 function supports asynchronous communications. It isn’t supposed to return anything directly, though it does report :noreply (or :stop) and updated state. In this case, it takes a very weak approach, but one that does well for a demonstration, calling IO:puts/1 to report on the number of calls:

def handle_cast(_msg, state) do

IO.puts("So far, calculated #{state.count} velocities.")

{:noreply, state}

end

The state doesn’t change, because asking for the number of times the process has calculated a fall velocity is not the same thing as actually calculating a fall velocity.

Until you have good reason to change them, you can leave handle_info/2, terminate/2, and code_change/3 alone.

Making a GenServer process run and calling it looks a little different than starting the processes you saw in Chapter 9.

iex(1)> c("drop_server.ex")

[DropServer,DropServer.State]

iex(2)> DropServer.start_link()

{:ok,#PID<0.46.0>}

iex(3)> GenServer.call(DropServer, 20)

{:ok,19.79898987322333}

iex(4)> GenServer.call(DropServer, 40)

{:ok,28.0}

iex(5)> GenServer.call(DropServer, 60)

{:ok,34.292856398964496}

iex(6)> GenServer.cast(DropServer, {})

So far, calculated 3 velocities.

:ok

The call to DropServer.start_link() sets up the process and makes it available. Then, you’re free to use GenServer.call or GenServer.cast to send it messages and get responses.

NOTE

While you can capture the pid, you don’t have to keep it around to use the process. Because start_link returns a tuple, if you want to capture the pid you can do something like {:ok, pid} = Drop.start_link().

Because of the way OTP calls GenServer functions, there’s an additional bonus — or perhaps a hazard — in that you can update code on the fly. For example, I tweaked the fall_velocity/1 function to lighten Earth’s gravity a little, using 9.1 as a constant instead of 9.8. Recompiling the code and asking for a velocity returns a different answer:

iex(7)> c("drop_server.ex")

drop_server.ex:1: redefining module DropServer

drop_server.ex:4: redefining module DropServer.State

[DropServer,DropServer.State]

iex(8)> GenServer.call(DropServer, 60)

{:ok,33.04542328371661}

This can be very convenient during the development phase, but be careful doing anything like this on a production machine. OTP has other mechanisms for updating code on the fly. There is also a built-in limitation to this approach: init gets called only when start_link sets up the service. It does not get called if you recompiled the code. If your new code requires any changes to the structure of its state, your code will break the next time it’s called.

A Simple Supervisor

When you started the DropServer module from the shell, you effectively made the shell the supervisor for the module — though the shell doesn’t really do any supervision. You can break the module easily:

iex(9)> GenServer.call(DropServer, -60)

=ERROR REPORT==== 28-Jun-2014::08:17:52 ===

** (EXIT from #PID<0.42.0>) an exception was raised:

** (ArithmeticError) bad argument in arithmetic expression

(stdlib) :math.sqrt(-1176.0)

drop_server.ex:44: DropServer.fall_velocity/1

drop_server.ex:20: DropServer.handle_call/3

(stdlib) gen_server.erl:580: :gen_server.handle_msg/5

(stdlib) proc_lib.erl:239: :proc_lib.init_p_do_apply/3

Interactive Elixir (1.0.0-rc2) - press Ctrl+C to exit (type h() ENTER for help)

08:52:00.459 [error] GenServer DropServer terminating

Last message: -60

State: %DropServer.State{count: 3}

** (exit) an exception was raised:

** (ArithmeticError) bad argument in arithmetic expression

(stdlib) :math.sqrt(-1176.0)

drop_server.ex:44: DropServer.fall_velocity/1

drop_server.ex:20: DropServer.handle_call/3

(stdlib) gen_server.erl:580: :gen_server.handle_msg/5

(stdlib) proc_lib.erl:239: :proc_lib.init_p_do_apply/3

The error message is nicely complete, even telling you the last message and the state, but when you go to call the service again, you can’t, because the IEx shell has restarted. You can restart it with DropServer.start_link/0 again, but you’re not always going to be watching your processes personally.

Instead, you want something that can watch over your processes and make sure they restart (or not) as appropriate. OTP formalizes the process management you saw in Example 9-10 with its supervisor behavior.

A basic supervisor needs to support only one callback function, init/1, and can also have a start_link function to fire it up. The return value of that init/1 function tells OTP which child processes your supervisor manages and how how you want to handle their failures. A supervisor for the drop module might look like Example 12-2, which is in ch12/ex2-drop-sup.

Example 12-2. A simple supervisor

defmodule DropSup do

use Supervisor

# convenience method for startup

def start_link do

Supervisor.start_link(__MODULE__, [], [{:name, __MODULE__}])

end

# supervisor callback

def init([]) do

child = [worker(DropServer, [], [])]

supervise(child, [{:strategy, :one_for_one}, {:max_restarts, 1},

{:max_seconds, 5}])

end

# Internal functions (none here)

end

The init/1 function’s job is to specify the process or processes that the supervisor is to keep track of, and specify how it should handle failure.

The worker/3 function specifies a module that the supervisor should start, its argument list, and any options to be given to the worker’s start_link function. In this example, there is only one child process to supervise, and the options are given as list of key/value tuples.

NOTE

You can also specify the options as a keyword list, which you would write this way:

supervise(child, strategy: :one_for_one, max_restarts: 1, max_seconds: 5)

The supervise/2 function takes the list of child processes as its first argument and a list of options as its second argument.

The :strategy of :one_for_one tells OTP that it should create a new child process every time a process that is supposed to be :permanent (the default) fails. You can also go with :one_for_all, which terminates and restarts all of the processes the supervisor oversees when one fails, or :rest_for_one, which restarts the process and any processes that began after the failed process had started.

NOTE

When you’re ready to take more direct control of how your processes respond to their environment, you might explore working with the dynamic functions Supervisor.start/2, Supervisor.terminate_child/2, Supervisor.restart_child/2, and Supervisor.delete_child/2, as well as the restart strategy :simple_one_for_one.

The next two values define how often the worker processes can crash before terminating the supervisor itself. In this case, it’s one restart every five seconds. Customizing these values lets you handle a variety of conditions but probably won’t affect you much initially. (Setting:max_restarts to zero means that the supervisor will just terminate if a worker has an error.)

The supervise function takes those arguments and creates a data structure that OTP will use. By default, this is a :permanent service, so the supervisor should always restart a child when it fails. You can specify a :restart option when defining the worker if you want to change this to a different value. The supervisor can wait five seconds before shutting down the worker completely; you can change this with the :shutdown option when defining the worker. More complex OTP applications can contain trees of supervisors managing other supervisors, which themselves manage other supervisors or workers. To create a child process that is a supervisor, you use the supervisor/3 function, whose arguments are the same as those of worker/3.

NOTE

OTP wants to know the dependencies so that it can help you upgrade software in place. It’s all part of the magic of keeping systems running without ever bringing them to a full stop.

Now that you have a supervisor process, you can set up the drop function by just calling the supervisor. However, running a supervisor from the shell using the start_link/0 function call creates its own set of problems; the shell is itself a supervisor, and will terminate processes that report errors. After a long error report, you’ll find that both your worker and the supervisor have vanished.

In practice this means that you need a way to test supervised OTP processes (that aren’t yet part of an application) directly from the shell. This method explicitly breaks the bond between the shell and the supervisor process by catching the pid of the supervisor (line 2) and then using theProcess.unlink/1 function to remove the link (line 3). Then you can call the process as usual with GenServer.call/2 and get answers. If you get an error (line 6), it’ll be okay. The supervisor will restart the worker, and you can make new calls successfully. The calls toProcess.whereis(DropServer) on lines 4 and 7 demonstrate that the supervisor has restarted DropServer with a new pid.

iex(1)> c("drop_sup.ex")

[DropSup]

iex(2)> {:ok, pid} = DropSup.start_link()

{:ok,#PID<0.44.0>}

iex(3)> Process.unlink(pid)

true

iex(4)> Process.whereis(DropServer)

#PID<0.45.0>

iex(5)> GenServer.call(DropServer,60)

{:ok,34.292856398964496}

iex(6)> GenServer.call(DropServer, -60)

** (exit) exited in: GenServer.call(DropServer, -60, 5000)

** (EXIT) an exception was raised:

** (ArithmeticError) bad argument in arithmetic expression

(stdlib) :math.sqrt(-1176.0)

drop_server.ex:44: DropServer.fall_velocity/1

drop_server.ex:20: DropServer.handle_call/3

(stdlib) gen_server.erl:580: :gen_server.handle_msg/5

(stdlib) proc_lib.erl:239: :proc_lib.init_p_do_apply/3

08:59:56.886 [error] GenServer DropServer terminating

Last message: -60

State: %DropServer.State{count: 1}

** (exit) an exception was raised:

** (ArithmeticError) bad argument in arithmetic expression

(stdlib) :math.sqrt(-1176.0)

drop_server.ex:44: DropServer.fall_velocity/1

drop_server.ex:20: DropServer.handle_call/3

(stdlib) gen_server.erl:580: :gen_server.handle_msg/5

(stdlib) proc_lib.erl:239: :proc_lib.init_p_do_apply/3

(elixir) lib/gen_server.ex:356: GenServer.call/3

iex(6)> GenServer.call(DropServer, 60)

{:ok,34.292856398964496}

iex(7)> Process.whereis(DropServer)

#PID<0.46.0>

NOTE

You can also open the Process Manager in Observer and whack away at worker processes through the Kill option on the Trace menu and watch them reappear.

This works, but it is only the tiniest taste of what supervisors can do. They can create child processes dynamically and manage their lifecycle in greater detail.

Packaging an Application with Mix

Elixir’s Mix tool “provides tasks for creating, compiling, testing (and soon deploying) Elixir projects.” In this section, you will use Mix to create an application for the drop supervisor and server that you have written.

Create a directory to hold your application, type mix new name, as in the following example:

$ mix new drop_app

* creating README.md

* creating .gitignore

* creating mix.exs

* creating config

* creating config/config.exs

* creating lib

* creating lib/drop_app.ex

* creating test

* creating test/test_helper.exs

* creating test/drop_app_test.exs

Your mix project was created successfully.

You can use mix to compile it, test it, and more:

cd drop_app

mix test

Run `mix help` for more commands.

NOTE

Make sure that the Elixir executable is in your $PATH variable so that Mix can find it.

Mix creates a set of files and directories for you. Change directory to the drop_app directory that Mix created. Then open up the mix.exs file in your favorite text editor.

defmodule DropApp.Mixfile do

use Mix.Project

def project do

[app: :drop_app,

version: "0.0.1",

elixir: "~> 1.0.0-rc2",

deps: deps]

end

# Configuration for the OTP application

#

# Type `mix help compile.app` for more information

def application do

[applications: [:logger]]

end

# Dependencies can be Hex packages:

#

# {:mydep, "~> 0.3.0"}

#

# Or git/path repositories:

#

# {:mydep, git: "https://github.com/elixir-lang/mydep.git", tag: "0.1.0"}

#

# Type `mix help deps` for more examples and options

defp deps do

[]

end

end

The project/0 function lets you name your application, give it a version number, and specify the dependencies for building the project.

The dependencies are returned by the deps/0 function. The commented example says that you need to have the mydep project version 0.3.0 or higher, and it is available via git at the specified URL. In addition to git:, you may specify the location of a dependency as a local file (path:)

In this example, the application doesn’t have any dependencies, so you may leave everything exactly as it is.

If you type the command mix compile, Mix will compile your empty project. If you look in your directory, you will see that Mix has created an _build directory for the compiled code.

$ mix compile

Compiled lib/drop_app.ex

Generated drop_app.app

$ ls

_build config lib mix.exs README.md test

An empty application isn’t very exciting, so copy the drop_server.ex and drop_sup.ex files that you wrote into the lib folder. Then start iex -S mix. Mix will compile the new files, and you can start using the server straightaway.

$ iex -S mix

Erlang/OTP 17 [erts-6.0] [source] [64-bit] [smp:2:2] [async-threads:10] [hipe] [kernel-poll:false]

Compiled lib/drop_sup.ex

Compiled lib/drop_server.ex

Generated drop_app.app

Interactive Elixir (1.0.0-rc2) - press Ctrl+C to exit (type h() ENTER for help)

iex(1)> {:ok, pid} = DropServer.start_link()

{:ok,#PID<0.60.0>}

The last steps you need to do are to write the application code itself and then tell Mix where everything is.

Inside mix.exs, change the application/0 function to look like this:

def application do

[ applications: [:logger],

registered: [:drop_app],

mod: {DropApp, []} ]

end

The :registered key is a list of all the names that your application registers (in this case, just :drop_app), and :mod is a tuple that gives the name of the module to be run when the application starts up and a list of any arguments to be passed to that module. :applications lists any applications that your application depends on at runtime.

Here is the code that we have added to the DropApp Module, which is in a file named drop_app.ex in the ch12/ex3-drop-app/drop_app/lib directory.

defmodule DropApp do

use Application

def start(_type, _args) do

IO.puts("Starting the app...") # show that app is really starting.

DropSup.start_link()

end

end

The start/2 function is required. The first argument tells how you want the virtual machine that Elixir runs on to handle application crashes. The second argument gives the arguments that you defined in the :mod key. The start/2 function should return a tuple of the form {:ok, pid}, which is exactly what DropSup.start_link/0 does.

If you type mix compile at the command prompt, Mix will generate a file _build/dev/lib/drop_app/ebin/drop_app.app. (If you look at that file, you will see an Erlang tuple that contains much of the information gleaned from the files you have already created.) You may then run the application from the command line.

$ elixir -pa _build/dev/lib/drop_app/ebin --app drop_app

Starting the app...

There is much, much more to learn. OTP deserves a book or several all on its own. Hopefully this chapter provides you with enough information to try some things out and understand those books. However, the gap between what this chapter can reasonably present and what you need to know to write solid OTP-based programs is, to say the least, vast.