The LFE Tutorial
Adapted from the Erlang "Getting Started" tutorial
Original Erlang version by the Erlang/OTP Team
LFE translation by Robert Virding & Duncan McGreggor
Copyright
© 1996-2020, Ericsson AB
© 2015, 2020-2023 Robert Virding, Duncan McGreggor
The contents of this file are subject to the Erlang Public License, Version 1.1, (the "License"); you may not use this file except in compliance with the License. You should have received a copy of the Erlang Public License along with this software. If not, it can be retrieved online at http://www.erlang.org/.
Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License.
Introduction
In this chapter we will provide brief introductions to Erlang, OTP, and LFE as well as discussion the dependencies for LFE development, and finally obtaining and building LFE from source.
About Erlang
Erlang is a programming language used to build massively scalable soft real-time systems with requirements on high availability. Some of its uses are in telecoms, banking, e-commerce, computer telephony and instant messaging. Erlang's runtime system has built-in support for concurrency, distribution and fault tolerance.
Most real-world Erlang applications are built using Erlang/OTP. OTP is set of Erlang libraries and design principles providing middleware to develop these systems. It includes its own distributed database, applications to interface towards other languages, debugging and release handling tools.
As an LFE programmer, you will have the ability and freedom to utilise Erlang and OTP libraries directly from a Lisp syntax.
About LFE
LFE, or "Lisp Flavoured Erlang", is a Lisp-2 (thus similar to Common Lisp, and MACLISP before that) that runs on top of the Erlang VM or "BEAM". It is the oldest sibling of the many BEAM languages that have been released since its inception (e.g., Elixir, Joxa, Luerl, Erlog, Haskerl, Clojerl, Hamler, etc.).
As a Lisp-2, LFE has different namespaces for functions and variables. Furthermore, LFE has Erlang's notion of arity, so functions with different arity may share the same name.
Lisps are great, but the thing that makes LFE so incredible is that it runs on the Erlang VM and is 100% compatible with Core Erlang.
Dependencies
In order to write LFE scripts, libraries, and applications, you will need to install the following:
- Erlang
makeand related developer toolsgit
Installing Erlang
First and foremost, you will need Erlang installed.
- On Mac OS X, this is as easy as executing
brew install erlang - On Ubuntu
apt-get install erlang.
You can also install Erlang from the various pre-built packages provided on the official Erlang download page or from the Erlang Solutions page (which supports many more package types).
For those who have the need of installing multiple versions of Erlang, there is also the kerl project.
Installing make
In order to use LFE, you will be calling make to compile it. To cover your bases, you'll want to make sure you have the basic development tools installed for your platform.
Ubuntu:
sudo apt-get install build-essential
CentOS/Fedora:
sudo yum groupinstall "Development Tools"
For Mac OS X, you will need to install the "Developer Tools" from the AppStore.
Installing git
You will also need git to continue with this quick-start. If git doesn't come installed on your system and it wasn't installed as part of your systems basic development tools package, you can download it here
or install it using your favourite OS package manager.
Installing LFE
Downloading the Source
The most recent version of LFE is always available here:
- https://github.com/rvirding/lfe/
With git in your PATH, you can download LFE:
git clone https://github.com/rvirding/lfe.git
cd lfe
You have two choices for the type of LFE build you want to do.
Now everything is ready to go!
Stable
Stable contains a version of LFE that is known to be fully reliable and used in production for years.
To build the latest stable release, make sure you are on the master branch:
git checkout master
Unstable
Unstable contains LFE that is under active development, has not been released, or has not had extensive use and testing in production environments.
To build the latest unstable LFE, make sure you are on the develop branch:
git checkout develop
LFE is just a set of Erlang libraries, so like an Erlang project, the source code needs to be compiled to .beam files. Running make in the lfe directory will do that:
make
Sequential Programming
In this chapter we will look at the basics of sequential programming in LFE.
The LFE REPL
Most operating systems have a command interpreter or shell -- Unix and Linux have many, while Windows has the Command Prompt. Likewise, Erlang has a shell where you can directly write bits of Erlang code and evaluate (run) them to see what happens. LFE has more than a shell: it's a full REPL (read-eval-print loop) like other Lisps, and it can do more than the Erlang shell can (including defining proper functions and Lisp macros).
Starting the LFE REPL
In your system terminal window where you changed directory to the clone of the LFE repository, you can start the LFE REPL by typing the following:
./bin/lfe
At which point you will see output something like this:
Erlang/OTP 26 [erts-14.0.2] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit] [dtrace]
..-~.~_~---..
( \\ ) | A Lisp-2+ on the Erlang VM
|`-.._/_\\_.-': | Type (help) for usage info.
| g |_ \ |
| n | | | Docs: http://docs.lfe.io/
| a / / | Source: http://github.com/lfe/lfe
\ l |_/ |
\ r / | LFE v2.1.2 (abort with ^G)
`-E___.-'
lfe>
Interactive LFE Code
Now let's multiply two numbers in the REPL by typing (* 2 21) at the > prompt:
lfe> (* 2 21)
Lisp stands for "LISt Processor" because nearly everything in Lisp is really just a list of things -- including the code itself. The lists in Lisps are created with parentheses, just like the expression above. As you can see, the multiplication operator goes first, followed by its arguments -- this is called prefix notation, due to the operator coming first.
In order to tell the REPL that you want it to evaluate your LFE code, you need to his the <ENTER> or <RETURN> key.
42
lfe>
It has correctly given you the answer: 42.
Now let's try a more complex calculation:
lfe> (* 2 (+ 1 2 3 4 5 6))
This expression has one nested inside the other. The first one to be executed is the inner-most, in this case, the addition operation. Just like we saw before, the operator comes first (the "addition" operator, in this case) and then all of the numbers to be added.
Hit <ENTER> to get your answer:
42
lfe>
Defining Variables and Functions
The LFE REPL allows you do set variables and define functions. Let's define a variable called multiplier:
lfe> (set multiplier 2)
2
lfe>
When we set the value for that variable, the REPL provided feedback on the expression entered, showing us the value. Now we can use it just like the number for which it stands:
lfe> (* multiplier (+ 1 2 3 4 5 6))
42
lfe>
The set form lets you define a variable; the defun form lets you define a function. Enter this in the REPL:
lfe> (defun double (x)
(* 2 x))
Now try it:
lfe> (double 21)
42
lfe>
As we can see, this function multiplies any given number by 2.
Leaving the REPL
To exit the REPL and shutdown the underlying Erlang system which started when you executed ./bin/lfe, simply exit:
lfe> (exit)
ok
At which point you will be presented with your regular system terminal prompt.
There are two other ways in which you may leave the REPL:
- Hitting
^ctwice in a row, or - Hitting
^gthenq
Modules and Functions
Creating a Simple Module
A programming language isn't much use if you can only run code from a REPL. So next we will write a small LFE program in a file on the file system. In the same directory that you started the LFE REPL, create a new file called tut1.lfe (the filename is important: be sure you type it just as we have) using your favourite text editor.
Here's the code to enter:
(defmodule tut1
(export all))
(defun double (x)
(* 2 x))
It's not hard to guess that this "program" doubles the value of numbers. We'll get back to the first two lines later. Let's compile the program. This can be done in the LFE REPL as shown below:
lfe> (c "tut1.lfe")
#(module tut1)
lfe>
The #(module tut1) tells you that the compilation was successful. If it said "error" instead, you have made some mistake in the text you entered and there will also be error messages to give you some idea as to what has gone wrong so you can change what you have written and try again.
Now lets run the program.
lfe> (tut1:double 108)
216
lfe>
As expected, 108 doubled is 216.
Now let's get back to the first two lines. LFE programs are written in files. Each file contains what we call an LFE module. The first line of code in the module tells LFE that we're defining a module and giving it a name:
(defmodule tut1
The name of our module is tut1 and the file which is used to store the module must have the same name as the module but with the .lfe extension. In our case the file name is tut1.lfe.
In LFE, whenever we use a function that has been defined in another module, we use the syntax, (module:function argument1 argument2 ...). So
lfe> (tut1:double 108)
means "call the function double in the module tut1 with the argument of 108.
The second line tells LFE which functions we will be exporting -- in this case, all of them (which is only one ...):
(export all))
If we wanted to be explicit about which functions were to be exported, we would have written:
(defmodule tut1
(export (double 1)))
That says "in the module tut1, please make available the function called double which takes one argument" (x in our example). By "make available" we mean that this function can be called from outside the module tut1.
A More Complicated Example
Now for a more complicated example, the factorial of a number (e.g. factorial of 4 is 4 * 3 * 2 * 1). Enter the following code in a file called tut2.lfe.
(defmodule tut2
(export (fac 1)))
(defun fac
((1) 1)
((n) (* n (fac (- n 1)))))
Compile the file
lfe> (c "tut2.lfe")
#(module tut2)
And now calculate the factorial of 4.
lfe> (tut2:fac 4)
24
The function fac contains two parts. The first part:
((1) 1)
says that the factorial of 1 is 1. Note that this part is a separate list in the function definition where the first element is a list of the arguments to the function and the rest is the body of the function. The second part:
((n) (* n (fac (- n 1)))))
says that the factorial of n is n multiplied by the factorial of n - 1. After this part which is the last part we end the function definition with the closing ).
A function can have many arguments. Let's expand the module tut2 with the rather stupid function to multiply two numbers:
(defmodule tut3
(export (fac 1) (mult 2)))
(defun fac
((1) 1)
((n) (* n (fac (- n 1)))))
(defun mult (x y)
(* x y))
Note that we have also had to expand the (export line with the information that there is another function mult with two arguments. Compile the file:
lfe> (c "tut3.lfe")
#(module tut3)
and try it out:
lfe> (tut3:mult 3 4)
12
In the example above the numbers are integers and the arguments in the functions in the code, n, x, y are called variables. Examples of variables could be number, shoe-size, age etc.
Note that when a function has only one part and all the arguments are variables then we can use the shorter form we saw in double and mult. This means that we could also have written mult as:
(defun mult
((x y) (* x y)))
Atoms
Atoms are another data type in LFE. They are words, for example charles, centimetre, inch and ok. Atoms are similar to symbols in Lisp except that are simply names, nothing else. They are not like variables which can have a value.
Enter the next program (file: tut4.lfe) which could be useful for converting from inches to centimetres and vice versa:
(defmodule tut4
(export (convert 2)))
(defun convert
((m 'inch) (/ m 2.54))
((n 'centimetre) (* n 2.54)))
Compile and test:
lfe> (c "tut4.lfe")
#(module tut4)
lfe> (tut4:convert 3 'inch)
1.1811023622047243
lfe> (tut4:convert 7 'centimetre)
17.78
Notice that atoms and variables look the same so we have to tell LFE when we want it to be an atom. We do this by quoting the atom with a ', for example 'inch and 'centimetre. We have to do this both when we use it as argument in a function definition and when we use it when calling a function, otherwise LFE will assume that it is a variable.
Also notice that we have introduced decimals (floating point numbers) without any explanation, but I guess you can cope with that.
See what happens if I enter something other than centimetre or inch in the convert function:
lfe> (tut4:convert 3 'miles)
exception error: function_clause
in (: tut4 convert 3 miles)
The two parts of the convert function are called its clauses. Here we see that miles is not part of either of the clauses. The LFE system can't match either of the clauses so we get an error message function_clause. The shell formats the error message nicely, but to see the actual error tuple we can do:
lfe> (catch (tut4:convert 3 'miles))
#(EXIT
#(function_clause
(#(tut2 convert (3 miles) (#(file "./tut2.lfe") #(line 4)))
#(lfe_eval eval_expr 2 (#(file "src/lfe_eval.erl") #(line 160)))
#(lfe_shell eval_form_1 2 (#(file "src/lfe_shell.erl") #(line 268)))
#(lists foldl 3 (#(file "lists.erl") #(line 1261)))
#(lfe_shell server_loop 1 (#(file "src/lfe_shell.erl") #(line 101))))))
Tuples
Now the tut4 program is hardly good programming style. Consider:
(tut4:convert 3 'inch)
Does this mean that 3 is in inches? or that 3 is in centimetres and we want to convert it to inches? So LFE has a way to group things together to make things more understandable. We call these tuples. Tuples are constructed and matched using (tuple ...), with literal tuples being written with #( ... ).
So we can write #(inch 3) to denote 3 inches and #(centimeter 5) to denote 5 centimetres. Now let's write a new program which converts centimetres to inches and vice versa (file tut5.lfe).
(defmodule tut5
(export (convert-length 1)))
(defun convert-length
(((tuple 'centimeter x)) (tuple 'inch (/ x 2.54)))
(((tuple 'inch y)) (tuple 'centimeter (* y 2.54))))
Compile and test:
(c "tut5.lfe")
#(module tut5)
lfe> (tut5:convert-length #(inch 5))
#(centimeter 12.7)
lfe> (tut5:convert-length (tut5:convert-length #(inch 5)))
#(inch 5.0)
Note that in the last call we convert 5 inches to centimetres and back again and reassuringly get back to the original value. I.e the argument to a function can be the result of another function. Pause for a moment and consider how that line (above) works. The argument we have given the function #(inch 5) is first matched against the first head clause of convert-length i.e. in ((tuple 'centimeter x)) where it can be seen that the pattern (tuple 'centimeter x) does not match #(inch 5) (the head is the first bit in the clause with a list of argument patterns). This having failed, we try the head of the next clause i.e. ((tuple 'inch y)), this pattern matches #(inch 5) and y gets the value 5.
We have shown tuples with two parts above, but tuples can have as many parts as we want and contain any valid LFE term. For example, to represent the temperature of various cities of the world we could write
#(Moscow #(C -10))
#(Cape-Town #(F 70))
#(Paris #(F 28))
Tuples have a fixed number of things in them. We call each thing in a tuple an element. So in the tuple #(Moscow #(C -10)), element 1 is Moscow and element 2 is #(C -10). We have chosen C meaning Celsius (or Centigrade) and F meaning Fahrenheit.
Lists
Whereas tuples group things together, we also want to be able to represent lists of things. Lists in LFE are surrounded by "(" and ")". For example a list of the temperatures of various cities in the world could be:
(#(Moscow #(C -10))
#(Cape-Town #(F 70)) #(Stockholm #(C -4))
#(Paris #(F 28)) #(London #(F 36)))
Note that this list was so long that it didn't fit on one line. This doesn't matter, LFE allows line breaks at all "sensible places" but not, for example, in the middle of atoms, integers etc.
A very useful way of looking at parts of lists is by using the constructor cons. It can also be used as a pattern to match against lists. This is best explained by an example using the shell.
lfe> (set (cons first rest) '(1 2 3 4 5))
(1 2 3 4 5)
lfe> first
1
lfe> rest
(2 3 4 5)
We see here that set also allows you to define variables using patterns. Using cons we could separate the first element of the list from the rest of list (first got the value 1 and rest the value (2 3 4 5)). We also see here that when we want to give a literal list we need to quote it with '. This stops LFE trying to evaluate the list as a function call in the same way as quoting an atom stops LFE trying to evaluate the atom as a variable.
Another example:
lfe> (set (cons e1 (cons e2 r)) '(1 2 3 4 5 6 7))
(1 2 3 4 5 6 7)
lfe> e1
1
lfe> e2
2
lfe> r
(3 4 5 6 7)
We see here nesting cons to get the first two elements from the list. Of course if we try to get more elements from the list than there are elements in the list we will get an error. Note also the special case of the list with no elements ().
lfe> (set (cons a (cons b c)) '(1 2))
(1 2)
lfe> a
1
lfe> b
2
lfe> c
()
In all examples above we have been using new variables names, not reusing old ones. While set does allow you to rebind variables normally a variable can only be given a value once in its context (scope).
The following example shows how we find the length of a list:
(defmodule tut6
(export (list-length 1)))
(defun list-length
((()) 0)
(((cons first rest))
(+ 1 (list-length rest))))
Compile (file tut4.lfe) and test:
lfe> (c "tut6.lfe")
#(module tut6)
lfe> (tut6:list-length '(1 2 3 4 5 6 7))
7
Explanation:
(defun list-length
((()) 0)
The length of an empty list is obviously 0.
(((cons first rest))
(+ 1 (list-length rest))))
The length of a list with the first element first and the remaining elements rest is 1 + the length of rest.
(Advanced readers only: This is not tail recursive, there is a better way to write this function).
In general we can say we use tuples where we would use "records" or "structs" in other languages and we use lists when we want to represent things which have varying sizes, (i.e. where we would use linked lists in other languages).
LFE does not have a string data type, instead strings can be represented by lists of Unicode characters. So the list (97 98 99) is equivalent to "abc". Note that we don't have to quote strings as we do lists. The LFE repl is "clever" and guesses the what sort of list we mean and outputs it in what it thinks is the most appropriate form, for example:
lfe> '(97 98 99)
"abc"
lfe> "abc"
"abc"
lfe> '"abc"
"abc"
Lists can also be surrounded by "[" and "]" instead of parentheses. They are equivalent but must match, for example:
lfe> '(a b c)
(a b c)
lfe> '[a b c]
(a b c)
lfe> '(a b c]
1: illegal ']'
This can be used to make list structures easier to read. For example, it is often used in function definitions for the list of arguments when there are multiple clauses:
(defun list-length
([()] 0)
([(cons first rest)]
(+ 1 (list-length rest))))
Property Lists and Maps
Property Lists
Property lists in Erlang and LFE are a simple way to create key-value pairs (we actually saw them in the last section, but didn't mention it). They have a very simple structure: a list of tuples, where the key is the first element of each tuple and is an atom. Very often you will see "options" for functions provided as property lists (this is similar to how other programming languages use keywords in function arguments).
Property lists can be created with just the basic data structures of LFE:
lfe> (set options (list (tuple 'debug 'true)
(tuple 'default 42)))
(#(debug true) #(default 42))
Or, more commonly, using quoted literals:
lfe> (set options '(#(debug true) #(default 42)))
(#(debug true) #(default 42))
There are convenience functions provided in the proplists module. In the last example, we define a default value to be used in the event that the given key is not found in the proplist:
lfe> (proplists:get_value 'default options)
42
lfe> (proplists:get_value 'poetry options)
undefined
lfe> (proplists:get_value 'poetry options "Vogon")
"Vogon"
Be sure to read the module documentation for more information. Here's an example of our options in action:
(defmodule tut61
(export (div 2) (div 3)))
(defun div (a b)
(div a b '()))
(defun div (a b opts)
(let ((debug (proplists:get_value 'debug opts 'false))
(ratio? (proplists:get_value 'ratio opts 'false)))
(if (and debug ratio?)
(io:format "Returning as ratio ...~n"))
(if ratio?
(++ (integer_to_list 1) "/" (integer_to_list 2))
(/ a b))))
Let's try our function without and then with various options:
lfe> (c "tut61.lfe")
#(module tut61)
lfe> (tut61:div 1 2)
0.5
lfe> (tut61:div 1 2 '(#(ratio true)))
"1/2"
lfe> (tut61:div 1 2 '(#(ratio true) #(debug true)))
Returning as ratio ...
"1/2"
Maps
As with property lists, maps are a set of key to value associations. You may create an association from "key" to value 42 in one of two ways: using the LFE core form map or entering a map literal:
lfe> (map "key" 42)
#M("key" 42)
lfe> #M("key" 42)
#M("key" 42)
We will jump straight into the deep end with an example using some interesting features. The following example shows how we calculate alpha blending using maps to reference colour and alpha channels. Save this code as the file tut7.lfe in the directory from which you have run the LFE REPL:
(defmodule tut7
(export (new 4) (blend 2)))
(defmacro channel? (val)
`(andalso (is_float ,val) (>= ,val 0.0) (=< ,val 1.0)))
(defmacro all-channels? (r g b a)
`(andalso (channel? ,r)
(channel? ,g)
(channel? ,b)
(channel? ,a)))
(defun new
((r g b a) (when (all-channels? r g b a))
(map 'red r 'green g 'blue b 'alpha a)))
(defun blend (src dst)
(blend src dst (alpha src dst)))
(defun blend
((src dst alpha) (when (> alpha 0.0))
(map-update dst
'red (/ (red src dst) alpha)
'green (/ (green src dst) alpha)
'blue (/ (blue src dst) alpha)
'alpha alpha))
((_ dst _)
(map-update dst 'red 0 'green 0 'blue 0 'alpha 0)))
(defun alpha
(((map 'alpha src-alpha) (map 'alpha dst-alpha))
(+ src-alpha (* dst-alpha (- 1.0 src-alpha)))))
(defun red
(((map 'red src-val 'alpha src-alpha)
(map 'red dst-val 'alpha dst-alpha))
(+ (* src-val src-alpha)
(* dst-val dst-alpha (- 1.0 src-alpha)))))
(defun green
(((map 'green src-val 'alpha src-alpha)
(map 'green dst-val 'alpha dst-alpha))
(+ (* src-val src-alpha)
(* dst-val dst-alpha (- 1.0 src-alpha)))))
(defun blue
(((map 'blue src-val 'alpha src-alpha)
(map 'blue dst-val 'alpha dst-alpha))
(+ (* src-val src-alpha)
(* dst-val dst-alpha (- 1.0 src-alpha)))))
Now let's try it out, first compiling it:
lfe> (c "tut7.lfe")
#(module tut7)
lfe> (set colour-1 (tut7:new 0.3 0.4 0.5 1.0))
#M(alpha 1.0 blue 0.5 green 0.4 red 0.3)
lfe> (set colour-2 (tut7:new 1.0 0.8 0.1 0.3))
#M(alpha 0.3 blue 0.1 green 0.8 red 1.0)
lfe> (tut7:blend colour-1 colour-2)
#M(alpha 1.0 blue 0.5 green 0.4 red 0.3)
lfe> (tut7:blend colour-2 colour-1)
#M(alpha 1.0 blue 0.38 green 0.52 red 0.51)
This example warrants some explanation.
First we define a couple macros to help with our guard tests. This is only here for convenience and to reduce syntax cluttering. Guards can be only composed of a limited set of functions, so we needed to use macros that would compile down to just the functions allowed in guards. A full treatment of Lisp macros is beyond the scope of this tutorial, but there is a lot of good material available online for learning macros, including Paul Graham's book "On Lisp."
(defun new
((r g b a) (when (all-channels? r g b a))
(map 'red r 'green g 'blue b 'alpha a)))
The function new/4 1 creates a new map term with and lets the keys red, green, blue and alpha be associated with an initial value. In this case we only allow for float values between and including 0.0 and 1.0 as ensured by the all-channels? and channel? macros.
By calling blend/2 on any colour term created by new/4 we can calculate the resulting colour as determined by the two maps terms.
The first thing blend/2 does is to calculate the resulting alpha channel.
(defun alpha
(((map 'alpha src-alpha) (map 'alpha dst-alpha))
(+ src-alpha (* dst-alpha (- 1.0 src-alpha)))))
We fetch the value associated with key alpha for both arguments using the (map 'alpha <var>) pattern. Any other keys in the map are ignored, only the key alpha is required and checked for.
This is also the case for functions red/2, blue/2 and green/2.
(defun red
(((map 'red src-val 'alpha src-alpha)
(map 'red dst-val 'alpha dst-alpha))
(+ (* src-val src-alpha)
(* dst-val dst-alpha (- 1.0 src-alpha)))))
The difference here is that we check for two keys in each map argument. The other keys are ignored.
Finally we return the resulting colour in blend/3.
(defun blend
((src dst alpha) (when (> alpha 0.0))
(map-update dst
'red (/ (red src dst) alpha)
'green (/ (green src dst) alpha)
'blue (/ (blue src dst) alpha)
'alpha alpha))
We update the dst map with new channel values. The syntax for updating an existing key with a new value is done with map-update form.
Notes
1 Note the use of the slash and number after the function name. We will be discussing this more in a future section, though before we get there you will see this again. Until we get to the full explanation, just know that the number represents the arity of a given function and this helps us be explicit about which function we mean.
The Erlang Standard Library
man Pages and Online Docs
Erlang has a lot of standard modules to help you do things which are directly callable from LFE. For example, the module io contains a lot of functions to help you perform various acts of formatted input/output. Depending upon your Erlang installation, you may have man pages available. From your operating system shell, you can found out by typing erl -man <module name> like so:
erl -man io
If you have man pages installed, that command would give output along these lines:
ERLANG MODULE DEFINITION io(3)
MODULE
io - Standard I/O Server Interface Functions
DESCRIPTION
This module provides an interface to standard Erlang IO
servers. The output functions all return ok if they are suc-
...
If your installation of Erlang doesn't have man pages, you can always find what you're looking for on the documentation web site. Here is the online man page for the io module.
Module and Function Tab-Completion in the REPL
From the LFE REPL, you have some other nice options for standard library discovery. Start up LFE to take a look:
./bin/lfe
Erlang/OTP 23 [erts-11.0.2] [source] [64-bit] [smp:12:12] ...
lfe>
Now, at the prompt, hit your <TAB> key. You should see something like this:
application application_controller application_master
beam_lib binary c
code code_server edlin
edlin_expand epp erl_distribution
erl_eval erl_parse erl_prim_loader
erl_scan erlang error_handler
error_logger error_logger_tty_h erts_internal
ets file file_io_server
file_server filename gb_sets
gb_trees gen gen_event
gen_server global global_group
group heart hipe_unified_loader
inet inet_config inet_db
inet_parse inet_udp init
io io_lib io_lib_format
kernel kernel_config lfe_env
lfe_eval lfe_init lfe_io
lfe_shell lists net_kernel
orddict os otp_ring0
prim_eval prim_file prim_inet
prim_zip proc_lib proplists
ram_file rpc standard_error
supervisor supervisor_bridge sys
unicode user_drv user_sup
zlib
These are all the modules available to you by default in the LFE REPL. Now type (g and hit <TAB>:
lfe> (g
gb_sets gb_trees gen gen_event
gen_server global global_group group
Let's keep going! Continue typing a full module, and then hit <TAB> again:
lfe> (gb_trees:
add/2 add_element/2 balance/1 del_element/2
delete/2 delete_any/2 difference/2 empty/0
filter/2 fold/3 from_list/1 from_ordset/1
insert/2 intersection/1 intersection/2 is_disjoint/2
is_element/2 is_empty/1 is_member/2 is_set/1
is_subset/2 iterator/1 largest/1 module_info/0
module_info/1 new/0 next/1 singleton/1
size/1 smallest/1 subtract/2 take_largest/1
take_smallest/1 to_list/1 union/1 union/2
Now you can see all the functions that are available in the module you have selected. This is a great feature, allowing for easy use as well as exploration and discovery.
Writing Output to a Terminal
It's nice to be able to do formatted output in these example, so the next example shows a simple way to use to use the lfe_io:format function. Of course, just like all other exported functions, you can test the lfe_io:format function in the repl:
lfe> (lfe_io:format "hello world~n" ())
hello world
ok
lfe> (lfe_io:format "this outputs one LFE term: ~w~n" '(hello))
this outputs one LFE term: hello
ok
lfe> (lfe_io:format "this outputs two LFE terms: ~w~w~n" '(hello world))
this outputs two LFE terms: helloworld
ok
lfe> (lfe_io:format "this outputs two LFE terms: ~w ~w~n" '(hello world))
this outputs two LFE terms: hello world
ok
The function format/2 (i.e. format with two arguments) takes two lists. The first one is nearly always a list written as a string between " ". This list is printed out as it stands, except that each ~w is replaced by a term taken in order from the second list. Each ~n is replaced by a new line. The lfe_io:format/2 function itself returns the atom ok if everything goes as planned. Like other functions in LFE, it crashes if an error occurs. This is not a fault in LFE, it is a deliberate policy. LFE has sophisticated mechanisms to handle errors which we will show later. As an exercise, try to make lfe_io:format crash, it shouldn't be difficult. But notice that although lfe_io:format crashes, the Erlang shell itself does not crash.
lfe> (lfe_io:format "this outputs one LFE term: ~w~n" 'hello)
exception error: badarg
in (: lfe_io fwrite1 "this outputs one LFE term: ~w~n" hello)
in (lfe_io format 3)
Example: Converting Temperature
Now for a larger example to consolidate what we have learnt so far. Assume we have a list of temperature readings from a number of cities in the world. Some of them are in Celsius (Centigrade) and some in Fahrenheit (as in the previous list). First let's convert them all to Celsius, then let's print out the data neatly. Save the following code to tut8.lfe:
(defmodule tut8
(export (format-temps 1)))
;; Only this function is exported
(defun format-temps
((())
;; No output for an empty list
'ok)
(((cons city rest))
(print-temp (f->c city))
(format-temps rest)))
(defun f->c
(((tuple name (tuple 'C temp)))
;; No conversion needed
(tuple name (tuple 'C temp)))
(((tuple name (tuple 'F temp)))
;; Do the conversion
(tuple name (tuple 'C (/ (* (- temp 32) 5) 9)))))
(defun print-temp
(((tuple name (tuple 'C temp)))
(lfe_io:format "~-15w ~w C~n" (list name temp))))
lfe> (c "tut8.lfe")
#(module tut8)
lfe> (tut8:format-temps
'(#(Moscow #(C 10))
#(Cape-Town #(F 70))
#(Stockholm #(C -4))
#(Paris #(F 28))
#(London #(F 36)))))
Moscow 10 C
Cape-Town 21.11111111111111 C
Stockholm -4 C
Paris -2.2222222222222223 C
London 2.2222222222222223 C
ok
Before we look at how this program works, notice that we have added a few comments to the code. A comment starts with a ; character and goes on to the end of the line. 1 Note as well that the (export (format-temps 1)) line only includes the function format-temps/1, the other functions are local functions, i.e. they are not visible from outside the module temp-convert.
When we call format-temps/1 the first time, the city variable gets the value #(Moscow #(C-10)) and the remainder of the list is assigned to the rest variable. Next, the f->c/1 function is called inside the print-temp/1 function, with f->c/1 getting passed #(Moscow #(C-10)).
Note that when we see function calls nested like (print-temp (f->c ...)) -- in other words when one function call is passed as the argument to another function -- we execute (evaluate) them from the inside out. We first evaluate (f->c city) which gives the value #(Moscow #(C 10)) as the temperature is already in Celsius and then we evaluate (print-temp #(Moscow #(C 10))). Note that the f->c/1 function works in a similar way to the convert-length/1 function we wrote in a previous section.
Next, print-temp/1 simply calls lfe_io:format/2 in a similar way to what has been described above. Note that ~-15w says to print the "term" with a field length (width) of 15 and left justify it.2
Now we call (format-temps rest) with the remainder of the list as an argument. This way of doing things is similar to the loop constructs in other languages. (Yes, this is recursion, but don't let that worry you). So the same format-temps/1 function is called again, this time city gets the value #(Cape-Town #(F 70)) and we repeat the same procedure as before. We go on doing this until the list becomes empty, i.e. (), which causes the first clause (format-temps '()) to match. This simply "returns" or "results in" the atom ok, so the program ends.
Notes
1 In LFE, the convention is that a comment starting with a single ; is reserved for comments at the end of a line of code; lines which start with a comment use two ;;, as above. There are also conventions for ;;; and ;;;; -- to learn more about these, see Program Development Using LFE - Rules and Conventions and LFE Style Guide.
2 LFE's lfe_io:format differs from the Erlang standard library io:format in that lfe_io displays its results using LFE-formatted data structures in its Lisp syntax; io uses the standard Erlang syntax. You may use either from LFE. lfe_io takes the same formatting parameters as io, so there should be no surprises if you're coming from Erlang. For more information, be sure to read the io:format documentation.
Matching and Guards and Scope of Variables
In the previous section we wrote a little example program for converting temperatures. In creating programs like that with special data structures (in our case, a list of cities and their temperatures), it's often useful to create utility functions which make working with our data more convenient. We will explore that below and use these functions to introduce some new concepts.
A Utility Function
In this case, it could be useful to find the maximum and minimum temperature in our data. We can add support for this by creating the necessary code little bit at a time. Let's start with creating functions for finding the maximum value of the elements of a property list:
(defmodule tut9
(export (list-max 1)))
(defun list-max
(((cons head tail))
(list-max tail head)))
(defun list-max
(('() results)
results)
(((cons head tail) result-so-far) (when (> head result-so-far))
(list-max tail head))
(((cons head tail) result-so-far)
(list-max tail result-so-far)))
Then, in the LFE REPL:
lfe> (c "tut9.lfe")
#(module tut9)
lfe> (tut9:list-max '(1 2 3 4 5 6 7 4 3 2 1))
7
Pattern Matching
Before we talk about pattern matching, let's clarify why we have two different functions with the same name. Note that the two list-max functions above each take a different number of arguments (parameters). Or, another way of saying it: they have different arity. In LFE, functions having the same name but differing in arity are actually different functions. Where we need to distinguish between these functions we write <function name>/<arity>, where <arity> is an integer representing the number of arguments that function takes. For our example above, we would write list-max/1 and list-max/2.
The next thing we should explain is the arguments for the list-max/2 function, since that probably looks pretty strange the first time you see it. If you look closely, you will see that there are three clauses in list-max/2 and each clause stats with the function arguments for that clause; in order, they are:
- an empty list and
results - a
consandresults-so-farwith something called a guard (more on that soon) - a
consandresults-so-farjust by itself
What each of these are doing is what is called pattern matching in LFE: if the arguments passed to list-max/2 match the first pattern, the first clause gets executed and the others don't. If the first one does not match, the second one is tried, and so on.
So what is being "matched" here? Well, the first clause will match if it's first argument is an empty list. The second and third will match if the first element is a list. The second has something more, though: let's take a closer look.
Guards
In the second clause of list-max/2 we see a new form: (when ...) which contains a comparison operation. The special form when is a something we can use with LFE patterns to limit a match. In this case we use it to say: "only use this function clause if head is greater than result-so-far. We call tests of this type a guard. If the guard isn't true (usually referred to as "if the guard fails"), then we try the next part of the function.
Stepping Through the Function
Now that we know how there can be two functions with the same name and how the arguments for list-max/2 are working, let's step through the functions above. They represent an example of walking through a list and "carrying" a value as we do so, in this case result-so-far is the variable that carries a value as we walk. list-max/1 simply assumes that the max value of the list is the head of the list and calls list-max/2 with the rest of the list and the value of the head of the list, in the above this would be (list-max '(2 3 4 5 6 7 4 3 2 1) 1). If we tried to use list-max/1 with an empty list or tried to use it with something which isn't a list at all, we would cause an error. The LFE philosophy is not to handle errors of this type in the function they occur, but to do so elsewhere. More about this later.
In list-max/2 we walk down the list and use head instead of result-so-far when head is greater than result-so-far. In this function, if head isn't greater than result-so-far then it must be smaller or equal to it, and the next clause is executed under this condition.
To change the above program to one which works out the minimum value of the element in a list, all we would need to do is to write < instead of > in the guard ... but it would be wise to change the name of the function to list-min :-).
Scope and let
In a function, the arguments that are passed to it are in scope or "accessible" for all of that function. In another function which has been passed its own arguments, only those are in scope; arguments from other functions are not available for use.
In the case of functions which are pattern matching on the arguments, like our list-max/2, we have three clauses, each with their own arguments and each with their own scope. The parentheses at the beginning of each clause mark this scope. In that case, the results variable is only available in the first clause; it is not in scope for the second two clauses.
But passing arguments to functions and pattern matching function arguments are not the only ways in which you can bind a value to a variable. There is a special form in LFE (and other Lisps) called let. Let's1 take a look, shall we? Here's another way we could have written list-max/2:
(defun list-max
(('() results)
results)
(((cons head tail) result-so-far) (when (> head result-so-far))
(let ((new-result-so-far head))
(list-max tail new-result-so-far)))
(((cons head tail) result-so-far)
(list-max tail result-so-far)))
We only made a change to the second clause: we assigned the value of head to the variable new-result-so-far. This assignment didn't involve any computation or new values, it was essentially just a "renaming" mostly for the sake of demonstration, and arguably to make it more clear the purpose of the value stored in head.
Notes
1 We are not going to apologise for that pun.
More About Lists
The cons Form
Remember how we used cons to "extract" head and tail values of a list when matching them? In the REPL, we would do it like so:
lfe> (set (cons head tail) (list 'Paris 'London 'Rome))
(Paris London Rome)
lfe> head
Paris
lfe> tail
(London Rome)
Well, that's not how cons started life1; it's original use was in "cons"tructing lists, not taking them apart. Here is some classic usage:
lfe> (cons 'Madrid tail)
(Madrid London Rome)
Let's look at a more involved example where we use conses to reverse the order of a list:
(defmodule tut10
(export all))
(defun reverse (list)
(reverse list '()))
(defun reverse
(((cons head tail) reversed-list)
(reverse tail (cons head reversed-list)))
(('() reversed-list)
reversed-list))
Then, in the REPL:
lfe> (c "tut10.lfe")
#(module tut10)
lfe> (tut10:reverse (list 1 2 3))
(3 2 1)
Consider how reversed-list is built: it starts as '(), we then successively take off the heads of the list that was provided and add these heads to the the reversed-list variable, as detailed by the following:
(reverse (cons 1 '(2 3)) '()) => (reverse '(2 3) (cons 1 '()))
(reverse (cons 2 '(3)) '(1)) => (reverse '(3) (cons 2 '(1)))
(reverse (cons 3 '()) (2 1)) => (reverse '() (cons 3 '(2 1)))
(reverse '() '(3 2 1)) => '(3 2 1)
The Erlang module lists contains a lot of functions for manipulating lists, for example for reversing them -- our work above was done for demonstration and pedagogical purposes. For serious applications, one should prefer functions in the Erlang standard library.2
Processing Lists
Now lets get back to the cities and temperatures, but take a more structured approach this time. First let's convert the whole list to Celsius as follows:
(defmodule tut11
(export (format-temps 1)))
(defun format-temps (cities)
(->c cities))
(defun ->c
(((cons (tuple name (tuple 'F temp)) tail))
(let ((converted (tuple name (tuple 'C (/ (* (- temp 32) 5) 9)))))
(cons converted (->c tail))))
(((cons city tail))
(cons city (->c tail)))
(('())
'()))
Now let's test this new function:
lfe> (c "tut11.lfe")
#(module tut11)
lfe> (tut11:format-temps
'(#(Moscow #(C 10))
#(Cape-Town #(F 70))
#(Stockholm #(C -4))
#(Paris #(F 28))
#(London #(F 36)))))
(#(Moscow #(C 10))
#(Cape-Town #(C 21.11111111111111))
#(Stockholm #(C -4))
#(Paris #(C -2.2222222222222223))
#(London #(C 2.2222222222222223)))
Let's look at this, bit-by-bit. In the first function:
(defun format-temps (cities)
(->c cities))
we see that format-temps/1 calls ->c/1. ->c/1 takes off the head of the List cities and converts it to Celsius if needed. The cons function is used to add the (maybe) newly converted city to the converted rest of the list:
(cons converted (->c tail))
or
(cons city (->c tail))
We go on doing this until we get to the end of the list (i.e. the list is empty):
(('())
'())
Now that we have converted the list, we should add a function to print it:
(defmodule tut12
(export (format-temps 1)))
(defun format-temps (cities)
(print-temps (->c cities)))
(defun ->c
(((cons (tuple name (tuple 'F temp)) tail))
(let ((converted (tuple name (tuple 'C (/ (* (- temp 32) 5) 9)))))
(cons converted (->c tail))))
(((cons city tail))
(cons city (->c tail)))
(('())
'()))
(defun print-temps
(((cons (tuple name (tuple 'C temp)) tail))
(io:format "~-15w ~w c~n" (list name temp))
(print-temps tail))
(('())
'ok))
Let's take a look:
lfe> (c "tut12.lfe")
#(module tut12)
lfe> (tut12:format-temps
'(#(Moscow #(C 10))
#(Cape-Town #(F 70))
#(Stockholm #(C -4))
#(Paris #(F 28))
#(London #(F 36)))))
'Moscow' 10 c
'Cape-Town' 21.11111111111111 c
'Stockholm' -4 c
'Paris' -2.2222222222222223 c
'London' 2.2222222222222223 c
ok
Utility Functions Revisited
Remember a few sections back when we created the utility function for finding the maximum value in a list? Let's put that into action now: we want to add a function which finds the cities with the maximum and minimum temperatures:
(defun find-max-min
(((cons city tail))
(find-max-min tail city city)))
(defun find-max-min
(((cons head tail) max-city min-city)
(find-max-min tail
(compare-max head max-city)
(compare-min head min-city)))
(('() max-city min-city)
(tuple max-city min-city)))
(defun compare-max
(((= (tuple name1 (tuple 'C temp1)) city1)
(= (tuple name2 (tuple 'C temp2)) city2))
(if (> temp1 temp2)
city1
city2)))
(defun compare-min
(((= (tuple name1 (tuple 'C temp1)) city1)
(= (tuple name2 (tuple 'C temp2)) city2))
(if (< temp1 temp2)
city1
city2)))
The Complete Example
(defmodule tut13
(export (format-temps 1)))
(defun format-temps (cities)
(let* ((converted (->c cities)))
(print-temps converted)
(print-max-min (find-max-min converted))))
(defun ->c
(((cons (tuple name (tuple 'F temp)) tail))
(let ((converted (tuple name (tuple 'C (/ (* (- temp 32) 5) 9)))))
(cons converted (->c tail))))
(((cons city tail))
(cons city (->c tail)))
(('())
'()))
(defun print-temps
(((cons (tuple name (tuple 'C temp)) tail))
(io:format "~-15w ~w c~n" (list name temp))
(print-temps tail))
(('())
'ok))
(defun find-max-min
(((cons city tail))
(find-max-min tail city city)))
(defun find-max-min
(((cons head tail) max-city min-city)
(find-max-min tail
(compare-max head max-city)
(compare-min head min-city)))
(('() max-city min-city)
(tuple max-city min-city)))
(defun compare-max
(((= (tuple name1 (tuple 'C temp1)) city1)
(= (tuple name2 (tuple 'C temp2)) city2))
(if (> temp1 temp2)
city1
city2)))
(defun compare-min
(((= (tuple name1 (tuple 'C temp1)) city1)
(= (tuple name2 (tuple 'C temp2)) city2))
(if (< temp1 temp2)
city1
city2)))
(defun print-max-min
(((tuple (tuple max-name (tuple 'C max-temp))
(tuple min-name (tuple 'C min-temp))))
(io:format "Max temperature was ~w c in ~w~n" (list max-temp max-name))
(io:format "Min temperature was ~w c in ~w~n" (list min-temp min-name))))
Let's try it out:
lfe> (c "tut13.lfe")
#(module tut13)
lfe> (tut13:format-temps
'(#(Moscow #(C 10))
#(Cape-Town #(F 70))
#(Stockholm #(C -4))
#(Paris #(F 28))
#(London #(F 36)))))
'Moscow' 10 c
'Cape-Town' 21.11111111111111 c
'Stockholm' -4 c
'Paris' -2.2222222222222223 c
'London' 2.2222222222222223 c
Max temperature was 21.11111111111111 c in 'Cape-Town'
Min temperature was -4 c in 'Stockholm'
ok
As you may have noticed, that program isn't the most efficient way of doing this, since we walk through the list of cities four times. But it is better to first strive for clarity and correctness and to make programs efficient only if really needed.
Notes
1 Way back in the prehistoric times when large, building-size computers still roamed the earth and the languages which ran on them were tiny and furry, Lisp came along with only a handful of forms: cons was one of them, and it was used to construct lists, one cell at a time.
2 More information about this lists module is available here.
Conditionals
In the module tut13.lfe, we saw our first conditional, the (if ...) form. We're going to spend the rest of this section discussing if, cond, case, as well as the use of guards and pattern matching to form conditional code branches.
The if Form
In the previous section, we wrote the function find-max-min/3 to work out the maximum and minimum temperature. This work was delegated to two helper functions:
compare-max/2compare-min/2
In both of those functions, we introduced the new if form. If works as follows:
(if <predicate>
<expression1>
<expression2>)
where <expression1> is executed if <predicate> evaluates to true and <expression2> is executed if <predicate> evaluates to false. If you have used other programming languages, then this will be quite familiar to you. If you have not, if should remind you a bit of the logic we looked at when discussing guards.
We can see it in action with the following LFE session in the REPL:
lfe> (if (=:= 1 1) "They are equal!" "They are *not* equal!")
"They are equal!"
lfe> (if (=:= 2 1) "They are equal!" "They are *not* equal!")
"They are *not* equal!"
Or -- you will be more familiar with this -- our code from the last section:
(if (< temp1 temp2)
city1
city2)
where, if temp1 is less than temp2, the value stored in city1 is returned.
So the if form works for two conditions. What about 3? 10? 100? Well, for the situations were we want to check multiple conditions, we'll need the cond form.
The cond Form
(cond (<predicate1> <expression1>)
(<predicate2> <expression2>)
(<predicate3> <expression3>)
...
(<predicaten> <expressionn>))
A given expression is only executed if its accompanying predicate evaluates to true. The cond returns the value of the expression for the first predicate that evaluates to true. Using cond, our temperature test would look like this:
(cond ((< temp1 temp2) city1)
((>= temp1 temp2) city2))
Here's an example which takes advantage of cond supporting more than two logic branches:
(cond ((> x 0) x)
((=:= x 0) 0)
((< x 0) (- x)))
Note that each predicate is an expression with it's own parentheses around it; on its left is the opening parenthesis for that particular branch of the cond.
Often times when using cond one needs a "default" or "fall-through" option to be used when no other condition is met. Since it's the last one, and we need it to evaluate to true we simply set the last condition to true when we need a default. Here's a rather silly example:
(cond ((lists:member x '(1 2 3)) "First three")
((=:= x 4) "Is four")
((>= x 5) "More than four")
('true "You chose poorly"))
Any number that is negative will be caught by the last condition.
In case you're wondering, yes: cond works with patterns as well. Let's take a look.
The Extended cond Form
When we talked about cond above, we only discussed the form as any Lisper would be familiar. However, LFE has extended cond with additional capabilities provided via pattern matching. LFE's cond has the following general form when this is taken into consideration:
(cond (<cond-clause1>)
(<cond-clause2>)
(<cond-clause3>)
...
(<cond-clausen>))
where each <cond-clause> could be either as it is in the regular cond, <predicate> <expression> or it could be (?= <pattern> [<guard>] <expression>) -- the latter being the extended form (with an optional guard). When using the extended form, instead of evaluating a predicate for its Boolean result, the data passed to the cond is matched against the defined patterns: if the pattern match succeeds, then the associated expression is evaluated. Here's an example:
(cond ((?= (cons head '()) x)
"Only one element")
((?= (list 1 2) x)
"Two element list")
((?= (list a _) (when (is_atom a)) x)
"List starts with an atom")
((?= (cons _ (cons a _)) (when (is_tuple a)) x)
"Second element is a tuple")
('true "Anything goes"))
That form is not that often used, but it can be very practical.
The case Form
The case form is useful for situations where you want to check for multiple possible values of the same expression. Without guards, the general form for case is the following:
(case <expression>
(<pattern1> <expression1>)
(<pattern2> <expression2>)
...
(<patternn> <expressionn>))
So we could rewrite the code for the non-extended cond above with the following case:
(case x
((cons head '())
"Only one element")
((list 1 2)
"Two element list")
((list 'a _)
"List starts with 'a'")
(_ "Anything goes"))
The following will happen with the case defined above:
- Any 1-element list will be matched by the first clause.
- A 2-element list of
1and2(in that order) will match the second clause. - Any 2-element list whose first element is the atom
awill match the third clause. - Anything not matching the first three clauses will be matched by the fourth.
With guards, the case has the following general form:
(case <expression>
(<pattern1> [<guard1>] <expression1>)
(<pattern2> [<guard2>] <expression2>)
...
(<patternn> [<guardn>] <expressionn>))
Let's update the previous example with a couple of guards:
(case x
((cons head '())
"Only one element")
((list 1 2)
"Two element list")
((list a _) (when (is_atom a))
"List starts with an atom")
((cons _ (cons a _)) (when (is_tuple a))
"Second element is a tuple")
(_ "Anything goes"))
This changes the logic of the previous example in the following ways:
- Any list whose first element is an atom will match the third clause.
- Any list whose second element is a tuple will match the fourth clause.
- Anything not matching the first four clauses will be matched by the fifth.
Function Heads as Conditionals
Another very common way to express conditional logic in LFE is through the use of pattern matching in function heads. This has the capacity to make code very concise while also remaining clear to read -- thus its prevalent use.
As we've seen, a regular LFE function takes the following form (where the arguments are optional):
(defun <function-name> ([<arg1> ... <argn>])
<body>)
When pattern matching in the function head, the form is as follows:
(defun <function-name>
((<pattern1>) [<guard1>]
<body1>)
((<pattern2>) [<guard2>]
<body2>)
...
((<patternn>) [<guardn>]
<bodyn>))
Note that simple patterns with no expressions are just regular function arguments. In other words <pattern1>, <pattern2>, etc., may be either a full pattern or they may be simple function arguments. The guards are optional.
Let's try this out by rewriting the silly case example above to use a function with pattern-matching in the function heads:
(defun check-val
(((cons head '()))
"Only one element")
(((list 1 2))
"Two element list")
(((list a _)) (when (is_atom a))
"List starts with an atom")
(((cons _ (cons a _))) (when (is_tuple a))
"Second element is a tuple")
((_) "Anything goes"))
If you run that in the REPL, you can test it out with the following:
lfe> (check-val '(1))
"Only one element"
lfe> (check-val '(a 1))
"List starts with an atom"
lfe> (check-val '(1 #(b 2)))
"Second element is a tuple"
lfe> (check-val 42)
"Anything goes"
And there you have LFE function definitions with much of the power of if, cond, and case!
Let's use some of these forms in actual code now ...
Example: Inches and Centimetres
[forthcoming]
[tutorial #14]
Example: Leap Years
[forthcoming]
[tutorial #15]
Built-in Functions
[forthcoming]
Ticket: https://github.com/lfe/tutorial/issues/7
Higher Order Functions
[forthcoming]
Refactor Temperature Conversion Example
[tutorial #16]
[tutorial #17]
Ticket: https://github.com/lfe/tutorial/issues/8
Concurrent Programming
In this chapter we will look at the basics of concurrent programming in LFE.
Processes
One of the main reasons for using LFE/Erlang instead of other functional languages is Erlang/LFE's ability to handle concurrency and distributed programming. By concurrency we mean programs which can handle several threads of execution at the same time. For example, modern operating systems would allow you to use a word processor, a spreadsheet, a mail client and a print job all running at the same time. Of course each processor (CPU) in the system is probably only handling one thread (or job) at a time, but it swaps between the jobs at such a rate that it gives the illusion of running them all at the same time. It is easy to create parallel threads of execution in an LFE program and it is easy to allow these threads to communicate with each other. In LFE we call each thread of execution a process.
(Aside: the term "process" is usually used when the threads of execution share no data with each other and the term "thread" when they share data in some way. Threads of execution in LFE share no data, that's why we call them processes).
The LFE BIF spawn is used to create a new process: (spawn module exported-function list-of-arguments). Consider the following module:
(defmodule tut18
(export (start 0) (say-something 2)))
(defun say-something
([what 0] 'done)
([what times]
(lfe_io:format "~p~n" (list what))
(say-something what (- times 1))))
(defun start ()
(spawn 'tut18 'say-something '(hello 3))
(spawn 'tut18 'say-something '(goodbye 3)))
lfe> (c "tut18.lfe")
#(module tut18)
lfe> (tut18:say-something 'hello 3)
hello
hello
hello
done
We can see that function say-something writes its first argument the number of times specified by second argument. Now look at the function start. It starts two LFE processes, one which writes "hello" three times and one which writes "goodbye" three times. Both of these processes use the function say-something. Note that a function used in this way by spawn to start a process must be exported from the module (i.e. in the (export ... ) at the start of the module).
lfe> (tut18:start)
<0.37.0>
hello
goodbye
lfe> hello
goodbye
hello
goodbye
Notice that it didn't write "hello" three times and then "goodbye" three times, but the first process wrote a "hello", the second a "goodbye", the first another "hello" and so forth. But where did the <0.37.0> come from? The return value of a function is of course the return value of the last "thing" in the function. The last thing in the function start is
(spawn 'tut18 'say-something '(goodbye 3))
spawn returns a process identifier, or pid, which uniquely identifies the process. So <0.37.0> is the pid of the spawn function call above. We will see how to use pids in the next example.
Note as well that we have used ~p instead of ~w in lfe_io:format. To quote the manual: "~p Writes the data with standard syntax in the same way as ~w, but breaks terms whose printed representation is longer than one line into many lines and indents each line sensibly. It also tries to detect lists of printable characters and to output these as strings".
Message Passing
In the following example we create two processes which send messages to each other a number of times.
(defmodule tut19
(export (start 0) (ping 2) (pong 0)))
(defun ping
((0 pong-pid)
(! pong-pid 'finished)
(lfe_io:format "Ping finished~n" ()))
((n pong-pid)
(! pong-pid (tuple 'ping (self)))
(receive
('pong (lfe_io:format "Ping received pong~n" ())))
(ping (- n 1) pong-pid)))
(defun pong ()
(receive
('finished
(lfe_io:format "Pong finished~n" ()))
((tuple 'ping ping-pid)
(lfe_io:format "Pong received ping~n" ())
(! ping-pid 'pong)
(pong))))
(defun start ()
(let ((pong-pid (spawn 'tut19 'pong ())))
(spawn 'tut19 'ping (list 3 pong-pid))))
lfe> (c "tut19.lfe")
#(module tut19)
lfe> (tut19:start)
<0.36.0>
lfe> Pong received ping
Ping received pong
Pong received ping
Ping received pong
Pong received ping
Ping received pong
Ping finished
Pong finished
The function start first creates a process, let's call it "pong":
(let ((pong-pid (spawn 'tut19 'pong ())))
This process executes (tut19:pong). pong-pid is the process identity of the "pong" process. The function start now creates another process "ping".
(spawn 'tut19 'ping (list 3 pong-pid))))
this process executes:
(tut19:ping (list 3 pong-pid))
<0.36.0> is the return value from the start function.
The process "pong" now does:
(receive
('finished
(lfe_io:format "Pong finished~n" ()))
((tuple 'ping ping-pid)
(lfe_io:format "Pong received ping~n" ())
(! ping-pid 'pong)
(pong)))
The receive construct is used to allow processes to wait for messages from other processes. It has the format:
(receive
(pattern1
actions1)
(pattern2
actions2)
....
(patternN
actionsN))
Messages between LFE processes are simply valid LFE terms. I.e. they can be lists, tuples, integers, atoms, pids etc.
Each process has its own input queue for messages it receives. New messages received are put at the end of the queue. When a process executes a receive, the first message in the queue is matched against the first pattern in the receive, if this matches, the message is removed from the queue and the actions corresponding to the the pattern are executed.
However, if the first pattern does not match, the second pattern is tested, if this matches the message is removed from the queue and the actions corresponding to the second pattern are executed. If the second pattern does not match the third is tried and so on until there are no more pattern to test. If there are no more patterns to test, the first message is kept in the queue and we try the second message instead. If this matches any pattern, the appropriate actions are executed and the second message is removed from the queue (keeping the first message and any other messages in the queue). If the second message does not match we try the third message and so on until we reach the end of the queue. If we reach the end of the queue, the process blocks (stops execution) and waits until a new message is received and this procedure is repeated.
Of course the LFE implementation is "clever" and minimises the number of times each message is tested against the patterns in each receive.
Now back to the ping pong example.
"Pong" is waiting for messages. If the atom finished is received, "pong" writes "Pong finished" to the output and as it has nothing more to do, terminates. If, however, it receives a message with the format:
#(ping ping-pid)
it writes "Pong received ping" to the output and sends the atom pong to the process "ping":
(! ping-pid 'pong)
Note how "!" is used to send messages. The syntax of "!" is:
(! pid message)
I.e. message (any LFE term) is sent to the process with identity pid.
After sending the message pong, to the process "ping", "pong" calls the pong function again, which causes it to get back to the receive again and wait for another message. Now let's look at the process "ping". Recall that it was started by executing:
(tut19:ping 3 pong-pid)
Looking at the function ping/2 we see that the second clause of ping/2 is executed since the value of the first argument is 3 (not 0) (first clause head is (0 pong-pid), second clause head is (n pong-pid), so n becomes 3).
The second clause sends a message to "pong":
(! pong-pid (tuple 'ping (self)))
(self) returns the pid of the process which executes (self), in this case the pid of "ping". (Recall the code for "pong", this will land up in the variable ping-pid in the receive previously explained).
"Ping" now waits for a reply from "pong":
(receive
('pong (lfe_io:format "Ping received pong~n" ())))
and writes "Ping received pong" when this reply arrives, after which "ping" calls the ping function again.
(ping (- n 1) pong-pid)
(- n 1) causes the first argument to be decremented until it becomes 0. When this occurs, the first clause of ping/2 will be executed:
(defun ping
((0 pong-pid)
(! pong-pid 'finished)
(lfe_io:format "Ping finished~n" ()))
The atom finished is sent to "pong" (causing it to terminate as described above) and "Ping finished" is written to the output. "Ping" then itself terminates as it has nothing left to do.
Registered Process Names
In the above example, we first created "pong" so as to be able to give the identity of "pong" when we started "ping". I.e. in some way "ping" must be able to know the identity of "pong" in order to be able to send a message to it. Sometimes processes which need to know each others identities are started completely independently of each other. Erlang thus provides a mechanism for processes to be given names so that these names can be used as identities instead of pids. This is done by using the register BIF:
(register some-atom pid)
We will now re-write the ping pong example using this and giving the name pong to the "pong" process:
(defmodule tut20
(export (start 0) (ping 1) (pong 0)))
(defun ping
((0)
(! 'pong 'finished)
(lfe_io:format "Ping finished~n" ()))
((n)
(! 'pong (tuple 'ping (self)))
(receive
('pong (lfe_io:format "Ping received pong~n" ())))
(ping (- n 1))))
(defun pong ()
(receive
('finished
(lfe_io:format "Pong finished~n" ()))
((tuple 'ping ping-pid)
(lfe_io:format "Pong received ping~n" ())
(! ping-pid 'pong)
(pong))))
(defun start ()
(let ((pong-pid (spawn 'tut20 'pong ())))
(register 'pong pong-pid)
(spawn 'tut20 'ping '(3))))
lfe> (c "tut20")
#(module tut20)
lfe> (tut20:start)
<0.36.0>
lfe> Pong received ping
Ping received pong
Pong received ping
Ping received pong
Pong received ping
Ping received pong
Ping finished
Pong finished
In the start/0 function,
(register 'pong pong-pid)
registers the "pong" process and gives it the name pong. In the "ping" process we can now send messages to pong by:
(! 'pong (tuple 'ping (self)))
so that ping/2 now becomes ping/1 as we don't have to use the argument pong-pid.
Distributed Programming
Now let's re-write the ping pong program with "ping" and "pong" on different computers. Before we do this, there are a few things we need to set up to get this to work. The distributed LFE/Erlang implementation provides a basic security mechanism to prevent unauthorised access to an Erlang system on another computer (manual). Erlang systems which talk to each other must have the same magic cookie. The easiest way to achieve this is by having a file called .erlang.cookie in your home directory on all machines which on which you are going to run Erlang systems communicating with each other (on Windows systems the home directory is the directory where pointed to by the $HOME environment variable - you may need to set this. On Linux or Unix you can safely ignore this and simply create a file called .erlang.cookie in the directory you get to after executing the command cd without any argument). The .erlang.cookie files should contain one line with the same atom. For example on Linux or Unix in the OS shell:
cd
cat > .erlang.cookie
Which shows this:
this_is_very_secret
Let's make it accessible only by the owner of the file:
chmod 400 .erlang.cookie
Note that this permission change is a requirement.
When you start an LFE/Erlang system which is going to talk to other LFE/Erlang systems, you must give it a name, e.g.:
lfe -sname my-name
We will see more details of this later (manual). If you want to experiment with distributed Erlang, but you only have one computer to work on, you can start two separate Erlang systems on the same computer but give them different names. Each Erlang system running on a computer is called an Erlang node.
(Note: erl -sname assumes that all nodes are in the same IP domain and we can use only the first component of the IP address, if we want to use nodes in different domains we use -name instead, but then all IP address must be given in full (manual).
Here is the ping pong example modified to run on two separate nodes:
(defmodule tut21
(export (start-ping 1) (start-pong 0) (ping 2) (pong 0)))
(defun ping
((0 pong-node)
(! (tuple 'pong pong-node) 'finished)
(lfe_io:format "Ping finished~n" ()))
((n pong-node)
(! (tuple 'pong pong-node) (tuple 'ping (self)))
(receive
('pong (lfe_io:format "Ping received pong~n" ())))
(ping (- n 1) pong-node)))
(defun pong ()
(receive
('finished
(lfe_io:format "Pong finished~n" ()))
((tuple 'ping ping-pid)
(lfe_io:format "Pong received ping~n" ())
(! ping-pid 'pong)
(pong))))
(defun start-pong ()
(register 'pong (spawn 'tut21 'pong ())))
(defun start-ping (pong-node)
(spawn 'tut21 'ping (list 3 pong-node)))
Let us assume we have two computers called gollum and kosken. We will start a node on kosken called ping and then a node on gollum called pong.
On kosken (on a Linux/Unix system):
lfe -sname ping
Erlang/OTP 17 [erts-6.0] [source-07b8f44] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
LFE Shell V6.0 (abort with ^G)
(ping@kosken)lfe>
On gollum:
lfe -sname pong
Erlang/OTP 17 [erts-6.0] [source-07b8f44] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
LFE Shell V6.0 (abort with ^G)
(pong@gollum)lfe>
Now start the "pong" process on gollum:
(pong@gollum)lfe> (tut21:start-pong)
true
and start the "ping" process on kosken (from the code above you will see that a parameter of the start-ping function is the node name of the Erlang system where "pong" is running):
(ping@kosken)lfe> (tut21:start-ping 'pong@gollum)
<0.41.0>
(ping@kosken)lfe> Ping received pong
Ping received pong
Ping received pong
Ping finished
Here we see that the ping pong program has run, on the "pong" side we see:
(pong@gollum)lfe>
Pong received ping
Pong received ping
Pong received ping
Pong finished
Looking at the tut21 code we see that the pong function itself is unchanged, the lines:
((tuple 'ping ping-pid)
(lfe_io:format "Pong received ping~n" ())
(! ping-pid 'pong)
work in the same way irrespective of on which node the "ping" process is executing. Thus Erlang pids contain information about where the process executes so if you know the pid of a process, the "!" operator can be used to send it a message if the process is on the same node or on a different node.
A difference is how we send messages to a registered process on another node:
(! (tuple 'pong pong-node) (tuple 'ping (self)))
We use a tuple #(registered-name node-name) instead of just the registered-name.
In the previous example, we started "ping" and "pong" from the shells of two separate Erlang nodes. spawn can also be used to start processes in other nodes. The next example is the ping pong program, yet again, but this time we will start "ping" in another node:
(defmodule tut22
(export (start 1) (ping 2) (pong 0)))
(defun ping
((0 pong-node)
(! (tuple 'pong pong-node) 'finished)
(lfe_io:format "Ping finished~n" ()))
((n pong-node)
(! (tuple 'pong pong-node) (tuple 'ping (self)))
(receive
('pong (lfe_io:format "Ping received pong~n" ())))
(ping (- n 1) pong-node)))
(defun pong ()
(receive
('finished
(lfe_io:format "Pong finished~n" ()))
((tuple 'ping ping-pid)
(lfe_io:format "Pong received ping~n" ())
(! ping-pid 'pong)
(pong))))
(defun start (ping-node)
(register 'pong (spawn 'tut22 'pong ()))
(spawn ping-node 'tut22 'ping (list 3 (node))))
The function node/0 returns the name of the current node.
Assuming an LFE system called ping (but not the "ping" process) has already been started on kosken, then on gollum we do:
(pong@gollum)lfe> (tut22:start 'ping@kosken)
<8524.50.0>
(pong@gollum)lfe> Pong received ping
Ping received pong
Pong received ping
Ping received pong
Pong received ping
Ping received pong
Ping finished
Pong finished
Notice we get all the output on gollum. This is because the io system finds out where the process is spawned from and sends all output there.
Although we do not show here there are no problems with running nodes distributed nodes where some run Erlang and some run LFE. The handling of io will also work the same across mixed LFE and Erlang nodes.
Example: Messenger
Now for a larger example. We will make an extremely simple "messenger". The messenger is a program which allows users to log in on different nodes and send simple messages to each other.
Before we start, let's note the following:
-
This example will just show the message passing logic no attempt at all has been made to provide a nice graphical user interface - this can of course also be done in LFE - but that's another tutorial.
-
This sort of problem can be solved more easily if you use the facilities in OTP, which will also provide methods for updating code on the fly etc. But again, that's another tutorial.
-
The first program we write will contain some inadequacies as regards handling of nodes which disappear, we will correct these in a later version of the program.
We will set up the messenger by allowing "clients" to connect to a central server and say who and where they are. I.e. a user won't need to know the name of the Erlang node where another user is located to send a message.
File messenger.lfe:
;;; Message passing utility.
;;; User interface:
;;; (logon name)
;;; One user at a time can log in from each Erlang node in the
;;; system messenger: and choose a suitable name. If the name
;;; is already logged in at another node or if someone else is
;;; already logged in at the same node, login will be rejected
;;; with a suitable error message.
;;; (logoff)
;;; Logs off anybody at at node
;;; (message to-name message)
;;; sends message to to-name. Error messages if the user of this
;;; function is not logged on or if to-name is not logged on at
;;; any node.
;;;
;;; One node in the network of Erlang nodes runs a server which maintains
;;; data about the logged on users. The server is registered as "messenger"
;;; Each node where there is a user logged on runs a client process registered
;;; as "mess-client"
;;;
;;; Protocol between the client processes and the server
;;; ----------------------------------------------------
;;;
;;; To server: (tuple client-pid 'logon user-name)
;;; Reply #(messenger stop user-exists-at-other-node) stops the client
;;; Reply #(messenger logged-on) logon was successful
;;;
;;; To server: (tuple client-pid 'logoff)
;;; Reply: #(messenger logged-off)
;;;
;;; To server: (tuple client-pid 'logoff)
;;; Reply: no reply
;;;
;;; To server: (tuple client-pid 'message-to to-name message) send a message
;;; Reply: #(messenger stop you-are-not-logged-on) stops the client
;;; Reply: #(messenger receiver-not-found) no user with this name logged on
;;; Reply: #(messenger sent) message has been sent (but no guarantee)
;;;
;;; To client: (tuple 'message-from name message)
;;;
;;; Protocol between the "commands" and the client
;;; ----------------------------------------------
;;;
;;; Started: (messenger:client server-node name)
;;; To client: logoff
;;; To client: (tuple 'message-to to-name message)
;;;
;;; Configuration: change the server-node() function to return the
;;; name of the node where the messenger server runs
(defmodule messenger
(export (start-server 0) (server 1) (logon 1) (logoff 0)
(message 2) (client 2)))
;;; Change the function below to return the name of the node where the
;;; messenger server runs
(defun server-node () 'messenger@renat)
;;; This is the server process for the "messenger"
;;; the user list has the format [{ClientPid1, Name1},{ClientPid22, Name2},...]
(defun server (user-list)
(receive
((tuple from 'logon name)
(let ((new-user-list (server-logon from name user-list)))
(server new-user-list)))
((tuple from 'logoff)
(let ((new-user-list (server-logoff from user-list)))
(server new-user-list)))
((tuple from 'message-to to message)
(server-transfer from to message user-list)
;;(lfe_io:format "list is now: ~p~n" (list user-list))
(server user-list))))
;;; Start the server
(defun start-server ()
(register 'messenger (spawn 'messenger 'server '(()))))
;;; Server adds a new user to the user list
(defun server-logon (from name user-list)
;; Check if logged on anywhere else
(if (lists:keymember name 2 user-list)
(progn ;Reject logon
(! from #(messenger stop user-exists-at-other-node))
user-list)
(progn ;Add user to the list
(! from #(messenger logged-on))
(cons (tuple from name) user-list))))
;;; Server deletes a user from the user list
(defun server-logoff (pid user-list)
(lists:keydelete pid 1 user-list))
;;; Server transfers a message between user
(defun server-transfer (from-pid to-name message user-list)
;; Check that the user is logged on and who he is
(case (lists:keyfind from-pid 1 user-list)
((tuple from-pid from-name)
(server-transfer from-pid from-name to-name message user-list))
('false
(! from-pid #(messenger stop you-are-not-logged-on)))))
;;; If the user exists, send the message
(defun server-transfer (from-pid from-name to-name message user-list)
;; Find the receiver and send the message
(case (lists:keyfind to-name 2 user-list)
((tuple to-pid to-name)
(! to-pid (tuple 'message-from from-name message))
(! from-pid #(messenger sent)))
('false
(! from-pid #(messenger receiver-not-found)))))
;;; User Commands
(defun logon (name)
(case (whereis 'mess-client)
('undefined
(let ((client (spawn 'messenger 'client (list (server-node) name))))
(register 'mess-client client)))
(_ 'already-logged-on)))
(defun logoff ()
(! 'mess-client 'logoff))
(defun message (to-name message)
(case (whereis 'mess-client) ;Test if the client is running
('undefined
'not-logged-on)
(_ (! 'mess-client (tuple 'message-to to-name message))
'ok)))
;;; The client process which runs on each server node
(defun client (server-node name)
(! (tuple 'messenger server-node) (tuple (self) 'logon name))
(await-result)
(client server-node))
(defun client (server-node)
(receive
('logoff
(! (tuple 'messenger server-node) (tuple (self) 'logoff))
(exit 'normal))
((tuple 'message-to to-name message)
(! (tuple 'messenger server-node)
(tuple (self) 'message-to to-name message))
(await-result))
((tuple 'message-from from-name message)
(lfe_io:format "Message from ~p: ~p~n" (list from-name message))))
(client server-node))
;;; Wait for a response from the server
(defun await-result ()
(receive
((tuple 'messenger 'stop why) ;Stop the client
(lfe_io:format "~p~n" (list why))
(exit 'normal))
((tuple 'messenger what) ;Normal response
(lfe_io:format "~p~n" (list what)))))
To use this program you need to:
- configure the server_node() function
- copy the compiled code (
essenger.beamto the directory on each computer where you start Erlang.
In the following example of use of this program, I have started nodes on four different computers, but if you don't have that many machines available on your network, you could start up several nodes on the same machine.
We start up four Erlang nodes, messenger@super, c1@bilbo, c2@kosken, c3@gollum.
First we start up a the server at messenger@super:
(messenger@super)lfe> (messenger:start-server)
true
Now Peter logs on at c1@bilbo:
(c1@bilbo)lfe> (messenger:logon 'peter)
true
logged-on
James logs on at c2@kosken:
(c2@kosken)lfe> (messenger:logon 'james)
true
logged-on
and Fred logs on at c3@gollum:
(c3@gollum)lfe> (messenger:logon 'fred)
true
logged-on
Now Peter sends Fred a message:
(c1@bilbo)lfe> (messenger:message 'fred "hello")
ok
sent
And Fred receives the message and sends a message to Peter and logs off:
Message from peter: "hello"
(c3@gollum)lfe> (messenger:message 'peter "go away, I'm busy")
ok
sent
(c3@gollum)lfe> (messenger:logoff)
logoff
James now tries to send a message to Fred:
(c2@kosken)lfe> (messenger:message 'fred "peter doesn't like you")
ok
receiver-not-found
But this fails as Fred has already logged off.
First let's look at some of the new concepts we have introduced.
There are two versions of the server-transfer function, one with four arguments (server-transfer/4) and one with five (server_transfer/5). These are regarded by LFE as two separate functions.
Note how we write the server function so that it calls itself, (server user-list) and thus creates a loop. The Erlang-LFE compiler is "clever" and optimises the code so that this really is a sort of loop and not a proper function call. But this only works if there is no code after the call, otherwise the compiler will expect the call to return and make a proper function call. This would result in the process getting bigger and bigger for every loop.
We use functions in the lists module. This is a very useful module and a study of the manual page is recommended (erl -man lists). (lists:keymember key position lists) looks through a list of tuples and looks at position in each tuple to see if it is the same as key. The first element is position 1. If it finds a tuple where the element at position is the same as key, it returns true, otherwise false.
> (lists:keymember 'a 2 '(#(x y z) #(b b b) #(b a c) #(q r s)))
true
> (lists:keymember 'p 2 '(#(x y z) #(b b b) #(b a c) #(q r s)))
false
lists:keydelete works in the same way but deletes the first tuple found (if any) and returns the remaining list:
> (lists:keymember 'a 2 '(#(x y z) #(b b b) #(b a c) #(q r s)))
(#(x y z) #(b b b) #(q r s))
lists:keyfind is like lists:keymember, but it returns the tuple found or the atom false:
> (lists:keyfind 'a 2 '(#(x y z) #(b b b) #(b a c) #(q r s)))
#(b a c)
> (lists:keyfind 'p 2 '(#(x y z) #(b b b) #(b a c) #(q r s)))
false
There are a lot more very useful functions in the lists module.
An LFE process will (conceptually) run until it does a receive and there is no message which it wants to receive in the message queue. I say "conceptually" because the LFE system shares the CPU time between the active processes in the system.
A process terminates when there is nothing more for it to do, i.e. the last function it calls simply returns and doesn't call another function. Another way for a process to terminate is for it to call exit/1. The argument to exit/1 has a special meaning which we will look at later. In this example we will do (exit 'normal) which has the same effect as a process running out of functions to call.
The BIF (whereis registered-name) checks if a registered process of name registered-name exists and return the pid of the process if it does exist or the atom undefined if it does not.
You should by now be able to understand most of the code above so I'll just go through one case: a message is sent from one user to another.
The first user "sends" the message in the example above by:
(messenger:message 'fred "hello")
After testing that the client process exists:
(whereis 'mess-client)
and a message is sent to mess-client:
(! 'mess-client #(message-to fred "hello"))
The client sends the message to the server by:
(! #(messenger messenger@renat) (tuple (self) 'message-to 'fred "hello"))
and waits for a reply from the server.
The server receives this message and calls:
(server-transfer from 'fred "hello" user-list)
which checks that the pid from is in the user-list:
(lists:keyfind from 1 user-list)
If keyfind returns the atom false, some sort of error has occurred and the server sends back the message:
(! from-pid #(messenger stop you-are-not-logged-on))
which is received by the client which in turn does (exit 'normal) and terminates. If keyfind returns (tuple from name) we know that the user is logged on and is his name (peter) is in variable name. We now call:
(server-transfer from 'peter 'fred "hello" user-list)
Note that as this is server-transfer/5 it is not the same as the previous function server_transfer/4. We do another keyfind on user-list to find the pid of the client corresponding to fred:
(lists:keyfind 'fred 2 user-list)
This time we use argument 2 which is the second element in the tuple. If this returns the atom false we know that fred is not logged on and we send the message:
(! from-pid #(messenger receiver-not-found)
which is received by the client, if keyfind returns:
(tuple to-pid 'fred)
we send the message:
(! to-pid #(message-from peter "hello"))
to Fred's client and the message:
(! from-pid #(messenger sent))
to peter's client.
Fred's client receives the message and prints it:
((tuple 'message-from from-name message)
(lfe_io:format "Message from ~p: ~p~n" (list from-name message))))
and Peter's client receives the message in the await-result function.
Robustness
There are several things which are wrong with the messenger example from the previous chapter. For example if a node where a user is logged on goes down without doing a log off, the user will remain in the server's user-list but the client will disappear thus making it impossible for the user to log on again as the server thinks the user already logged on.
Or what happens if the server goes down in the middle of sending a message leaving the sending client hanging for ever in the await-result function?
Timeouts
Before improving the messenger program we will look into some general principles, using the ping pong program as an example. Recall that when "ping" finishes, it tells "pong" that it has done so by sending the atom finished as a message to "pong" so that "pong" could also finish. Another way to let "pong" finish, is to make "pong" exit if it does not receive a message from ping within a certain time, this can be done by adding a timeout to pong as shown in the following example:
(defmodule tut19
(export (start-ping 1) (start-pong 0) (ping 2) (pong 0)))
(defun ping
((0 pong-node)
(lfe_io:format "Ping finished~n" ()))
((n pong-node)
(! (tuple 'pong pong-node) (tuple 'ping (self)))
(receive
('pong (lfe_io:format "Ping received pong~n" ())))
(ping (- n 1) pong-node)))
(defun pong ()
(receive
((tuple 'ping ping-pid)
(lfe_io:format "Pong received ping~n" ())
(! ping-pid 'pong)
(pong))
(after 5000
(lfe_io:format "Pong timed out~n" ()))))
(defun start-pong ()
(register 'pong (spawn 'tut19 'pong ())))
(defun start-ping (pong-node)
(spawn 'tut19 'ping (list 3 pong-node)))
After we have compiled this and copied the tut19.beam file to the necessary directories:
On (pong@kosken):
(pong@kosken)lfe> (tut19:start-pong)
true
Pong received ping
Pong received ping
Pong received ping
Pong timed out
On (ping@gollum):
(ping@renat)lfe> (tut19:start-ping 'pong@kosken)
<0.40.0>
Ping received pong
Ping received pong
Ping received pong
Ping finished
The timeout is set in:
(defun pong ()
(receive
((tuple 'ping ping-pid)
(lfe_io:format "Pong received ping~n" ())
(! ping-pid 'pong)
(pong))
(after 5000
(lfe_io:format "Pong timed out~n" ()))))
We start the timeout (after 5000) when we enter receive. The timeout is cancelled if #(ping ping-pid) is received. If #(ping ping-pid) is not received, the actions following the timeout will be done after 5000 milliseconds. after must be last in the receive, i.e. preceded by all other message reception specifications in the receive. Of course we could also call a function which returned an integer for the timeout:
(after (pong-timeout)
In general, there are better ways than using timeouts to supervise parts of a distributed Erlang system. Timeouts are usually appropriate to supervise external events, for example if you have expected a message from some external system within a specified time. For example, we could use a timeout to log a user out of the messenger system if they have not accessed it, for example, in ten minutes.
Error Handling
Before we go into details of the supervision and error handling in an LFE system, we need see how LFE processes terminate, or in LFE terminology, exit.
A process which executes (exit 'normal) or simply runs out of things to do has a normal exit.
A process which encounters a runtime error (e.g. divide by zero, bad match, trying to call a function which doesn't exist etc) exits with an error, i.e. has an abnormal exit. A process which executes (exit reason) where reason is any LFE term except the atom normal, also has an abnormal exit.
An LFE process can set up links to other LFE processes. If a process calls (link other-pid) it sets up a bidirectional link between itself and the process called other-pid. When a process terminates, it sends something called a signal to all the processes it has links to.
The signal carries information about the pid it was sent from and the exit reason.
The default behaviour of a process which receives a normal exit is to ignore the signal.
The default behaviour in the two other cases (i.e. abnormal exit) above is to bypass all messages to the receiving process and to kill it and to propagate the same error signal to the killed process' links. In this way you can connect all processes in a transaction together using links and if one of the processes exits abnormally, all the processes in the transaction will be killed. As we often want to create a process and link to it at the same time, there is a special BIF, spawn_link which does the same as spawn, but also creates a link to the spawned process.
Now an example of the ping pong example using links to terminate "pong":
(defmodule tut20
(export (start 1) (ping 2) (pong 0)))
(defun ping (n pong-pid)
(link pong-pid)
(ping1 n pong-pid))
(defun ping1
((0 pong-pid)
(exit 'ping))
((n pong-pid)
(! pong-pid (tuple 'ping (self)))
(receive
('pong (lfe_io:format "Ping received pong~n" ())))
(ping1 (- n 1) pong-pid)))
(defun pong ()
(receive
((tuple 'ping ping-pid)
(lfe_io:format "Pong received ping~n" ())
(! ping-pid 'pong)
(pong))))
(defun start (ping-node)
(let ((pong-pid (spawn 'tut20 'pong ())))
(spawn ping-node 'tut20 'ping (list 3 pong-pid))))
(s1@bill)lfe> (tut20:start 's2@kosken)
Pong received ping
<5627.43.0>
Ping received pong
Pong received ping
Ping received pong
Pong received ping
Ping received pong
This is a slight modification of the ping pong program where both processes are spawned from the same start/1 function, where the "ping" process can be spawned on a separate node. Note the use of the link BIF. "Ping" calls (exit `ping) when it finishes and this will cause an exit signal to be sent to "pong" which will also terminate.
It is possible to modify the default behaviour of a process so that it does not get killed when it receives abnormal exit signals, but all signals will be turned into normal messages on the format #(EXIT from-pid reason) and added to the end of the receiving processes message queue. This behaviour is set by:
(process_flag 'trap_exit 'true)
There are several other process flags, see erlang manual. Changing the default behaviour of a process in this way is usually not done in standard user programs, but is left to the supervisory programs in OTP (but that's another tutorial). However we will modify the ping pong program to illustrate exit trapping.
(defmodule tut21
(export (start 1) (ping 2) (pong 0)))
(defun ping (n pong-pid)
(link pong-pid)
(ping1 n pong-pid))
(defun ping1
((0 pong-pid)
(exit 'ping))
((n pong-pid)
(! pong-pid (tuple 'ping (self)))
(receive
('pong (lfe_io:format "Ping received pong~n" ())))
(ping1 (- n 1) pong-pid)))
(defun pong ()
(process_flag 'trap_exit 'true)
(pong1))
(defun pong1 ()
(receive
((tuple 'ping ping-pid)
(lfe_io:format "Pong received ping~n" ())
(! ping-pid 'pong)
(pong1))
((tuple 'EXIT from reason)
(lfe_io:format "Pong exiting, got ~p~n"
(list (tuple 'EXIT from reason))))))
(defun start (ping-node)
(let ((pong-pid (spawn 'tut21 'pong ())))
(spawn ping-node 'tut21 'ping (list 3 pong-pid))))
(s1@bill)lfe> (tut21:start 's2@kosken)
<5627.44.0>
Pong received ping
Ping received pong
Pong received ping
Ping received pong
Pong received ping
Ping received pong
Pong exiting, got #(EXIT <5627.44.0> ping)
Example: Robust Messenger
Now we return to the messenger program and add changes which make it more robust:
;;; Message passing utility.
;;; User interface:
;;; (logon name)
;;; One user at a time can log in from each Erlang node in the
;;; system messenger: and choose a suitable name. If the name
;;; is already logged in at another node or if someone else is
;;; already logged in at the same node, login will be rejected
;;; with a suitable error message.
;;; (logoff)
;;; Logs off anybody at at node
;;; (message to-name message)
;;; sends message to to-name. Error messages if the user of this
;;; function is not logged on or if to-name is not logged on at
;;; any node.
;;;
;;; One node in the network of Erlang nodes runs a server which maintains
;;; data about the logged on users. The server is registered as "messenger"
;;; Each node where there is a user logged on runs a client process registered
;;; as "mess-client"
;;;
;;; Protocol between the client processes and the server
;;; ----------------------------------------------------
;;;
;;; To server: (tuple client-pid 'logon user-name)
;;; Reply #(messenger stop user-exists-at-other-node) stops the client
;;; Reply #(messenger logged-on) logon was successful
;;;
;;; When the client terminates for some reason
;;; To server: (tuple 'EXIT client-pid reason)
;;;
;;; To server: (tuple client-pid 'message-to to-name message) send a message
;;; Reply: #(messenger stop you-are-not-logged-on) stops the client
;;; Reply: #(messenger receiver-not-found) no user with this name logged on
;;; Reply: #(messenger sent) message has been sent (but no guarantee)
;;;
;;; To client: (tuple 'message-from name message)
;;;
;;; Protocol between the "commands" and the client
;;; ----------------------------------------------
;;;
;;; Started: (messenger:client server-node name)
;;; To client: logoff
;;; To client: (tuple 'message-to to-name message)
;;;
;;; Configuration: change the server-node() function to return the
;;; name of the node where the messenger server runs
(defmodule messenger
(export (start-server 0) (server 0)
(logon 1) (logoff 0) (message 2) (client 2)))
;;; Change the function below to return the name of the node where the
;;; messenger server runs
(defun server-node () 'messenger@renat)
;;; This is the server process for the "messenger"
;;; the user list has the format [{ClientPid1, Name1},{ClientPid22, Name2},...]
(defun server ()
(process_flag 'trap_exit 'true)
(server ()))
(defun server (user-list)
(receive
((tuple from 'logon name)
(let ((new-user-list (server-logon from name user-list)))
(server new-user-list)))
((tuple 'EXIT from _)
(let ((new-user-list (server-logoff from user-list)))
(server new-user-list)))
((tuple from 'message-to to message)
(server-transfer from to message user-list)
(lfe_io:format "list is now: ~p~n" (list user-list))
(server user-list))))
;;; Start the server
(defun start-server ()
(register 'messenger (spawn 'messenger 'server '())))
;;; Server adds a new user to the user list
(defun server-logon (from name user-list)
;; Check if logged on anywhere else
(if (lists:keymember name 2 user-list)
(progn ;Reject logon
(! from #(messenger stop user-exists-at-other-node))
user-list)
(progn ;Add user to the list
(! from #(messenger logged-on))
(link from)
(cons (tuple from name) user-list))))
;;; Server deletes a user from the user list
(defun server-logoff (pid user-list)
(lists:keydelete pid 1 user-list))
;;; Server transfers a message between user
(defun server-transfer (from-pid to-name message user-list)
;; Check that the user is logged on and who he is
(case (lists:keyfind from-pid 1 user-list)
((tuple from-pid from-name)
(server-transfer from-pid from-name to-name message user-list))
('false
(! from-pid #(messenger stop you-are-not-logged-on)))))
;;; If the user exists, send the message
(defun server-transfer (from-pid from-name to-name message user-list)
;; Find the receiver and send the message
(case (lists:keyfind to-name 2 user-list)
((tuple to-pid to-name)
(! to-pid (tuple 'message-from from-name message))
(! from-pid #(messenger sent)))
('false
(! from-pid #(messenger receiver-not-found)))))
;;; User Commands
(defun logon (name)
(case (whereis 'mess-client)
('undefined
(let ((client (spawn 'messenger 'client (list (server-node) name))))
(register 'mess-client client)))
(_ 'already-logged-on)))
(defun logoff ()
(! 'mess-client 'logoff))
(defun message (to-name message)
(case (whereis 'mess-client) ;Test if the client is running
('undefined
'not-logged-on)
(_ (! 'mess-client (tuple 'message-to to-name message))
'ok)))
;;; The client process which runs on each server node
(defun client (server-node name)
(! (tuple 'messenger server-node) (tuple (self) 'logon name))
(await-result)
(client server-node))
(defun client (server-node)
(receive
('logoff
(exit 'normal))
((tuple 'message-to to-name message)
(! (tuple 'messenger server-node)
(tuple (self) 'message-to to-name message))
(await-result))
((tuple 'message-from from-name message)
(lfe_io:format "Message from ~p: ~p~n" (list from-name message))))
(client server-node))
;;; Wait for a response from the server
(defun await-result ()
(receive
((tuple 'messenger 'stop why) ;Stop the client
(lfe_io:format "~p~n" (list why))
(exit 'normal))
((tuple 'messenger what) ;Normal response
(lfe_io:format "~p~n" (list what)))
(after 5000
(lfe_io:format "No response from server~n" ())
(exit 'timeout))))
We have added the following changes:
The messenger server traps exits. If it receives an exit signal, #(EXIT from reason) this means that a client process has terminated or is unreachable because:
- the user has logged off (we have removed the "logoff" message),
- the network connection to the client is broken,
- the node on which the client process resides has gone down, or
- the client processes has done some illegal operation.
If we receive an exit signal as above, we delete the tuple, #(from name) from the servers user-list using the server-logoff function. If the node on which the server runs goes down, an exit signal (automatically generated by the system), will be sent to all of the client processes: #(EXIT messenger-pid noconnection) causing all the client processes to terminate.
We have also introduced a timeout of five seconds in the await-result function. I.e. if the server does not reply within five seconds (5000 ms), the client terminates. This is really only needed in the logon sequence before the client and server are linked.
An interesting case is if the client was to terminate before the server links to it. This is taken care of because linking to a non-existent process causes an exit signal, #(EXIT from noproc), to be automatically generated as if the process terminated immediately after the link operation.
Records
[forthcoming]
Ticket: https://github.com/lfe/tutorial/issues/9
Modularising
[forthcoming]
Ticket: https://github.com/lfe/tutorial/issues/9
Header Files
[forthcoming]
Ticket: https://github.com/lfe/tutorial/issues/9
Records
A record is defined as:
(defrecord <name-of-record> <field-name-1> <field-name-2> ...)
The defrecord macro creates a number of new macro for creating, matching and accessing the fields of the record.
For example:
(defrecord message-to to-name message)
The record data is a tuple which is exactly equivalent to:
#(message-to <to-name> <message>)
Creating record is done with a make-<record-name> macro, and is best illustrated by an example:
(make-message-to message "hello" to-name 'fred)
This will create the tuple:
#(message-to fred "hello")
Note that you don't have to worry about the order you assign values to the various parts of the records when you create it. The advantage of using records is that by placing their definitions in header files you can conveniently define interfaces which are easy to change. For example, if you want to add a new field to the record, you will only have to change the code where the new field is used and not at every place the record is referred to. If you leave out a field when creating a record, it will get the value of the atom undefined. (manual)
Pattern matching with records is very similar to creating records. For example inside a function clause, case or receive:
(match-message-to to-name the-name message the-message)
will match a message-to record and extract the to-name field in to the variable the-name and the message field in to the variable the-message. It is equivalent to writing:
(tuple 'message-to the-name the-message)
Accessing the fields of a record is done through macros which are created when the record is defined. For example the variable my-message contains a message-to record then
(message-to-message my-message)
will return the value of the message field of my-message and
(set-message-to-message my-message "goodbye")
will return a new record where the message field now has the value "goodbye".
Macros
As LFE code is just lists so it seems natural that there should be some way to evaluate data structures. In this chapter we will look at some ways of doing this.
Eval
The form eval takes an LFE data structure and evaluates it as an expression and then returns the value:
lfe> (eval 15)
15
lfe> (eval '(+ 1 2 3 4))
10
lfe> (eval '(list 'a 'b 'c))
(a b c)
Using eval is one way way to merge lists and code. However, it is not a very good way:
-
It is inefficient as the input expression is evaluated by the LFE interpreter,
lfe_eval. This is much slower than running compiled code. -
The expression is evaluated without a lexical context. So calling
evalinside aletdoes not allow the evaluated expression to refer to variables bound by thelet:
lfe> (set expr '(* x y)) ; Expression to evaluate
(* x y)
lfe> (let ((x 17) (y 42)) (eval expr))
exception error: #(unbound_symb x)
Well, this is not quite true. If we "reverse" the code and build a let expression which imports and binds the variables and then call eval on it we can access the variables:
lfe> (eval (list 'let (list (list 'x 17) (list 'y 42)) expr))
714
Macros
The most common way to write programs in LFE that write programs is by defining macros. Macros work by transformation. When you define a macro you say how a call to it should be translated, the macro expansion, which is then done automatically by the compile. The code generated by the macro then becomes an integral part of the program.
Macros are usually defined with the defmacro form. It resembles a defun but instead of defining the value a call returns it defines how the call should be expanded. For example a macro unless which returns an expression which evaluates the body if the test is false:
(defmacro unless (test body)
(list 'if (list 'not test) body))
So if we type into the top-level REPL:
lfe> (unless (> 3 4) 'yes)
yes
lfe> (unless (is_number 'foo) 'number)
number
To test a macro and look at its expansion we can use the function macroexpand which takes a macro call and generates its expansion1:
lfe> (macroexpand '(unless (> 3 4) 'yes) $ENV)
(if (not (> 3 4)) 'yes)
If a macro call expands into another macro call then the compiler or the top-level REPL it will keep expanding until the expansion is no longer a macro. It is the expansion of the macro call which is then inserted into the code. So in the example of unless it is the resultant if form which is then inserted into the code.
Notes
1 The extra argument $ENV is needed as this is where the REPL keeps its locally defined functions and macros and we need to tell macroexpand where to look.
The Backquote Macro
The backquote macro makes it possible to build lists and tuples from templates. Used by itself a backquote is equivalent to a regular quote:
lfe> `(a b c)
(a b c)
Like a regular quote, a backquote alone protects its arguments from evaluation. The advantage of backquote is that it is possible to turn on evaluation inside forms which are backquoted using , (comma or "unquote") and ,@ (comma-at or "unquote splice").1 When something is prefixed with a comma it will be evaluated. For example:
lfe> (set (tuple a b) #(1 2))
#(1 2)
lfe> `(a is ,a and b is ,b)
(a is 1 and b is 2)
lfe> `#(a ,a b ,b)
#(a 1 b 2)
Quoting works with both lists and tuples. The backquote actually expands to an expression which builds the structure the templates describes. For example, the following
`(a is ,a and b is ,b)
expands to
(list 'a 'is a 'and b 'is b)
and
`(a . ,a)
expands to
(cons 'a a)
This:
`#(a ,a b ,b)
expands to
(tuple 'a a 'b b)
They are very useful in macros as we can write a macro definitions which look like the expansions they produce. For example we could define the unless from the previous section as:
(defmacro unless
((cons test body) `(if (not ,test) (progn ,@body))))
Here we have extended it allow multiple forms in the body. Comma-at is like comma but splices its argument which should be a list. So for example:
lfe> (macroexpand '(unless (test x) (first-do) (second-do)) $ENV)
(if (not (test x)) (progn (first-do) (second-do)))
As the backquote macro expands to the expression which would build the template it is also very useful in patterns as we can use a template to describe the pattern. Here is the Converting Temperature example rewritten to use backquote in both the patterns and constructors:
(defun f->c
((`#(,name #(C ,temp)))
;; No conversion needed
`#(,name #(C ,temp)))
((`#(,name #(F ,temp)))
;; Do the conversion
`#(,name #(C ,(/ (* (- temp 32) 5) 9)))))
(defun print-temp
((`#(,name #(C ,temp)))
(lfe_io:format "~-15w ~w C~n" `(,name ,temp))))
Using the backquote macro also makes it much easier to build expressions which we can evaluate with eval. So if we want to import values into the expression to evaluate we can do it like this:
lfe> (set expr '(* x y)) ;Expression to evaluate
(* x y)
lfe> (eval `(let ((x 17) (y 42)) ,expr))
714
Notes
1 In LFE the backquote is a normal macro and is expanded at the same time as other macros. When they are parsed `thing becomes (backquote thing), ,thing becomes (comma thing) and ,@thing becomes (comma-at thing).
Conclusion
[forthcoming]
Ticket: https://github.com/lfe/tutorial/issues/10
Feedback and Docs Bugs
If you would like to provide feedback about this guide, we would welcome the chance to improve the experience for everyone. Please create a ticket in the Github issue tracker. Be sure to give a full description so that we can best help you!