LFE MACHINE MANUAL

Adatped from multiple sources
by Duncan McGreggor and Robert Virding

Published by Cowboys 'N' Beans Books

https://github.com/cnbbooks ◈ http://cnbb.pub/ ◈ info@cnbb.pub

First electronic edition published: 2020

Portions © 1974, David Moon

Portions © 1978-1981, Daniel Weinreb and David Moon

Portions © 1979-1984, Douglas Adams

Portions © 1983, Kent Pitman

Portions © 1992-1993, Peter Norvig and Kent Pitman

Portions © 2003-2020, Ericsson AB

Portions © 2008-2012, Robert Virding

Portions © 2010-2020, Steve Klabnik and Carol Nichols

Portions © 2013-2020, Robert Virding and Duncan McGreggor

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License

Preface

The original Lisp Machine Manual, the direct spiritiaul ancestor of the LFE Machine Manual, described both the language and the "operating system" of the Lisp Machine. The language was a dialect of Lisp called Zetalisp. Zetalisp was a direct descendant of MACLISP, created as a systems programming language for the MIT Lisp machines. This is of special note since Erlang was created as a systems programming language too. One of it's co-creators, Robert Virding, created Lisp Flavoured Erlang (LFE) based upon his expereinces with Franz Lisp (which based largely upon MACLISP), Portable Standard Lisp (itself an experiment in systems programming), and ultimately in an implementation he made of Lisp Machine Flavors on top of VAX/VMS where he extensively utilized the Lisp Machine Manual.

As such, LFE has a very strong inheritance of systems programming from both parents, as it were. First and foremost, it is a BEAM language written on top of the Erlang VM and from which it strays very little. Secondly, it is a Lisp dialect. It is, however, entirely a systems programming language.

Which brings us back to Zetalisp and the Lisp Machine Manual. It seemed only fitting to base the LFE manual upon the fantastic work and docuentation that was done on Lisp systems programming in the 70s and 80s, work that so many of us treasure and adore and to which we still defer. Thus the machine that is OTP in the context and syntax of the LFE Lisp dialect is extensively documented in the LFE MACHINE MANUAL.

About the Cover

The LFE "Chineual" cover is based upon the Lisp Machine Manual covers of the early 80s. The Lisp Machine Manual editions we have seen from 1977 and 1979 had only hand-typed title pages, no covers, so we're not sure if the famous graphic/typographic design occurred any earlier than 1982. We've also been unable to discover who the original designer was, but would love to give them credit, should we find out.

The Original

The Software Preservation Group has this image on their site:

Bitsavers has a 3rd edition of the Chineual with the full cover

The LFE Edition

Whole Cover

Back Cover

The Spine

Dedication

TBD

Forward

TBD

Acknowledgments

TBD

[gonna be a long list ...]

PART I

Introduction

Far out in the uncharted backwaters of the unfashionable end of computer science known as "distributed systems programming" lies a small red e. Orbitting this at a distance roughly proportional to the inverse of the likelihood of it being noticed is an utterly insignificant little green mug filled with the morning beverage stimulant equivalent of That Old Janx Spirit. Upon that liquid floats a little yellow 𝛌 whose adherents are so amazingly primitive that they still think conss, car, and cdr are pretty neat ideas.

This is their book.

Their language, Lisp Flavoured Erlang (henceforth "LFE"), lets you use the archaic and much-beloved S-expressions to write some of the most advanced software on the planet. LFE is a general-purpose, concurrent, functional Lisp whose underlying virtual machine (Erlang) was designed to create distributed, fault-tolerant, soft-realtime, highly-availale, always-up, hot-swappable appliances, applications, and services. In addition to fashionable digital watches, LFE sports immutable data, pattern-matching, eager evaluation, and dynamic typing.

This manual will not only teach you what all of that means and why you want it in your breakfast cereal, but also: how to create LFE programs; what exactly are the basic elements of the language; the ins-and-outs of extremely efficient and beautiful clients and servers; and much, much more.

Note, however, that the first chapter is a little different than most other books, and is in fact different from the rest of the chapters in this manual. We wrote this chapter with two guiding thoughts: firstly and foremost, we wanted to provide some practical examples of code-in-action as a context in which a programmer new to LFE could continuously refer -- from the very beginning through to the successful end -- while learning the general principles of that language; secondly, most programming language manuals are dry references to individual pieces of a language or system, not representatives of the whole, and we wanted to provide a starting place for those who learn well via examples, who would benefit from a view toward that whole. For those who have already seen plenty of LFE examples, don't need this, and would rather get to down to the nitty-gritty, rest assured we desire your experience to be guilt-free and thus encourage you to jump into next chapter immediately!

This book is divided into 6 parts with the following foci:

• Introductory material
• Core data types and capabilities
• The basics of LFE code and projects
• Advanced language features
• The machine that is OTP
• Concluding thoughts and summaries

There is a lot of material in this book, so just take it section by section. If at any time you feel overwhelmed, simply set down the book, take a deep breath, fix yourself a cuppa, and don't panic.

Welcome to the LFE MACHINE MANUAL, the definitive LFE reference.

About LFE

What LFE Isn't

Just to clear the air and set some expectations, here's what you're not going to find in LFE:

• An implementation of Scheme
• An implementation of Common Lisp
• An implementation of Clojure

As such, you will not find the following:

• A Scheme-like single namespace
• CL packages or munged names faking packages
• Access to Java libraries

What LFE Is!

Here's what you can expect of LFE:

• A proper Lisp-2, based on the features and limitations of the Erlang VM
• Compatibility with vanilla Erlang and OTP
• It runs on the standard Erlang VM

Furthermore, as a result of Erlang's influence (and LFE's compatibility with it), the following hold:

• there is no global data
• data is not mutable
• only the standard Erlang data types are used
• you get pattern matching and guards
• you have access to Erlang functions and modules
• LFE has a compiler/interpreter
• functions with declared arity and fixed number of arguments
• Lisp macros

Prerequisites

Anyone coming to LFE should have experience programming in another language, ideally a systems programming language, especially if that language was found lacking. If the corageous reader is attmping to use LFE as a means of entering the study of computer science, we might offer several other paths of study which may bear fruit more quickly and with less pain.

No prior Lisp experience is required, but that would certinaly be helpful. The same goes for Erlang/OTP (or any of the suite of BEAM languages). The reader with experience writing concurrent applications, wrestling with fault-tolerance, or maintaining highly-available applications and services does receive bonus points for preparedness. Such well-prepared readers landing here may have, in fact, done so due to a quest for a distributed Lisp. For those whom this does apply, your quest has found its happy end.

This book assumes the reader has the following installed upon their system:

• a package manager for easily installing software (in particular, development tools and supporting libraries)
• git, make, and other core open source software development tools
• a modern version of Erlang (as of the writing of this book, that would include versions 19 through 23); the rebar3 documentation has great suggestions on what to use here, depending upon your need
• the rebar3 build tool for Erlang (and other BEAM languages); see its docs for installation instructions

Conventions

Typography

Key Entry

We use the angle bracket convention to indicate typing actual key on the keyboard. For instance, when the reader sees <ENTER> they should interpret this as an actual key they should type. Note that all keys are given in upper-case. If the reader is expected to use an upper-case "C" instead of a lower-case "c", they will be presented with the key combination <SHIFT><C>.

Code

Color syntax highlighting is used in this text to display blocks of code. The formatting of this display is done in such a way as to invoke in the mind of the reader the feeling of a terminal, thus making an obvious visual distinction in the text. For instance:

(defun fib
  ((0) 0)
  ((1) 1)
  ((n)
    (+ (fib (- n 1))
       (fib (- n 2)))))

Examples such as this one are readily copied and may be pasted without edit into a file or even the LFE REPL itself.

For interactive code, we display the default LFE prompt the reader will see when in the REPL:

lfe> (integer_to_list 42 2)
;; "101010"

We also distinguish the output from the entered LFE code using code comments displayed afer the command.

For shell commands, the commands to enter at the prompt are prefixed by a $ for the prompt. Input and any relevant output are provided as comment strings:

$ echo "I am excited to learn LFE"
# I am excited to learn LFE

LiffyBot

This is a severly hoopy frood. With an attitude. He pops up from time to time, generally with good advice. Or simply as a marker for something the authors hope you will pay special note.

Messages of Note

From time to time you will see call-out boxes, aimed at drawing your attention to something of note. There are four differnt types of these:

• ones that share useful info (blue)
• ones that highlight something of a momentus nature (green)
• ones that offer warnings to tred carefully (orange)
• ones that beg you not to follow a particular path (red)

These messages will take the following forms:

Development Setup

rebar3 Configuration

Having followed the notes and linked instructions in the Prerequisites section, you are ready to add global support for the LFE rebar3 plugin.

First, unless you have configured other rebar3 plugins on your system, you will need to create the configuration directory and the configuration file:

$ mkdir ~/.config/rebar3
$ touch ~/.config/rebar3/rebar.config

Next, open up that file in your favourite editor, and give it these contents:

{plugins, [
  {rebar3_lfe,
    {git, "https://github.com/lfe-rebar3/rebar3_lfe.git", {branch, "master"}}}
]}.

If you already have a rebar.config file with a plugins entry, then simply add a comma after the last plugin listed and paste the {rebar3_lfe, {...}} line from above (with no trailing comma!).

A Quick Test with the REPL

With the LFE rebar3 plugin successfully configured, you should be able to start up the LFE REPL anywhere on your system with the following:

$ rebar3 lfe repl

Erlang/OTP 23 [erts-11.0] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [hipe]

   ..-~.~_~---..
  (      \\     )    |   A Lisp-2+ on the Erlang VM
  |`-.._/_\\_.-':    |   Type (help) for usage info.
  |         g |_ \   |
  |        n    | |  |   Docs: http://docs.lfe.io/
  |       a    / /   |   Source: http://github.com/rvirding/lfe
   \     l    |_/    |
    \   r     /      |   LFE v1.3-dev (abort with ^G)
     `-E___.-'

lfe>

Exit out of the REPL for now by typing <CTRL><G> and thn <Q>.

'Hello, World!'

Hello-World style introductory programs are intended to give the potential programmer for that language a sense of what it is like to write a minimalist piece of software with the language in question. In particular, it should show off the minimum capabilitiues of the language, essentially, what you could be in for should you decide upon this path.

In the case of LFE/OTP, this is extremely misleading. But more on that in the OTP version of the Hello-World program.

In this section we will concede to conventional practice and produce a minimal Hello-World that does what many other languages' Hello-World programs do.

From the REPL

As previously demonstrated, it is possible to start up the LFE 'read-eval-print loop' (REPL) using rebar3:

$ rebar3 lfe repl

Once you are at the LFE prompt, you may write a simple LFE "program" like the following:

lfe> (io:format "~p~n" (list "Hello, World!"))

Or, for the terminally lazy:

lfe> (io:format "~p~n" '("Hello, World!"))

While technically a program, it is not a very interesting one: we didn't create a function of our own, nor did we run it from outside the LFE interactive programming environment.

Let's address one of those points right now. Try this:

lfe> (defun hello-world ()
lfe>   (io:format "~p~n" '("Hello, World!")))

This is a simple function definition in LFE.

We can run it by calling it:

lfe> (hello-world)
;; "Hello, World!"
;; ok

When we execute our hello-world function, it prints our message to standard-output and then lets us know everything really quite fine with a friendly ok. LFE displays ok as output for functions that do not return a value.

Now let's address the other point: running a Hello-World programming from outside LFE.

Hit <CTRL-G><Q> to exit the REPL and get back to your terminal.

From the Command Line

From your system shell prompt, run the following to create a new project that will let us run a Hello-World program from the command line:

$ rebar3 new lfe-main hello-world
$ cd ./hello-world

Once in the project directory, you can actually just do this:

$ rebar3 lfe run

You will see code getting downloaded and compiled, and then your script will run, generating the following output:

Running script '/usr/local/bin/rebar3' with args [] ...
'hello-world

When you created a new LFE project of type 'main', a Hello-World function was automatically generated for you, one that's even simpler than what we created in the previous section:

(defun my-fun ()
  'hello-world)

The other code that was created when we executed rebar3 new lfe-main hello-world was a script meant to be used by LFE with LFE acting as a shell interpreter:

#!/usr/bin/env lfescript

(defun main (args)
  (let ((script-name (escript:script_name)))
    (io:format "Running script '~s' with args ~p ...~n" `(,script-name ,args))
    (io:format "~p~n" `(,(hello-world:my-fun)))))

You may be wondering about the args argument to the main function, and the fact that the printed output for the args when we ran this was []. Let's try something:

$ rebar3 lfe run -- Fenchurch 42

Running script '/usr/local/bin/rebar3' with args [<<"Fenchurch">>,<<"42">>] ...
'hello-world'

We have to provide the two dashes to let rebar3 know that we're done with it, that the subsequent argsuments are not for it, but rather for the program we want it to start for us. Using it causes everything after the -- to be passed as arguments to our script.

As for the code itself, it's tiny. But there is a lot going on just with these two files. Have no fear, though: the remainder of this book will explore all of that and more. For now, know that the main function in the executable is calling the hello-world module's my-fun function, which takes no arguments. To put another way, what we really have here is a tiny, trivial library project with the addition of a script that calls a function from that library.

For now just know that an executable file which starts with #!/usr/bin/env lfescript and contains a main function accepting one argument is an LFE script capable of being executed from the command line -- a we have shown!

LFE/OTP 'Hello, World!'

What have been demonstrated so far are fairly vanilla Hello-World examples; there's nothing particularly interesting about them, which puts them solidly iin the company of the millions of other Hello-World programs. As mentioned before, this approach is particularly vexing in the case of LFE/OTP, since it lures the prospective developer into the preconception that BEAM languages are just like other programming languages. They most decidfedluy are not.

What makes them, and in this particular case LFE, special is OTP. There's really nothing quite like it, certainly not baked into the heart and soul of another programming language. Most useful applications you will write in LFE/OTP will be composed of some sort of long-running service or server, something that manages that server and restarts it in the event of errors, and lastly, a context that contains both, the latter usually referred to as the "application" itself.

As such, a real Hello-World in LFE would be honest and let the prospective developer know what they are in for (and what power will be placed at their fingertips). That is what we will show now, a real LFE Hello-World example.

If you are still in the directory of the previous Hello-World project, let's get out of that:

$ cd ../

Now we're going to create a new project, one utilising the some very basic OTP patterns:

$ rebar3 new lfe-app hello-otp-world
$ cd ./hello-otp-world

We won't look at the code for this right now, since there are chapters dedicated to that in the second half of the book. But let's brush the surface with a quick run in the REPL:

$ rebar3 lfe repl

To start your new hello-world application, use the OTP application module:

lfe> (application:ensure_all_started 'hello-otp-world)
;; #(ok (hello-otp-world))

That message lets you know that not only was the hello-otp-word application and server started without issue, any applications upon which it depends were also started. Furthermore, there is a supervisor for our server, and it has started as well. Should our Hello-World server crash for any reason, the supervisor will restart it.

To finish the demonstration, and display the clichéd if classic message:

(hello-otp-world:echo "Hello, OTP World!")
;; "Hello, OTP World!"

And that, dear reader, is a true LFE/OTP Hello-World program, complete with message-passing and pattern-matching!

Feel free to poke around in the code that was generated for you, but know that eventually all its mysteries will be revealed, and by the end of this book this program's magic will just seem like ordinary code to you, ordinary, dependable, fault-tolerant, highly-availble, massively-concurrent code.

Walk-through: An LFE Guessing Game

Now that you've seen some LFE in action, let's do something completely insane: write a whole game before we even know the language!

We will follow the same patterns established in the Hello-World examples, so if you are still in one of the Hello-World projects, change directory and then create a new LFE project:

$ cd ../
$ rebar3 new lfe-app guessing-game
$ cd ./guessing-game

We will create this game by exploring functions in the REPL and then saving the results in a file. Open up your generated project in your favourite code-editing application, and then open up a terminal from your new project directory, and start the REPL:

$ rebar3 lfe repl

Planning the Game

We've created our new project, but before we write even a single atom of code, let's take a moment to think about the problem and come up with a nice solution. Byt doing this, we increase our chances of making something both useful and elegant. As long as what we write remains legible and meets our needs, the less we write the better. This sort of practice elegance will make the code easier to maintain and reduce the chance for bugs (by the simple merrit of there being less code in which a bug may arise; the more code, the greater opportunities for bugs).

Our first step will be making sure we understand the problem and devising some minimal abstractions. Next, we'll think about what actually need to happen in the game. With that in hand, we will know what state we need to track. Then, we're off to the races: all the code will fall right into place and we'll get to play our game.

Key Abstractions

In a guessing game, there are two players: one who has the answer, and one who seeks the answer. Our code and data should clearly model these two players.

Actions

The player with the answer needs to peform the following actions:

1. At the beginning of the game, state the problem and tell the other player to start guessing
2. Receive the other player's guess
3. Check the guess against the answer
4. Report back to the other player on the provided guess
5. End the game if the guess was correct

The guessing player needs to take only one action:

1. guess!

State

We need to track the state of the game. Based upon the actions we've examined, the overall state is very simple. Through the course of the game, will only need to preserve the answer that will be guessed.

Code Explore

Now that we've thought through our problem space clearly and cleanly, let's do some code exploration and start defining some functions we think we'll need.

We've already generated an OTP application using the LFE rebar3 plugin, and once we've got our collection of functions that address the needed game features, we can plug those into the application.

We'll make those changes in the code editor you've opened, and we'll explore a small set of possible functions to use for this using the project REPL session you've just started.

Getting User Input

How do we get user input in LFE? Like this!

lfe> (io:fread "Guess number: " "~d")

This will print the prompt Guess number: and then await your input and the press of the <ENTER> key. The input you provide needs to match the format type given in the second argument. In this case, the ~d tells us that this needs to be a decimal (base 10) integer.

fe> (io:fread "Guess number: " "~d")
;; Guess number: 42
;; #(ok "*")

If we try typing something that is not a base 10 integer, we get an error:

lfe> (io:fread "Guess number: " "~d")
;; Guess number: forty-two
;; #(error #(fread integer))

With correct usage, how do we capture the value in a variable? The standard way to do this in LFE is destructuring via pattern matching. The following snippet extracts the value and then prints the extracted value in the REPL:

lfe> (let ((`#(ok (,value)) (io:fread "Guess number: " "~d")))
lfe>   (io:format "Got: ~p~n" `(,value)))
;; Guess number: 42
;; Got: 42
;; ok

We'll talk a lot more about pattern matching in the future, as well as the meaning of the backtick and commas. For now,let's keep pottering in the REPL with these explorations, and make a function for this:

lfe> (defun guess ()
lfe>   (let ((`#(ok (,value)) (io:fread "Guess number: " "~d")))
lfe>     (io:format "You guessed: ~p~n" `(,value))))

And call it:

lfe> (guess)
;; Guess number: 42
;; You guessed: 42
;; ok

Checking the Input

In LFE there are several ways in which you can perform checks on values:

• the if form
• the cond form
• the case form
• pattern-matching and/or guards in function heads

The last one is commonly used in LFE when passing messages / data between functions. Our initial, generated project code is already doing this, and given the game state data we will be working with, this feels like a good fit what we need to implement.

Normally records are used for application data, but since we just care about the value of two integers (the number selected for the answer and the number guessed by the player), we'll keep things simple in this game:

(set answer 42)

Let's create a function with a guard:

lfe> (defun check
lfe>   ((guess) (when (< guess answer))
lfe>    (io:format "Guess is too low~n")))

The extra parenthesis around the function's arguments is due to the use of the pattern-matching form of function definition we're using here. We need this form, since we're going to use a guard. The when after the function args is called a "guard" in LFE. As you might imagine, we could use any number of these.

lfe> (check 10)
;; Guess is too low
;; ok

Let's add some more guards for the other checks we want to perform:

lfe> (defun check
lfe>   ((guess) (when (< guess answer))
lfe>    (io:format "Guess is too low~n"))
lfe>   ((guess) (when (> guess answer))
lfe>    (io:format "Guess is too high~n"))
lfe>   ((guess) (when (== guess answer))
lfe>    (io:format "Correct!~n")))

lfe> (check 10)
;; Guess is too low
;; ok

lfe> (check 100)
;; Guess is too high
;; ok

lfe> (check 42)
;; Correct!
;; ok

This should give a very general sense of what is possible.

Integrating into an Application

We're only going to touch one of the files that was generated when you created the guessing-game project: ./src/guessing-game.lfe. You can ignore all the others. Once we've made all the changes summarized below, we will walk through this file at a high level, discussing the changes and how those contribute to the completion of the game.

First though, we need to reflect on the planning we just did, remembering the actions and states that we want to support. There's also another thing to consider, since we're writing this as is an always-up OTP app. With some adjustments for state magagement, it could easily be turned into something that literally millions of users could be accessing simultaneouslyi. So: how does a game that is usually implemented as a quick CLI toy get transformed in LFE/OTP such that it can be run as a server?

In short, we'll use OTP's gen_server capability ("behaviour") and the usual message-passing practices. As such, the server will need to be able to process the following messages:

• #(start-game true) (create a record to track game state)
• #(stop-game true) (clear the game state)
• #(guess n)
  • check for guess equal to the answer
  • greater than the answer, and
  • less than the answer

We could have just used atoms for the first two, and done away with the complexity of using tuples for those, but symmetry is nice :-)

To create the game, we're going to need to perform the following integration tasks:

• Update the handle_cast function to process the commands and guards we listed above
• Create API functions that cast the appropriate messages
• Update the export form in the module definition
• Set the random seed so that the answers are different every time you start the application

handle_cast

The biggest chunk of code that needs to be changed is the handle_cast function. Since our game doesn't return values, we'll be using handle_cast. (If we needed to have data or results returned to us in the REPL, we would have used handle_call instead. Note that both are standard OTP gen_server callback functions.)

The generated project barely populates this function and the function isn't of the form that supports patten-matching (which we need here) so we will essentially be replacing what was generated. In the file ./src/guessing-game.lfe, change this:

(defun handle_cast (_msg state)
  `#(noreply ,state))

to this:

(defun handle_cast
  ((`#(start-game true) _state)
   (io:format "Guess the number I have chosen, between 1 and 10.~n")
   `#(noreply ,(random:uniform 10)))
  ((`#(stop-game true) _state)
   (io:format "Game over~n")
   '#(noreply undefined))
  ((`#(guess ,n) answer) (when (== n answer))
   (io:format "Well-guessed!!~n")
   (stop-game)
   '#(noreply undefined))
  ((`#(guess ,n) answer) (when (> n answer))
   (io:format "Your guess is too high.~n")
   `#(noreply ,answer))
  ((`#(guess ,n) answer) (when (< n answer))
   (io:format "Your guess is too low.~n")
   `#(noreply ,answer))
  ((_msg state)
   `#(noreply ,state)))

That is a single function in LFE, since for every match the arity of the function remains the same. It is, however, a function with six different and separate arguement-body forms: one for each pattern and/or guard.

These patterns are matched:

1. start
2. stop
3. guess (three times)
4. any

For the three guess patterns (well, one pattern, really) since there are three different guards we want placed on them:

1. guess is equal
2. guess is greater
3. guess is less

Note that the pattern for the function argument in these last three didn't change, only the guard is different beptween them.

Finally, there's the original "pass-through" or "match-any" pattern (this is used to prevent an error in the event of an unexpected message type).

Game API

In order to send a message to a running OTP server, you use special OTP functions for the type of server you are running. Our game is running a gen_server so we'll be using that OTP module to send messages, in particular we'll be calling gen_server:cast. However, creating messages and sending them via the appropriate gen_server function can get tedious quickly, so it is common practice to create API functions that do these things for you.

In our case, we want to go to the section with the heading ;;; our server API and add the following:

(defun start-game ()
  (gen_server:cast (SERVER) '#(start-game true)))

(defun stop-game ()
  (gen_server:cast (SERVER) '#(stop-game true)))

(defun guess (n)
  (gen_server:cast (SERVER) `#(guess ,n)))

Functions in LFE are private by default, so simply adding these functions doesn't make them publicly accessible. As things now stand these will not be usable outside their module; if we want to use them, e.g., from the REPL, we need to export them.

Go to the top of the guessing-game module and update the "server API" sectopm of the exports, chaning this:

    ;; server API
    (pid 0)
    (echo 1)))

to this:

    ;; server API
    (pid 0)
    (echo 1)
    (start-game 0)
    (stop-game 0)
    (guess 1)))

The final form of your module definition should look like this:

(defmodule guessing-game
  (behaviour gen_server)
  (export
    ;; gen_server implementation
    (start_link 0)
    (stop 0)
    ;; callback implementation
    (init 1)
    (handle_call 3)
    (handle_cast 2)
    (handle_info 2)
    (terminate 2)
    (code_change 3)
    ;; server API
    (pid 0)
    (echo 1)
    (start-game 0)
    (stop-game 0)
    (guess 1)))

Now our game functions are public, and we'll be able to use them from the REPL.

Finishing Touches

There is one last thing we can do to make our game more interesting. Right now, the game will work. But every time we start up the REPL and kick off a new game, the same "random" number will be selected for the answer. In order to make things interesting, we need to generate a random seed when we initialize our server.

We want to only do this once, though -- not every time the game starts, and certainly not every time a user guesses! When the LFE server supervisor starts our game server, one functions is called and called only once: init/1. That's where we want to make the change to support a better-than-default random seed.

Let's change that function:

(defun init (state)
  `#(ok ,state))

to this:

(defun init (state)
  (random:seed (erlang:phash2 `(,(node)))
               (erlang:monotonic_time)
               (erlang:unique_integer))
  `#(ok ,state))

Now we're ready to play!

Playing the Game

If you are still in the REPL, quit out of it so that rebar3 can rebuild our changed module. Then start it up again:

$ rebar3 lfe repl

Once at the LFE propmpt, start up the application:

lfe> (application:ensure_all_started 'guessing-game)

With the application and all of its dependencies started, we're ready to start the game and play it through:

lfe> (guessing-game:start-game)
;; ok
;; Guess the number I have chosen, between 1 and 10.

lfe> (guessing-game:guess 10)
;; ok
;; Your guess is too high.

lfe> (guessing-game:guess 1)
;; ok
;; Your guess is too low.

lfe> (guessing-game:guess 5)
;; ok
;; Your guess is too low.

lfe> (guessing-game:guess 7)
;; ok
;; Your guess is too low.

lfe> (guessing-game:guess 8)
;; ok
;; Well-guessed!!
;; Game over

Success! You've just done something pretty amazing, if still mysterious: you've not only created your first OTP application running a generic server, you've successully run it through to completion!

Until we can dive into all the details of what you've seen in this walkthrough, much of what you've just written will seem strange and maybe even overkill. For now, though, we'll mark a placeholder for those concepts: the next section will briefly review what you've done and indicate which parts of this book will provide the remaining missing pieces.

Review

We've got the whole rest of the book ahead of us to cover much of what you've seen in the sample project we've just created with our guessing game. In the coming pages, you will revisit every aspect of what you've seen so far in lots of detail with correspondingly useful insttruction on these matters.

That being said, it would be unfair to not at least read through the code together and mention the high-level concepts involved. Since we only touched the code in one file, that will be the one that gets the most of our attention for this short review, but let's touch on the others here, too.

Project Files

rebar.config

This is the file you need in every LFE project you will write in order to take advantage of the features (and time-savings!) that rebar3 provides. For this project, the two important parts are:

1. the entry for dependencies (only LFE in this case), and
2. the plugins entry for the LFE rebar3 plugin.

Project setup will be covered in Chapter XXX, section XXX.

Source Files

The source files for our sample program in this walkthrough are for an OTP application. OTP-based projects will be covered in Chapter XXX, section XXX.

.app.src

This file is mostly used for application metadata. Most of what our app uses in this file is pretty self-explanatory. Every LFE application will have one of these in the project source code. Every LFE library and application needs this file.

guessing-game-app.lfe

This is the top-level file for our game, an OTP application. It only exports two functions: one to start the app and the other to stop it. The application is responsible for starting up whatever supervisors all your services/servers need. For this sample application, only one supervisor is needed (with a very simple supervision tree).

guessing-game-sup.lfe

This module is a little more invloved and has all the configuration and code necessary to properly set up a supervisor for our server. When something goes wrong with our server, the restart strategy defined by our supervisor will kick in and get things back up and running again. This is one of the key secrets to OTP's wizardry, and we will be covering this in great detail later.

src/guessing-game.lfe

This is the last file we'll look at, and is the one we'll cover in the most detail right now. Here's the entire content of what we created for our game:

(defmodule guessing-game
  (behaviour gen_server)
  (export
   ;; gen_server implementation
   (start_link 0)
   (stop 0)
   ;; callback implementation
   (init 1)
   (handle_call 3)
   (handle_cast 2)
   (handle_info 2)
   (terminate 2)
   (code_change 3)
   ;; server API
   (pid 0)
   (echo 1)
   (start-game 0)
   (stop-game 0)
   (guess 1)))

;;; ----------------
;;; config functions
;;; ----------------

(defun SERVER () (MODULE))
(defun initial-state () '#())
(defun genserver-opts () '())
(defun unknown-command () #(error "Unknown command."))

;;; -------------------------
;;; gen_server implementation
;;; -------------------------

(defun start_link ()
  (gen_server:start_link `#(local ,(SERVER))
                         (MODULE)
                         (initial-state)
                         (genserver-opts)))

(defun stop ()
  (gen_server:call (SERVER) 'stop))

;;; -----------------------
;;; callback implementation
;;; -----------------------

(defun init (state)
  (random:seed (erlang:phash2 `(,(node)))
               (erlang:monotonic_time)
               (erlang:unique_integer))
  `#(ok ,state))

(defun handle_cast
  ((`#(start-game true) _state)
   (io:format "Guess the number I have chosen, between 1 and 10.~n")
   `#(noreply ,(random:uniform 10)))
  ((`#(stop-game true) _state)
   (io:format "Game over~n")
   '#(noreply undefined))
  ((`#(guess ,n) answer) (when (== n answer))
   (io:format "Well-guessed!!~n")
   (stop-game)
   '#(noreply undefined))
  ((`#(guess ,n) answer) (when (> n answer))
   (io:format "Your guess is too high.~n")
   `#(noreply ,answer))
  ((`#(guess ,n) answer) (when (< n answer))
   (io:format "Your guess is too low.~n")
   `#(noreply ,answer))
  ((_msg state)
   `#(noreply ,state)))

(defun handle_call
  (('stop _from state)
   `#(stop shutdown ok state))
  ((`#(echo ,msg) _from state)
   `#(reply ,msg state))
  ((message _from state)
   `#(reply ,(unknown-command) ,state)))

(defun handle_info
  ((`#(EXIT ,_from normal) state)
   `#(noreply ,state))
  ((`#(EXIT ,pid ,reason) state)
   (io:format "Process ~p exited! (Reason: ~p)~n" `(,pid ,reason))
   `#(noreply ,state))
  ((_msg state)
   `#(noreply ,state)))

(defun terminate (_reason _state)
  'ok)

(defun code_change (_old-version state _extra)
  `#(ok ,state))

;;; --------------
;;; our server API
;;; --------------

(defun pid ()
  (erlang:whereis (SERVER)))

(defun echo (msg)
  (gen_server:call (SERVER) `#(echo ,msg)))

(defun start-game ()
  (gen_server:cast (SERVER) '#(start-game true)))

(defun stop-game ()
  (gen_server:cast (SERVER) '#(stop-game true)))

(defun guess (n)
  (gen_server:cast (SERVER) `#(guess ,n)))

The beginning of the file opens with a declaration of the module: not only its name, but the public functions we want to expose as part of our API. This will be covered in Chapter XXX, section XXX.

Next, we have a few constant functions. Functions are necessary here due to the fact that LFE does not have global variables. This will be covered in Chapter XXX, section XXX.

Then we define the functions that will be used as this module's implementation of a generic OTP server. There is some boilerplate here that will be discussed when we dive into LFE/OTP. This will be covered in Chapter XXX, section XXX.

After that, we define the functions that are used by the OTP machinery that will run our server. Here you see several examples of pattern matching function heads in LFE, a very powerful feature that lends itself nicely to consise and expressive code. This will be covered in Chapter XXX, section XXX.

Lastly, we define our own API. Most of these functions simply send messages to our running server. More on this in Chapter XXX, section XXX.

The LFE REPL

We briefly introduced the REPL in the first version of the Hello-World example we wrote, stating that it was an acronym for 'read-eval-print loop' and how to start it with rebar3. As an LFE developer, this is one of the primnary tools -- arguably the most powerful -- at your disposal, so we're going to do a more thorough job of introducing its capabilities in this section.

A basic REPL can be implemented with just four functions; such an implementation could be started with the following:

(LOOP (PRINT (EVAL (READ))))

LFE has implemented most these functions for us already (and quite robustly), but we could create our own very limited REPL (single lines with no execution context or environment) within the LFE REPL using the following convenience wrappers:

(defun read ()
  (case (io:get_line "myrepl> ")
    ("quit\n" "quit")
    (str (let ((`#(ok ,expr) (lfe_io:read_string str)))
           expr))))

(defun print (result)
  (lfe_io:format "~p~n" `(,result))
  result)

(defun loop
  (("quit")
   'good-bye)
  ((code)
   (loop (print (eval (read))))))

Now we can start our custom REPL inside the LFE REPL:

lfe> (loop (print (eval (read))))

This gives us a new prompt:

myrepl>

At this prompt we can evaluate basic LFE expressions:

myrepl> (+ 1 2 3)
;; 6
myrepl> (* 2 (lists:foldl #'+/2 0 '(1 2 3 4 5 6)))
;; 42
myrepl> quit
;; good-bye
lfe>

Note that writing an evaluator is the hard part, and we've simply re-used the LFE evaluator for this demonstration.

Now that we've explored some of the background of REPLs and Lisp interpreters, let's look more deeply into the LFE REPL and how to best take advantage of its power when using the machine that is LFE and OTP.

Core Features

In keeping with the overall herritage of LFE, its REPL is both a Lisp REPL as well as an Erlang shell. In fact, when support was added for the LFE REPL to the rebar3_lfe plugin, it utilised all of the plumbing for the Erlang shell support in rebar3.

For the developer, though, this means that the LFE REPL holds a dual set of features, multiplying the set of features available to just the Erlang shell. These features include the following support:

• Evaluation of Lisp S-expressions
• Definition of functions, completely with LFE tail-recursion support
• Definition of records and use of record-specific support functions/macros
• Creation of LFE macros via standard Lisp-2 syntax
• Macro examination and debugging with various expansion macros
• The ability to start the LFE REPL in distribution mode, complete with Erlang cookie, and thus to not only access remote LFE and Erlang nodes, but to be accessed as a remote node itself (for nodes that have been granted access
• Access to the Erlang JCL and the ability to start separate, LFE shells running concurrently

Unsupported

The following capabilities are not supported in the LFE REPL:

• Module definitions; these are a file-based feature in LFE, just as with Erlang.

Starting LFE

The lfe executable

While this book focuses upon the use of rebar3 and its LFE plugin -- due entirely to the amount of time it saves through various features it supports -- LFE may be used quite easily without it.

To use LFE and its REPL without rebar3, you'll need to clone the repo, e.g.:

$ cd ~/lab
$ git clone https://github.com/rvirding/lfe.git
$ cd lfe

Since you have read the earlier section on dependencies, you already have Erlang, make, and your system build tools installed. As such, all you have to do is run the following to build LFE:

$ make

This will generate an executable in ./bin and you can start the LFE REPL by calling it:

$ ./bin/lfe

Erlang/OTP 23 [erts-11.0.2] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [hipe] [dtrace]

   ..-~.~_~---..
  (      \\     )    |   A Lisp-2+ on the Erlang VM
  |`-.._/_\\_.-':    |   Type (help) for usage info.
  |         g |_ \   |
  |        n    | |  |   Docs: http://docs.lfe.io/
  |       a    / /   |   Source: http://github.com/rvirding/lfe
   \     l    |_/    |
    \   r     /      |   LFE v1.3-dev (abort with ^G)
     `-E___.-'

lfe>

If you opt to install LFE system-wide with make install, then you can start the REPL from anywhere by simply executing lfe.

Via rebar3 lfe repl

As demonstrated earlier on several occasions, you can start the LFE REPL with the rebar3 LFE plugin (and this is what we'll do in the rest of this manual):

$ rebar3 lfe repl

Since you have updated your global rebar3 settings (in the "Prerequisites" section, after following the instructions on the rebar3 site), you may also start the LFE REPL from anywhere on your machine using the rebar3 command.

readline Support

History

Tab-Completion

(help)

As you gain familiarity with the LFE REPL, one of the most useful and convenient references will be the summary of functions, commands, and variablese that come with the LFE REPL.

To see these, simple call the help or h function:

lfe> (help)

That will result in the following being displyed to your terminal:

LFE shell built-in functions

(c file)       -- compile and load code in <file>
(cd dir)       -- change working directory to <dir>
(clear)        -- clear the REPL output
(doc mod)      -- documentation of a module
(doc mod:mac)  -- documentation of a macro
(doc m:f/a)    -- documentation of a function
(ec file)      -- compile and load code in erlang <file>
(ep expr)      -- print a term in erlang form
(epp expr)     -- pretty print a term in erlang form
(exit)         -- quit - an alias for (q)
(flush)        -- flush any messages sent to the shell
(h)            -- an alias for (help)
(help)         -- help info
(i)            -- information about the system
(i pids)       -- information about a list of pids
(l module)     -- load or reload <module>
(ls)           -- list files in the current directory
(ls dir)       -- list files in directory <dir>
(m)            -- which modules are loaded
(m mod)        -- information about module <mod>
(p expr)       -- print a term
(pp expr)      -- pretty print a term
(pid x y z)    -- convert <x>, <y> and <z> to a pid
(pwd)          -- print working directory
(q)            -- quit - shorthand for init:stop/0
(regs)         -- information about registered processes

LFE shell built-in commands

(reset-environment)             -- reset the environment to its initial state
(run file)                      -- execute all the shell commands in a <file>
(set pattern expr)
(set pattern (when guard) expr) -- evaluate <expr> and match the result with
                                   pattern binding
(slurp file)                    -- slurp in a LFE source <file> and makes
                                   everything available in the shell
(unslurp)                       -- revert back to the state before the last
                                   slurp

LFE shell built-in variables

+/++/+++      -- the tree previous expressions
*/**/***      -- the values of the previous expressions
-             -- the current expression output
$ENV          -- the current LFE environment

ok

Most of those are documented in stdlib reference for their Erlang counterparts, so be sure to reference that information for details on many of the above.

Those not covered in that Erlang reference manual, or those that are different in their LFE versionsm, include:

• Built-in Functions
  • Compilation functions
  • LFE code documentation
  • Printing and pretty-printing
• Built-in commands
• Built-in variables

REPL Functions

Most of the LFE REPL functions are documented in stdlib reference for their Erlang counterparts. This section documents where the LFE REPL help diverges from the Erlang Shell help.

Compilation

If you view the Erlang reference manual documentation for compiling files in the shell, you will see differences from what is show in the LFE help text. In particular, (c) is for compiling LFE modules and (ec) needs to be used for compiling Erlang source files.

In both cases, the resulting .beam files are compiled to the current working directory and not to an ebin directory. These .beam files will be found by LFE, since the current working directory is included in the path, but you'll likely want to perform some cleanup afterward.

Documentation

You may access the documentation for LFE modules, macros, and functions in the REPL via the doc function. For instance, the Common Lisp compatibility module's documentation:

lfe> (doc cl)
;; ____________________________________________________________
;; cl
;;
;; LFE Common Lisp interface library.
;;
;; ok

That module's cond macro documentation:

lfe> (doc cl:cond)
;; ____________________________________________________________
;; cond
;; args
;; CL compatible cond macro.
;;
;; ok

That module's pairlis/2 function documentation:

lfe> (doc cl:pairlis/2)
;; ____________________________________________________________
;; pairlis/2
;; keys values
;; Make an alist from pairs of keys values.
;;
;; ok

Documentation for Erlang modules and fucntions is available via the Command Interface

Printing Data

LFE Formatting

LFE provides some nice convenience functions for displaying data structions in the REPL. Let's say we had a data structure defined thusly:

lfe> (set data `(#(foo bar baz) #(quux quuz) #(corge grault garply)
lfe> #(plugh xyzzy) #(flurb nirf) #(zobod zirfid)))

We can print our data with the following:

lfe> (p data)
;; (#(foo bar baz) #(quux quuz) #(corge grault garply) #(plugh xyzzy) #(flurb nirf) #(zobod zirfid))
;; ok

Or we can pretty-print it:

lfe> (pp data)
;;(#(foo bar baz)
;; #(quux quuz)
;; #(corge grault garply)
;; #(plugh xyzzy)
;; #(flurb nirf)
;; #(zobod zirfid))
;; ok

Erlang Formatting

The same may be done for displaying data in the Erlang format:

lfe> (ep data)
;; [{foo,bar,baz},{quux,quuz},{corge,grault,garply},{plugh,xyzzy},{flurb,nirf},{zobod,zirfid}]
;; ok

lfe> (epp data)
;; [{foo,bar,baz},
;;  {quux,quuz},
;;  {corge,grault,garply},
;;  {plugh,xyzzy},
;;  {flurb,nirf},
;;  {zobod,zirfid}]
;; ok

REPL Commands

The LFE REPL provides several useful commands users:

(reset-environment)             -- reset the environment to its initial state
(run file)                      -- execute all the shell commands in a <file>
(set pattern expr)
(set pattern (when guard) expr) -- evaluate <expr> and match the result with
                                   pattern binding
(slurp file)                    -- slurp in a LFE source <file> and makes
                                   everything available in the shell
(unslurp)                       -- revert back to the state before the last
                                   slurp

These are fairly self-explanatory, with the possible exception of clarifying how run and slurp differ:

• Calling (run "some/file.lfe") will cause the LFE REPL to read the contents of that file and then execute every line in that file as if they had been typed at the terminal. This is a convenient way of duplicating REPL state between sessions. (If you haven't kept track of your entries, you can always open up your BEAM history file and create an .lfe file with all the required commands!)
• Calling (slurp "some/other/file.lfe") will place all functions, records, and macros defined in that file into the LFE environment, allowing you to call them without a module: prefix. Note that the code is not executed, but is instead placed into the current environment, ready for use.

Special Variables

LFE shell built-in variables

+/++/+++      -- the tree previous expressions
*/**/***      -- the values of the previous expressions
-             -- the current expression output
$ENV          -- the current LFE environment

Most of these variables are taken directly from Common Lisp and have the same exact meaning. From the Common Lisp HyperSpec for +,++,+++:

The variables +, ++, and +++ are maintained by the Lisp read-eval-print loop to save forms that were recently evaluated.

The value of + is the last form that was evaluated, the value of ++ is the previous value of +, and the value of +++ is the previous value of ++.

And for *,**,***:

The variables *, **, and *** are maintained by the Lisp read-eval-print loop to save the values of results that are printed each time through the loop.

The value of * is the most recent primary value that was printed, the value of ** is the previous value of *, and the value of *** is the previous value of **.

Lastly, for -:

The value of - is the form that is currently being evaluated by the Lisp read-eval-print loop.

The $ENV variable in the LFE REPL is a critical tool for debugging particularly tricky issues in the REPL (especially useful when creating complex macros).

Command Interface

While many of the functions listed in the LFE (help) have their documentation in the Erlang Command Interface module (CIM), not everything in the CIM has been provided in the LFE REPL, some of which can be useful at times.

Here are some of the more useful functions you may with to be aware of from that module:

• (c:bt pid) - Stack backtrace for a process. Equivalent to (erlang:process_display pid 'backtrace).
• (c:h mod) - Print the documentation for mod
• (c:h mod fun) - Print the documentation for all mod:fun
• (c:h mod fun arity) - Print the documentation for mod:fun/arity
• (c:lm) - Reloads all currently loaded modules that have changed on disk (see (c:mm)). Returns the list of results from calling (l mod) for each such loaded module.
• (c:memory) - Memory allocation information. Equivalent to (erlang:memory).
• (c:mm) -
• (c:ni) - Display system information, listing information about all nodes on the network
• (c:nl mod) - Loads Module on all nodes
• (c:nregs) - Displays information about all registered processes for all nodes in the network.
• (c:uptime) - Prints the node uptime (as specified by (erlang:statistics 'wall_clock)) in human-readable form.
• (c:xm mod) - Finds undefined functions, unused functions, and calls to deprecated functions in a module by calling xref:m/1.

Job Control

When in the LFE REPL, a special mode is accessible upon typing <CTRL-G>:

lfe>
User switch command
 -->

At the JCL --> prompt, you may get help text by typing ?:

 --> ?
  c [nn]            - connect to job
  i [nn]            - interrupt job
  k [nn]            - kill job
  j                 - list all jobs
  s [shell]         - start local shell
  r [node [shell]]  - start remote shell
  q                 - quit erlang
  ? | h             - this message
 -->

Running Multiple Shells

[ also: custom prompts ]

Files

REPL-Only

slurp

unslurp

Compiling LFE Files

Compiling Erlang Files

REPL & Module

include-lib

include-file

Module Loading

l and code:ensure_loaded

Variables

In LFE, variables are implemented with atoms. Atoms are used in the language for such things as naming functions and macros, use as keywords and in data structures. As you'll find out when reading about atoms, they are evaluated as simply themselves. However, when they are used as variable names, they evaulate to the value of which they were assigned.

There are two contexts for setting variables:

• in the LFE REPL
• in LFE modules

This distinction is important, not only because the contexts use different forms, but because like Erlang, LFE does not support global variables.

This chpater will also introduce the reader to pattern-matching in LFE, as it applies to variable assignment; a fuller discussion is presented in a dedicated chapter as well in the chapters that cover forms which support pattern-matching (i.e., compound data types, functions, etc.).

Lastly we'll talk more about some of the LFE nuances around global variables.

Bindings

In the REPL

To set a variable in the LFE REPL, use the set macro:

lfe> (set answer 42)
42

In the language itself, LFE doesn't support global variables -- a valuable feature inherited from Erlang. However, in order for a REPL experience to be useful, an environment must be maintained in which the user may write expressions and then refer to them later. This environment is essentially a mechanism for global state in the context of a single user running a single Erlang VM. If we set a variable called answer, that variable will be available to us as long as the REPL process continues or until we reset the environment.

Setting another value with the same variable name is allowed: it merely replaces the assignment in the current REPL environment:

lfe> (set answer "forty-two")
"forty-two"

With a variable assigned with set it may be used at any time in the REPL environment where it was defined:

lfe> (++ "The answer is " answer)
"The answer is forty-two"

Attempting to use a variable that has not been defined results in an error:

lfe> (++ "The question was " question)
** exception error: symbol question is unbound
  in lfe_eval:eval_error/1 (src/lfe_eval.erl, line 1292)
  in lists:map/2 (lists.erl, line 1243)
  in lists:map/2 (lists.erl, line 1243)

If you don't have any need to update the environment with data that you only need for a specific calculation, you may use the let form:

lfe> (let ((short-lived-value (* 2 (+ 1 2 3 4 5 6))))
lfe>   (io:format "The answer is ~p~n" `(,short-lived-value)))
The answer is 42
ok

Let's make sure that variable wasn't saved to our environment:

lfe> short-lived-value
** exception error: symbol short-lived-value is unbound

The lexical scope for the short-lived-value is within the let only and is not available outside of that scope.

In Functions and Macros

Within functions, variables are lexically scoped and bound with let and let*. One may also define lexically scoped functions inside other fucntions, and this is done with flet and fletrec (the latter required for defining recursive functions inside another function). These will be covered in detail later in the book.

We've seen let used above in the REPL; the same applies inside functions:

(defun display-answer ()
  (let ((answer (* 2 (+ 1 2 3 4 5 6))))
    (io:format "The answer is ~p~n" `(,answer))))

This is a straight-forward case of assignment; but what if we needed to assign a varaible that depended upon another variable. Using let, you'd have to do this:

(defun display-answer ()
  (let ((data '(1 2 3 4 5 6)))
    (let ((answer (* 2 (lists:sum data))))
      (io:format "The answer is ~p~n" `(,answer)))))

However, as with other Lisps, LFE provides a convenience macro for this: let*. Here's how it's used:

(defun display-answer ()
  (let* ((data '(1 2 3 4 5 6))
         (answer (* 2 (lists:sum data))))
    (io:format "The answer is ~p~n" `(,answer))))

Lexical scoping helps one isolate unrelated data or calculations, even in the same function: multiple let or let* blocks may be declared in a function and none of the bound variables in one block will be available to another block. Attempting to do so will result in an error.

In Modules

In LFE, one cannot bind variables at the module-level, only functions and macros. This is part of the "no global variables" philosophy (and practice!) of Erlang and LFE. Module-level bindings are done with defun for functions and defmacro for macros. The creation of modules, functions, and macros will all be covered in detail later in the book.

Shadowing

One shadows a variable in one scope when, at a higher scope, that variable was also defined. Here's an annotated example:

(defun shadow-demo ()
  (let ((a 5))
    (io:format "~p~n" `(,a))    ; prints 5
    (let ((a 'foo))             ; 'a' here shadows 'a' in the previous scope
      (io:format "~p~n" `(,a))) ; prints foo
    (io:format "~p~n" `(,a)))   ; prints 5; the shadow binding is out of scope
  (let ((a 42))
    (io:format "~p~n" `(,a))))  ; prints 42 - new scope, no shadowing

Shadowing also may occur at the module-level with the definition of functions, and the shadowing could be of functions at one of several levels. Here's a run-down on function shadowing in modules, from the highest or "outermost" to the lowest or "innermost":

• Predefined Erlang built-in functions (BIFs) may be shadowed by any of the following
• Predefined LFE BIFs may be shadowed by any of the following
• Module imports may shadow any of the above via aliasing
• Functions defined in a module may shadow any of the above
• Functions defined inside a function (e.g., via flet or fletrec) may shadow any of the above

Note that to shadow functions in LFE, functions must match both in name as well as arity (number of arguments).

The hd Erlang BIF returns the "head" of a list (the first item). Here's an example of shadowing it in the REPL. Here's the BIF at work:

lfe> (hd '(a b c d e))
a

Next, paste this into the REPL:

(defun hd (_)
  ;; part of the pun here is that the same function in Lisp is called 'car'
  "My other car is The Heart of Gold.")

The hd function takes one argument (a list), so our function also needs to take one. However, since we don't do anything with that, we use the "don't care" variable _.

Now let's call hd again:

lfe> (hd '(a b c d e))
"My other car is The Heart of Gold."

Shadowed!

Note that, like many other Lisps, LFE has the car function, but since this is a core form, it can't be shadowed (see the next section).

The Unshadowable

Shadowing does not apply to the supported LFE core forms. It may appear that your code is shadowing those forms, but the compiler will always use the core meaning and never an alternative. It does this silently, without warning -- so take care and do not be surprised!

Pattern-matching Preview

Global Variables Revisted

The Process Dictionary

ETS Tables

State and OTP Servers

External Databases

Primitive Types

This chapter covers the basic types of data available to LFE, upon which primatives rest all LFE libraries and applications.

• Integers
• Floats
• Atoms
• Booleans
• Characters

Each of these types has an LFE test function of the form TYPENAMEp (which wrap the respective Erlang is_TYPENAME function). These are used to perform type checks (especially common in guard expressions). These predicate functions will be covered in their respective type sections.

Integers

Integers in LFE may be either positive or negative and are by default base 10. Like Erlang, LFE does not have a maximum size of integer (in contrast to languages like C). This is accomplished via automatic conversion to larger (or smaller) internal representations (including the use of bignums).

lfe> 42
42
lfe> -42
-42
lfe> 1764
1764
lfe> 150130937545296561928688012959677941476701514734130607701636390322176
150130937545296561928688012959677941476701514734130607701636390322176

Bases

Several bases are supported via special syntax:

lfe> #b101010 ; binary
42
lfe> #o52 ; octal
42
lfe> #d42 ; decimal (explicit base 10)
42)
lfe> #x2a ; hexadecimal
42

Generic bases are supported, too:

lfe> #36r16
42

The number after the hash # is the base and may be any positive integer from 2 through 36. The number after the radix r is the actual value and must only bve comprised of integers allowed for the given base.

Converting between bases in LFE is most easily done via the integer_to_list Erlang function. For example:

lfe> (integer_to_list 1000 2)
"1111101000"
lfe> (integer_to_list 1000 8)
"1750"
lfe> (integer_to_list 1000 10)
"1000"
lfe> (integer_to_list 1000 16)
"3E8"
lfe> (integer_to_list 1000 36)
"RS"

If, for whatever reason, you want your base 10 integrer as a list (Erlang/LFE string), you can do that with this:

lfe> (integer_to_list 1000)
"1000"

Conversion to the binary type is also supported:

lfe> (integer_to_binary 1000)
#"1000"
lfe> (integer_to_binary 1000 2)
#"1111101000"
lfe> (integer_to_binary 1000 8)
#"1750"

The results above show LFE's literal representations of binary data; this will be covered in the chapter on "Bytes and Binaries".

Arithmetic Operators

Integers may be operated upon with the following:

lfe> (+ 1)
1
lfe> (+ 1 2)
3
lfe> (+ 1 2 3)
6
lfe> (- 1 2 3)
-4
lfe> (* 1 2 3)
6
lfe> (/ 1 2 3)
0.16666666666666666

Note that the division operator returns a float; floats will be covered in the next section.

Integer division is supported with a 2-arity function:

lfe> (div 1 2)
0
lfe> (div 10 2)
5

LFE also supports the remainder operation:

lfe> (rem 10 3)
1

As with any functional programming language, these operations may be composed (to any depth):

lfe> (div (* 12 (+ 1 2 3 4 5 6)) 6)
42

Mathematical Functions

The auto-loaded erlang module has several mathematical functions and is accessible in LFE without having to type the erlang: module prefix in the function calls. These include the following:

lfe> (abs -42)
42
lfe> (min 1 2)
1
lfe> (max 1 2)
2

Additional maths functions are provided via the math module. Since this module is not auto-loaded, in order to auto-complete it in the REPL you will need to load it:

lfe> (code:ensure_loaded 'math)
#(module math)

Now you can hit <TAB> after typing the following:

lfe> (math:

Which gives:

acos/1         acosh/1        asin/1         asinh/1        atan/1
atan2/2        atanh/1        ceil/1         cos/1          cosh/1
erf/1          erfc/1         exp/1          floor/1        fmod/2
log/1          log10/1        log2/1         module_info/0  module_info/1
pi/0           pow/2          sin/1          sinh/1         sqrt/1
tan/1          tanh/1

lfe> (round (math:pow 42 42))
150130937545296561928688012959677941476701514734130607701636390322176

The documentation for these functions is limited (available here) due in part to the fact that these are C library wrappers. Those that aren't documented should be self-explanatory for anyone who has used simular mathematical functions in other programming language libraries.

Predicates

To test if a value is an integer, we will first include some code:

lfe> (include-lib "lfe/include/cl.lfe")

That adds Common Lisp inspired functions and macros to our REPL session.

lfe> (integerp 42)
true
lfe> (integerp 42.24)
false
lfe> (integerp "forty-two")
false

If you prefer the Clojure-style of predicates:

lfe> (include-lib "lfe/include/clj.lfe")
lfe> (integer? 42)
true
lfe> (integer? "forty-two")
false

Of course there is always the Erlang predicate, usable without having to do any includes:

lfe> (is_integer 42)
true
lfe> (is_integer "forty-two")
false

Floats

Real numbers in LFE are represented using the stadnard floating point numbers.

lfe> 42.42
42.42
lfe> (/ 10 3)
3.3333333333333335
lfe> (math:pow 42 42)
1.5013093754529656e68

Note that the ~1.5e68 above is the floating point equivalent of scientific notation, namely 1.5 x 1068. LFE follows the 64-bit standard for float representation given by IEEE 754-1985.

Converting

An integer may be converted to a float explicitly:

lfe> (float 42)
42.0

Or, as with integers, to binaries:

lfe> (float_to_binary 42.42)
#"4.24200000000000017053e+01"
lfe> (float_to_binary 42.42 '(#(scientific 10)))
#"4.2420000000e+01"
lfe> (float_to_binary 42.42 '(#(scientific 20)))
#"4.24200000000000017053e+01"
lfe> (float_to_binary 42.42 '(#(decimals 10)))
#"42.4200000000"
lfe> (float_to_binary 42.42 '(#(decimals 10) compact))
#"42.42"

Or to lists (LFE and Erlang strings):

lfe> (float_to_list 42.42 '(#(scientific 10)))
"4.2420000000e+01"
lfe> (float_to_list 42.42 '(#(scientific 20)))
"4.24200000000000017053e+01"
lfe> (float_to_list 42.42 '(#(decimals 10)))
"42.4200000000"
lfe> (float_to_list 42.42 '(#(decimals 10) compact))
"42.42"

Formatting

If you need to round floating point numbers to a specific precision, you'll want to use the format function from either the io, io_lib, or lfe_io modules. If just want to print a value using Erlang syntax and formatting, the io module is what you want. If you prefer LFE syntax and formatting for your output, you'll want to use the lfe_io module. If you want to use the data or store it in a variable, you'll need the io_lib library.

For default precision:

lfe> (io_lib:format "~f" `(,(math:pow 42 42)))
"150130937545296561929000000000000000000000000000000000000000000000000.000000"

Two decimal places:

lfe> (io_lib:format "~.2f" `(,(math:pow 42 42)))
"150130937545296561929000000000000000000000000000000000000000000000000.00"

20 decimal places:

lfe> (io_lib:format "~.20f" `(,(math:pow 42 42)))
"150130937545296561929000000000000000000000000000000000000000000000000.00000000000000000000"

Arithmetic Operators & Mathematical Functions

Floats use most of the same operators and functions as integers, so be sure to review these subsections in the "Integers" section.

Others include:

lfe> (abs -42)
42
lfe> (ceil 42.1)
43
lfe> (floor 42.1)
42
lfe> (round 42.4)
42
lfe> (round 42.5)
43
lfe> (min 1 2)
1
lfe> (max 1 2)
2

Predicates

To test if a value is an integer, we will first include some code:

lfe> (include-lib "lfe/include/cl.lfe")

That adds Common Lisp inspired functions and macros to our REPL session.

lfe> (floatp 42.42)
true
lfe> (floatp 42)
false
lfe> (floatp "forty-two.forty-two")
false

If you prefer the Clojure-style of predicates:

lfe> (include-lib "lfe/include/clj.lfe")
lfe> (float? 42.42)
true
lfe> (float? "forty-two.forty-two")
false

Of course there is always the Erlang predicate, usable without having to do any includes:

lfe> (is_float 42.42)
true
lfe> (is_float "forty-two.forty-two")
false

Atoms

The cloest analog in LFE to what most Lisp dialects call symbols is the Erlang atom. Just as with Lisps do with symbols, LFE uses atoms for its variable and function names. Atoms are literals and constants, which means that their value is the same as their name and once created, cannot be changed.

Some basic examples of atoms:

lfe> 'zark
zark
lfe> 'zarking-amazing
zarking-amazing

Slightly less straight-forward examples which start with non-alphanumeric characters:

lfe> ':answer
:answer
lfe> '42answer
42answer
lfe> '42.0e42answer
42.0e42answer
lfe> '42°C
42°C

Standard LFE atom names may be comprised of all the latin-1 character set except the following:

• control character
• whitespace
• the various brackets
• double quotes
• semicolon

Of these, only |, \, ', ,, and # may not be the first character of the symbol's name (but they are allowed as subsequent letters).

Non-standard atom names may be created using atom quotes:

lfe> '|symbol name with spaces|
|symbol name with spaces|
lfe> '|'with staring quote!|
|'with staring quote!|
lfe> '|| ; <-- empty atoms are supported too
||
lfe> '|really weird atom: '#[]{}()<>";\||
|really weird atom: '#[]{}()<>";\||

In this case the name can contain any character of in the range from 0 to 255, and even no character at all.

In the case of atoms, it is important to understand a little something about their internals. In particular, Erlang and LFE atoms are global for each instance of a running Erlang virtual machine. Atoms are maintained by the VM in a table and are not garbage collected. By default, the Erlang atom table allows for a maximum of 1,048,576 entries.

As Symbols

The following code shows LFE's use of atoms in variable names. First, let's use a function for a slighly different purpose than designed: calling list_to_existing_atom on a string will only return a result if an atom of the same name already exists in the atom table. Otherwise, it will return an error:

lfe> (list_to_existing_atom "zaphod")
** exception error: bad argument
  in (erlang : list_to_existing_atom "zaphod")

This confirms that there is no zaphod atom in the atom table.

Now let's create a variable, assigning a value to it, and then use our indirect means of checkihg the atom table:

lfe> (set zaphod "frood")
"frood"
lfe> (list_to_existing_atom "zaphod")
zaphod

And here's an example showing LFE's use of atoms in function names using the same approach as above:

lfe> (list_to_existing_atom "beez")
** exception error: bad argument
  in (erlang : list_to_existing_atom "beez")

lfe> (defun beez (x) x)
beez
lfe> (list_to_existing_atom "beez")
beez

Converting

Atoms may be converted to strings and bitstrings, and vice versa.

(atom_to_binary 'trisha)
#"trisha"
(atom_to_binary 'mcmillan 'latin1)
#"mcmillan"
lfe> (atom_to_list 'trillian)
"trillian"

Note that the first one above is only available in Erlang 23.0 and above.

Some more examples for encoding:

lfe> (atom_to_binary '42°C 'latin1)
#B(52 50 176 67)
lfe> (atom_to_binary '42°C 'utf8)
#"42°C"

Functions that convert atoms only if they already exist in the atom table:

lfe> (binary_to_existing_atom #"trisha")
trisha
lfe> (binary_to_existing_atom #"trisha" 'latin1)
trisha
lfe> (list_to_existing_atom "trisha")
trisha

Operators

The only operators you may use on atoms are the comparison operators, e.g.:

lfe> (> 'a 'b)
false
lfe> (< 'a 'b)
true
lfe> (=:= 'a 'a)
true

Predicates

To test if a value is an atom, we will first include some code:

lfe> (include-lib "lfe/include/cl.lfe")

That adds Common Lisp inspired functions and macros to our REPL session.

lfe> (atomp 'arthur)
true
lfe> (atomp 42)
false
lfe> (atomp "forty-two.forty-two")
false

If the atom in question has been used in a function name definition:

If you prefer the Clojure-style of predicates:

lfe> (include-lib "lfe/include/clj.lfe")
lfe> (atom? 'dent)
true
lfe> (atom? "Ford")
false

Of course there is always the Erlang predicate, usable without having to do any includes:

lfe> (is_atom 'arthur)
true
lfe> (is_atom "forty-two.forty-two")
false

Caveats

As mentioned above (and documented), one needs to take care when creating atoms. By default, the maximum number of atoms that the Erlang VM will allow is 1,048,576; any more than that, and the VM will crash.

The first rule of thumb is not to write any code that generates large numbers of atoms. More explicitly useful, there are some handy functions for keeping track of the atom table, should you have the need.

lfe> (erlang:memory 'atom_used)
244562
lfe> (erlang:system_info 'atom_count)
9570
lfe> (erlang:system_info 'atom_limit)
1048576

Note that support for easily extracting the current atom data from system_info -- as demonstrated by the last two function calls above -- were added in Erlang 20; should you be running an older version of Erlang, you will need to parse the (system_info 'info) bitstring.

The default atom table size may be overridden during startup by passing a value with the +t options:

$ lfe +t 200000001

lfe> (erlang:system_info 'atom_limit)
200000001

Booleans

Strictly speaking, LFE has no Boolean type, just like Erlang. Instead, the atoms true and false are treated as Booleans

lfe> (== 42 "the question")
false
lfe> (== 42 42)
true
lfe> (> 1 2)
false
lfe> (< 1 2)
true

Operators

The standard logical operators are available to LFE Booleans:

lfe> (not 'true)
false
lfe> (not 'false)
true
lfe> (and 'true 'false)
false
lfe> (or 'true 'false)
true
lfe> (and 'true 'true 'true)
true
lfe> (and 'true 'true 'false)
false
lfe> (or 'false 'false 'false)
false
lfe> (or 'false 'false 'true)
true
lfe> (xor 'true 'true)
false
lfe> (xor 'true 'false)
true
lfe> (xor 'false 'false)
false
lfe> (xor 'false 'true)
true

With the and and or Boolean oprators, every argument is evaluated. To accomodate situations where complex and possible expensive logical expressions comprise the arguments to Boolean operators, short-circuit versions of these functions are also provided:

• with andalso, returns as soon as the first 'false is encountered;
• with orelse, returns as soon as the first 'true is encountered.

To demonstrate this, we'll define a boolean function that prints to standard out when it is evaluated:

(defun hey (x)
  (io:format "Made it here!~n") x)

Short-circuit demonstration of andalso:

lfe> (andalso 'true 'true 'false (hey 'true))
false
lfe> (andalso 'false 'true 'true (hey 'true))
false
lfe> (andalso 'true 'true 'true (hey 'true))
Made it here!
true

Short-circuit demonstration of orelse:

lfe> (orelse 'false 'false 'true (hey 'true))
true
lfe> (orelse 'true 'false 'false (hey 'true))
true
lfe> (orelse 'false 'false 'false (hey 'true))
Made it here!
true

Predicates

To test if a value is a Boolean, we will first include some code:

lfe> (include-lib "lfe/include/cl.lfe")

That adds Common Lisp inspired functions and macros to our REPL session.

lfe> (booleanp 'true)
true
lfe> (booleanp 'false)
true
lfe> (booleanp 'arthur)
false
lfe> (booleanp 42)
false

If the atom in question has been used in a function name definition:

If you prefer the Clojure-style of predicates:

lfe> (include-lib "lfe/include/clj.lfe")
lfe> (boolean? 'true)
true
lfe> (boolean? 'false)
true
lfe> (boolean? 'arthur)
false

Of course there is always the Erlang predicate, usable without having to do any includes:

lfe> (is_boolean 'true)
true
lfe> (is_boolean 'false)
true
lfe> (is_boolean 'arthur)
false

Note that, since LFE Booleans are also atoms, these are valid as well:

lfe> (atomp 'true)
true
lfe> (atom? 'false)
true
lfe> (is_atom 'true)
true

Characters

Characters in LFE Are represented internally by integers, however a literal syntax is offered for convenience:

lfe> #\a
97
lfe> #\A
65
lfe> #\ü
252
lfe> #\Æ
198

Converting

Since a character literal and integer are the same thing as far as LFE is concerned, there is no such thing as converting between a "char" and "ord" like there is in some other languages.

However, one can format an integer as a string by telling the class of format functions that the input is "character" type:

lfe> (io_lib:format "~c" `(198))
"Æ"

For merely printing to standard out instead of returning a value, one may use:

lfe> (lfe_io:format "~c~n" `(198))
Æ
ok
lfe> (io:format "~c~n" `(198))
Æ
ok

Operators

All operations that are valid for integers are valid for characters.

Predicates

All predicates that are valid for integers are valid for characters.

Primitive Types

This chapter covers the basic types of data available to LFE, upon which primatives rest all LFE libraries and applications.

• Integers
• Floats
• Atoms
• Booleans
• Characters

Each of these types has an LFE test function of the form TYPENAMEp (which wrap the respective Erlang is_TYPENAME function). These are used to perform type checks (especially common in guard expressions). These predicate functions will be covered in their respective type sections.

Lists

Bytes and Binaries

Tuples

Property Lists

Maps

Arrays

Dicts

Records

Pattern Matching

Generic Sequence Functions

Expressions

Functions

Closures

Evaluation

Flow of Control

Processes

Messages and Their Passing

Objects and Flavors

I/O

Accessing Files

Modules

Packages

Scripting with LFE

Creating LFE Projects

Using rebar3

Project Layout Conventions

Errors and Debugging

Writing Unit Tests

The Common Test Framework

The Propr Test Framework

The Compiler

Macros

Distributed LFE

Ports and Port Drivers

Servers

Clients

OTP Behaviours

Applications

Releases

Tables and Databases

Example OTP Project

Epilogue

Afterword

Glossary

Bibliography

Appendices

The Origins of Lisp

The Origins of Erlang

An Overview of LFE

Versions

Current

The current version of the LFE MACHINE MANUAL is written against LFE 2.0-dev.

Previous

After the LFE 2.0 release, you will see a link to content created as part of that effort here. At which point the "Current" version will be incremented.

Feedback and Documantation Bugs

If you would like to provide feedback about this guide, we would welcome the chance to improve the experience for everyone. Please create a ticket in the Github issue tracker. Be sure to give a full description so that we can best help you!