LFE MACHINE MANUAL

LFE MACHINE MANUAL

Adatped from multiple sources
by Duncan McGreggor and Robert Virding

Copyright

Published by Cowboys ‘N’ Beans Books

https://github.com/cnbbooks ◈ http://cnbb.pub/ ◈ info@cnbb.pub

First electronic edition published: 2020

Portions © 1974, David Moon

Portions © 1978-1981, Daniel Weinreb and David Moon

Portions © 1979-1984, Douglas Adams

Portions © 1983, Kent Pitman

Portions © 1992-1993, Peter Norvig and Kent Pitman

Portions © 2003-2020, Ericsson AB

Portions © 2008-2012, Robert Virding

Portions © 2010-2020, Steve Klabnik and Carol Nichols

Portions © 2013-2023, Robert Virding and Duncan McGreggor

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License

About the Cover

The LFE “Chinenual” cover is based upon the Lisp Machine Manual covers of the early 80s. The Lisp Machine Manual editions we have seen from 1977 and 1979 had only hand-typed title pages, no covers, so we’re not sure if the famous graphic/typographic design occurred any earlier than 1982. We’ve also been unable to discover who the original designer was, but would love to give them credit, should we find out.

The Original

The Software Preservation Group has this image on their site:

Bitsavers has a 3rd edition of the Chinenual with the full cover

The LFE Edition

Whole Cover

Back Cover

The Spine

Dedication

To all LFE Community members, Lispers, programmers as well as all our friends and families.

Preface

The original Lisp Machine Manual, the direct spiritiaul ancestor of the LFE Machine Manual, described both the language and the “operating system” of the Lisp Machine. The language was a dialect of Lisp called Zetalisp. Zetalisp was a direct descendant of MACLISP, created as a systems programming language for the MIT Lisp machines. This is of special note since Erlang was created as a systems programming language too. One of it’s co-creators, Robert Virding, created Lisp Flavoured Erlang (LFE) based upon his expereinces with Franz Lisp (which based largely upon MACLISP), Portable Standard Lisp (itself an experiment in systems programming), and ultimately in an implementation he made of Lisp Machine Flavors on top of VAX/VMS where he extensively utilized the Lisp Machine Manual.

As such, LFE has a very strong inheritance of systems programming from both parents, as it were. First and foremost, it is a BEAM language written on top of the Erlang VM and from which it strays very little. Secondly, it is a Lisp dialect. It is, however, entirely a systems programming language.

Which brings us back to Zetalisp and the Lisp Machine Manual. It seemed only fitting to base the LFE manual upon the fantastic work and docuentation that was done on Lisp systems programming in the 70s and 80s, work that so many of us treasure and adore and to which we still defer. Thus the machine that is OTP in the context and syntax of the LFE Lisp dialect is extensively documented in the LFE MACHINE MANUAL.

Forward

The original Lisp programming language was implemented nearly 65 years ago¹ at MIT (Massachusetts Institute of Technology), as part of the work done by John McCarthy and Marvin Minsky at the then-nascent AI (Artificial Intelligence) Lab. In the intervening half-century, the original Lisp evolved and experienced significant transformation in a technological diaspora originally fuelled by an explosion of research in the field of artificial intelligence. Through this, the industry witnessed 40 years of development where countless independent Lisp implementations were created. A small sampling of these number such favourites as Lisp 1.5, MacLisp, ZetaLisp, Scheme, Common Lisp, and ISLISP. However, the early 1990s saw the beginning of what would become the AI winter, and Lisp declined into obscurity – even notoriety – as graduating computer scientists were ushered into the “enterprise” world of Java, never to look back.

Except some did. By the early to mid-2000s, groundwork was being laid for what is now being recognized as a “Lisp renaissance.” The rediscovery of Lisp in the new millennium has led to the creation of a whole new collection of dialects, many implemented on top of other languages: Clojure on Java; LFE (Lisp Flavoured Erlang) and Joxa on Erlang; Hy on Python; Gherkin on Bash. New languages such as Go and Rust have several Lisp implementations, while JavaScript seems to gain a new Lisp every few years. While all of these ultimately owe their existence to the first Lisp, conceived in 1956² and defined in 1958³, they represent a new breed and a new techno-ecological niche for Lisp: bringing the power of meta-programming and the clarity of near syntaxlessness⁴ to established language platforms. Whereas in the past Lisp has been an either-or choice, the new era of Lisps represents a symbiotic relationship between Lisp and the language platforms or VMs (virtual machines) upon which they are implemented; you now can have both, without leaving behind the accumulated experiences and comfort of your preferred platform.

Just as the Lisp implementations of the 1960s were greatly impacted by the advent of time-sharing computer systems, the new Lisps mentioned above like Clojure and LFE have been deeply influenced not only by their underlying virtual machines, but – more importantly – by the world view which the creators and maintainers of those VMs espoused. For the Erlang ecosystem of BEAM (Bogdan/Björn’s Erlang Abstract Machine) languages, the dominant world view is the primacy of highly-concurrent, fault-tolerant, soft real-time, distributed systems. Erlang was created with these requirements in mind, and LFE inherits this in full. As such, LFE is more than a new Lisp; it is a language of power designed for creating robust services and systems. This point bears some discussion in order to properly prepare the intrepid programming language enthusiast who wishes to travel through the dimensions of LFE.

Unlike languages whose prototypical users were developers working in an interactive shell engaged in such tasks as solving math problems, Erlang’s prototypical “user” was a telecommunications device in a world where downtime was simply unacceptable. As such, Erlang’s requirements and constraints were very unusual when compared to most other programming languages of its generation.⁵ Erlang, and thus its constellation of dialects, was designed from the start to be a programming language for building distributed systems, one where applications created with it could survive network and systems catastrophes, millions of processes could be sustained on a single machine, where tens and hundreds of thousands of simultaneous network connections could be supported, where even a live, production deployment could have its code updated without downtime. Such is the backdrop against which the Erlang side of the LFE story unfolds – not only in the teaching and learning of it, but in its day-to-day use, and over time, in the minds of its developers.

When bringing new developers up to speed, this perspective is often overlooked or set aside for later. This is often done intentionally, since one doesn’t want to overwhelm or discourage a newcomer by throwing them into the “deep end” of distributed systems theory and the critical minutia of reliability. However, if we ignore the strengths of LFE when teaching it, we do our members as well as ourselves a disservice that leads to much misunderstanding and frustration: “Why is LFE so different? You can do X so much more simply in language Y”. It should be stated quite clearly in all introductory materials that the BEAM languages are not like other programming languages; in many ways, the less you rely upon your previous experiences with C, Java, Python, Ruby, etc., the better off you will be.

When compared to mainstream programming languages, Erlang’s development is akin to the divergent evolution of animals on a continent which has been completely isolated for hundreds of millions of years. For instance, programming languages have their “Hello, world”s and their “first project”s. These are like lap dogs for newcomers to the programming world, a distant and domesticated version of their far more powerful ancestors. Though each language has its own species of puppy to help new users, they are all loyal and faithful companions which share a high percentage of common genetic history: this is how you print a line, this is how you create a new project. Erlang – the Down Under of programming languages – has its “Hello, world”s and “first project”s, too. But in this case, the lapdog does not count the wolf in its ancestral line. It’s not even a canid.⁶ It’s a thylacine⁷ with terrifying jaws and and unfamiliar behaviours. Its “hello world” is sending messages to thousands of distributed peers and to nodes in supervised, monitored hierarchies. It has just enough familiarity to leave one feeling mystified by the differences and with the understanding that one is in the presence of something mostly alien.

That is the proper context for learning Erlang when coming from another programming language.

In LFE, we take that a step further by adding Lisp to the mix, supplementing a distributed programming language platform with the innovation laboratory that is Lisp. Far from making the learning process more difficult, this algebraic, time-honoured syntax provides an anchoring point, a home base for future exploration: it is pervasive and familiar, with very few syntactical rules to remember. We have even had reports of developers more easily learning Erlang via LFE.

In summary, by the end of this book we hope to have opened the reader’s eyes to a new world of promise: distributed systems programming with a distinctly 1950s flavour – and a capacity to create applications that will thrive for another 50 years in our continually growing, decentralized technological world.

Notes

1. The first draft of this forward which was written in 2015 said “almost 60 years ago” but was never published. Today, at the end of 2023, this content is finally seeing the light of day, with the origins of Lisp receding further into the past … ↩

2. See McCarthy’s 1979 paper History of Lisp, in particular the section entitled “LISP prehistory - Summer 1956 through Summer 1958”. ↩

3. Ibid., section “The implementation of LISP”. ↩

4. As you learn and then take advantage of Lisp’s power, you will find yourself regularly creating new favourite features that LFE lacks. The author has gotten so used to this capability that he has applied this freedom to other areas of life. He hopes that you can forgive the occasional English language hack. ↩

5. In fact, in the 1980s when Erlang was born, these features were completely unheard of in mainstream languages. Even today, the combination of features Erlang/OTP (Open Telecom Platform) provides is rare; an argument can be made that Erlang (including its dialects) is still the only language which provides them all. ↩

6. The family of mammals that includes dogs, wolves, foxes, and jackals, among others. ↩

7. An extinct apex predator and marsupial also known as the Tasmanian tiger or Tasmanian wolf. ↩

Acknowledgments

Thank you Robert Virding for all that you have done to make Lisp Flavoured Erlang a reality.

Part I - Getting Started

Every programming journey begins with a single step, and in the Lisp tradition, that step is often taken at the REPL—the Read-Eval-Print Loop that transforms programming from a batch process into an interactive conversation with the computer. Part I of this manual guides you through your first steps in LFE (Lisp Flavored Erlang), from understanding what makes this language unique to building your first complete application.

LFE occupies a fascinating position in the programming language landscape. It brings the elegant syntax and powerful abstractions of Lisp to the robust, fault-tolerant world of the Erlang Virtual Machine (BEAM). This combination offers something remarkable: a language that supports both the reflective, code-as-data philosophy of Lisp and the “let it crash” resilience of actor-model concurrency. Whether you’re drawn to LFE from the Lisp side or the Erlang side, you’ll discover that this synthesis creates new possibilities for building distributed, fault-tolerant systems.

We begin with the traditional “Hello, World!” programs, but in three distinct flavors that showcase LFE’s versatility. The REPL version demonstrates immediate interactivity—type an expression, see the result instantly. The main function approach shows how to create standalone programs, while the LFE/OTP version introduces you to the powerful supervision trees and process management that make Erlang applications famously reliable.

The heart of Part I is the guessing game walk-through, a complete project that evolves from simple procedural code to a proper (if very simple) OTP application. This progression mirrors the journey many LFE programs take: starting as interactive explorations at the REPL, growing into functions and modules, and ultimately becoming supervised processes that can handle failures gracefully and scale across multiple machines.

Finally, we dive deep into the REPL itself—not just as a place to test code fragments, but as a powerful development environment. The REPL is where LFE’s dual nature as both Lisp and Erlang becomes most apparent. You can manipulate code as data using traditional Lisp techniques, while simultaneously spawning processes, sending messages, and exploring the vast ecosystem of Erlang and OTP libraries.

By the end of Part I, you’ll have not just written your first LFE programs, but developed an intuition for the LFE way of thinking—interactive, incremental, and designed for systems that never stop running. You’ll understand how to leverage both the immediate feedback of Lisp development and the industrial-strength reliability of the BEAM platform, setting the foundation for the deeper explorations that follow in the subsequent parts of this manual.

Introduction

Far out in the uncharted backwaters of the unfashionable end of computer science known as “distributed systems programming” lies a small red e. Orbitting this at a distance roughly proportional to the inverse of the likelihood of it being noticed is an utterly insignificant little green mug filled with the morning beverage stimulant equivalent of That Old Janx Spirit. Upon that liquid floats a little yellow 𝛌 whose adherents are so amazingly primitive that they still think cons, car, and cdr are pretty neat ideas.

This is their book.

Their language, Lisp Flavoured Erlang (henceforth “LFE”), lets you use the archaic and much-beloved S-expressions to write some of the most advanced software on the planet. LFE is a general-purpose, concurrent, functional Lisp whose underlying virtual machine (Erlang) was designed to create distributed, fault-tolerant, soft-realtime, highly-availale, always-up, hot-swappable appliances, applications, and services. In addition to fashionable digital watches, LFE sports immutable data, pattern-matching, eager evaluation, and dynamic typing.

This manual will not only teach you what all of that means and why you want it in your breakfast cereal, but also: how to create LFE programs; what exactly are the basic elements of the language; the ins-and-outs of extremely efficient and beautiful clients and servers; and much, much more.

Note, however, that the first chapter is a little different than most other books, and is in fact different from the rest of the chapters in this manual. We wrote this chapter with two guiding thoughts: firstly and foremost, we wanted to provide some practical examples of code-in-action as a context in which a programmer new to LFE could continuously refer – from the very beginning through to the successful end – while learning the general principles of that language; secondly, most programming language manuals are dry references to individual pieces of a language or system, not representatives of the whole, and we wanted to provide a starting place for those who learn well via examples, who would benefit from a view toward that whole. For those who have already seen plenty of LFE examples and would rather get to down to the nitty-gritty, rest assured we desire your experience to be guilt-free and thus encourage you to jump into next chapter immediately!

This book is divided into 6 parts with the following foci:

• Introductory material
• Core data types and capabilities
• The basics of LFE code and projects
• Advanced language features
• The machine that is OTP
• Concluding thoughts and summaries

There is a lot of material in this book, so just take it section by section. If at any time you feel overwhelmed, simply set down the book, take a deep breath, fix yourself a cuppa, and don’t panic.

Welcome to the LFE MACHINE MANUAL, the definitive LFE reference.

About LFE

LFE is a modern programming language of two lineages, as indicated by the expansion of its acronym: Lisp Flavoured Erlang. In this book we aim to provide the reader with a comprehensive reference for LFE and will therefore explore both parental lines. Though the two language branches which ultimately merged in LFE are separated by nearly 30 years, and while LFE was created another 20 years after that, our story of their origins reveals that age simply doesn’t matter. More significantly, LFE has unified Lisp and Erlang with a grace that is both simple and natural. This chapter is a historical overview on how Lisp, Erlang, and ultimately LFE came to be, thus providing a broad context for a complete learning experience.

What Is LFE?

LFE is a Lisp dialect which is heavily flavoured by the programming language virtual machine upon which it rests, the Erlang VM.¹ Lisps are members of a programming language family whose typical distinguishing visual characteristic is the heavy use of parentheses and what is called prefix notation.² To give you a visual sense of the LFE language, here is some example code:

(defun remsp
  (('())
   '())
  (((cons #\  tail))
   (remsp tail))
  (((cons head tail))
   (cons head (remsp tail))))

This function removes spaces from a string that it passed to it. We will postpone explanation and analysis of this code, but in a few chapters you will have the knowledge necessary to understand this bit of LFE.

Besides the parentheses and prefix notation, other substantial features which LFE shares with Lisp languages include the interchangeability of data with code and the ability to write code which generates new code using the same syntax as the rest of the language. Examples of other Lisps include Common Lisp, Scheme, and Clojure.

Erlang, on the other hand, is a language inspired most significantly by Prolog and whose virtual machine supports not only Erlang and LFE, but also newer BEAM languages including Joxa, Elixir, and Erlog. BEAM languages tend to focus on the key characteristics of their shared VM: fault-tolerance, massive scalability, and the ability to easily build soft real-time systems.

One way of describing LFE is as a programming language which unites these two. LFE is a happy mix of the serious, fault-tolerant philosophy of Erlang combined with the flexibility and extensibility offered by Lisp dialects. It is a homoiconic distributed systems programming language with Lisp macros which you will soon come to know in much greater detail.

A Brief History

To more fully understand the nature of LFE, we need to know more about Lisp and Erlang – how they came to be and even more importantly, how they are used today. To do this we will cast our view back in time: first we’ll look at Lisp, then we’ll review the circumstances of Erlang’s genesis, and finally conclude the section with LFE’s beginnings.

The Origins of Lisp

Lisp, originally spelled LISP, is an acronym for “LISt Processor”. The language’s creator, John McCarthy, was inspired by the work of his colleagues who in 1956 created IPL (Information Processing Language), an assembly programming language based upon the idea of manipulating lists. Though initially intrigued with their creation, McCarthy’s interests in artificial intelligence required a higher-level language than IPL with a more general application of its list-manipulating features. So after much experimentation with FORTRAN, IPL, and heavy inspiration from Alonzo Church’s lambda calculus,³ McCarthy created the first version of Lisp in 1958.

In the 1950s and 1960s programming languages were actually created on paper, due to limited computational resources. Volunteers, grad students, and even the children of language creators were used to simulate registers and operations of the language. This is how the first version of Lisp was “run”.⁴ Furthermore, there were two kinds of Lisp: students in the AI lab wrote a form called S-expressions, which was eventually input into an actual computer. These instructions had the form of nested lists, the syntax that eventually became synonymous with Lisp. The other form, called M-expressions, was used when McCarthy gave lectures or presented papers.⁵ These had a syntax which more closely resembles what programmers expect. This separation was natural at the time: for over a decade programmers had entered instructions using binary or machine language while describing these efforts in papers using natural language or pseudocode. McCarthy’s students programmed entirely in S-expressions and as their use grew in popularity, the fate of M-expressions was sealed: they were never implemented.⁶

The Lisp 1.5 programmer’s manual, first published in 1962, used M-expressions extensively to introduce and explain the language. Here is an example function⁷ for removing spaces from a string input defined using M-expressions:

remsp[string] = [null[string]→F;
              eq[car[string];" "]→member[cdr[string]];
              T→cons[car[string];remsp[cdr[string]]]]

The corresponding S-expression is what the Lisp programmer would actually enter into the IBM 704 machine that was used by the AI lab at MIT:⁸

DEFINE ((
  (REMSP (LAMBDA (STRING)
    (COND ((NULL STRING)
           F)
          ((EQ (CAR STRING) " ")
           (REMSP (CDR STRING)))
          (T
           (CONS (CAR STRING)
                 (REMSP (CDR STRING)))))))))

The period from 1958 to 1962, when Lisp 1.5 was released, marked the beginning of a new era in computer science. Since then Lisp dialects have made an extraordinary impact on the design and theory of other programming languages, changing the face of computing history perhaps more than any other language group. Language features that Lisp pioneered include such significant examples as: homoiconicity, conditional expressions, recursion, meta-programming, meta-circular evaluation, automatic garbage collection, and first class functions. A classic synopsis of these accomplishments was made by the computer scientist of great renown, Edsger Dijkstra in his 1972 Turing Award lecture, where he said the following about Lisp:

“With a few very basic principles at its foundation, [Lisp] has shown a remarkable stability. Besides that, Lisp has been the carrier for a considerable number of, in a sense, our most sophisticated computer applications. Lisp has jokingly been described as ‘the most intelligent way to misuse a computer’. I think that description a great compliment because it transmits the full flavour of liberation: it has assisted a number of our most gifted fellow humans in thinking previously impossible thoughts.”

Lisp usage is generally described as peaking in the 80s and early 90s, experiencing an adoption setback with the widespread view that problems in artificial intelligence were far more difficult to solve than originally anticipated.⁹ Another problem which faced Lisp was related hardware requirements: specialized architectures were developed in order to provide sufficient computational power to its users. These were expensive with few vendors, and a slow product cycle.

In the midst of this Lisp cold-spell, two seminal Lisp books were published: On Lisp, and ANSI Common Lisp, both by famed entrepreneur Paul Graham.¹⁰ Despite a decade of decline, these events helped catalyse a new appreciation for the language by a younger generation of programmers and within a few short years, the number of Lisp books and Lisp-based languages began growing, giving the world the likes of Practical Common Lisp and Let Over Lambda in the case of the former, and Clojure and LFE, in the case of the latter.

Constructing Erlang

Erlang was born in the heart of Ericsson’s Computer Science Laboratory,¹¹ just outside of Stockholm, Sweden.¹² The lab had the general aim “to make Ericsson software activities as efficient as possible through the purposeful exploitation of modern technology.” The Erlang programming language was the lab’s crowning achievement, but the effort leading up to this was extensive with a great many people engaged in the creation of many prototypes and the use of numerous of programming languages.

One example of this is the work that Nabiel Elshiewy and Robert Virding did in 1986 with Parlog, a concurrent logic programming language based on Prolog. Though this work was eventually abandoned, you can read the paper The Phoning Philosopher’s Problem or Logic Programming for Telecommunications Applications and see the impact its features had on the future development of Erlang. The paper provides some examples of Parlog usage; using that for inspiration we can envision what our space-removing program would looking like:¹³

remsp([]) :-
  [].

remsp([$ |Tail]) :-
  remsp(Tail).

remsp([Head|Tail]) :-
  [Head|remsp(Tail)].

Another language that was part of this department-wide experimentation was Smalltalk. Joe Armstrong started experimenting with it in 1985 to model a telephone exchanges and used this to develop a telephony algebra with it. A year later, when his colleague Roger Skagervall showed him the equivalence between this and logic programming, Prolog began its rise to prominence, and the first steps were made towards the syntax of Erlang as the world now knows it. In modern Erlang, our program has the following form:

remsp([]) ->
  [];
remsp([$ |Tail]) ->
  remsp(Tail);
remsp([Head|Tail]) ->
  [Head|remsp(Tail)].

The members of the Ericsson lab who were tasked with building the next generation telephone exchange system, and thus involved with the various language experiments made over the course of a few years, came to the following conclusions:

• Small languages seemed better at succinctly addressing the problem space.
• The functional programming paradigm was appreciated, if sometimes viewed as awkward.
• Logic programming provided the most elegant solutions in the given problem space.
• Support for concurrency was viewed as essential.

If these were the initial guideposts for Erlang development, its guiding principles were the following:

• To handle high-concurrency
• To handle soft real-time constraints
• To support non-local, distributed computing
• To enable hardware interaction
• To support very large scale software systems
• To support complex interactions
• To provide non-stop operation
• To allow for system updates without downtime
• To allow engineers to create systems with only seconds of down-time per year
• To easily adapt to faults in both hardware and software

These were accomplished using such features as immutable data structures, light weight processes, no shared memory, message passing, supervision trees, and heartbeats. Furthermore, having adopted message-passing as the means of providing high-concurrency, Erlang slowly evolved into the exemplar of a programming paradigm that it essentially invented and even today, dominates: concurrent functional programming.

After four years of development from the late 80s into the early 90s, Erlang matured to the point where it was adopted for large projects inside Ericsson. In 1998 it was released as open source software, and has since seen growing adoption in the wider world of network- and systems-oriented programming.

The Birth of LFE

One of the co-inventors of Erlang, and part of the Lab’s early efforts in language experimentation was Robert Virding. Virding first encountered Lisp in 1980 when he started his PhD in theoretical physics at Stockholm University. His exposure to the language came as a result of the physics department’s use in performing symbolic algebraic computations. Despite this, he spent more time working on micro-processor programming and didn’t dive into it until a few years later when he was working at Ericsson’s Computer Science Laboratory. One of the languages evaluated for use in building telephony software was Lisp, but to do so properly required getting to know it in-depth – both at the language level as well as the operating system level.¹⁴ It was in this work that Virding’s passion for Lisp blossomed and he came to appreciate deeply its functional nature, macros, and homoiconicity – all excellent and time-saving tools for building complicated systems.

Though the work on Lisps did not become the focus of Erlang development, the seeds of LFE were planted even before Erlang itself had come to be. After 20 years of contributions to the Erlang programming language, these began to bear fruit. In 2007 Virding decided to do something fun in his down time: to see what a Lisp would look like if written on top of the Prolog-inspired Erlang VM. After several months of hacking, he announced a first version of LFE to the Erlang mail list in early 2008.

A few years latter, when asked about the origins of LFE and the motivating elements behind his decision to start the project, Virding shared the following on the LFE mail list:

• It had always been a goal of Robert’s to make a Lisp which could fully interact with Erlang/OTP, to see what it would look like and how it would run.
• He was looking for some interesting programming projects that were not too large to do in his spare time.
• He thought it would be a fun, open-ended problem to solve with many interesting parts.
• He likes implementing languages.

We showed an example of LFE at the beginning of this chapter; in keeping with our theme for each language subsection, we present it here again, though in a slightly altered form:

(defun remsp
  (('())
   '())
  ((`(32 . ,tail))
   (remsp tail))
  ((`(,head . ,tail))
   (cons head (remsp tail))))

What is LFE Good For?

Very few languages have the powerful capabilities which Erlang offers – both in its standard library as well as the set of Erlang libraries, frameworks, and patterns that are provided in OTP. This covers everything from fault-tolerance, scalability, soft real time capacity, and high-availability to proper design, component assembly, and deployment in distributed environments.

Similarly, despite the impact that Lisp has had on so many programming languages, its full suite of features is still essentially limited to Lisp dialects. This includes the features we have already mentioned: the ability to treat code as data, easily generate new code from data, as well as the interrelated power of writing macros – the last allows developers to modify the language to suit their needs. These rare features from two different language branches are unified in LFE and there is no well-established language that provides the union of these.

As such, LFE gives developers everything they need to envision, prototype, and then build distributed applications – ones with unique requirements that no platform provides and which can be delivered thanks to LFE’s language-building capabilities.

To paraphrase and augment the opening of Chapter 1 in Designing for Scalability with Erlang/OTP:

“You need to implement a fault tolerant, scalable soft real time system with requirements for high availability. It has to be event driven and react to external stimulus, load and failure. It must always be responsive. You also need language-level features that don't exist yet. You would like to encode your domain's best practices and design patterns seamlessly into your chosen platform.”

LFE has everything you need to realize this dream … and so much more.

In Summary

What LFE Is

Here’s what you can expect of LFE:

• A proper Lisp-2, based on the features and limitations of the Erlang VM
• Compatibility with vanilla Erlang and OTP
• It runs on the standard Erlang VM

Furthermore, as a result of Erlang’s influence (and LFE’s compatibility with it), the following hold:

• there is no global data
• data is not mutable
• only the standard Erlang data types are used
• you get pattern matching and guards
• you have access to Erlang functions and modules
• LFE has a compiler/interpreter
• functions with declared arity and fixed number of arguments
• Lisp macros

What LFE Isn’t

Just to clear the air and set some expectations, we’ll go a step further. Here’s what you’re not going to find in LFE:

• An implementation of Scheme
• An implementation of Common Lisp
• An implementation of Clojure

As such, you will not find the following:

• A Scheme-like single namespace
• CL packages or munged names faking packages
• Access to Java libraries

Notes

1. Robert Virding, the creator of LFE and one of the co-creators of the Erlang programming language, has previously stated that, were he to start again, he would name his Lisp dialect EFL, since it truly is a Lisp with an Erlang flavour, rather than the other way round. ↩

2. We will be covering prefix notation when we cover symbolic expressions later in the book. ↩

3. Alonzo Church was one of McCarthy’s professors at Princeton. McCarthy did not use all of the lambda calculus when creating Lisp, as there were many esoteric aspects for which he had no practical need. ↩

4. In the case of Lisp, university students were the primary computer hardware … and sometimes even high school students (see REPL footnote below). ↩

5. This approach was not uncommon at the time: the ALGOL 58 specification defined a syntax for the language reference, one for publications, and a third for implementation. ↩

6. The single greatest contributor to the ascendance of the S-expression is probably the invention of the REPL by L Peter Deutsch, which allowed for interactive Lisp programming. This was almost trivial in S-expressions, whereas a great deal of effort would have been required to support a similar functionality for M-expressions. ↩

7. The function we use in this chapter to demonstrate various syntaxes and dialects was copied from the cover of Byte Magazine’s August 1979 issue which focused on Lisp and had part of a Lisp 1.5 program on its cover. ↩

8. The formatting applied to the S-expression version of the function is a modern convention, added here for improved readability. There was originally no formatting, since there was no display – a keypunch was used to enter text on punchcards, 80 characters at a time. As such, a more historically accurate representation would perhaps be: DEFINE (((REMSP (LAMBDA (STRING) (COND ((NULL STRING) F) ((EQ (CAR STRING) " ") (REMSP (CDR STRING))) (T (CONS (CAR STRING) (REMSP (CDR STRING))))))))) ↩

9. This time period is commonly referred to as the “AI winter”. ↩

10. Paul Graham sold his Lisp-based e-commerce startup to Yahoo! In 1998. ↩

11. The majority of this section’s content was adapted from Joe Armstrong’s paper “A History of Erlang” by, written for the HOPL III conference in 2007. ↩

12. The Computer Science Laboratory operated from 1982 to 2002 in Älvsjö, Stockholm. ↩

13. We’ve taken the liberty of envisioning the Parlog of 1986 as one that supported pattern matching on characters. ↩

14. One of Virding’s project aims was to gain a deeper understanding of Lisp internals. As part of this, he ported the Lisp Machine Lisp object framework Flavors to Portable Standard Lisp running on UNIX. His work on this project contributed to his decision to use Flavour as part of the name for LFE (spelling divergence intentional). ↩

Prerequisites

Anyone coming to LFE should have experience programming in another language, ideally a systems programming language, especially if that language was found lacking. If the corageous reader is attmping to use LFE as a means of entering the study of computer science, we might offer several other paths of study which may bear fruit more quickly and with less pain.

No prior Lisp experience is required, but that would certinaly be helpful. The same goes for Erlang/OTP (or any of the suite of BEAM languages). The reader with experience writing concurrent applications, wrestling with fault-tolerance, or maintaining highly-available applications and services does receive bonus points for preparedness. Such well-prepared readers landing here may have, in fact, done so due to a quest for a distributed Lisp. For those whom this does apply, your quest has found its happy end.

This book assumes the reader has the following installed upon their system:

• a package manager for easily installing software (in particular, development tools and supporting libraries)
• git, make, and other core open source software development tools
• a modern version of Erlang (as of the writing of this book, that would include versions 19 through 23); the rebar3 documentation has great suggestions on what to use here, depending upon your need
• the rebar3 build tool for Erlang (and other BEAM languages); see its docs for installation instructions

Conventions

Typography

Key Entry

We use the angle bracket convention to indicate typing actual key on the keyboard. For instance, when the reader sees <ENTER> they should interpret this as an actual key they should type. Note that all keys are given in upper-case. If the reader is expected to use an upper-case “C” instead of a lower-case “c”, they will be presented with the key combination <SHIFT><C>.

Code

Color syntax highlighting is used in this text to display blocks of code. The formatting of this display is done in such a way as to invoke in the mind of the reader the feeling of a terminal, thus making an obvious visual distinction in the text. For instance:

(defun fib
  ((0) 0)
  ((1) 1)
  ((n)
    (+ (fib (- n 1))
       (fib (- n 2)))))

Examples such as this one are readily copied and may be pasted without edit into a file or even the LFE REPL itself.

For interactive code, we display the default LFE prompt the reader will see when in the REPL:

lfe> (integer_to_list 42 2)
;; "101010"

We also distinguish the output from the entered LFE code using code comments displayed afer the command.

For shell commands, the commands to enter at the prompt are prefixed by a $ for the prompt. Input and any relevant output are provided as comment strings:

$ echo "I am excited to learn LFE"
# I am excited to learn LFE

LiffyBot

This is a severly hoopy frood. With an attitude. He pops up from time to time, generally with good advice. Or simply as a marker for something the authors hope you will pay special note.

Messages of Note

From time to time you will see call-out boxes, aimed at drawing your attention to something of note. There are four differnt types of these:

• ones that share useful info (blue)
• ones that highlight something of a momentus nature (green)
• ones that offer warnings to tred carefully (orange)
• ones that beg you not to follow a particular path (red)

These messages will take the following forms:

Development Setup

rebar3 Configuration

Having followed the notes and linked instructions in the Prerequisites section, you are ready to add global support for the LFE rebar3 plugin.

First, unless you have configured other rebar3 plugins on your system, you will need to create the configuration directory and the configuration file:

$ mkdir ~/.config/rebar3
$ touch ~/.config/rebar3/rebar.config

Next, open up that file in your favourite editor, and give it these contents:

{plugins, [
  {rebar3_lfe, "0.5.5"}
]}.

{deps, [
  {lfe, "2.2.0"}
]}.

If you already have a rebar.config file with a plugins entry, then simply add a comma after the last plugin listed and paste the {rebar3_lfe, ...} line from above (with no trailing comma!). When a new version of rebar3_lfe is released, you can follow the instructions in the rebar3_lfe repo to upgrade.

A Quick Test with the REPL

With the LFE rebar3 plugin successfully configured, you should be able to start up the LFE REPL anywhere on your system with the following:

$ rebar3 lfe repl

Erlang/OTP 23 [erts-11.0] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [hipe]

   ..-~.~_~---..
  (      \\     )    |   A Lisp-2+ on the Erlang VM
  |`-.._/_\\_.-':    |   Type (help) for usage info.
  |         g |_ \   |
  |        n    | |  |   Docs: http://docs.lfe.io/
  |       a    / /   |   Source: http://github.com/rvirding/lfe
   \     l    |_/    |
    \   r     /      |   LFE v1.3-dev (abort with ^G)
     `-E___.-'

lfe>

Exit out of the REPL for now by typing <CTRL><G> and then <Q>.

‘Hello, World!’

Hello-World style introductory programs are intended to give the prospective programmer for that language a sense of what it is like to write a minimalist piece of software with the language in question. In particular, it should show off the minimum capabilitiues of the language. Practically, this type of program should sigify to the curious coder what they could be in for, should they decide upon this particular path.

In the case of LFE/OTP, a standard Hello-World program (essentially a “print” statement) is extremely misleading; more on that in the OTP version of the Hello-World program. Regardless, we concede to conventional practice and produce a minimal Hello-World that does what many other languages’ Hello-World programs do. We do, however, go further afterwards …

From the REPL

As previously demonstrated, it is possible to start up the LFE ‘read-eval-print loop’ (REPL) using rebar3:

$ rebar3 lfe repl

Once you are at the LFE prompt, you may write a simple LFE “program” like the following:

lfe> (io:format "~p~n" (list "Hello, World!"))

Or, for the terminally lazy:

lfe> (io:format "~p~n" '("Hello, World!"))

While technically a program, it is not a very interesting one: we didn’t create a function of our own, nor did we run it from outside the LFE interactive programming environment.

Let’s address one of those points right now. Try this:

lfe> (defun hello-world ()
lfe>   (io:format "~p~n" '("Hello, World!")))

This is a simple function definition in LFE.

We can run it by calling it:

lfe> (hello-world)
;; "Hello, World!"
;; ok

When we execute our hello-world function, it prints our message to standard-output and then lets us know everything really quite fine with a friendly ok.

Now let’s address the other point: running a Hello-World programming from outside LFE.

Hit <CTRL-G><Q> to exit the REPL and get back to your terminal.

From the Command Line

From your system shell prompt, run the following to create a new project that will let us run a Hello-World program from the command line:

$ rebar3 new lfe-main hello-world
$ cd ./hello-world

Once in the project directory, you can actually just do this:

$ rebar3 lfe run

You will see code getting downloaded and compiled, and then your script will run, generating the following output:

Running script '/usr/local/bin/rebar3' with args [] ...
'hello-world

When you created a new LFE project of type ‘main’, a Hello-World function was automatically generated for you, one that’s even simpler than what we created in the previous section:

(defun my-fun ()
  'hello-world)

The other code that was created when we executed rebar3 new lfe-main hello-world was a script meant to be used by LFE with LFE acting as a shell interpreter:

#!/usr/bin/env lfescript

(defun main (args)
  (let ((script-name (escript:script_name)))
    (io:format "Running script '~s' with args ~p ...~n" `(,script-name ,args))
    (io:format "~p~n" `(,(hello-world:my-fun)))))

You may be wondering about the args argument to the main function, and the fact that the printed output for the args when we ran this was []. Let’s try something:

$ rebar3 lfe run -- Fenchurch 42

Running script '/usr/local/bin/rebar3' with args [<<"Fenchurch">>,<<"42">>] ...
'hello-world'

We have to provide the two dashes to let rebar3 know that we’re done with it, that the subsequent argsuments are not for it, but rather for the program we want it to start for us. Using it causes everything after the -- to be passed as arguments to our script.

As for the code itself, it’s tiny. But there is a lot going on just with these two files. Have no fear, though: the remainder of this book will explore all of that and more. For now, know that the main function in the executable is calling the hello-world module’s my-fun function, which takes no arguments. To put another way, what we really have here is a tiny, trivial library project with the addition of a script that calls a function from that library.

For now just know that an executable file which starts with #!/usr/bin/env lfescript and contains a main function accepting one argument is an LFE script capable of being executed from the command line – as we have shown!

LFE/OTP ‘Hello, World!’

What have been demonstrated so far are fairly vanilla Hello-World examples; there’s nothing particularly interesting about them, which puts them solidly in the company of the millions of other Hello-World programs. As mentioned before, this approach is particularly vexing in the case of LFE/OTP, since it lures the prospective developer into the preconception that BEAM languages are just like other programming languages. They most decidedly are not.

What makes them, and in this particular case LFE, special is OTP. There’s nothing quite like it, certainly not another language with OTP’s feature set baked into its heart and soul. Most useful applications you will write in LFE/OTP will be composed of some sort of long-running service or server, something that manages that server and restarts it in the event of errors, and lastly, a context that contains both – usually referred to as the “application” itself.

As such, a real Hello-World in LFE would be honest and let the prospective developer know what they are in for (and what power will be placed at their fingertips). That is what we will show now, an LFE OTP Hello-World example.

If you are still in the directory of the previous Hello-World project, let’s get out of that:

cd ../

Now we’re going to create a new project, one utilising the some very basic OTP patterns:

rebar3 new lfe-app hello-otp-world
cd ./hello-otp-world

We won’t look at the code for this right now, since there are chapters dedicated to that in the second half of the book. But let’s brush the surface with a quick run in the REPL:

rebar3 lfe repl

To start your new hello-world application, use the OTP application module:

lfe> (application:ensure_all_started 'hello-otp-world)
;; #(ok (hello-otp-world))

That message lets you know that not only was the hello-otp-word application and server started without issue, any applications upon which it depends were also started. Furthermore, there is a supervisor for our server, and it has started as well. Should our Hello-World server crash for any reason, the supervisor will restart it.

To finish the demonstration, and display the clichéd if classic message:

(hello-otp-world:echo "Hello, OTP World!")
;; "Hello, OTP World!"

And that, dear reader, is a true LFE/OTP Hello-World program, complete with message-passing and pattern-matching!

Feel free to poke around in the code that was generated for you, but know that eventually all its mysteries will be revealed, and by the end of this book, that program’s magic will just seem like ordinary code to you, ordinary, dependable, fault-tolerant, highly-availble, massively-concurrent code.

Walk-through: An LFE Guessing Game

Now that you’ve seen some LFE in action, let’s do something completely insane: write a whole game before we even know the language!

We will follow the same patterns established in the Hello-World examples, so if you are still in one of the Hello-World projects, change directory and then create a new LFE project:

$ cd ../
$ rebar3 new lfe-app guessing-game
$ cd ./guessing-game

We will create this game by exploring functions in the REPL and then saving the results in a file. Open up your generated project in your favourite code-editing application, and then open up a terminal from your new project directory, and start the REPL:

$ rebar3 lfe repl

Planning the Game

We’ve created our new project, but before we write even a single atom of code, let’s take a moment to think about the problem and come up with a nice solution. By doing this, we increase our chances of making something both useful and elegant. As long as what we write remains legible and meets our needs, the less we write the better. This sort of practice elegance will make the code easier to maintain and reduce the chance for bugs (by the simple merrit of there being less code in which a bug may arise; the more code, the greater opportunities for bugs).

Our first step will be making sure we understand the problem and devising some minimal abstractions. Next, we’ll think about what actually need to happen in the game. With that in hand, we will know what state we need to track. Then, we’re off to the races: all the code will fall right into place and we’ll get to play our game.

Key Abstractions

In a guessing game, there are two players: one who has the answer, and one who seeks the answer. Our code and data should clearly model these two players.

Actions

The player with the answer needs to peform the following actions:

1. At the beginning of the game, state the problem and tell the other player to start guessing
2. Receive the other player’s guess
3. Check the guess against the answer
4. Report back to the other player on the provided guess
5. End the game if the guess was correct

The guessing player needs to take only one action:

1. guess!

State

We need to track the state of the game. Based upon the actions we’ve examined, the overall state is very simple. Through the course of the game, will only need to preserve the answer that will be guessed.

Code Explore

Now that we’ve thought through our problem space clearly and cleanly, let’s do some code exploration and start defining some functions we think we’ll need.

We’ve already generated an OTP application using the LFE rebar3 plugin, and once we’ve got our collection of functions that address the needed game features, we can plug those into the application.

We’ll make those changes in the code editor you’ve opened, and we’ll explore a small set of possible functions to use for this using the project REPL session you’ve just started.

Getting User Input

How do we get user input in LFE? Like this!

lfe> (io:fread "Guess number: " "~d")

This will print the prompt Guess number: and then await your input and the press of the <ENTER> key. The input you provide needs to match the format type given in the second argument. In this case, the ~d tells us that this needs to be a decimal (base 10) integer.

fe> (io:fread "Guess number: " "~d")
;; Guess number: 42
;; #(ok "*")

If we try typing something that is not a base 10 integer, we get an error:

lfe> (io:fread "Guess number: " "~d")
;; Guess number: forty-two
;; #(error #(fread integer))

With correct usage, how do we capture the value in a variable? The standard way to do this in LFE is destructuring via pattern matching. The following snippet extracts the value and then prints the extracted value in the REPL:

lfe> (let ((`#(ok (,value)) (io:fread "Guess number: " "~d")))
lfe>   (io:format "Got: ~p~n" `(,value)))
;; Guess number: 42
;; Got: 42
;; ok

We’ll talk a lot more about pattern matching in the future, as well as the meaning of the backtick and commas. For now,let’s keep pottering in the REPL with these explorations, and make a function for this:

lfe> (defun guess ()
lfe>   (let ((`#(ok (,value)) (io:fread "Guess number: " "~d")))
lfe>     (io:format "You guessed: ~p~n" `(,value))))

And call it:

lfe> (guess)
;; Guess number: 42
;; You guessed: 42
;; ok

Checking the Input

In LFE there are several ways in which you can perform checks on values:

• the if form
• the cond form
• the case form
• pattern-matching and/or guards in function heads

The last one is commonly used in LFE when passing messages / data between functions. Our initial, generated project code is already doing this, and given the game state data we will be working with, this feels like a good fit what we need to implement.

Normally records are used for application data, but since we just care about the value of two integers (the number selected for the answer and the number guessed by the player), we’ll keep things simple in this game:

(set answer 42)

Let’s create a function with a guard:

lfe> (defun check
lfe>   ((guess) (when (< guess answer))
lfe>    (io:format "Guess is too low~n")))

The extra parenthesis around the function’s arguments is due to the use of the pattern-matching form of function definition we’re using here. We need this form, since we’re going to use a guard. The when after the function args is called a “guard” in LFE. As you might imagine, we could use any number of these.

lfe> (check 10)
;; Guess is too low
;; ok

Let’s add some more guards for the other checks we want to perform:

lfe> (defun check
lfe>   ((guess) (when (< guess answer))
lfe>    (io:format "Guess is too low~n"))
lfe>   ((guess) (when (> guess answer))
lfe>    (io:format "Guess is too high~n"))
lfe>   ((guess) (when (== guess answer))
lfe>    (io:format "Correct!~n")))

lfe> (check 10)
;; Guess is too low
;; ok

lfe> (check 100)
;; Guess is too high
;; ok

lfe> (check 42)
;; Correct!
;; ok

This should give a very general sense of what is possible.

Integrating into an Application

We’re only going to touch one of the files that was generated when you created the guessing-game project: ./src/guessing-game.lfe. You can ignore all the others. Once we’ve made all the changes summarized below, we will walk through this file at a high level, discussing the changes and how those contribute to the completion of the game.

First though, we need to reflect on the planning we just did, remembering the actions and states that we want to support. There’s also another thing to consider, since we’re writing this as is an always-up OTP app. With some adjustments for state magagement, it could easily be turned into something that literally millions of users could be accessing simultaneouslyi. So: how does a game that is usually implemented as a quick CLI toy get transformed in LFE/OTP such that it can be run as a server?

In short, we’ll use OTP’s gen_server capability (“behaviour”) and the usual message-passing practices. As such, the server will need to be able to process the following messages:

• #(start-game true) (create a record to track game state)
• #(stop-game true) (clear the game state)
• #(guess n)
  • check for guess equal to the answer
  • greater than the answer, and
  • less than the answer

We could have just used atoms for the first two, and done away with the complexity of using tuples for those, but symmetry is nice :-)

To create the game, we’re going to need to perform the following integration tasks:

• Update the handle_cast function to process the commands and guards we listed above
• Create API functions that cast the appropriate messages
• Update the export form in the module definition
• Set the random seed so that the answers are different every time you start the application

handle_cast

The biggest chunk of code that needs to be changed is the handle_cast function. Since our game doesn’t return values, we’ll be using handle_cast. (If we needed to have data or results returned to us in the REPL, we would have used handle_call instead. Note that both are standard OTP gen_server callback functions.)

The generated project barely populates this function and the function isn’t of the form that supports patten-matching (which we need here) so we will essentially be replacing what was generated. In the file ./src/guessing-game.lfe, change this:

(defun handle_cast (_msg state)
  `#(noreply ,state))

to this:

(defun handle_cast
  ((`#(start-game true) _state)
   (io:format "Guess the number I have chosen, between 1 and 10.~n")
   `#(noreply ,(random:uniform 10)))
  ((`#(stop-game true) _state)
   (io:format "Game over~n")
   '#(noreply undefined))
  ((`#(guess ,n) answer) (when (== n answer))
   (io:format "Well-guessed!!~n")
   (stop-game)
   '#(noreply undefined))
  ((`#(guess ,n) answer) (when (> n answer))
   (io:format "Your guess is too high.~n")
   `#(noreply ,answer))
  ((`#(guess ,n) answer) (when (< n answer))
   (io:format "Your guess is too low.~n")
   `#(noreply ,answer))
  ((_msg state)
   `#(noreply ,state)))

That is a single function in LFE, since for every match the arity of the function remains the same. It is, however, a function with six different and separate arguement-body forms: one for each pattern and/or guard.

These patterns are matched:

1. start
2. stop
3. guess (three times)
4. any

For the three guess patterns (well, one pattern, really) since there are three different guards we want placed on them:

1. guess is equal
2. guess is greater
3. guess is less

Note that the pattern for the function argument in these last three didn’t change, only the guard is different between them.

Finally, there’s the original “pass-through” or “match-any” pattern (this is used to prevent an error in the event of an unexpected message type).

Game API

In order to send a message to a running OTP server, you use special OTP functions for the type of server you are running. Our game is running a gen_server so we’ll be using that OTP module to send messages, in particular we’ll be calling gen_server:cast. However, creating messages and sending them via the appropriate gen_server function can get tedious quickly, so it is common practice to create API functions that do these things for you.

In our case, we want to go to the section with the heading ;;; our server API and add the following:

(defun start-game ()
  (gen_server:cast (SERVER) '#(start-game true)))

(defun stop-game ()
  (gen_server:cast (SERVER) '#(stop-game true)))

(defun guess (n)
  (gen_server:cast (SERVER) `#(guess ,n)))

Functions in LFE are private by default, so simply adding these functions doesn’t make them publicly accessible. As things now stand these will not be usable outside their module; if we want to use them, e.g., from the REPL, we need to export them.

Go to the top of the guessing-game module and update the “server API” sectopm of the exports, chaning this:

    ;; server API
    (pid 0)
    (echo 1)))

to this:

    ;; server API
    (pid 0)
    (echo 1)
    (start-game 0)
    (stop-game 0)
    (guess 1)))

The final form of your module definition should look like this:

(defmodule guessing-game
  (behaviour gen_server)
  (export
    ;; gen_server implementation
    (start_link 0)
    (stop 0)
    ;; callback implementation
    (init 1)
    (handle_call 3)
    (handle_cast 2)
    (handle_info 2)
    (terminate 2)
    (code_change 3)
    ;; server API
    (pid 0)
    (echo 1)
    (start-game 0)
    (stop-game 0)
    (guess 1)))

Now our game functions are public, and we’ll be able to use them from the REPL.

Finishing Touches

There is one last thing we can do to make our game more interesting. Right now, the game will work. But every time we start up the REPL and kick off a new game, the same “random” number will be selected for the answer. In order to make things interesting, we need to generate a random seed when we initialize our server.

We want to only do this once, though – not every time the game starts, and certainly not every time a user guesses! When the LFE server supervisor starts our game server, one functions is called and called only once: init/1. That’s where we want to make the change to support a better-than-default random seed.

Let’s change that function:

(defun init (state)
  `#(ok ,state))

to this:

(defun init (state)
  (random:seed (erlang:phash2 `(,(node)))
               (erlang:monotonic_time)
               (erlang:unique_integer))
  `#(ok ,state))

Now we’re ready to play!

Playing the Game

If you are still in the REPL, quit out of it so that rebar3 can rebuild our changed module. Then start it up again:

$ rebar3 lfe repl

Once at the LFE propmpt, start up the application:

lfe> (application:ensure_all_started 'guessing-game)

With the application and all of its dependencies started, we’re ready to start the game and play it through:

lfe> (guessing-game:start-game)
;; ok
;; Guess the number I have chosen, between 1 and 10.

lfe> (guessing-game:guess 10)
;; ok
;; Your guess is too high.

lfe> (guessing-game:guess 1)
;; ok
;; Your guess is too low.

lfe> (guessing-game:guess 5)
;; ok
;; Your guess is too low.

lfe> (guessing-game:guess 7)
;; ok
;; Your guess is too low.

lfe> (guessing-game:guess 8)
;; ok
;; Well-guessed!!
;; Game over

Success! You've just done something pretty amazing, if still mysterious: you've not only created your first OTP application running a generic server, you've successully run it through to completion!

Until we can dive into all the details of what you've seen in this walkthrough, much of what you've just written will seem strange and maybe even overkill. For now, though, we'll mark a placeholder for those concepts: the next section will briefly review what you've done and indicate which parts of this book will provide the remaining missing pieces.

Review

We’ve got the whole rest of the book ahead of us to cover much of what you’ve seen in the sample project we’ve just created with our guessing game. In the coming pages, you will revisit every aspect of what you’ve seen so far in lots of detail with correspondingly useful instructions on these matters.

That being said, it would be unfair to not at least read through the code together and mention the high-level concepts involved. Since we only touched the code in one file, that will be the one that gets the most of our attention for this short review, but let’s touch on the others here, too.

Project Files

rebar.config

This is the file you need in every LFE project you will write in order to take advantage of the features (and time-savings!) that rebar3 provides. For this project, the two important parts are:

1. the entry for dependencies (only LFE in this case), and
2. the plugins entry for the LFE rebar3 plugin.

Project setup will be covered in Chapter XXX, section XXX.

Source Files

The source files for our sample program in this walkthrough are for an OTP application. OTP-based projects will be covered in Chapter XXX, section XXX.

.app.src

This file is mostly used for application metadata. Most of what our app uses in this file is pretty self-explanatory. Every LFE application will have one of these in the project source code. Every LFE library and application needs this file.

guessing-game-app.lfe

This is the top-level file for our game, an OTP application. It only exports two functions: one to start the app and the other to stop it. The application is responsible for starting up whatever supervisors all your services/servers need. For this sample application, only one supervisor is needed (with a very simple supervision tree).

guessing-game-sup.lfe

This module is a little more invloved and has all the configuration and code necessary to properly set up a supervisor for our server. When something goes wrong with our server, the restart strategy defined by our supervisor will kick in and get things back up and running again. This is one of the key secrets to OTP’s wizardry, and we will be covering this in great detail later.

src/guessing-game.lfe

This is the last file we’ll look at, and is the one we’ll cover in the most detail right now. Here’s the entire content of what we created for our game:

(defmodule guessing-game
  (behaviour gen_server)
  (export
   ;; gen_server implementation
   (start_link 0)
   (stop 0)
   ;; callback implementation
   (init 1)
   (handle_call 3)
   (handle_cast 2)
   (handle_info 2)
   (terminate 2)
   (code_change 3)
   ;; server API
   (pid 0)
   (echo 1)
   (start-game 0)
   (stop-game 0)
   (guess 1)))

;;; ----------------
;;; config functions
;;; ----------------

(defun SERVER () (MODULE))
(defun initial-state () '#())
(defun genserver-opts () '())
(defun unknown-command () #(error "Unknown command."))

;;; -------------------------
;;; gen_server implementation
;;; -------------------------

(defun start_link ()
  (gen_server:start_link `#(local ,(SERVER))
                         (MODULE)
                         (initial-state)
                         (genserver-opts)))

(defun stop ()
  (gen_server:call (SERVER) 'stop))

;;; -----------------------
;;; callback implementation
;;; -----------------------

(defun init (state)
  (random:seed (erlang:phash2 `(,(node)))
               (erlang:monotonic_time)
               (erlang:unique_integer))
  `#(ok ,state))

(defun handle_cast
  ((`#(start-game true) _state)
   (io:format "Guess the number I have chosen, between 1 and 10.~n")
   `#(noreply ,(random:uniform 10)))
  ((`#(stop-game true) _state)
   (io:format "Game over~n")
   '#(noreply undefined))
  ((`#(guess ,n) answer) (when (== n answer))
   (io:format "Well-guessed!!~n")
   (stop-game)
   '#(noreply undefined))
  ((`#(guess ,n) answer) (when (> n answer))
   (io:format "Your guess is too high.~n")
   `#(noreply ,answer))
  ((`#(guess ,n) answer) (when (< n answer))
   (io:format "Your guess is too low.~n")
   `#(noreply ,answer))
  ((_msg state)
   `#(noreply ,state)))

(defun handle_call
  (('stop _from state)
   `#(stop shutdown ok state))
  ((`#(echo ,msg) _from state)
   `#(reply ,msg state))
  ((message _from state)
   `#(reply ,(unknown-command) ,state)))

(defun handle_info
  ((`#(EXIT ,_from normal) state)
   `#(noreply ,state))
  ((`#(EXIT ,pid ,reason) state)
   (io:format "Process ~p exited! (Reason: ~p)~n" `(,pid ,reason))
   `#(noreply ,state))
  ((_msg state)
   `#(noreply ,state)))

(defun terminate (_reason _state)
  'ok)

(defun code_change (_old-version state _extra)
  `#(ok ,state))

;;; --------------
;;; our server API
;;; --------------

(defun pid ()
  (erlang:whereis (SERVER)))

(defun echo (msg)
  (gen_server:call (SERVER) `#(echo ,msg)))

(defun start-game ()
  (gen_server:cast (SERVER) '#(start-game true)))

(defun stop-game ()
  (gen_server:cast (SERVER) '#(stop-game true)))

(defun guess (n)
  (gen_server:cast (SERVER) `#(guess ,n)))

The beginning of the file opens with a declaration of the module: not only its name, but the public functions we want to expose as part of our API. This will be covered in Chapter XXX, section XXX.

Next, we have a few constant functions. Functions are necessary here due to the fact that LFE does not have global variables. This will be covered in Chapter XXX, section XXX.

Then we define the functions that will be used as this module’s implementation of a generic OTP server. There is some boilerplate here that will be discussed when we dive into LFE/OTP. This will be covered in Chapter XXX, section XXX.

After that, we define the functions that are used by the OTP machinery that will run our server. Here you see several examples of pattern matching function heads in LFE, a very powerful feature that lends itself nicely to consise and expressive code. This will be covered in Chapter XXX, section XXX.

Lastly, we define our own API. Most of these functions simply send messages to our running server. More on this in Chapter XXX, section XXX.

The LFE REPL

We briefly introduced the REPL in the first version of the Hello-World example we wrote, stating that it was an acronym for ‘read-eval-print loop’ and how to start it with rebar3. As an LFE developer, this is one of the primnary tools – arguably the most powerful – at your disposal, so we’re going to do a more thorough job of introducing its capabilities in this section.

A basic REPL can be implemented with just four functions; such an implementation could be started with the following:

(LOOP (PRINT (EVAL (READ))))

LFE has implemented most these functions for us already (and quite robustly), but we could create our own very limited REPL (single lines with no execution context or environment) within the LFE REPL using the following convenience wrappers:

(defun read ()
  (case (io:get_line "myrepl> ")
    ("quit\n" "quit")
    (str (let ((`#(ok ,expr) (lfe_io:read_string str)))
           expr))))

(defun print (result)
  (lfe_io:format "~p~n" `(,result))
  result)

(defun loop
  (("quit")
   'good-bye)
  ((code)
   (loop (print (eval (read))))))

Now we can start our custom REPL inside the LFE REPL:

lfe> (loop (print (eval (read))))

This gives us a new prompt:

myrepl>

At this prompt we can evaluate basic LFE expressions:

myrepl> (+ 1 2 3)
;; 6
myrepl> (* 2 (lists:foldl #'+/2 0 '(1 2 3 4 5 6)))
;; 42
myrepl> quit
;; good-bye
lfe>

Note that writing an evaluator is the hard part, and we’ve simply re-used the LFE evaluator for this demonstration.

Now that we’ve explored some of the background of REPLs and Lisp interpreters, let’s look more deeply into the LFE REPL and how to best take advantage of its power when using the machine that is LFE and OTP.

Core Features

In keeping with the overall herritage of LFE, its REPL is both a Lisp REPL as well as an Erlang shell. In fact, when support was added for the LFE REPL to the rebar3_lfe plugin, it utilised all of the plumbing for the Erlang shell support in rebar3.

For the developer, though, this means that the LFE REPL holds a dual set of features, multiplying the set of features available to just the Erlang shell. These features include the following support:

• Evaluation of Lisp S-expressions
• Definition of functions, completely with LFE tail-recursion support
• Definition of records and use of record-specific support functions/macros
• Creation of LFE macros via standard Lisp-2 syntax
• Macro examination and debugging with various expansion macros
• The ability to start the LFE REPL in distribution mode, complete with Erlang cookie, and thus to not only access remote LFE and Erlang nodes, but to be accessed as a remote node itself (for nodes that have been granted access
• Access to the Erlang JCL and the ability to start separate, LFE shells running concurrently

Unsupported

The following capabilities are not supported in the LFE REPL:

• Module definitions; these are a file-based feature in LFE, just as with Erlang.

Starting LFE

The lfe executable

While this book focuses upon the use of rebar3 and its LFE plugin – due entirely to the amount of time it saves through various features it supports – LFE may be used quite easily without it.

To use LFE and its REPL without rebar3, you’ll need to clone the repo, e.g.:

cd ~/lab
git clone https://github.com/lfe/lfe.git
cd lfe

Since you have read the earlier section on dependencies, you already have Erlang, make, and your system build tools installed. As such, all you have to do is run the following to build LFE:

make

This will generate an executable in ./bin and you can start the LFE REPL by calling it:

./bin/lfe

Erlang/OTP 28 [erts-16.0] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit] [dtrace]

   ..-~.~_~---..
  (      \\     )    |   A Lisp-2+ on the Erlang VM
  |`-.._/_\\_.-':    |   Type (help) for usage info.
  |         g |_ \   |
  |        n    | |  |   Docs: http://docs.lfe.io/
  |       a    / /   |   Source: http://github.com/lfe/lfe
   \     l    |_/    |
    \   r     /      |   LFE v2.2.0 (abort with ^G)
     `-E___.-'

lfe>

If you opt to install LFE system-wide with make install, then you can start the REPL from anywhere by simply executing lfe.

Via rebar3 lfe repl

As demonstrated earlier on several occasions, you can start the LFE REPL with the rebar3 LFE plugin (and this is what we’ll do in the rest of this manual):

rebar3 lfe repl

Since you have updated your global rebar3 settings (in the “Prerequisites” section, after following the instructions on the rebar3 site), you may also start the LFE REPL from anywhere on your machine using the rebar3 command.

readline Support

The LFE REPL, being built atop the Erlang shell, can benefit from GNU Readline support to provide enhanced line editing capabilities including command history, keyboard shortcuts, and tab completion. This section covers how to enable and configure these features for optimal development experience.

Built-in History Support (Erlang/OTP 20+)

Starting with Erlang/OTP 20, persistent shell history is available out of the box, though it’s disabled by default. This feature provides basic command history persistence across sessions.

Enabling Shell History

To enable persistent history for all Erlang-based shells (including LFE), add the following to your shell’s configuration file (~/.bashrc, ~/.zshrc, etc.):

# Enable Erlang shell history
export ERL_AFLAGS="-kernel shell_history enabled"

Advanced History Configuration

You can customize the history behavior using additional kernel parameters:

# Complete history configuration
export ERL_AFLAGS="+pc unicode"
ERL_AFLAGS="$ERL_AFLAGS -kernel shell_history enabled"
ERL_AFLAGS="$ERL_AFLAGS -kernel shell_history_path '\"$HOME/.erl_history\"'"
ERL_AFLAGS="$ERL_AFLAGS -kernel shell_history_file_bytes 1048576"
ERL_AFLAGS="$ERL_AFLAGS -kernel shell_history_drop '[\"q().\",\"init:stop().\",\"halt().\"]'"
export ERL_AFLAGS

Where:

• shell_history_path - Custom path for history file (default: ~/.erlang-history.*))
• shell_history_file_bytes - Maximum history file size in bytes (default: 512KB, minimum: 50KB)
• shell_history_drop - Commands to exclude from history (e.g., exit commands)

Built-in Tab Completion

The Erlang shell (and by extension, LFE) includes sophisticated built-in tab completion functionality through the edlin and edlin_expand modules. LFE extends this with its own lfe_edlin_expand module to provide Lisp-aware completion capabilities. Starting with Erlang/OTP 26, the auto-completion feature has been vastly improved, supporting auto-completion of variables, record names, record field names, map keys, function parameter types, and file names.

LFE-Specific Tab Completion

LFE provides enhanced tab completion that understands Lisp syntax and LFE-specific constructs. When you type in the LFE REPL, tab completion works seamlessly with:

• Module names: Type (code and press TAB to see code and code_server
• Function names with arities: Type (code: and press TAB to see all available functions with their arities displayed in a paged format
• LFE syntax: Completion works properly with LFE’s S-expression syntax
• Standard Erlang modules: Full access to Erlang’s standard library with completion

What Tab Completion Covers

The standard expansion function is able to expand strings to valid Erlang terms, including module names, and automatically add commas or closing parentheses when no other valid expansion is possible. Specifically, tab completion works for:

• Module names: Type li and press TAB to complete to lists:
• Function names: Type lists: and press TAB to see all available functions
• Variables: Complete previously defined shell variables by typing the first letter(s) and pressing TAB
• Record names: Type # and TAB to see available records
• Record fields: Complete field names within record syntax
• Map keys: Complete keys when working with maps
• Built-in functions (BIFs): Complete standard Erlang functions
• File names: When working with file operations

Using Tab Completion

• Single TAB: Autocomplete the current word, or show 5 lines of possible completions
• Double TAB: Output all possible tab completions
• Navigation: Use move_expand_up and move_expand_down to navigate through completion lists in the expand area

LFE Tab Completion Examples

Here are some examples of LFE’s tab completion in action:

;; Module completion
lfe> (code<TAB>
code           code_server

;; Function completion with paged display
lfe> (code:<TAB>
add_path/1                 add_path/2                 add_patha/1
add_patha/2                add_paths/1                add_paths/2
add_pathsa/1               add_pathsa/2               add_pathsz/1
add_pathsz/2               add_pathz/1                add_pathz/2
...
rows 1 to 7 of 8

;; LFE syntax-aware completion
lfe> (defun my-func<TAB>
;; Will complete user-defined functions and LFE constructs

The completion system understands LFE’s parenthesized syntax and provides contextually appropriate suggestions.

Customizing Tab Completion

You can customize tab completion by setting an expand function using io:setopts/1,2 with the {expand_fun, expand_fun()} option:

;; In LFE, you would set this via Erlang interop
(: io setopts `(#(expand_fun ,(lambda (input)
                               ;; Custom completion logic here
                               '#(yes "completion" ("option1" "option2"))))))

Full Readline Integration

For a more complete readline experience with features like reverse search (Ctrl-R), customizable key bindings, and improved line editing beyond the built-in capabilities, you have several options.

Option 1: Using rlwrap

rlwrap (readline wrapper) is a utility that adds GNU Readline capabilities to any command-line program. It’s the most straightforward way to add full readline support to LFE.

Installing rlwrap

On most Unix-like systems:

# Ubuntu/Debian
sudo apt-get install rlwrap

# macOS (Homebrew)
brew install rlwrap

# CentOS/RHEL/Fedora
sudo yum install rlwrap  # or dnf install rlwrap

Basic Usage

# Start LFE with readline support
rlwrap rebar3 lfe repl

# Or if using the lfe executable directly
rlwrap lfe

Advanced rlwrap Configuration

For optimal LFE/Lisp experience, you can customize rlwrap with specific options:

# Enhanced configuration for Lisp-like languages
rlwrap -r -m -q '"' -b "(){}[],^%#@\\;:'" rebar3 lfe repl

Where:

• -r puts all words seen on input and output on the completion list
• -m enables multi-line editing
• -q '"' handles double quotes specially
• -b "(){}[],^%#@\\;:'" defines characters that break words for completion

Creating a Wrapper Script

For convenience, create a wrapper script in your $PATH (e.g., ~/bin/lfe-repl):

#!/bin/bash
# LFE REPL with readline support

# Basic version
rlwrap rebar3 lfe repl

# Or enhanced version with better Lisp support
rlwrap -r -m -q '"' -b "(){}[],^%#@\\;:'" rebar3 lfe repl "$@"

Make it executable:

chmod +x ~/bin/lfe-repl

Tab Completion vs rlwrap

Important Note: Using rlwrap with the -a option (always use readline) will disable the Erlang shell’s built-in tab completion functionality. This is a trade-off between rlwrap’s enhanced features (like persistent history and colored prompts) and the shell’s native completion capabilities.

If you prioritize tab completion:

• Use the built-in shell history (ERL_AFLAGS="-kernel shell_history enabled")
• Use rlwrap without the -a flag, though this may provide a less integrated experience

If you prioritize rlwrap’s features:

• Accept that built-in tab completion will be disabled
• Consider creating custom completion files for rlwrap (advanced)

Option 2: Terminal-Specific Configuration

Many modern terminal emulators provide their own line editing features that can complement or replace readline functionality.

Using .inputrc Configuration

The ~/.inputrc file configures GNU Readline behavior system-wide. Here’s a useful configuration for Lisp development:

# ~/.inputrc - GNU Readline configuration

# Use vi or emacs editing mode
set editing-mode emacs

# Enable case-insensitive completion
set completion-ignore-case on

# Show completion matches immediately
set show-all-if-ambiguous on
set show-all-if-unmodified on

# Enable colored completion
set colored-stats on
set colored-completion-prefix on

# Better history search
"\e[A": history-search-backward
"\e[B": history-search-forward

# Alternative: use Ctrl-P/Ctrl-N for history search
# "\C-p": history-search-backward
# "\C-n": history-search-forward

# Disable terminal bell
set bell-style none

# Enable bracketed paste mode (safe pasting)
set enable-bracketed-paste on

# Show mode indicator for vi mode (if using vi editing-mode)
set show-mode-in-prompt on

After editing ~/.inputrc, you can reload it with:

bind -f ~/.inputrc

Environment Variable Summary

For a complete readline-enabled LFE environment, add these to your shell configuration:

# ~/.bashrc, ~/.zshrc, etc.

# Enable Erlang shell history
export ERL_AFLAGS="-kernel shell_history enabled"

# Optional: customize history location and behavior
# export ERL_AFLAGS="$ERL_AFLAGS -kernel shell_history_path '\"$HOME/.lfe_history\"'"
# export ERL_AFLAGS="$ERL_AFLAGS -kernel shell_history_file_bytes 1048576"

# Create an alias for LFE with readline
alias lfe-repl='rlwrap rebar3 lfe repl'

# Or for the lfe executable
alias lfe='rlwrap lfe'

Troubleshooting

History Not Persisting

If command history isn’t being saved between sessions:

1. Ensure shell_history is enabled in ERL_AFLAGS
2. Check that the history directory is writable
3. Wait a moment before exiting the shell (history is written asynchronously)
4. Try setting a custom history path with proper permissions

rlwrap Issues

If rlwrap isn’t working correctly:

1. Verify rlwrap is installed: which rlwrap
2. Check that your terminal supports readline properly
3. Try with minimal options first: rlwrap rebar3 lfe repl
4. On some systems, you may need to omit the -q option

Tab Completion Problems

If tab completion behaves unexpectedly:

1. Try adjusting the word-breaking characters in rlwrap’s -b option
2. Check your ~/.inputrc configuration
3. Some terminal emulators may require specific settings for completion

Performance Issues

If the REPL feels sluggish with readline enabled:

1. Reduce history file size with shell_history_file_bytes
2. Use fewer rlwrap options for minimal overhead
3. Consider using built-in history only without rlwrap for simple use cases

(help)

As you gain familiarity with the LFE REPL, one of the most useful and convenient references will be the summary of functions, commands, and variablese that come with the LFE REPL.

To see these, simply call the help or h function:

lfe> (help)

That will result in the following being displyed to your terminal:

LFE shell built-in functions

(c file)       -- compile and load code in <file>
(cd dir)       -- change working directory to <dir>
(clear)        -- clear the REPL output
(doc mod)      -- documentation of a module
(doc mod:mac)  -- documentation of a macro
(doc m:f/a)    -- documentation of a function
(ec file)      -- compile and load code in erlang <file>
(ep expr)      -- print a term in erlang form
(epp expr)     -- pretty print a term in erlang form
(exit)         -- quit - an alias for (q)
(flush)        -- flush any messages sent to the shell
(h)            -- an alias for (help)
(help)         -- help info
(i)            -- information about the system
(i pids)       -- information about a list of pids
(l module)     -- load or reload <module>
(ls)           -- list files in the current directory
(ls dir)       -- list files in directory <dir>
(m)            -- which modules are loaded
(m mod)        -- information about module <mod>
(p expr)       -- print a term
(pp expr)      -- pretty print a term
(pid x y z)    -- convert <x>, <y> and <z> to a pid
(pwd)          -- print working directory
(q)            -- quit - shorthand for init:stop/0
(regs)         -- information about registered processes

LFE shell built-in commands

(reset-environment)             -- reset the environment to its initial state
(run file)                      -- execute all the shell commands in a <file>
(set pattern expr)
(set pattern (when guard) expr) -- evaluate <expr> and match the result with
                                   pattern binding
(slurp file)                    -- slurp in a LFE source <file> and makes
                                   everything available in the shell
(unslurp)                       -- revert back to the state before the last
                                   slurp

LFE shell built-in variables

+/++/+++      -- the tree previous expressions
*/**/***      -- the values of the previous expressions
-             -- the current expression output
$ENV          -- the current LFE environment

ok

Most of those are documented in stdlib reference for their Erlang counterparts, so be sure to reference that information for details on many of the above.

Those not covered in that Erlang reference manual, or those that are different in their LFE versionsm, include:

• Built-in Functions
  • Compilation functions
  • LFE code documentation
  • Printing and pretty-printing
• Built-in commands
• Built-in variables

REPL Functions

Most of the LFE REPL functions are documented in stdlib reference for their Erlang counterparts. This section documents where the LFE REPL help diverges from the Erlang Shell help.

Compilation

If you view the Erlang reference manual documentation for compiling files in the shell, you will see differences from what is shown in the LFE help text. In particular, (c) is for compiling LFE modules and (ec) needs to be used for compiling Erlang source files.

In both cases, the resulting .beam files are compiled to the current working directory and not to an ebin directory. These .beam files will be found by LFE, since the current working directory is included in the path, but you’ll likely want to perform some cleanup afterward.

Documentation

You may access the documentation for LFE modules, macros, and functions in the REPL via the doc function. For instance, the Common Lisp compatibility module’s documentation:

lfe> (doc cl)
;; ____________________________________________________________
;; cl
;;
;; LFE Common Lisp interface library.
;;
;; ok

That module’s cond macro documentation:

lfe> (doc cl:cond)
;; ____________________________________________________________
;; cond
;; args
;; CL compatible cond macro.
;;
;; ok

That module’s pairlis/2 function documentation:

lfe> (doc cl:pairlis/2)
;; ____________________________________________________________
;; pairlis/2
;; keys values
;; Make an alist from pairs of keys values.
;;
;; ok

Documentation for Erlang modules and fucntions is available via the Command Interface

Printing Data

LFE Formatting

LFE provides some nice convenience functions for displaying data structions in the REPL. Let’s say we had a data structure defined thusly:

lfe> (set data `(#(foo bar baz) #(quux quuz) #(corge grault garply)
lfe> #(plugh xyzzy) #(flurb nirf) #(zobod zirfid)))

We can print our data with the following:

lfe> (p data)
;; (#(foo bar baz) #(quux quuz) #(corge grault garply) #(plugh xyzzy) #(flurb nirf) #(zobod zirfid))
;; ok

Or we can pretty-print it:

lfe> (pp data)
;;(#(foo bar baz)
;; #(quux quuz)
;; #(corge grault garply)
;; #(plugh xyzzy)
;; #(flurb nirf)
;; #(zobod zirfid))
;; ok

Erlang Formatting

The same may be done for displaying data in the Erlang format:

lfe> (ep data)
;; [{foo,bar,baz},{quux,quuz},{corge,grault,garply},{plugh,xyzzy},{flurb,nirf},{zobod,zirfid}]
;; ok

lfe> (epp data)
;; [{foo,bar,baz},
;;  {quux,quuz},
;;  {corge,grault,garply},
;;  {plugh,xyzzy},
;;  {flurb,nirf},
;;  {zobod,zirfid}]
;; ok

REPL Commands

The LFE REPL provides several useful commands users:

(reset-environment)             -- reset the environment to its initial state
(run file)                      -- execute all the shell commands in a <file>
(set pattern expr)
(set pattern (when guard) expr) -- evaluate <expr> and match the result with
                                   pattern binding
(slurp file)                    -- slurp in a LFE source <file> and makes
                                   everything available in the shell
(unslurp)                       -- revert back to the state before the last
                                   slurp

These are fairly self-explanatory, with the possible exception of clarifying how run and slurp differ:

• Calling (run "some/file.lfe") will cause the LFE REPL to read the contents of that file and then execute every line in that file as if they had been typed at the terminal. This is a convenient way of duplicating REPL state between sessions. (If you haven’t kept track of your entries, you can always open up your BEAM history file and create an .lfe file with all the required commands!)
• Calling (slurp "some/other/file.lfe") will place all functions, records, and macros defined in that file into the LFE environment, allowing you to call them without a module: prefix. Note that the code is not executed, but is instead placed into the current environment, ready for use.

Special Variables

LFE shell built-in variables

+/++/+++      -- the tree previous expressions
*/**/***      -- the values of the previous expressions
-             -- the current expression output
$ENV          -- the current LFE environment

Most of these variables are taken directly from Common Lisp and have the same exact meaning. From the Common Lisp HyperSpec for +,++,+++:

The variables +, ++, and +++ are maintained by the Lisp read-eval-print loop to save forms that were recently evaluated.

The value of + is the last form that was evaluated, the value of ++ is the previous value of +, and the value of +++ is the previous value of ++.

And for *,**,***:

The variables *, **, and *** are maintained by the Lisp read-eval-print loop to save the values of results that are printed each time through the loop.

The value of * is the most recent primary value that was printed, the value of ** is the previous value of *, and the value of *** is the previous value of **.

Lastly, for -:

The value of - is the form that is currently being evaluated by the Lisp read-eval-print loop.

The $ENV variable in the LFE REPL is a critical tool for debugging particularly tricky issues in the REPL (especially useful when creating complex macros).

Command Interface

While many of the functions listed in the LFE (help) have their documentation in the Erlang Command Interface module (CIM), not everything in the CIM has been provided in the LFE REPL, some of which can be useful at times.

Here are some of the more useful functions you may with to be aware of from that Erlang module:

• (c:bt pid) - Stack backtrace for a process. Equivalent to (erlang:process_display pid 'backtrace).
• (c:h mod) - Print the documentation for mod
• (c:h mod fun) - Print the documentation for all mod:fun
• (c:h mod fun arity) - Print the documentation for mod:fun/arity
• (c:lm) - Reloads all currently loaded modules that have changed on disk (see (c:mm)). Returns the list of results from calling (l mod) for each such loaded module.
• (c:memory) - Memory allocation information. Equivalent to (erlang:memory).
• (c:mm) -
• (c:ni) - Display system information, listing information about all nodes on the network
• (c:nl mod) - Loads Module on all nodes
• (c:nregs) - Displays information about all registered processes for all nodes in the network.
• (c:uptime) - Prints the node uptime (as specified by (erlang:statistics 'wall_clock)) in human-readable form.
• (c:xm mod) - Finds undefined functions, unused functions, and calls to deprecated functions in a module by calling xref:m/1.

Job Control

Thanks to Erlang, when working in the LFE REPL you have access to a powerful job control system that allows you to manage multiple shell sessions, interrupt running processes, and switch between different contexts. This system is accessed through Job Control Language (JCL) mode, which becomes available when you press <CTRL-G>.

Entering JCL Mode

When in the LFE REPL, typing <CTRL-G> will detach the current job and activate JCL mode:

lfe> (set ltuae (* 2 (+ 1 2 3 4 5 6)))
42
lfe>
User switch command (enter 'h' for help)
 -->

At the JCL --> prompt, you may get help text by typing ? or h:

 --> h
  c [nn]            - connect to job
  i [nn]            - interrupt job
  k [nn]            - kill job
  j                 - list all jobs
  s [shell]         - start local shell
  r [node [shell]]  - start remote shell
  q                 - quit erlang
  ? | h             - this message
 -->

Understanding Jobs

In the context of the Erlang/LFE shell, a job refers to a single evaluator process along with any local processes it spawns. When you start the LFE REPL, you begin with one job that acts as your interactive shell session.

Each job maintains its own:

• Variable bindings
• Process dictionary
• Record definitions
• Evaluation context

Only the currently connected job can perform operations with standard I/O, while detached jobs are blocked from using standard I/O (though they can continue running background computations).

JCL Commands

Each JCL command serves a specific purpose in managing your shell sessions:

• c [nn] - Connect to job: Connects to job number nn or the current job if no number is specified. This resumes the standard shell and allows you to interact with that job’s evaluation context.

• i [nn] - Interrupt job: Stops the evaluator process for job number nn or the current job, but preserves the shell process. Variable bindings and the process dictionary are maintained, so you can reconnect to the job later. This is particularly useful for interrupting infinite loops or long-running computations.

• k [nn] - Kill job: Completely terminates job number nn or the current job. All spawned processes in the job are killed (provided they haven’t changed their group leader and are on the local machine). This permanently destroys the job’s context.

• j - List all jobs: Displays all current jobs with their numbers and descriptions. The currently connected job is marked with an asterisk (*).

• s [shell] - Start local shell: Creates a new job with a fresh shell environment. If no shell module is specified, it starts the default shell (LFE shell in our case).

• r [node [shell]] - Start remote shell: Starts a remote job on the specified node. This is used in distributed Erlang to control applications running on other nodes in the network.

• q - Quit Erlang: Completely exits the Erlang runtime system. Note that this command may be disabled if Erlang was started with the +Bi system flag.

Running Multiple Shells: A Practical Example

Here’s a practical demonstration of using multiple jobs to maintain separate evaluation contexts:

$ lfe
lfe> (set ltuae (* 2 (+ 1 2 3 4 5 6)))
42
lfe> ^G
User switch command (enter 'h' for help)
 --> j
   1* {lfe_shell,start,[]}
 --> s lfe_shell
 --> j
   1  {lfe_shell,start,[]}
   2* {lfe_shell,start,[]}
 --> c 2
lfe> ltuae
** exception error: symbol ltuae is unbound

lfe> (set arch (erlang:system_info 'system_architecture))
"aarch64-apple-darwin24.4.0"
lfe> ^G
User switch command (enter 'h' for help)
 --> j
   1  {lfe_shell,start,[]}
   2* {lfe_shell,start,[]}
 --> c 1
lfe> ltuae
42
lfe> arch
** exception error: symbol arch is unbound

lfe> ^G
User switch command (enter 'h' for help)
 --> c 2
lfe> arch
"aarch64-apple-darwin24.4.0"

This example demonstrates how each job maintains its own variable bindings:

• Job 1 has ltuae defined but not arch
• Job 2 has arch defined but not ltuae
• Each job represents a completely separate evaluation environment

Common Use Cases

1. Interrupting Infinite Loops

If your code gets stuck in an infinite loop:

lfe> (defun infinite-loop () (infinite-loop))
infinite-loop
lfe> (infinite-loop)
;; Process gets stuck here - press Ctrl-G
User switch command
 --> i
 --> c
lfe> ;; Back to a responsive shell

2. Experimental Development

Use separate jobs for different experiments:

;; Job 1: Working on feature A
lfe> (set feature-a-data '(1 2 3 4))

;; Switch to Job 2 for feature B
User switch command
 --> s lfe_shell
 --> c 2
lfe> (set feature-b-data '(a b c d))

;; Switch back to continue feature A work
User switch command
 --> c 1
lfe> feature-a-data
(1 2 3 4)

3. Remote Development

Connect to remote nodes in distributed systems:

User switch command
 --> r other_node@other_host
;; Now connected to a shell on the remote node

Important Notes

• Variable Isolation: Each job maintains completely separate variable bindings and evaluation contexts
• Process Preservation: Using i (interrupt) preserves the job’s state, while k (kill) destroys it permanently
• I/O Blocking: Only the connected job can use standard I/O; detached jobs will block if they attempt I/O operations
• Record Definitions: Each job has its own set of record definitions, though some default records may be loaded automatically

Customizing JCL Behavior

The behavior of the shell escape key (Ctrl-G) can be modified using the STDLIB application variable shell_esc:

• jcl (default): Activates JCL mode
• abort: Terminates the current shell and starts a new one instead of entering JCL mode

This is set with: erl -stdlib shell_esc abort

Job Control in LFE provides a powerful way to manage multiple development contexts, handle problematic code execution, and work with distributed systems, making it an essential tool for effective LFE development.

Files

So far, everything we’ve looked at in the REPL involves typing (or pasting) code. When wanting to use the REPL to experiment with more complicated code, there’s a better, time-honoured way: files. There are several ways you can use file-based code in the REPL:

• evaluation
• compilation
• including
• loading

Each of these is covered in more detail in the following sub-sections.

Evaluation: : slurp and unslurp

LFE provides a unique and powerful way to evaluate file contents directly in the REPL through the slurp and unslurp commands. This feature allows you to temporarily import function and macro definitions from files into your current REPL session.

Using slurp

The slurp function reads an LFE source file and makes all functions and macros defined in that file available directly in the shell, without requiring module prefixes:

lfe> (slurp "examples/my-functions.lfe")
#(ok -no-mod-)
lfe> $ENV
;; Shows all the new function and macro definitions from the file

Key characteristics of slurp:

• Only one file can be slurped at a time
• Slurping a new file automatically removes all data from the previously slurped file
• Functions and macros become available without module prefixes
• The code is evaluated in the current REPL environment
• Variable bindings from the file are added to your current session

Using unslurp

The unslurp command reverts the REPL back to the state before the last slurp, removing all function and macro definitions that were imported:

lfe> (unslurp)
ok
lfe> $ENV
;; Back to the original environment state

This is particularly useful when experimenting with different versions of functions or when you want to clean up your REPL environment.

Practical Example

Let’s say you have a file called math-utils.lfe:

(defun double (x)
  (* x 2))

(defun square (x)
  (* x x))

(defmacro when-positive (x body)
  `(if (> ,x 0) ,body 'not-positive))

After slurping this file:

lfe> (slurp "math-utils.lfe")
#(ok -no-mod-)
lfe> (double 21)
42
lfe> (square 6)
36
lfe> (when-positive 5 'yes)
yes
lfe> (when-positive -1 'yes)
not-positive

Compilation

The LFE REPL provides several functions for compiling source files and loading the resulting modules.

Compiling LFE Files

Use the c function to compile and load LFE modules:

lfe> (c "my-module.lfe")
#(module my-module)

This function:

• Compiles the LFE source file
• Loads the resulting .beam file into the current session
• Makes the module’s exported functions available for use
• Places the compiled .beam file in the current working directory

You can also provide compilation options:

lfe> (c "my-module.lfe" '(debug_info export_all))
#(module my-module)

Common compilation options include:

• debug_info - Include debugging information
• export_all - Export all functions (useful for development)
• warn_unused_vars - Warn about unused variables

Compiling Erlang Files

To compile Erlang source files from the LFE REPL, use the ec function:

lfe> (ec "utility.erl")
#(module utility)

This allows you to work with existing Erlang code from within your LFE development session.

Working with Compiled Modules

Once a module is compiled and loaded, you can call its functions using the standard module:function syntax:

lfe> (my-module:some-function arg1 arg2)
result

You can also check which modules are currently loaded:

lfe> (m)
;; Lists all loaded modules
lfe> (m 'my-module)
;; Shows information about a specific module

Cleanup

If you compile a module in the REPL and forget to clean up the resulting .beam file, you may encounter mysterious issues later, naming continuing to see old behaviour from the module compiled in your current working directory instead of the .beam file most recently updated.

Inclusion

LFE supports including header files and library files, which is essential for larger projects and when working with Erlang/OTP libraries.

include-lib

The include-lib directive allows you to include files from installed OTP applications or other libraries:

(include-lib "kernel/include/file.hrl")
(include-lib "stdlib/include/qlc.hrl")

This searches for the include file in the standard library locations and makes the definitions available in your code.

include-file

The include-file directive includes files using relative or absolute paths:

(include-file "local-definitions.lfe")
(include-file "../shared/common.lfe")
(include-file "/absolute/path/to/file.lfe")

Include File Content

Include files typically contain:

• Record definitions
• Macro definitions
• Constant definitions
• Type specifications

Example include file (records.lfe):

(defrecord person
  name
  age
  email)

(defmacro debug (msg)
  `(io:format "DEBUG: ~p~n" (list ,msg)))

After including this file, you can use the record and macro definitions in your code:

lfe> (include-file "records.lfe")
debug

;; In LFE, when including a file in the REPL, the last
;; function defined in the file is printed to stdout.

lfe> (make-person name "Robert" age 54 email "robert@lfe.io")
#(person "Robert" 54 "robert@lfe.io")
lfe>

lfe> (debug "oops")
DEBUG: "oops"
ok

Loading

LFE provides several ways to load pre-compiled modules into your REPL session.

Using l (Load Module)

The l function loads or reloads a specific module:

lfe> (l 'my-module)
#(module my-module)

This is useful when:

• You’ve recompiled a module and want to reload it
• You want to load a module that exists but isn’t currently loaded
• You’re working with hot code reloading during development

Using code:ensure_loaded

For more control over the loading process, you can use Erlang’s code server directly:

lfe> (code:ensure_loaded 'my-module)
#(module my-module)

This function:

• Loads the module if it’s not already loaded
• Does nothing if the module is already loaded
• Returns error information if the module can’t be found

Checking Module Status

You can check what modules are currently loaded and get information about them:

;; List all loaded modules
lfe> (m)

;; Get information about a specific module
lfe> (m 'lists)

;; Check if a module is loaded
lfe> (code:is_loaded 'my-module)
#(file "/path/to/my-module.beam")

Hot Code Reloading

During development, you can reload modules without restarting the REPL:

;; Edit and recompile your module file
lfe> (c "my-module.lfe")
#(module my-module)

;; The module is automatically reloaded with new code
lfe> (my-module:updated-function)

Part II - Code as Data

In the Lisp tradition, all program code is written as s-expressions, or parenthesized lists Lisp, making the primary representation of programs also a data structure in a primitive type of the language itself. This fundamental principle—code as data—means that before we can understand how LFE programs execute, we must first master the data structures from which they are constructed.

Part II explores the foundational data types and structures that form the vocabulary of LFE programming. Every LFE program, from the simplest function call to the most complex macro, is ultimately composed of the primitive types and data structures covered in these chapters. We begin with the atomic elements—integers, floats, atoms, and characters—and progress through increasingly sophisticated composite structures like lists, tuples, maps, and records.

The interchangeability of code and data gives Lisp its instantly recognizable syntax, and LFE inherits this property. When you write (+ 1 2 3) in LFE, you are simultaneously creating a list data structure and expressing a computation. The list contains the atom + followed by three integers, but it also represents the addition of those numbers. This duality is not accidental—it is the essence of homoiconicity that makes LFE programs both readable as data and executable as code.

Understanding these data structures deeply is crucial because data representing code can be passed between the meta and base layer of the program. Pattern matching, one of LFE’s most powerful features, allows you to decompose and analyze these structures with precision. By the end of Part II, you will have the tools to construct, manipulate, and understand any LFE data structure—knowledge that directly translates to understanding LFE code itself.

Variables

In LFE, variables are implemented with atoms. Atoms are used in the language for such things as naming functions and macros, use as keywords and in data structures. As you’ll find out when reading about atoms, they are evaluated as simply themselves. However, when they are used as variable names, they evaulate to the value of which they were assigned.

There are two contexts for setting variables:

• in the LFE REPL
• in LFE modules

This distinction is important, not only because the contexts use different forms, but because like Erlang, LFE does not support global variables.

This chpater will also introduce the reader to pattern-matching in LFE, as it applies to variable assignment; a fuller discussion is presented in a dedicated chapter as well in the chapters that cover forms which support pattern-matching (i.e., compound data types, functions, etc.).

Lastly we’ll talk more about some of the LFE nuances around global variables.

Bindings

In the REPL

To set a variable in the LFE REPL, use the set macro:

lfe> (set answer 42)
42

In the language itself, LFE doesn’t support global variables – a valuable feature inherited from Erlang. However, in order for a REPL experience to be useful, an environment must be maintained in which the user may write expressions and then refer to them later. This environment is essentially a mechanism for global state in the context of a single user running a single Erlang VM. If we set a variable called answer, that variable will be available to us as long as the REPL process continues or until we reset the environment.

Setting another value with the same variable name is allowed: it merely replaces the assignment in the current REPL environment:

lfe> (set answer "forty-two")
"forty-two"

With a variable assigned with set it may be used at any time in the REPL environment where it was defined:

lfe> (++ "The answer is " answer)
"The answer is forty-two"

Attempting to use a variable that has not been defined results in an error:

lfe> (++ "The question was " question)
** exception error: symbol question is unbound
  in lfe_eval:eval_error/1 (src/lfe_eval.erl, line 1292)
  in lists:map/2 (lists.erl, line 1243)
  in lists:map/2 (lists.erl, line 1243)

If you don’t have any need to update the environment with data that you only need for a specific calculation, you may use the let form:

lfe> (let ((short-lived-value (* 2 (+ 1 2 3 4 5 6))))
lfe>   (io:format "The answer is ~p~n" `(,short-lived-value)))
The answer is 42
ok

Let’s make sure that variable wasn’t saved to our environment:

lfe> short-lived-value
** exception error: symbol short-lived-value is unbound

The lexical scope for the short-lived-value is within the let only and is not available outside of that scope.

In Functions and Macros

Within functions, variables are lexically scoped and bound with let and let*. One may also define lexically scoped functions inside other functions, and this is done with flet and fletrec (the latter required for defining recursive functions inside another function). These will be covered in detail later in the book.

We’ve seen let used above in the REPL; the same applies inside functions:

(defun display-answer ()
  (let ((answer (* 2 (+ 1 2 3 4 5 6))))
    (io:format "The answer is ~p~n" `(,answer))))

This is a straight-forward case of assignment; but what if we needed to assign a varaible that depended upon another variable. Using let, you’d have to do this:

(defun display-answer ()
  (let ((data '(1 2 3 4 5 6)))
    (let ((answer (* 2 (lists:sum data))))
      (io:format "The answer is ~p~n" `(,answer)))))

However, as with other Lisps, LFE provides a convenience macro for this: let*. Here’s how it’s used:

(defun display-answer ()
  (let* ((data '(1 2 3 4 5 6))
         (answer (* 2 (lists:sum data))))
    (io:format "The answer is ~p~n" `(,answer))))

Lexical scoping helps one isolate unrelated data or calculations, even in the same function: multiple let or let* blocks may be declared in a function and none of the bound variables in one block will be available to another block. Attempting to do so will result in an error.

In Modules

In LFE, one cannot bind variables at the module-level, only functions and macros. This is part of the “no global variables” philosophy (and practice!) of Erlang and LFE. Module-level bindings are done with defun for functions and defmacro for macros. The creation of modules, functions, and macros will all be covered in detail later in the book.

Shadowing

One shadows a variable in one scope when, at a higher scope, that variable was also defined. Here’s an annotated example:

(defun shadow-demo ()
  (let ((a 5))
    (io:format "~p~n" `(,a))    ; prints 5
    (let ((a 'foo))             ; 'a' here shadows 'a' in the previous scope
      (io:format "~p~n" `(,a))) ; prints foo
    (io:format "~p~n" `(,a)))   ; prints 5; the shadow binding is out of scope
  (let ((a 42))
    (io:format "~p~n" `(,a))))  ; prints 42 - new scope, no shadowing

Shadowing also may occur at the module-level with the definition of functions, and the shadowing could be of functions at one of several levels. Here’s a run-down on function shadowing in modules, from the highest or “outermost” to the lowest or “innermost”:

• Predefined Erlang built-in functions (BIFs) may be shadowed by any of the following
• Predefined LFE BIFs may be shadowed by any of the following
• Module imports may shadow any of the above via aliasing
• Functions defined in a module may shadow any of the above
• Functions defined inside a function (e.g., via flet or fletrec) may shadow any of the above

Note that to shadow functions in LFE, functions must match both in name as well as arity (number of arguments).

The hd Erlang BIF returns the “head” of a list (the first item). Here’s an example of shadowing it in the REPL. Here’s the BIF at work:

lfe> (hd '(a b c d e))
a

Next, paste this into the REPL:

(defun hd (_)
  ;; part of the pun here is that the same function in Lisp is called 'car'
  "My other car is The Heart of Gold.")

The hd function takes one argument (a list), so our function also needs to take one. However, since we don’t do anything with that, we use the “don’t care” variable _.

Now let’s call hd again:

lfe> (hd '(a b c d e))
"My other car is The Heart of Gold."

Shadowed!

Note that, like many other Lisps, LFE has the car function, but since this is a core form, it can’t be shadowed (see the next section).

The Unshadowable

Shadowing does not apply to the supported LFE core forms. It may appear that your code is shadowing those forms, but the compiler will always use the core meaning and never an alternative. It does this silently, without warning – so take care and do not be surprised!

Pattern-matching Preview

Pattern matching is deeply integral to Erlang, and thus LFE. Great swaths of the language touch upon it, so therefore it is a farily large subject to cover in LFE. We will be revisiting pattern matching throughout the rest of the book, but for now we will provide a brief preview in the context of variables and binding values to them.

In the previous section, we saw in-REPL binding like this:

lfe> (set answer 42)
42

But what if it wasn’t a simple integer we wanted to assign? What if we wanted to bind something in a data structure, like the tuple #(answer 42)? This is one of the ways in which pattern matching is used in Erlang:

lfe> (set (tuple 'answer answer) `#(answer 42))
#(answer 42)
lfe> answer
42

If we want to capture the key name, too?

lfe> (set (tuple key answer) `#(answer 42))
#(answer 42)
lfe> answer
42
lfe> key
answer

And if we don’t care about the key at all?

lfe> (set (tuple _ answer) `#(answer 42))
#(answer 42)
lfe> answer
42

We can do the same thing in many other LFE forms, but here’s a quick example of variable assignment with pattern matching in a let form, again borrowing from the previous section:

(defun display-answer ()
  (let (((tuple _ answer) `#(answer 42)))
    (io:format "The answer is ~p~n" `(,answer))))

Pattern matching can be used in may places in LFE, but things really start getting interesting when you define function heads to extract specific values! We will cover examples of that later in the book, in addition to many others.

Global Variables Revisited

In traditional programming languages, global variables are widely considered dangerous and bug-prone. They introduce several critical problems:

• Race conditions: Multiple threads can modify global variables simultaneously, leading to unpredictable state and difficult-to-reproduce bugs
• Hidden side effects: Functions that modify globals create non-obvious dependencies, making code harder to understand and maintain
• Debugging difficulties: When any function can modify a global variable, tracing the source of bugs becomes extremely challenging
• Testing challenges: Global state creates hidden dependencies that make isolated unit testing nearly impossible

Erlang fundamentally solves these problems through immutable data. When you create a data structure in Erlang, its contents cannot be modified in-place. Operations create entirely new data structures, preserving the original. For example, updating a tuple element creates a new tuple rather than modifying the existing one. This eliminates race conditions, makes data flow explicit and predictable, and enables safer concurrent programming without the pitfalls of shared mutable state.

However, real-world applications still need to maintain state across function calls or share data between processes. Erlang provides several safe alternatives, each with its own trade-offs and appropriate use cases.

The Process Dictionary

The process dictionary is a key-value storage mechanism unique to each Erlang process. It provides simple functions like put/2, get/1, and erase/1 for storing and retrieving values within a process’s lifetime.

Benefits: The process dictionary offers quick access to process-local data without passing it explicitly through function parameters. Each process has its own isolated dictionary, preventing interference between processes. It’s particularly useful for cross-cutting concerns like request IDs in logging or maintaining parse state in complex recursive operations.

Weaknesses: Using the process dictionary breaks functional programming principles by introducing hidden mutable state. This makes code less transparent, harder to test, and more difficult to debug, as data flow becomes implicit rather than explicit. Most Erlang style guides recommend avoiding it except in specific cases where its convenience outweighs these concerns, such as instrumenting third-party libraries or maintaining debug context.

ETS Tables

ETS (Erlang Term Storage) provides in-memory database tables that can be shared across multiple processes. ETS supports several table types: set (unique keys), ordered_set (keys in sorted order), bag (multiple objects per key), and duplicate_bag (allowing duplicate objects).

Benefits: ETS tables provide extremely fast access—constant time O(1) for set tables and logarithmic time O(log N) for ordered_set tables. They support concurrent access with atomic and isolated operations, making them ideal for caching, session storage, counters, and high-performance in-memory data structures. Tables can store large amounts of data efficiently within the Erlang runtime, and access patterns don’t require message passing between processes.

Weaknesses: ETS tables have no automatic garbage collection—they persist until explicitly deleted or their owner process terminates. Select and match operations can be expensive as they typically scan the entire table unless properly indexed. Memory management requires careful attention, as tables exist outside normal process memory and don’t benefit from Erlang’s generational garbage collector. Overly complex match specifications can also become difficult to maintain.

State and OTP Servers

While huge chunks of this book are dedicated to OTP servers and state management, here’s the essential concept: gen_server is Erlang’s standard behavior for implementing stateful server processes.

A gen_server initializes state through an init/1 callback and maintains that state by passing it through callback functions like handle_call/3 and handle_cast/2. State updates work by creating new copies of data structures with the desired changes, then returning that new immutable data as the current state. This approach combines the benefits of persistent, mutable-seeming state with the safety guarantees of immutable data.

Benefits: OTP servers provide structured, predictable state management with built-in support for supervision, debugging, and code upgrades. State changes are centralized in a single process, making the system easier to reason about. The gen_server behavior handles message queuing, timeouts, and system messages automatically.

Weaknesses: All requests must pass through a single process, which can become a bottleneck under high load. Each state update creates new data structures, which can impact performance for very large or frequently-updated state. Care must be taken to avoid blocking operations that could make the server unresponsive.

External Databases

Erlang applications often connect to external databases for persistent storage. Mnesia, Erlang’s built-in distributed database, provides a native solution with strong integration into the Erlang ecosystem. It offers ACID transactions, table replication across nodes, and the ability to store tables in RAM, on disk, or both.

For other databases, Erlang has drivers and libraries for PostgreSQL, MySQL, MongoDB, Redis, and more. These typically use connection pools and message-passing patterns to maintain safety, with processes dedicated to managing database connections and handling queries asynchronously.

Benefits: External databases provide durable persistence across system restarts, ACID transaction guarantees, and the ability to handle datasets larger than available RAM. They enable data sharing across multiple applications and programming languages, and often include sophisticated query capabilities, indexes, and analytical tools. Databases like PostgreSQL offer mature backup, replication, and disaster recovery solutions.

Weaknesses: Database operations introduce network latency and I/O overhead, becoming a potential bottleneck in high-throughput systems. They create external dependencies that can affect system reliability and require careful connection management to avoid resource leaks. The impedance mismatch between Erlang’s term-based data model and SQL’s relational model can complicate data mapping. Database connections are limited resources that must be pooled and managed carefully.

Choosing the Right Approach

Each alternative serves different needs:

• Process Dictionary: Use sparingly for cross-cutting concerns or when integrating with third-party code
• ETS Tables: Ideal for shared, high-performance in-memory data that multiple processes need to access
• OTP Servers: The default choice for managing application state with clear ownership and structure
• External Databases: Essential for persistent data, large datasets, or sharing data across systems

The key insight is that Erlang doesn’t prevent you from maintaining state—it just ensures you do so safely and explicitly, avoiding the pitfalls of traditional global variables.

Primitive Types

This chapter covers the basic types of data available to LFE, upon which primatives rest all LFE libraries and applications.

• Integers
• Floats
• Atoms
• Booleans
• Characters

Each of these types has an LFE test function of the form TYPENAMEp (which wrap the respective Erlang is_TYPENAME function). These are used to perform type checks (especially common in guard expressions). These predicate functions will be covered in their respective type sections.

Integers

Integers in LFE may be either positive or negative and are by default base 10. Like Erlang, LFE does not have a maximum size of integer (in contrast to languages like C). This is accomplished via automatic conversion to larger (or smaller) internal representations (including the use of bignums).

lfe> 42
42
lfe> -42
-42
lfe> 1764
1764
lfe> 150130937545296561928688012959677941476701514734130607701636390322176
150130937545296561928688012959677941476701514734130607701636390322176

Bases

Several bases are supported via special syntax:

lfe> #b101010 ; binary
42
lfe> #o52 ; octal
42
lfe> #d42 ; decimal (explicit base 10)
42)
lfe> #x2a ; hexadecimal
42

Generic bases are supported, too:

lfe> #36r16
42

The number after the hash # is the base and may be any positive integer from 2 through 36. The number after the radix r is the actual value and must only bve comprised of integers allowed for the given base.

Converting between bases in LFE is most easily done via the integer_to_list Erlang function. For example:

lfe> (integer_to_list 1000 2)
"1111101000"
lfe> (integer_to_list 1000 8)
"1750"
lfe> (integer_to_list 1000 10)
"1000"
lfe> (integer_to_list 1000 16)
"3E8"
lfe> (integer_to_list 1000 36)
"RS"

If, for whatever reason, you want your base 10 integrer as a list (Erlang/LFE string), you can do that with this:

lfe> (integer_to_list 1000)
"1000"

Conversion to the binary type is also supported:

lfe> (integer_to_binary 1000)
#"1000"
lfe> (integer_to_binary 1000 2)
#"1111101000"
lfe> (integer_to_binary 1000 8)
#"1750"

The results above show LFE’s literal representations of binary data; this will be covered in the chapter on “Bytes and Binaries”.

Arithmetic Operators

Integers may be operated upon with the following:

lfe> (+ 1)
1
lfe> (+ 1 2)
3
lfe> (+ 1 2 3)
6
lfe> (- 1 2 3)
-4
lfe> (* 1 2 3)
6
lfe> (/ 1 2 3)
0.16666666666666666

Note that the division operator returns a float; floats will be covered in the next section.

Integer division is supported with a 2-arity function:

lfe> (div 1 2)
0
lfe> (div 10 2)
5

LFE also supports the remainder operation:

lfe> (rem 10 3)
1

As with any functional programming language, these operations may be composed (to any depth):

lfe> (div (* 12 (+ 1 2 3 4 5 6)) 6)
42

Mathematical Functions

The auto-loaded erlang module has several mathematical functions and is accessible in LFE without having to type the erlang: module prefix in the function calls. These include the following:

lfe> (abs -42)
42
lfe> (min 1 2)
1
lfe> (max 1 2)
2

Additional maths functions are provided via the math module. Since this module is not auto-loaded, in order to auto-complete it in the REPL you will need to load it:

lfe> (code:ensure_loaded 'math)
#(module math)

Now you can hit <TAB> after typing the following:

lfe> (math:

Which gives:

acos/1         acosh/1        asin/1         asinh/1        atan/1
atan2/2        atanh/1        ceil/1         cos/1          cosh/1
erf/1          erfc/1         exp/1          floor/1        fmod/2
log/1          log10/1        log2/1         module_info/0  module_info/1
pi/0           pow/2          sin/1          sinh/1         sqrt/1
tan/1          tanh/1

lfe> (round (math:pow 42 42))
150130937545296561928688012959677941476701514734130607701636390322176

The documentation for these functions is limited (available here) due in part to the fact that these are C library wrappers. Those that aren’t documented should be self-explanatory for anyone who has used simular mathematical functions in other programming language libraries.

Predicates

To test if a value is an integer, we will first include some code:

lfe> (include-lib "lfe/include/cl.lfe")

That adds Common Lisp inspired functions and macros to our REPL session.

lfe> (integerp 42)
true
lfe> (integerp 42.24)
false
lfe> (integerp "forty-two")
false

If you prefer the Clojure-style of predicates:

lfe> (include-lib "lfe/include/clj.lfe")
lfe> (integer? 42)
true
lfe> (integer? "forty-two")
false

Of course there is always the Erlang predicate, usable without having to do any includes:

lfe> (is_integer 42)
true
lfe> (is_integer "forty-two")
false

Floats

Real numbers in LFE are represented using the stadnard floating point numbers.

lfe> 42.42
42.42
lfe> (/ 10 3)
3.3333333333333335
lfe> (math:pow 42 42)
1.5013093754529656e68

Note that the ~1.5e68 above is the floating point equivalent of scientific notation, namely 1.5 x 1068. LFE follows the 64-bit standard for float representation given by IEEE 754-1985.

Converting

An integer may be converted to a float explicitly:

lfe> (float 42)
42.0

Or, as with integers, to binaries:

lfe> (float_to_binary 42.42)
#"4.24200000000000017053e+01"
lfe> (float_to_binary 42.42 '(#(scientific 10)))
#"4.2420000000e+01"
lfe> (float_to_binary 42.42 '(#(scientific 20)))
#"4.24200000000000017053e+01"
lfe> (float_to_binary 42.42 '(#(decimals 10)))
#"42.4200000000"
lfe> (float_to_binary 42.42 '(#(decimals 10) compact))
#"42.42"

Or to lists (LFE and Erlang strings):

lfe> (float_to_list 42.42 '(#(scientific 10)))
"4.2420000000e+01"
lfe> (float_to_list 42.42 '(#(scientific 20)))
"4.24200000000000017053e+01"
lfe> (float_to_list 42.42 '(#(decimals 10)))
"42.4200000000"
lfe> (float_to_list 42.42 '(#(decimals 10) compact))
"42.42"

Formatting

If you need to round floating point numbers to a specific precision, you’ll want to use the format function from either the io, io_lib, or lfe_io modules. If just want to print a value using Erlang syntax and formatting, the io module is what you want. If you prefer LFE syntax and formatting for your output, you’ll want to use the lfe_io module. If you want to use the data or store it in a variable, you’ll need the io_lib library.

For default precision:

lfe> (io_lib:format "~f" `(,(math:pow 42 42)))
"150130937545296561929000000000000000000000000000000000000000000000000.000000"

Two decimal places:

lfe> (io_lib:format "~.2f" `(,(math:pow 42 42)))
"150130937545296561929000000000000000000000000000000000000000000000000.00"

20 decimal places:

lfe> (io_lib:format "~.20f" `(,(math:pow 42 42)))
"150130937545296561929000000000000000000000000000000000000000000000000.00000000000000000000"

Arithmetic Operators & Mathematical Functions

Floats use most of the same operators and functions as integers, so be sure to review these subsections in the “Integers” section.

Others include:

lfe> (abs -42)
42
lfe> (ceil 42.1)
43
lfe> (floor 42.1)
42
lfe> (round 42.4)
42
lfe> (round 42.5)
43
lfe> (min 1 2)
1
lfe> (max 1 2)
2

Predicates

To test if a value is a float, we will first include some code:

lfe> (include-lib "lfe/include/cl.lfe")

That adds Common Lisp inspired functions and macros to our REPL session.

lfe> (floatp 42.42)
true
lfe> (floatp 42)
false
lfe> (floatp "forty-two.forty-two")
false

If you prefer the Clojure-style of predicates:

lfe> (include-lib "lfe/include/clj.lfe")
lfe> (float? 42.42)
true
lfe> (float? "forty-two.forty-two")
false

Of course there is always the Erlang predicate, usable without having to do any includes:

lfe> (is_float 42.42)
true
lfe> (is_float "forty-two.forty-two")
false

Atoms

The cloest analog in LFE to what most Lisp dialects call symbols is the Erlang atom. Just as with Lisps do with symbols, LFE uses atoms for its variable and function names. Atoms are literals and constants, which means that their value is the same as their name and once created, cannot be changed.

Some basic examples of atoms:

lfe> 'zark
zark
lfe> 'zarking-amazing
zarking-amazing

Slightly less straight-forward examples which start with non-alphanumeric characters:

lfe> ':answer
:answer
lfe> '42answer
42answer
lfe> '42.0e42answer
42.0e42answer
lfe> '42°C
42°C

Standard LFE atom names may be comprised of all the latin-1 character set except the following:

• control character
• whitespace
• the various brackets
• double quotes
• semicolon

Of these, only |, \, ', ,, and # may not be the first character of the symbol’s name (but they are allowed as subsequent letters).

Non-standard atom names may be created using atom quotes:

lfe> '|symbol name with spaces|
|symbol name with spaces|
lfe> '|'with staring quote!|
|'with staring quote!|
lfe> '|| ; <-- empty atoms are supported too
||
lfe> '|really weird atom: '#[]{}()<>";\||
|really weird atom: '#[]{}()<>";\||

In this case the name can contain any character of in the range from 0 to 255, and even no character at all.

In the case of atoms, it is important to understand a little something about their internals. In particular, Erlang and LFE atoms are global for each instance of a running Erlang virtual machine. Atoms are maintained by the VM in a table and are not garbage collected. By default, the Erlang atom table allows for a maximum of 1,048,576 entries.

As Symbols

The following code shows LFE’s use of atoms in variable names. First, let’s use a function for a slighly different purpose than designed: calling list_to_existing_atom on a string will only return a result if an atom of the same name already exists in the atom table. Otherwise, it will return an error:

lfe> (list_to_existing_atom "zaphod")
** exception error: bad argument
  in (erlang : list_to_existing_atom "zaphod")

This confirms that there is no zaphod atom in the atom table.

Now let’s create a variable, assigning a value to it, and then use our indirect means of checkihg the atom table:

lfe> (set zaphod "frood")
"frood"
lfe> (list_to_existing_atom "zaphod")
zaphod

And here’s an example showing LFE’s use of atoms in function names using the same approach as above:

lfe> (list_to_existing_atom "beez")
** exception error: bad argument
  in (erlang : list_to_existing_atom "beez")

lfe> (defun beez (x) x)
beez
lfe> (list_to_existing_atom "beez")
beez

Converting

Atoms may be converted to strings and bitstrings, and vice versa.

(atom_to_binary 'trisha)
#"trisha"
(atom_to_binary 'mcmillan 'latin1)
#"mcmillan"
lfe> (atom_to_list 'trillian)
"trillian"

Note that the first one above is only available in Erlang 23.0 and above.

Some more examples for encoding:

lfe> (atom_to_binary '42°C 'latin1)
#B(52 50 176 67)
lfe> (atom_to_binary '42°C 'utf8)
#"42°C"

Functions that convert atoms only if they already exist in the atom table:

lfe> (binary_to_existing_atom #"trisha")
trisha
lfe> (binary_to_existing_atom #"trisha" 'latin1)
trisha
lfe> (list_to_existing_atom "trisha")
trisha

Operators

The only operators you may use on atoms are the comparison operators, e.g.:

lfe> (> 'a 'b)
false
lfe> (< 'a 'b)
true
lfe> (=:= 'a 'a)
true

Predicates

To test if a value is an atom, we will first include some code:

lfe> (include-lib "lfe/include/cl.lfe")

That adds Common Lisp inspired functions and macros to our REPL session.

lfe> (atomp 'arthur)
true
lfe> (atomp 42)
false
lfe> (atomp "forty-two.forty-two")
false

If the atom in question has been used in a function name definition:

If you prefer the Clojure-style of predicates:

lfe> (include-lib "lfe/include/clj.lfe")
lfe> (atom? 'dent)
true
lfe> (atom? "Ford")
false

Of course there is always the Erlang predicate, usable without having to do any includes:

lfe> (is_atom 'arthur)
true
lfe> (is_atom "forty-two.forty-two")
false

Caveats

As mentioned above (and documented), one needs to take care when creating atoms. By default, the maximum number of atoms that the Erlang VM will allow is 1,048,576; any more than that, and the VM will crash.

The first rule of thumb is not to write any code that generates large numbers of atoms. More explicitly useful, there are some handy functions for keeping track of the atom table, should you have the need.

lfe> (erlang:memory 'atom_used)
244562
lfe> (erlang:system_info 'atom_count)
9570
lfe> (erlang:system_info 'atom_limit)
1048576

Note that support for easily extracting the current atom data from system_info – as demonstrated by the last two function calls above – were added in Erlang 20; should you be running an older version of Erlang, you will need to parse the (system_info 'info) bitstring.

The default atom table size may be overridden during startup by passing a value with the +t options:

$ lfe +t 200000001

lfe> (erlang:system_info 'atom_limit)
200000001

Booleans

Strictly speaking, LFE has no Boolean type, just like Erlang. Instead, the atoms true and false are treated as Booleans

lfe> (== 42 "the question")
false
lfe> (== 42 42)
true
lfe> (> 1 2)
false
lfe> (< 1 2)
true

Operators

The standard logical operators are available to LFE Booleans:

lfe> (not 'true)
false
lfe> (not 'false)
true
lfe> (and 'true 'false)
false
lfe> (or 'true 'false)
true
lfe> (and 'true 'true 'true)
true
lfe> (and 'true 'true 'false)
false
lfe> (or 'false 'false 'false)
false
lfe> (or 'false 'false 'true)
true
lfe> (xor 'true 'true)
false
lfe> (xor 'true 'false)
true
lfe> (xor 'false 'false)
false
lfe> (xor 'false 'true)
true

With the and and or Boolean oprators, every argument is evaluated. To accomodate situations where complex and possible expensive logical expressions comprise the arguments to Boolean operators, short-circuit versions of these functions are also provided:

• with andalso, returns as soon as the first 'false is encountered;
• with orelse, returns as soon as the first 'true is encountered.

To demonstrate this, we’ll define a boolean function that prints to standard out when it is evaluated:

(defun hey (x)
  (io:format "Made it here!~n") x)

Short-circuit demonstration of andalso:

lfe> (andalso 'true 'true 'false (hey 'true))
false
lfe> (andalso 'false 'true 'true (hey 'true))
false
lfe> (andalso 'true 'true 'true (hey 'true))
Made it here!
true

Short-circuit demonstration of orelse:

lfe> (orelse 'false 'false 'true (hey 'true))
true
lfe> (orelse 'true 'false 'false (hey 'true))
true
lfe> (orelse 'false 'false 'false (hey 'true))
Made it here!
true

Predicates

To test if a value is a Boolean, we will first include some code:

lfe> (include-lib "lfe/include/cl.lfe")

That adds Common Lisp inspired functions and macros to our REPL session.

lfe> (booleanp 'true)
true
lfe> (booleanp 'false)
true
lfe> (booleanp 'arthur)
false
lfe> (booleanp 42)
false

If the atom in question has been used in a function name definition:

If you prefer the Clojure-style of predicates:

lfe> (include-lib "lfe/include/clj.lfe")
lfe> (boolean? 'true)
true
lfe> (boolean? 'false)
true
lfe> (boolean? 'arthur)
false

Of course there is always the Erlang predicate, usable without having to do any includes:

lfe> (is_boolean 'true)
true
lfe> (is_boolean 'false)
true
lfe> (is_boolean 'arthur)
false

Note that, since LFE Booleans are also atoms, these are valid as well:

lfe> (atomp 'true)
true
lfe> (atom? 'false)
true
lfe> (is_atom 'true)
true

Characters

Characters in LFE Are represented internally by integers, however a literal syntax is offered for convenience:

lfe> #\a
97
lfe> #\A
65
lfe> #\ü
252
lfe> #\Æ
198

Converting

Since a character literal and integer are the same thing as far as LFE is concerned, there is no such thing as converting between a “char” and “ord” like there is in some other languages.

However, one can format an integer as a string by telling the class of format functions that the input is “character” type:

lfe> (io_lib:format "~c" `(198))
"Æ"

For merely printing to standard out instead of returning a value, one may use:

lfe> (lfe_io:format "~c~n" `(198))
Æ
ok
lfe> (io:format "~c~n" `(198))
Æ
ok

Operators

All operations that are valid for integers are valid for characters.

Predicates

All predicates that are valid for integers are valid for characters.

Cons Cells

Cons cells are the fundamental building blocks of lists in both Lisp and Erlang, and by extension, LFE. Understanding cons cells is essential to mastering list manipulation and pattern matching in LFE. While the concept originates from Lisp, Erlang (and thus LFE) adapts it in ways that make it particularly powerful for functional programming and recursive operations.

At first encounter, cons cells might seem like an unnecessarily complicated way to think about lists—after all, why not just have an array-like structure that holds multiple values? The answer lies in the elegance and efficiency they provide for functional programming. Think of a cons cell as a simple container with exactly two compartments: one holds a value (the “head” or “first element”), and the other holds a pointer to the rest of the list (the “tail”). This structure creates a chain—like a linked list in other languages—but with an important difference: in functional programming, these chains are immutable. When you “add” an element to the front of a list, you’re not modifying the original list; you’re creating a new cons cell that points to the existing list. This makes operations like prepending elements incredibly fast (O(1) constant time) and allows multiple “versions” of a list to safely share structure in memory.

The real power of cons cells emerges when combined with pattern matching. Instead of writing imperative code that indexes into lists or checks for empty conditions, you can write function definitions that naturally express “here’s what to do with an empty list, and here’s what to do with a list that has at least one element.” The cons cell structure makes this pattern matching both intuitive and efficient: destructuring a list into its head and tail is a natural operation that matches how the data is actually stored in memory. As you work through the following sections, you’ll see how this simple two-slot structure enables elegant recursive algorithms, efficient list manipulation, and a programming style that feels natural once the initial conceptual hurdle is overcome.

Lisp Cons Cells

In Lisp dialects, a cons cell is a fundamental data structure that holds two values or pointers to values. These two slots are traditionally called the CAR (Contents of the Address Register) and the CDR (Contents of the Decrement Register), names that derive from the original Lisp implementation on IBM 704 hardware.

The cons function constructs these memory objects, and the expression “to cons x onto y” means to construct a new cons cell with x in the car slot and y in the cdr slot.

Structure and Notation

A simple cons cell holding two values can be represented in dotted pair notation:

(cons 'a 'b)  ; Creates a cons cell
=> (a . b)    ; Dotted pair notation

Lists in Lisp are built by having the car slot contain an element and the cdr slot point to another cons cell or to nil (the empty list). This creates a singly-linked list structure:

(cons 1 (cons 2 (cons 3 nil)))
=> (1 2 3)

Visual Representation

Here’s a diagram showing how the list (1 2 3) is constructed from cons cells:

graph LR
    A["cons cell 1"] -->|car| V1["1"]
    A -->|cdr| B["cons cell 2"]
    B -->|car| V2["2"]
    B -->|cdr| C["cons cell 3"]
    C -->|car| V3["3"]
    C -->|cdr| N["nil"]

    style A fill:#e1f5ff
    style B fill:#e1f5ff
    style C fill:#e1f5ff
    style V1 fill:#fff4e1
    style V2 fill:#fff4e1
    style V3 fill:#fff4e1
    style N fill:#ffe1e1

Each cons cell contains two pointers: the car points to the element value, and the cdr points to the next cons cell (or nil for the last cell).

Erlang Cons Cells

Erlang lists are built as sequences of cons cells, with each cell composed of a value and a pointer to another cons cell or empty list. While structurally similar to Lisp cons cells, Erlang’s implementation and usage patterns differ in important ways.

The Pipe Operator

In Erlang, the cons operator is represented by the pipe symbol (|), which separates the head of a list from its tail. The syntax [Head | Tail] constructs or pattern-matches a cons cell.

% Building a list with cons
[1 | [2, 3]]      % => [1, 2, 3]
[1 | [2 | [3]]]   % => [1, 2, 3]
[1 | [2 | [3 | []]]]  % => [1, 2, 3]

Pattern Matching and Head/Tail

Every function operating on lists in Erlang is defined in terms of two primitives: head and tail, which return the first element and the rest of the list respectively. Pattern matching with cons cells provides an elegant idiom for recursive list operations, where you can extract the head and tail in function definitions.

% Pattern matching to extract head and tail
[Head | Tail] = [1, 2, 3].
% Head => 1
% Tail => [2, 3]

This makes recursive list processing natural and efficient:

length([]) -> 0;
length([_Head | Tail]) -> 1 + length(Tail).

Proper vs Improper Lists

A proper list in Erlang ends with an empty list as its last cell. When the tail of the last cons cell contains something other than another cons cell or the empty list, you have an improper list:

[1 | 2]      % Improper list - tail is not a list
[1 | [2]]    % Proper list

While improper lists are valid Erlang terms, most standard list functions expect proper lists.

LFE Cons Cells

LFE provides Lisp-style syntax for working with Erlang’s cons cell implementation, giving you the best of both worlds: familiar Lisp notation with Erlang’s powerful pattern matching.

The cons Function

In LFE, the cons function creates a new list by prepending an element to an existing list:

lfe> (cons 1 '())
(1)
lfe> (cons 1 '(2 3))
(1 2 3)
lfe> (cons 1 (cons 2 (cons 3 '())))
(1 2 3)

Pattern Matching with Cons

LFE allows pattern matching on cons cells in function definitions, making recursive list processing elegant and readable:

lfe> (defun my-length
  (('()) 0)
  (((cons _ tail)) (+ 1 (my-length tail))))
my-length
lfe> (my-length '(a b c d))
4

In this example:

• The first clause matches the empty list () and returns 0
• The second clause uses (cons _ tail) to extract the tail, ignoring the head with _
• The function recursively processes the tail

Common Patterns

Here are some common patterns for working with cons cells in LFE:

Accessing the head and tail:

lfe> (set (cons head tail) '(1 2 3 4))
(1 2 3 4)
lfe> head
1
lfe> tail
(2 3 4)

Building lists incrementally:

lfe> (defun build-list (n)
  (build-list n '()))

lfe> (defun build-list
  ((0 acc) acc)
  ((n acc) (build-list (- n 1) (cons n acc))))

lfe> (build-list 5)
(1 2 3 4 5)

Pattern matching in let bindings:

lfe> (let (((cons first (cons second rest)) '(a b c d e)))
  (list first second rest))
(a b (c d e))

Recursive list transformation:

lfe> (defun double-all
  (('()) '())
  (((cons h t)) (cons (* 2 h) (double-all t))))

lfe> (double-all '(1 2 3 4 5))
(2 4 6 8 10)

Using Backtick Syntax

LFE also supports backtick (quasiquote) syntax for pattern matching, which can be more concise:

lfe> (set `(,first ,second . ,rest) '(1 2 3 4 5))
(1 2 3 4 5)
lfe> first
1
lfe> second
2
lfe> rest
(3 4 5)

The dot (.) in the pattern (,first ,second . ,rest) represents the cons operator, separating the explicitly matched elements from the remaining tail.

List Construction vs Traversal

One crucial performance consideration: prepending to a list with cons is O(1), but appending to the end requires traversing the entire list and is O(n):

; Fast - O(1)
lfe> (cons 0 '(1 2 3))
(0 1 2 3)

; Slow for large lists - O(n)
lfe> (++ '(1 2 3) '(4))
(1 2 3 4)

This is why many recursive functions in LFE build lists in reverse order using an accumulator, then reverse the final result:

(defun map-helper (f lst acc)
  (case lst
    ('() (lists:reverse acc))
    ((cons h t) (map-helper f t (cons (funcall f h) acc)))))

(defun my-map (f lst)
  (map-helper f lst '()))

Predicates

To check if a value is a list (a chain of cons cells ending in []), you can use the standard predicates:

lfe> (is_list '(1 2 3))
true
lfe> (is_list '())
true
lfe> (is_list 42)
false

With Common Lisp-style predicates:

lfe> (include-lib "lfe/include/cl.lfe")
lfe> (listp '(1 2 3))
true
lfe> (listp '())
true

Or Clojure-style:

lfe> (include-lib "lfe/include/clj.lfe")
lfe> (list? '(1 2 3))
true

Summary

Cons cells are the foundation of list processing in LFE. Understanding how they work—as pairs of values forming linked structures—is essential for effective functional programming. The ability to pattern match on cons cells makes LFE code both elegant and efficient, allowing you to express complex list operations with clarity and precision.

The key insights to remember:

• Lists are chains of cons cells terminating in the empty list ()
• Pattern matching with (cons head tail) is the idiomatic way to destructure lists
• Prepending with cons is fast; building lists in reverse and then reversing is a common pattern
• Proper lists always end in [], while improper lists end in other values

Lists and Strings

In most programming languages, lists and strings are distinct creatures—separate types with their own operations, behaviors, and internal representations. In LFE, however, these two concepts share a rather intimate relationship: a string is a list. More precisely, a string is simply a list of integers, where each integer represents the character code of a printable character. This might seem peculiar at first, perhaps even a bit unsettling if you’re accustomed to languages where strings are special, privileged types with their own dedicated machinery.

But consider the elegance of this approach: if strings are just lists, then all the wonderful things you can do with lists—pattern matching, recursive processing, higher-order functions—work equally well on strings. Want to reverse a string? Use the same function you’d use to reverse any list. Want to filter characters? Use the same filter function you’d use on a list of numbers or atoms. This unification means fewer concepts to learn, more code reuse, and a consistency that permeates the entire language.

The practical implication is straightforward: when you see "hello" in LFE, you’re looking at syntactic sugar for (104 101 108 108 111)—a list of five integers corresponding to the ASCII values of those letters. The REPL helpfully displays lists of printable character codes as strings, but underneath, it’s lists all the way down. This chapter explores both lists and strings, first separately to understand their individual characteristics, then together to appreciate how this design choice shapes the way you write LFE code.

Lists

In the beginning, there was the cons cell. And from the cons cell came forth lists, which were chains of cons cells stretching out across memory like improbability particles across the fabric of spacetime—each pointing to the next with what can only be described as dogged determination. And lo, these lists were good, for they were immutable, they were elegant, and most importantly, they made recursion look less like wizardry and more like afternoon tea.

Lists are the fundamental sequential data structure in LFE, inherited from both Erlang’s practical heritage and Lisp’s theoretical elegance. They are singly-linked lists built from cons cells (as explored in the previous chapter), making them ideally suited for functional programming patterns where you process data from front to back, building new structures as you go rather than mutating old ones.

Creating Lists

The most straightforward way to create a list is with the quote:

lfe> '(1 2 3 4 5)
(1 2 3 4 5)
lfe> '(arthur ford trillian zaphod)
(arthur ford trillian zaphod)
lfe> '()
()

You can also use the list function:

lfe> (list 1 2 3 4 5)
(1 2 3 4 5)
lfe> (list 'arthur 'ford 'trillian 'zaphod)
(arthur ford trillian zaphod)
lfe> (list)
()

Lists can contain elements of mixed types:

lfe> '(42 "meaning" of life 42.0)
(42 "meaning" of life 42.0)
lfe> (list 'atom 42 3.14 "string" #b101010 '(nested list))
(atom 42 3.14 "string" 42 (nested list))

Building lists with cons:

lfe> (cons 1 '())
(1)
lfe> (cons 1 (cons 2 (cons 3 '())))
(1 2 3)
lfe> (cons 'first '(second third))
(first second third)

List Operations

Accessing elements:

lfe> (car '(1 2 3 4 5))
1
lfe> (cdr '(1 2 3 4 5))
(2 3 4 5)
lfe> (cadr '(1 2 3 4 5))  ; equivalent to (car (cdr ...))
2
lfe> (cddr '(1 2 3 4 5))  ; equivalent to (cdr (cdr ...))
(3 4 5)

LFE provides Common Lisp-style accessor combinations up to four levels deep:

lfe> (caddr '(1 2 3 4))   ; third element
3
lfe> (cadddr '(1 2 3 4))  ; fourth element
4

Finding length:

lfe> (length '(1 2 3 4 5))
5
lfe> (length '())
0
lfe> (length '(a (nested (list structure)) here))
3

Appending lists:

lfe> (++ '(1 2 3) '(4 5 6))
(1 2 3 4 5 6)
lfe> (++ '(a) '(b) '(c) '(d))
(a b c d)
lfe> (append '(1 2) '(3 4) '(5 6))
(1 2 3 4 5 6)

Note that ++ is the operator form while append is the function form. Both are O(n) operations where n is the total length of all lists except the last one.

Reversing lists:

lfe> (lists:reverse '(1 2 3 4 5))
(5 4 3 2 1)
lfe> (lists:reverse '())
()

Taking and dropping elements:

lfe> (lists:sublist '(1 2 3 4 5) 3)
(1 2 3)
lfe> (lists:sublist '(1 2 3 4 5) 2 3)
(2 3 4)
lfe> (lists:nthtail 2 '(1 2 3 4 5))
(3 4 5)

Membership testing:

lfe> (lists:member 3 '(1 2 3 4 5))
(3 4 5)
lfe> (lists:member 6 '(1 2 3 4 5))
false

Note that lists:member returns the tail of the list starting with the found element, or false if not found.

Finding elements:

lfe> (lists:nth 1 '(a b c d))
a
lfe> (lists:nth 3 '(a b c d))
c

Be aware that lists:nth uses 1-based indexing (Erlang convention), unlike 0-based indexing common in many languages.

Sorting:

lfe> (lists:sort '(5 2 8 1 9 3))
(1 2 3 5 8 9)
lfe> (lists:sort '(zaphod arthur ford trillian))
(arthur ford trillian zaphod)
lfe> (lists:sort (lambda (a b) (> a b)) '(5 2 8 1 9 3))
(9 8 5 3 2 1)

Higher-Order List Functions

Mapping:

lfe> (lists:map (lambda (x) (* x 2)) '(1 2 3 4 5))
(2 4 6 8 10)
lfe> (lists:map #'atom_to_list/1 '(arthur ford trillian))
("arthur" "ford" "trillian")

Filtering:

lfe> (lists:filter (lambda (x) (> x 3)) '(1 2 3 4 5 6))
(4 5 6)
lfe> (lists:filter #'is_atom/1 '(1 arthur 2 ford 3))
(arthur ford)

Folding (reducing):

lfe> (lists:foldl #'+/2 0 '(1 2 3 4 5))
15
lfe> (lists:foldl (lambda (x acc) (cons x acc)) '() '(1 2 3))
(3 2 1)
lfe> (lists:foldr (lambda (x acc) (cons x acc)) '() '(1 2 3))
(1 2 3)

The difference between foldl (fold left) and foldr (fold right) is the direction of traversal. foldl is tail-recursive and generally more efficient, while foldr processes from right to left.

Flatmapping:

lfe> (lists:flatmap (lambda (x) (list x (* x 2))) '(1 2 3))
(1 2 2 4 3 6)

Zipping:

lfe> (lists:zip '(1 2 3) '(a b c))
((1 a) (2 b) (3 c))
lfe> (lists:zip3 '(1 2 3) '(a b c) '(x y z))
((1 a x) (2 b y) (3 c z))

List Comprehensions

LFE supports powerful list comprehensions that combine filtering, mapping, and Cartesian products:

lfe> (lc ((<- x '(1 2 3 4 5))) (* x 2))
(2 4 6 8 10)

lfe> (lc ((<- x '(1 2 3 4 5 6)) (> x 3)) (* x x))
(16 25 36)

lfe> (lc ((<- x '(1 2 3)) (<- y '(a b))) (tuple x y))
(#(1 a) #(1 b) #(2 a) #(2 b) #(3 a) #(3 b))

The <- operator draws elements from a list, and optional guard conditions filter elements before the body expression is evaluated.

Pattern Matching with Lists

List pattern matching is one of the most elegant features in LFE:

lfe> (set (list first second) '(1 2))
(1 2)
lfe> first
1
lfe> second
2

lfe> (set (cons head tail) '(1 2 3 4))
(1 2 3 4)
lfe> head
1
lfe> tail
(2 3 4)

lfe> (set `(,a ,b . ,rest) '(1 2 3 4 5))
(1 2 3 4 5)
lfe> a
1
lfe> b
2
lfe> rest
(3 4 5)

Predicates

To test if a value is a list, first include the Common Lisp compatibility library:

lfe> (include-lib "lfe/include/cl.lfe")
lfe> (listp '(1 2 3))
true
lfe> (listp '())
true
lfe> (listp 42)
false

Clojure-style predicates:

lfe> (include-lib "lfe/include/clj.lfe")
lfe> (list? '(1 2 3))
true
lfe> (list? "string")
false

Standard Erlang predicate:

lfe> (is_list '(1 2 3))
true
lfe> (is_list '())
true
lfe> (is_list 42)
false

Performance Considerations

Understanding the performance characteristics of list operations is crucial for writing efficient LFE code:

• Prepending (cons) is O(1) — constant time, very fast
• Appending (++, append) is O(n) — requires traversing the entire first list
• Length is O(n) — must traverse the entire list
• Accessing by index is O(n) — must traverse to the position
• Reversing is O(n) — must traverse the entire list

For these reasons, idiomatic LFE code often:

• Builds lists in reverse order then reverses once at the end
• Uses cons instead of append when possible
• Avoids repeated length calculations
• Uses pattern matching instead of index access

Example of the reverse-and-accumulate pattern:

(defun process-list (lst)
  (process-list-helper lst '()))

(defun process-list-helper
  (('() acc) (lists:reverse acc))
  (((cons h t) acc)
   (let ((processed (* h 2)))
     (process-list-helper t (cons processed acc)))))

Common Patterns

Collecting results:

(defun collect-evens
  (('()) '())
  (((cons h t)) (when (== (rem h 2) 0))
   (cons h (collect-evens t)))
  (((cons _ t))
   (collect-evens t)))

Processing pairs:

(defun process-pairs
  (('()) '())
  (((list x)) (list x))
  (((list x y . rest))
   (cons (+ x y) (process-pairs rest))))

Searching:

(defun find-first
  ((_ '()) 'not-found)
  ((pred (cons h t))
   (case (funcall pred h)
     ('true h)
     ('false (find-first pred t)))))

Strings

In LFE, as in Erlang, strings are simply lists of integers representing character codes. This might seem unusual if you’re coming from languages with dedicated string types, but it’s a perfectly sensible representation that fits naturally with list processing. A string is just a list where each element happens to be an integer in the valid character range.

Creating Strings

String literals are enclosed in double quotes:

lfe> "Hello, World!"
"Hello, World!"
lfe> "The answer is 42"
"The answer is 42"
lfe> ""
""

Under the hood, these are lists of integers:

lfe> (== "ABC" '(65 66 67))
true
lfe> (== "ABC" (list 65 66 67))
true

You can see this more clearly by forcing LFE to show the underlying representation:

lfe> (io:format "~w~n" '("ABC"))
[65,66,67]
ok

Character literals make this relationship explicit:

lfe> (list #\A #\B #\C)
"ABC"
lfe> #\A
65

String Operations

Since strings are lists, all list operations work on strings:

lfe> (++ "Hello, " "World!")
"Hello, World!"
lfe> (lists:reverse "stressed")
"desserts"
lfe> (length "Hello")
5

Accessing characters:

lfe> (car "Hello")
72
lfe> (cdr "Hello")
"ello"
lfe> (lists:nth 2 "Hello")
101

Case conversion:

lfe> (string:uppercase "hello world")
"HELLO WORLD"
lfe> (string:lowercase "HELLO WORLD")
"hello world"
lfe> (string:titlecase "hello world")
"Hello World"

Trimming whitespace:

lfe> (string:trim "  hello  ")
"hello"
lfe> (string:trim "  hello  " 'leading)
"hello  "
lfe> (string:trim "  hello  " 'trailing)
"  hello"

Splitting and joining:

lfe> (string:split "one,two,three" "," 'all)
("one" "two" "three")
lfe> (string:split "a:b:c:d" ":")
("a" "b:c:d")
lfe> (string:split "a:b:c:d" ":" 'all)
("a" "b" "c" "d")

lfe> (string:join '("one" "two" "three") ", ")
"one, two, three"

Searching:

lfe> (string:find "hello world" "world")
"world"
lfe> (string:find "hello world" "universe")
'nomatch

lfe> (string:str "hello world" "world")
7
lfe> (string:str "hello world" "universe")
0

Note that string:str returns 1-based position or 0 if not found.

Replacing:

lfe> (string:replace "hello world" "world" "universe")
"hello universe"
lfe> (string:replace "a,b,c,d" "," ":" 'all)
"a:b:c:d"

Checking prefixes and suffixes:

lfe> (string:prefix "hello world" "hello")
"world"
lfe> (string:prefix "hello world" "goodbye")
'nomatch

lfe> (lists:prefix "hello" "hello world")
true
lfe> (lists:prefix "goodbye" "hello world")
false

lfe> (lists:suffix "world" "hello world")
true

String Predicates

Testing for empty strings:

lfe> (string:is_empty "")
true
lfe> (string:is_empty "hello")
false

Testing if all characters match a predicate:

lfe> (lists:all (lambda (c) (and (>= c #\a) (=< c #\z))) "hello")
true
lfe> (lists:all (lambda (c) (and (>= c #\a) (=< c #\z))) "Hello")
false

Formatting Strings

The format family of functions provides powerful string interpolation:

lfe> (io_lib:format "The answer is ~p" '(42))
"The answer is 42"

lfe> (io_lib:format "~s ~s ~p" '("Hello" "world" 123))
"Hello world 123"

lfe> (io_lib:format "~.2f" '(3.14159))
"3.14"

lfe> (io_lib:format "~10.2.0f" '(3.14159))
"      3.14"

Common format specifiers:

• ~p — pretty-print any term
• ~s — string
• ~w — write in Erlang term format
• ~c — character
• ~f — float
• ~e — exponential notation
• ~.Nf — float with N decimal places
• ~n — newline

For console output, use lfe_io:format or io:format:

lfe> (lfe_io:format "Hello, ~s!~n" '("World"))
Hello, World!
ok

lfe> (io:format "The answer is ~p~n" '(42))
The answer is 42
ok

Converting Between Types

Atoms to strings:

lfe> (atom_to_list 'hello)
"hello"
lfe> (atom_to_list 'hello-world)
"hello-world"

Strings to atoms:

lfe> (list_to_atom "hello")
hello
lfe> (list_to_existing_atom "hello")
hello

Numbers to strings:

lfe> (integer_to_list 42)
"42"
lfe> (integer_to_list 42 16)
"2A"

lfe> (float_to_list 3.14159)
"3.14159000000000009237e+00"
lfe> (float_to_list 3.14159 '(#(decimals 2)))
"3.14"

Strings to numbers:

lfe> (list_to_integer "42")
42
lfe> (list_to_integer "2A" 16)
42

lfe> (list_to_float "3.14159")
3.14159

Binaries to strings:

lfe> (binary_to_list #"Hello")
"Hello"
lfe> (list_to_binary "Hello")
#"Hello"

Unicode Support

LFE strings can contain Unicode characters, though the integer representation may be surprising:

lfe> "Hello, 世界"
"Hello, 世界"

lfe> (length "Hello, 世界")
9

lfe> (lists:nth 8 "Hello, 世界")
19990

For proper Unicode string handling, especially with grapheme clusters, use the Unicode-aware functions in the string module:

lfe> (string:length "Hello, 世界")
9

lfe> (string:slice "Hello, 世界" 7 2)
"世界"

The modern string module (Erlang 20+) handles Unicode correctly and should be preferred over older list-based string functions when working with international text.

Common Patterns

Building strings incrementally:

(defun build-message (name age)
  (++ "Name: " name ", Age: " (integer_to_list age)))

Processing each character:

(defun count-vowels
  (("") 0)
  (((cons c rest))
   (let ((is-vowel (lists:member c "aeiouAEIOU")))
     (if is-vowel
         (+ 1 (count-vowels rest))
         (count-vowels rest)))))

Filtering characters:

(defun remove-spaces
  (("") "")
  (((cons #\space rest)) (remove-spaces rest))
  (((cons c rest)) (cons c (remove-spaces rest))))

Using list comprehensions on strings:

lfe> (lc ((<- c "hello")) (- c 32))
"HELLO"

lfe> (lc ((<- c "hello world") (=/= c #\space)) c)
"helloworld"

Performance Notes

String operations have the same performance characteristics as list operations because they are list operations:

• Concatenation with ++ is O(n) in the length of the first string
• Repeated concatenation in a loop can be O(n²)
• For building large strings, accumulate in a list and call lists:flatten once
• Consider using binaries for large text or when performance is critical

Example of efficient string building:

(defun build-csv-row (items)
  (lists:flatten
    (lists:join ","
      (lists:map #'format-item/1 items))))

Summary

Strings in LFE are lists of character codes—simple, elegant, and slightly peculiar if you’re new to this representation. This design choice means:

• All list operations work on strings
• Strings naturally integrate with pattern matching
• String manipulation is intuitive if you think in terms of list processing
• Performance characteristics match those of lists
• Unicode requires some care but is fully supported

For most string operations in modern code, prefer the string module’s Unicode-aware functions. For performance-critical text processing with large strings, consider using binaries (covered in a later chapter). But for everyday string manipulation, list-based strings remain a perfectly reasonable and often elegant choice.

Remember: in LFE, as in life, it’s not about having a separate type for everything—it’s about recognizing that sometimes, a list of integers masquerading as text is exactly what you need.

Bits, Bytes, and Binaries in LFE

Or: How I Learned to Stop Worrying and Love the Bit Stream

Introduction

In most programming languages, manipulating binary data at the bit level is rather like performing surgery with a pickaxe. It’s technically possible, and if you squint hard enough while wielding your bit shifts and masking operations, you might even get something resembling the desired result. But it won’t be pretty, and there’s a non-zero chance you’ll need therapy afterward.

LFE, inheriting Erlang’s particularly sophisticated relationship with binary data, takes a different approach. Instead of making you grovel through arcane incantations of bitwise operations (though those are available if you’re feeling nostalgic), it provides what can only be described as pattern matching on steroids. Or perhaps pattern matching that’s been to finishing school and learned proper table manners. The result is that tasks which would normally require careful bit-shuffling and a comprehensive insurance policy can be accomplished with code that looks suspiciously like the specification of the data format you’re working with.

This is not an accident. It’s the kind of elegant design that makes you wonder what other programming languages have been doing with their time.

Why Binaries Matter

Let us consider, for a moment, the humble byte. Eight bits, representing values from 0 to 255 (or -128 to 127 if you’re feeling signed about it). Unremarkable in isolation, perhaps, but stack enough of them together and you have:

• Network protocol packets pretending to be well-behaved structured data
• Image files storing millions of pixels that somehow form pictures of cats
• Audio streams carrying the complete works of humanity’s musical genius (and also whatever’s popular on the charts)
• Video codecs performing computational miracles that would have made our ancestors believe in magic
• Encrypted data that looks like someone spilled alphabet soup into a blender

In the world of telecommunications, distributed systems, and general data wrangling—which is to say, the world Erlang and LFE were designed for—the ability to efficiently pack, unpack, and pattern-match binary data isn’t merely convenient. It’s essential. It’s the difference between “this will work” and “this will work while handling millions of messages per second without breaking a sweat.”

The LFE Advantage

What makes LFE’s approach to binaries particularly noteworthy is that it treats binary data as a first-class citizen with its own syntax and pattern-matching capabilities. You don’t convert things to binaries as an afterthought; you work with them directly, naturally, and—dare we say it—elegantly.

Consider this: in many languages, if you wanted to parse an MPEG audio frame header (which consists of 11 bits of sync word, 2 bits of version ID, 2 bits of layer description, and so on), you’d be looking at code that resembles a particularly obtuse mathematical proof. In LFE, you write something that looks like:

(binary (11 (sync bits)) (2 (version bits)) (2 (layer bits)) ...)

And that’s it. You’ve just pattern-matched against the bit-level structure of the data. The code reads like the specification. This is the kind of thing that makes programmers weep with joy, or at least nod approvingly while adjusting their glasses.

What You’ll Learn

In the chapters that follow, we’ll explore:

• The fundamentals: What binaries are, how they’re represented in memory, and why the BEAM VM is particularly good at handling them
• Basic construction and manipulation: Creating binaries with the #B(...) syntax and using built-in functions to dissect them
• The bit syntax: The full glory of size specifications, type qualifiers, and pattern matching at arbitrary bit boundaries
• Pattern matching: Because if you can pattern-match lists and tuples, why not pattern-match the individual bits in a TCP packet?
• Binary comprehensions: List comprehensions for binaries, because sometimes you need to transform a stream of bytes and regular loops are just so… imperative
• Real-world applications: From parsing network protocols to implementing codec algorithms, because theory is nice but practice pays the bills

By the end of this exploration, you’ll be able to look at a binary data format specification and translate it almost directly into working LFE code. You’ll understand why binary pattern matching is one of the secret weapons that makes Erlang and LFE particularly well-suited for building telecommunications systems, distributed databases, and anything else that involves pushing large quantities of structured binary data through pipes at high velocity.

More importantly, you’ll have in your toolkit an approach to binary data manipulation that doesn’t feel like medieval torture. And in the grand scheme of programming, that’s worth quite a lot.

A Note on Bitstrings

Before we dive in, a clarification: when the number of bits in your data is evenly divisible by 8, we call it a binary. When it’s not—when you’re dealing with, say, 13 bits or 47 bits—we call it a bitstring. Most operations work identically on both, but we’ll use the more specific term when the distinction matters. Think of it as the difference between a byte-aligned data structure and one that wandered off the alignment grid to explore more interesting territory.

The convention is simple: binaries are the well-behaved, byte-aligned data structures your file I/O operations expect. Bitstrings are their more adventurous cousins that hang out at odd bit boundaries, usually because some protocol designer thought it would be clever to pack flags into 3 bits instead of wasting a whole byte.

Both are equally valid. Both are equally supported. And both, as we shall see, are equally elegant to work with in LFE.

Right, then. Shall we begin?

What Binaries Are

Before we start constructing and dissecting binaries with reckless abandon, it’s worth pausing to understand what they actually are—in memory, on the page, and in relation to their slightly less respectable cousins, the bitstrings. The sections that follow examine the nature of the beast, how the runtime represents binaries, how the REPL displays them, the binary/bitstring distinction, why you’d reach for a binary instead of a list, and the immutability that makes all of it safe to pass between processes without a second thought.

The Nature of the Beast

A binary, in the LFE (and Erlang) universe, is fundamentally a reference to a chunk of raw, untyped memory. If you’re coming from a background in higher-level languages where everything must be wrapped in three layers of abstraction and a bow tie before it’s considered respectable, this might feel a bit like meeting someone who still knows how to start a fire without matches. Primitive, perhaps, but undeniably useful when you need to cook dinner.

Binaries were originally conceived for a rather specific purpose: loading code over networks in the Erlang runtime system. Imagine trying to send compiled programs between machines in the 1980s, when network bandwidth was measured in units that would make modern developers weep. You needed something efficient, something compact, something that wouldn’t spend more time being transferred than actually running. Thus, the binary was born—a data structure optimized for moving large quantities of bits around without the overhead of, say, converting everything to strings and hoping for the best.

What the designers discovered, rather like scientists who set out to invent a better glue and accidentally created Post-It notes, was that binaries were extraordinarily useful for many things. Socket-based communication. Protocol parsing. Codec implementation. Any situation where you need to deal with data in its raw, unadorned, byte-level glory.

Memory Representation

In memory, a binary is essentially a sequence of bytes (which are, if we’re being pedantic, sequences of 8 bits each). The system represents them efficiently—more efficiently, in fact, than lists. This might come as a surprise if you’ve been taught that lists are the fundamental data structure and everything else is just lists in disguise. But consider: a list in Erlang (and thus LFE) is a linked list, where each element requires not only its own storage but also a pointer to the next element. A binary is more like a C array: a contiguous block of memory with minimal overhead.

For small binaries (less than 64 bytes), the data is typically stored directly in the process heap. For larger binaries, they’re stored in a shared memory area and processes keep references to them. This means that when you send a large binary in a message between processes, you’re not copying the entire thing—you’re just passing a reference. It’s rather like instead of photocopying a lengthy document and mailing it, you simply tell someone where the filing cabinet is.

This reference-based approach has pleasant implications for garbage collection as well. Large binaries can be shared among many processes without being duplicated, and they’re only freed when the last reference disappears. It’s communism for data structures, but the kind that actually works.

Visual Representation

When the REPL prints a binary, it uses the notation #B(...) (or sometimes <<...>> if you’re looking at Erlang documentation). The contents are displayed as integers, each representing a byte:

lfe> #B(1 2 3)
#B(1 2 3)

If the binary happens to contain bytes that form printable ASCII characters, the shell might helpfully display it as a string:

lfe> #B(72 101 108 108 111)
#"Hello"

Note the #"..." notation for binary strings. This is the shell being clever, recognizing that the bytes 72 101 108 108 111 correspond to the ASCII codes for “Hello” and presenting them in a more readable format. The bytes haven’t changed—only their cosmetic presentation has been upgraded for human consumption.

This is similar to how the number 42 and the string "42" are different beasts, even though they might occasionally be mistaken for each other at parties. One is a numeric value; the other is a sequence of character codes that, when interpreted by a human visual system, evoke the concept of forty-two. The distinction matters more than you might think.

The Binary/Bitstring Distinction

As mentioned in our introduction, binaries are the special case of bitstrings where the number of bits is evenly divisible by 8. A bitstring with 16 bits? That’s a binary. A bitstring with 15 bits? That’s still a bitstring, but it’s not quite respectable enough to join the binary club. It’s the data structure equivalent of showing up to a formal dinner in jeans and a t-shirt—perfectly valid, but people will notice.

The important point is that most operations work identically on both. Pattern matching doesn’t care if you’re byte-aligned or wandering off into the wilderness of arbitrary bit boundaries. The system handles both with equal grace, which is more than can be said for most programming languages’ relationship with non-byte-aligned data.

You can verify what you’re dealing with using the predicates (binary? x) and (bitstring? x):

lfe> (binary? #B(1 2 3))
true
lfe> (bitstring? #B(1 2 3))
true
lfe> (let ((b (binary (1 (size 9)))))  ; 9 bits
      (tuple (binary? b) (bitstring? b)))
#(false true)

Note that all binaries are bitstrings, but not all bitstrings are binaries. It’s a classic is-a relationship, the kind that object-oriented programmers spend their entire careers thinking about but which here just… works, without requiring seventeen layers of inheritance and a PhD in type theory.

Why Not Just Use Lists?

A reasonable question. After all, you could represent binary data as a list of integers from 0 to 255. Many languages force you to do exactly that. But consider:

1. Space efficiency: A list needs space for both the data and the links between elements. A binary just needs space for the data. For large data structures, this difference can be substantial enough to matter, even in the age of abundant RAM.

2. Access patterns: Lists are optimized for sequential access from the front. Binaries can be indexed directly. If you need to extract the 47th byte, you don’t have to walk through the first 46 elements holding hands with yourself.

3. Message passing: When you send a binary in a message, you’re sending a reference. When you send a list, you’re sending… well, the entire list, links and all. For large data structures, one of these is significantly less expensive than the other.

4. Pattern matching performance: Pattern matching on binary structures is highly optimized in the BEAM VM. It’s the kind of optimization that the Erlang team spent years perfecting because they needed it for telecommunications. You get the benefits of their decades of effort for free.

Lists are wonderful data structures. They have their place, and that place is wherever you need flexible, dynamic, recursively-defined collections of things. But for raw bytes and bits? Binaries are simply better suited to the task, in the same way that a screwdriver is better suited to turning screws than is a particularly determined hammer.

Immutability

An Important Property: Immutability

Like all data structures in LFE, binaries are immutable. Once created, they cannot be modified. If you need to “change” a binary, you create a new one. This might seem inefficient until you realize that the BEAM VM is rather clever about this—it can often share unchanged portions of the data structure and only allocate new space for the parts that actually differ.

This immutability has profound implications for concurrent programming, which is rather the point of Erlang and LFE. You never need to worry about another process modifying a binary out from under you, because it simply cannot happen. The binary you have is the binary you’ll always have, unchanging and eternal, like a particularly stubborn philosophy professor or the value of e.

In practice, this means you can pass binaries between processes with abandon, secure in the knowledge that they won’t suddenly transform into something else when you’re not looking. In a world of concurrent processes and message passing, this guarantee is worth its weight in gold. Or at least in silicon.

Basic Binary Syntax

In the beginning, there was #B(...). Or possibly <<...>> if you’re consulting Erlang documentation and wondering why everyone keeps using those cheerful angle brackets. Both notations refer to the same underlying concept, though LFE, having a certain aesthetic sensibility about parentheses, prefers the former. It’s the difference between wearing a properly tailored suit and showing up in whatever was clean that morning—functionally equivalent, but one makes a better impression at job interviews.

The simplest possible binary is the empty one:

lfe> #B()
#B()

This binary contains exactly nothing, which might seem like an odd thing to create deliberately, but philosophers have been creating nothing deliberately for millennia and they’re considered rather clever. In practice, the empty binary occasionally proves useful as a starting point for accumulation patterns, or when you need to pass a binary but have no actual data. It’s the binary equivalent of a polite nod of acknowledgment.

Constructing Simple Binaries

To create a binary containing actual data, you simply list the values between the parentheses:

> #B(1 2 3)
#B(1 2 3)

Each integer you provide becomes a byte in the resulting binary. The values must be in the range 0 to 255, because that’s what fits in 8 bits, and attempting to stuff larger numbers into bytes is a bit like trying to fit an elephant into a phone booth—technically possible if you’re willing to ignore conventional constraints like physics, but generally frowned upon by those who have to clean up afterward.

> #B(42)
#B(42)
> #B(255)
#B(255)
> #B(256)
; This will result in an error, because 256 requires 9 bits

The system will politely inform you when you’ve exceeded the byte boundary, typically by throwing an exception. It’s being helpful, really. Better to know at compile time than to discover later that your carefully constructed binary has been truncated in mysterious and exciting ways.

Binary Strings

A particularly convenient special case involves text. Since strings in many contexts are just sequences of character codes, you can create a binary directly from string literals:

lfe> #"Hello"
#"Hello"

This is exactly equivalent to:

lfe> #B(72 101 108 108 111)
#"Hello"

The ASCII codes for H, e, l, l, and o are 72, 101, 108, 108, and 111, respectively. You can verify this with a character code lookup, or by asking any passing computer scientist, though the latter approach may result in a lengthy discussion about character encodings and why Unicode was necessary and why UTF-8 is simultaneously brilliant and a source of ongoing existential dread.

Binary strings are particularly efficient for storing text that you don’t plan to manipulate character-by-character. If you’re reading a file and want to keep its contents in memory without converting everything to a list (which, remember, has pointer overhead for every element), binary strings are your friend. A friend who doesn’t talk much and just does their job efficiently, like a particularly competent butler.

The syntax #"..." is exactly like the list string syntax "...", except the result is a binary instead of a list. Everything about escape sequences works identically:

lfe> #"Line 1\nLine 2"
#"Line 1\nLine 2"
lfe> #"Tab\there"
#"Tab\there"

You can even use the hexadecimal escape sequences:

lfe> #"\x48\x65\x6c\x6c\x6f"
#"Hello"

Though one might reasonably ask why you’d want to write “Hello” in hexadecimal, unless you’re being deliberately obscure for job security purposes. Security through obscurity might not work for cryptography, but it works reasonably well for ensuring you’re the only person who can maintain your code.

Mixing Integers and Strings

The #B(...) syntax allows you to mix integers and strings freely:

lfe> #B(1 2 "Hello" 3 4)
#B(1 2 72 101 108 108 111 3 4)

Note how the string “Hello” was expanded into its constituent byte values. The binary doesn’t remember that those bytes once belonged to a string—it just stores the raw values. The distinction between “a byte containing 72” and “the letter H” exists only in our meat-based interpretation systems, not in the silicon.

This mixing capability is occasionally useful when you need to construct a binary containing both protocol-specific numeric values and actual text data. Which is to say, it’s useful frequently, because protocols are designed by people who can never quite decide whether they want to use magic numbers or human-readable identifiers.

Nested Binaries

You can nest binaries within binary construction:

lfe> (let ((b1 #B(1 2))
        (b2 #B(3 4)))
    #B(b1 b2))
#B(1 2 3 4)

This concatenates the binaries, flattening them into a single contiguous sequence. It’s the binary equivalent of appending lists, except more efficient because you’re not walking linked structures and rebuilding chains of cons cells. You’re just stacking memory blocks end-to-end like a particularly orderly game of digital Tetris.

The nested binary must, of course, be bound to a variable. Attempting to write #B(#B(1 2)) will confuse the parser, much like attempting to say “this sentence is false” will confuse certain types of philosophers and most logical paradox detection systems.

Variables in Binary Construction

You can use variables in binary construction, as we just demonstrated:

lfe> (let ((x 42))
    #B(x))
#B(42)

The variable is evaluated and its value is inserted into the binary. This works as long as the value is something sensible—an integer in the correct range, another binary, or a string. Attempting to insert tuples, lists, or atoms directly will result in the system politely declining your request, typically by throwing a bad argument exception.

If you want to insert complex terms into binaries, you’ll need to serialize them first, which we’ll discuss in a later chapter. For now, just remember: binaries like numbers and other binaries. They’re not particularly sociable with the more structured data types, preferring to maintain their raw, untyped aesthetic.

Efficiency

When you construct a binary, the system allocates the appropriate amount of memory and copies the data into it. This is a relatively fast operation, but it’s still worth being aware of. If you’re building up a large binary piece by piece, you might want to accumulate the pieces in a list and construct the final binary all at once:

(let ((pieces (list #B(1 2) #B(3 4) #B(5 6))))
  (list-to-binary pieces))

This is more efficient than repeatedly appending to a binary, because immutable data structures mean that each append operation creates a new binary. Better to collect everything and do one allocation than to do many allocations as you build up the structure incrementally.

Of course, premature optimization is the root of all evil (or at least the root of spending three weeks optimizing code that runs once at startup and accounts for 0.0001% of your execution time). But when dealing with large binaries in performance-critical code, this distinction can matter. The BEAM VM is fast, but physics still applies, and copying gigabytes of data byte-by-byte is still gigabytes of data that need to be copied.

The List-to-Binary Bridge

Speaking of list-to-binary, this BIF (Built-In Function) is your primary tool for converting list-based data into binary form:

lfe> (list-to-binary '(72 101 108 108 111))
#"Hello"

It accepts what Erlang calls an “iolist”—a structure that can be a flat list of integers (0-255) and binaries, or a nested structure of the same. The nesting can be arbitrarily deep, because sometimes data structures grow organically and you don’t want to flatten everything just to convert it to a binary:

lfe> (list-to-binary (list #B(1 2) (list 3 4) #B(5 6)))
#B(1 2 3 4 5 6)

Note how the nested list (list 3 4) was automatically flattened. The system understands that you want all the bytes, regardless of how they were structured in your list. It’s remarkably forgiving, which is pleasant when you’re dealing with data that was accumulated through various means and hasn’t been tidied up into a perfectly flat structure.

Summary

Basic binary construction in LFE is refreshingly straightforward: put integers and strings between #B( and ), and out comes a binary. Variables work. Nesting works. It’s the kind of intuitive design that makes you wonder why other languages make it so complicated. The answer, as is often the case, involves historical accidents, committee decisions, and the fact that once something is in a language specification, removing it requires an act of international treaty negotiation and possibly divine intervention.

For now, we have what we need: a simple, expressive syntax for creating binaries. In the next sections, we’ll explore what to do with them once they exist, which is where things get interesting enough that people wrote their doctoral theses on it. Not because it’s complicated—it isn’t—but because it’s powerful in ways that aren’t immediately obvious until you try to accomplish the same things in languages that don’t have bit syntax and realize you’ve been sent to extract data from a binary format armed only with a toothpick and determination.

Binary BIFs

The Erlang runtime system, in its infinite wisdom (or at least its well-considered pragmatism developed over three decades of telecommunications infrastructure), provides a collection of Built-In Functions—BIFs, to those in the know—for manipulating binaries. These are not functions written in Erlang or LFE themselves, but rather direct calls into the runtime’s implementation in C, which means they’re fast in the way that things are fast when they don’t have to worry about abstractions or politeness and can just get on with the business of moving bytes around memory.

Think of BIFs as the difference between asking someone nicely to complete a task and simply reaching into their brain and adjusting the relevant neurons directly. More efficient, certainly, though probably less socially acceptable in most cultures.

list-to-binary: The Great Flattener

We’ve already met list-to-binary, but it deserves a more formal introduction:

(list-to-binary list-of-bytes-and-binaries)

This function takes what Erlang calls an “iolist”—a deeply recursive definition that essentially means “a list containing integers (0-255), binaries, or other iolists.” The function then flattens this entire structure into a single contiguous binary, removing all the list overhead and leaving you with just the raw bytes.

lfe> (list-to-binary '(1 2 3))
#B(1 2 3)
lfe> (list-to-binary (list #B(1 2) 3 (list 4 5) #B(6)))
#B(1 2 3 4 5 6)

Note how nesting is handled transparently. The function doesn’t care how your data is structured—it cares only about the bytes, and those it will find no matter how deeply you’ve hidden them in nested lists. It’s like having a particularly determined detective who will find what they’re looking for even if you’ve buried it under seventeen layers of bureaucratic paper trails and misdirection.

binary-to-list: The Decomposer

The inverse operation is equally straightforward:

(binary-to-list binary)

This extracts each byte from the binary and produces a list of integers:

lfe> (binary-to-list #B(1 2 3))
(1 2 3)
lfe> (binary-to-list #"Hello")
(72 101 108 108 111)

Why would you want to convert a space-efficient binary into a less-efficient list? Perhaps you need to manipulate individual elements and the list structure makes that more convenient. Perhaps you’re interfacing with code that expects lists. Perhaps you’re being deliberately perverse for artistic purposes. The function doesn’t judge; it merely transforms.

There’s also a variant that converts a slice of the binary:

(binary-to-list binary start end)

Where start and end are byte positions (1-indexed, because someone, somewhere, decided that counting should start at 1 despite computers preferring 0, and we’re all just living with the consequences). This allows you to extract a portion of the binary without converting the entire thing to a list first.

split-binary: The Precision Slicer

When you need to divide a binary into two parts at a specific position:

(split-binary binary position)

This returns a tuple containing two binaries—the part before the position and the part from the position onward:

lfe> (split-binary #B(1 2 3 4 5 6 7 8 9 10) 3)
#(#B(1 2 3) #B(4 5 6 7 8 9 10))

Note that the position is 0-indexed (unlike the list version), because different functions in the standard library have different opinions about whether counting should start at 0 or 1, and consistency is apparently for lesser languages. This is the kind of thing that makes perfect sense when you understand the historical context and which committee member was arguing for what during the standardization process, and makes no sense whatsoever if you expect software design to be guided by principles like “intuitive interface design” or “not making users remember special cases.”

The first binary contains position bytes, and the second contains everything else. If you try to split at a position larger than the binary’s size, you’ll get an error of the “what did you expect to happen?” variety. If you split at position 0, you get an empty binary and the original binary. If you split at the size of the binary, you get the original binary and an empty binary. Edge cases are handled with all the enthusiasm of someone filing tax returns, which is to say, correctly but joylessly.

term-to-binary: The Universal Serializer

This is where things get interesting enough that people write papers about it:

(term-to-binary any-erlang-term)

This function takes any Erlang/LFE term—atoms, integers, lists, tuples, maps, binaries, other terms nested arbitrarily deep—and converts it into a binary representation. This binary uses the “Erlang external term format,” which is precisely specified and designed to be efficient both in space and encoding/decoding time.

lfe> (term-to-binary '#(test 12 true (1 2 3)))
#B(131 104 4 100 0 4 116 101 115 116 97 12 100 0 4 116 114
   117 101 107 0 3 1 2 3)

Those numbers might look like line noise, but they’re actually a carefully structured representation of the tuple #(test 12 true (1 2 3)). The format includes type tags, length specifications, and the actual data, all packed efficiently into as few bytes as possible.

This is extraordinarily useful for:

• Storing complex data structures in files
• Sending data structures over network sockets
• Implementing persistence layers
• Debugging (though the binary output is not particularly human-readable without tools)
• Implementing distributed Erlang (which uses this format for inter-node communication)

The format is version-stable, meaning binaries created with older versions of Erlang can generally be read by newer versions. This is the kind of forward compatibility that database designers dream about and which distributed systems rely upon to avoid version upgrade nightmares where everything must be upgraded simultaneously or face the consequences (those consequences typically involving angry phone calls at 3 AM and rapidly updating one’s résumé).

binary-to-term: The Reanimator

The inverse of term-to-binary:

(binary-to-term binary-containing-encoded-term)

This takes a binary in the Erlang external term format and reconstructs the original term:

lfe> (let ((bin (term-to-binary '#(cat dog))))
    (binary-to-term bin))
#(cat dog)

The term that emerges is not “the same” as the original in any referential sense—it’s a new term, freshly allocated, with all the same structure and values. But for all practical purposes, it’s equivalent. It’s the teleportation of data: the original is scanned, destroyed (well, garbage collected eventually), and an identical copy is created elsewhere. Philosophers might object. Data structures do not care.

It’s worth noting that binary-to-term will cheerfully reconstruct any valid Erlang term, including atoms. If you’re decoding binaries from untrusted sources, you might want to be careful about this, as atoms are never garbage collected and creating too many of them will eventually exhaust the atom table. The function has options to prevent atom creation during decoding, but that’s a security discussion beyond our current scope, which is primarily concerned with the mechanics of how this all works rather than all the exciting ways it can go wrong if you’re not paying attention.

byte-size: The Measurer

To find out how many bytes are in a binary:

(byte-size binary)

Simple enough:

lfe> (byte-size #B(1 2 3 4 5))
5

For bitstrings that aren’t byte-aligned, this returns the number of bytes required to contain all the bits, rounded up:

lfe> (let ((bs (binary (1 (size 9)))))  ; 9 bits
    (byte-size bs))
2

Nine bits require two bytes to store, because bytes, being stubbornly eight bits each, don’t come in fractions despite what one might wish when dealing with data that refuses to align itself properly.

bit-size: The Precise Measurer

For the exact number of bits:

(bit-size bitstring)

This works on both binaries and bitstrings:

lfe> (bit-size #B(1 2 3))
24
lfe> (let ((bs (binary (1 (size 9)))))
    (bit-size bs))
9

The distinction matters when you’re working with protocols or formats that pack data at non-byte boundaries. Which is to say, nearly all modern compression formats and most networking protocols designed by people who believed that saving a single bit per packet, when multiplied by billions of packets, might add up to something worth optimizing.

Summary

These BIFs form the basic toolkit for binary manipulation in LFE. They’re not particularly exciting—they convert things, split things, measure things, and serialize things. But combined with the bit syntax (which we’ll explore next), they provide the foundation for remarkably sophisticated binary data manipulation.

The key insight is that the Erlang VM has spent decades optimizing these operations. When you call term-to-binary, you’re not running some slow interpretation loop—you’re calling hand-optimized C code that knows exactly how to serialize Erlang terms efficiently. When you split a binary, you’re not copying data unnecessarily—the VM is using sophisticated memory management to share unchanged portions when possible.

This is the kind of optimization that only comes from real-world use at scale. The Erlang VM wasn’t designed in an academic laboratory to be theoretically optimal—it was designed in telecommunications companies to handle millions of phone calls without falling over, which tends to focus one’s attention rather effectively on what actually matters versus what sounds good in papers.

You get all of this for free, simply by using the provided BIFs. It’s one of those situations where the lazy option (calling provided functions rather than implementing your own) is also the optimal option. This almost never happens in software engineering, which makes it worth celebrating when it does.

Bit Syntax Fundamentals

In Which We Descend Into Particular Detail

The bit syntax is where LFE’s approach to binary data transforms from “convenient” into “why doesn’t every language do this?” It’s a notation for describing binary data at arbitrary bit-level granularity, and it does something that would be remarkable if it weren’t so obviously sensible: it makes code look like the specification of the binary format you’re working with.

This is worth pausing to appreciate. In most languages, if someone hands you a protocol specification that says “the packet consists of a 3-bit flag field, followed by a 13-bit length field, followed by an 8-bit type identifier,” your next step involves a calculator, several bitwise operators, a growing sense of dread, and possibly a therapeutic discussion with a rubber duck about whether this career path was really the right choice.

In LFE, you write something very close to: (binary (flag (size 3)) (length (size 13)) (type (size 8))). The code reads like the spec. This is not an accident. This is deliberate design by people who had to implement telecommunications protocols and decided that life was too short for bit-masking gymnastics.

The General Form

The bit syntax uses the binary form (or #B(...) for construction) with segments that can be annotated:

(binary segment1 segment2 ... segmentN)

Each segment can be:

• A simple value: 42
• A value with size: (42 (size 16))
• A value with type: (42 integer)
• A value with size and type: (42 (size 16) integer)
• A value with various other qualifiers we’ll explore momentarily

The pattern is consistent: value first, then optional annotations. The annotations provide additional information about how to interpret or pack the bits for that segment.

Segments: The Atomic Unit

A segment is a sequence of bits in a binary. These sequences need not be aligned on byte boundaries—you can have a 3-bit segment followed by a 5-bit segment, and the system will pack them tightly together because it’s not 1970 anymore and we have the technology to handle bits that don’t come in groups of eight.

lfe> (binary (5 (size 4)) (5 (size 4)))
#B(85)

Wait, what? Where did 85 come from?

Let’s examine this with a magnifying glass and possibly a stiff drink: The integer 5 in binary is 0101. We’ve asked for two 4-bit segments, each containing 5. So we get: 0101 0101, which when interpreted as a byte is… 85 in decimal. Or if you’re feeling hexadecimal, that’s 0x55. Or if you’re feeling ASCII, that’s “U”.

The REPL, in its infinite helpfulness, notices that 85 is a printable ASCII character and shows us <<"U">> (in Erlang notation) or #"U" (in LFE notation). The bits haven’t changed—only our perception of them has been subject to the REPL’s well-meaning interference.

lfe> (list-to-binary (integer-to-list 85))
#"U"

Yes, indeed. 85 is “U”. This is the kind of thing you either find fascinating or deeply troubling, depending on your relationship with the arbitrary nature of character encodings and numeric representations.

Size Specifications

The size specification tells the system how many bits (or bytes, depending on the unit) a segment should occupy:

(value (size N))

lfe> (binary (42 (size 16)))
#B(0 42)

Here we’re storing the number 42 in 16 bits (2 bytes). In big-endian order (which is the default, and we’ll discuss endianness shortly), this becomes 0000000000101010 in binary, or 0x00 0x2A in hexadecimal, which the REPL kindly displays as #B(0 42).

The size can be larger than strictly necessary:

lfe> (binary (42 (size 32)))
#B(0 0 0 42)

Four bytes for the number 42, because you asked for 32 bits and 32 bits you shall receive. The system pads with zeros in the most significant positions, because that’s how you represent small positive integers in larger spaces without changing their value—unlike padding your résumé, where additional zeros in unexpected places tend to have different effects on value.

If you specify a size smaller than required, the system will truncate:

lfe> (binary (258 (size 8)))  ; 258 = 0b100000010, needs 9 bits
#B(2)  ; Only the lower 8 bits: 0b00000010 = 2

The top bits are simply discarded, like baggage exceeding the weight limit on a particularly strict airline. Whether this is what you intended or a subtle bug that will manifest six months later in production is between you and whatever testing regime you’ve implemented. The system neither knows nor cares about your intentions—it simply does what you asked, which is all we can reasonably expect from deterministic computational systems.

Default Sizes

If you don’t specify a size, the system assumes defaults based on the type:

• Integers: 8 bits (1 byte)
• Floats: 64 bits (8 bytes)
• Binaries/bitstrings: The entire binary

lfe> (binary 42)  ; Same as (binary (42 (size 8)))
#B(42)
lfe> (binary 1.5)  ; 64-bit float
#B(63 248 0 0 0 0 0 0)

That floating-point representation looks like line noise, doesn’t it? That’s IEEE 754 binary64 format, which is how computers have agreed to represent floating-point numbers ever since the IEEE standardization committee decided that “whatever hack your hardware happens to implement” was an insufficient specification for portable floating-point arithmetic. The details are fascinating if you’re into that sort of thing, and utterly terrifying if you’re trying to understand why 0.1 + 0.2 doesn’t exactly equal 0.3 in most programming languages.

Type Annotations

You can explicitly specify the type of a segment:

(value type-specifier)

The available types include:

• integer (the default)
• float
• binary
• bitstring
• bytes (synonym for binary)
• bits (synonym for bitstring)
• utf8, utf16, utf32 (for Unicode text)

lfe> (binary (1.23 float))
#B(63 243 174 20 122 225 71 174)
lfe> (binary (42 integer))
#B(42)
lfe> (binary (#"hello" binary))
#"hello"

When you specify binary as a type, you’re saying “this segment is itself a binary, and should be included in its entirety.” This is how you concatenate binaries within the bit syntax:

lfe> (let ((b1 #B(1 2))
        (b2 #B(3 4)))
    (binary (b1 binary) (b2 binary)))
#B(1 2 3 4)

The binary type annotation is required here because otherwise the system wouldn’t know whether you meant “include the binary” or “include the integer that represents some aspect of the binary” or possibly “make me a sandwich.” Explicit is better than implicit, as some Python programmer probably said once, though that principle is more universally applicable than its origin might suggest.

Combining Size and Type

You can specify both size and type:

lfe> (binary (1.23 (size 32) float))
#B(63 157 112 164)
lfe> (binary (1.23 (size 64) float))
#B(63 243 174 20 122 225 71 174)

The 32-bit float is less precise than the 64-bit version, which is why they’re different. This is the fundamental trade-off in floating-point representation: more bits give you more precision, at the cost of more bits. When you have millions of floating-point numbers and limited bandwidth or storage, those extra 32 bits per number can add up to “can we afford the hardware to store all this?” versus “it barely fits.” Compression algorithms have entire doctoral theses dedicated to this topic.

For integers, specifying both size and type looks like:

lfe> (binary (42 (size 16) integer))
#B(0 42)

Though since integer is the default type, the type annotation is typically omitted unless you’re being exceptionally explicit for documentation purposes, or your team has a style guide written by someone who believes that verbosity correlates with correctness (a belief that is often tested in code review sessions where “can we make this simpler?” meets “but the specification says…”).

Variables in Segments

You can use variables in segment specifications:

lfe> (let ((x 42)
        (y 17))
    (binary (x (size 8)) (y (size 8))))
#B(42 17)

The variable must be bound at the point it’s used. Attempting to use unbound variables will result in the usual “unbound variable” error, which is the compiler’s polite way of saying “I have no idea what you’re talking about and neither, I suspect, do you.”

More interestingly, you can use variables for sizes in pattern matching, which allows extraordinarily flexible binary parsing:

(let (((binary (size (size 8)) (data (size size) binary)) some-packet))
  data)

Here, the first byte tells us the size, and then we extract exactly that many bytes into data. The size is used both to extract a value (the first 8 bits) and to control the size of a subsequent segment. This kind of self-describing binary format is common in network protocols, where the packet tells you how long it is so you know when you’ve received all of it.

This is pattern matching at its finest: you’re not writing parsing code, you’re describing the structure of the data, and the system figures out how to extract the values. It’s declarative programming for binary formats, which sounds like it should be impossible until you realize it works perfectly well and has done so for decades.

Philosophical Implications

The bit syntax represents a particular philosophy of language design: rather than forcing you to write imperative code to pick apart binary data bit by bit, the language provides a notation that describes the structure of the data. You say what you want, not how to get it.

This is a subtle but profound shift. It means that:

1. Your code becomes documentation of the binary format
2. The compiler can optimize the operations because it knows the structure
3. Errors become compile-time issues rather than runtime surprises (when possible)
4. Adding new fields or changing layouts is a matter of updating the description, not rewriting parsing logic

This is why Erlang became popular in telecommunications despite not being taught in many computer science programs—it solves real problems in ways that feel almost unfairly simple once you understand the syntax. The bit syntax is one of those features that seems complex when you first encounter it (hence this chapter), but once understood, makes you wonder how anyone parses binary protocols without it.

The answer, of course, is “with great difficulty and numerous bugs.” But we don’t have to live that way. We have bit syntax. And that, in the grand scheme of programming language features, is worth celebrating.

Type Specifiers

On the Nature of Types in Binary Segments

If the bit syntax is a language for describing binary structures, then type specifiers are its vocabulary—or possibly its accent marks and articles, depending on how metaphorically inclined you’re feeling today. They modify and qualify segments, telling the system not just “there are bits here” but rather “these bits mean something specific, please interpret them accordingly.”

The full specification of a segment can be positively baroque:

(value (size N) type-spec1 type-spec2 ... type-specN)

Where each type-spec can modify how the value is interpreted or stored. The order doesn’t matter (except to pedants and to the parser, which cares deeply about syntax but not about your feelings), and you can omit any specifiers you don’t need, in which case sensible defaults are applied. The defaults are “sensible” in the way that most defaults are sensible: reasonable for the common case, potentially surprising in edge cases, and the source of occasional debugging sessions where you discover that what you thought was happening wasn’t actually happening because of an implicit assumption you didn’t realize you’d made.

Welcome to software engineering. We have coffee, because we need it.

The Type Itself

The fundamental type specifiers tell the system what kind of value the bits represent:

integer

The default type. Bits are interpreted as a signed or unsigned integer (depending on the sign specifier, which we’ll discuss in a moment). If no size is specified, 8 bits is assumed, because bytes remain the atomic unit of traditional computing despite our best efforts to think in arbitrary bit quantities.

lfe> (binary (42 integer))
#B(42)
lfe> (binary (42 (size 16) integer))
#B(0 42)

Integers are straightforward, which is pleasant. They’re numbers. They fit in a specified number of bits. There’s no floating-point imprecision to worry about, no encoding ambiguities, just bits in a row meaning what bits in a row have meant since boolean algebra was invented: numbers, but in base 2, because base 10 is for humans and we’re talking to silicon now.

float

Floating-point numbers, represented in IEEE 754 format. These require either 32 bits (single precision) or 64 bits (double precision). If you specify any other size, the system will object strenuously, because IEEE 754 doesn’t define formats for arbitrary bit widths and inventing new ones would be the kind of creative interpretation that ends with incompatible data and sad engineers.

lfe> (binary (3.14159 (size 32) float))
#B(64 73 15 209)
lfe> (binary (3.14159 (size 64) float))
#B(64 9 33 245 249 84 68 45)

The 32-bit version has less precision, which you’d notice if you were doing extensive calculations with it, or if you were a physicist worried about whether your simulation of quantum mechanics was accurate to within acceptable rounding error. For drawing circles on computer screens, 32 bits is typically fine. For landing spacecraft on distant planets, you might want to reconsider.

The default size for floats is 64 bits, because at some point someone decided that double precision should be the default and everyone else went along with it, probably because arguing about default floating-point precision is the kind of thing that makes standardization committees long for retirement.

binary

This specifies that the segment is itself a complete binary that should be included in its entirety:

lfe> (let ((inner #B(1 2 3)))
    (binary (inner binary)))
#B(1 2 3)

The size specifier, if provided, must be divisible by 8, because binaries are byte-aligned by definition. If you try (binary (#B(1) (size 7) binary)), the system will inform you of your error, typically at compile time, which is better than runtime, which is better than never.

When pattern matching, the binary type consumes the rest of the binary unless you specify a size:

lfe> (let (((binary (head (size 8)) (tail binary)) #B(1 2 3 4 5)))
    (list head tail))
(1 #B(2 3 4 5))

This is extraordinarily useful for protocol parsing where you need to extract a header and then process the rest of the data separately. It’s also useful for implementing streaming processing, where you take a bit, do something with it, then recursively process the remainder, which is the functional programming way of saying “loop” without admitting that you’re essentially looping.

bitstring

Like binary, but doesn’t require byte alignment. This lets you work with data at arbitrary bit boundaries:

lfe> (let (((binary (bits1 (size 3) bitstring) (bits2 (size 5) bitstring))
         #B(156)))  ; 10011100 in binary
    (list (bit-size bits1) (bit-size bits2)))
(3 5)

Here we’ve split a byte into a 3-bit chunk and a 5-bit chunk. The result isn’t two integers—it’s two bitstrings, which are like binaries but more flexible about their alignment, like yoga practitioners of the data structure world.

bytes and bits

Synonyms for binary and bitstring, respectively, provided for those who prefer the plural form or who learned the syntax from documentation that used these terms. Language designers, in an unusual fit of generosity, sometimes provide multiple ways to say the same thing, presumably to reduce the learning curve for people coming from different backgrounds, or possibly just to give everyone something to argue about in code reviews.

utf8, utf16, utf32

These specify Unicode encodings. They’re used for handling text in various Unicode Transformation Formats:

lfe> (binary (#\α utf8))
#B(206 177)
lfe> (binary (#\α utf16))
#B(3 177)
lfe> (binary (#\α utf32))
#B(0 0 3 177)

The Greek letter alpha (α, Unicode code point U+03B1) is encoded differently in each format. UTF-8 uses 2 bytes, UTF-16 uses 2 bytes, and UTF-32 uses 4 bytes. This is because the UTF formats have different properties: UTF-8 is space-efficient for ASCII and variable-length, UTF-16 is space-efficient for most modern scripts but still variable-length, and UTF-32 is fixed-length but wastes space for anything that isn’t using the full Unicode range.

The choice of encoding is a complex topic involving trade-offs between space efficiency, parsing complexity, and whether your data is mostly ASCII or includes substantial non-Latin scripts. Entire flame wars have been fought over this topic, with participants taking positions ranging from “UTF-8 for everything” to “UTF-16 for everything” to “why not just use ASCII and pretend the rest of the world doesn’t exist?” (The last position is generally held by people who don’t realize they’re advocating for essentially 1960s-era computing.)

For these types, the size is determined by the encoding and cannot be explicitly specified, because suggesting that UTF-8 should use a different number of bytes than UTF-8 specifies would be creating a new encoding, and we have enough encodings already, thank you.

Sign

Integers can be signed or unsigned:

unsigned

The default. All bits contribute to the magnitude:

lfe> (let (((binary (x unsigned)) #B(255)))
    x)
255

Eight bits can represent 0 through 255 when unsigned, which is the full range of byte values and why bytes are often thought of as “0 to 255” rather than “-128 to 127,” even though both ranges contain exactly 256 distinct values and are therefore equivalent in information-theoretic terms if not in convenience-of-interpretation terms.

signed

The first bit becomes the sign bit (0 for positive, 1 for negative), using two’s complement representation:

lfe> (let (((binary (x signed)) #B(255)))
    x)
-1

The same bit pattern 11111111 means 255 when unsigned and -1 when signed. This is because two’s complement representation uses that pattern for -1, which makes arithmetic simpler at the hardware level but means that interpreting integers requires knowing whether they’re signed, which is one of those bits of context that must be maintained externally to the data itself.

This sign specification only matters for pattern matching (extracting values). When constructing binaries, the bit pattern is determined by the value and size:

lfe> (binary (-1 (size 8)))
#B(255)  ; Two's complement representation of -1 in 8 bits

Endianness

When a value spans multiple bytes, the byte order matters:

big (big-endian)

The default. Most significant byte first:

lfe> (binary (16#1234 (size 16) big))
#B(18 52)  ; 0x12 0x34

Called “big-endian” because the big end (most significant byte) comes first. Also called “network byte order” because network protocols standardized on this representation, probably because the alternative was every machine interpreting every other machine’s data incorrectly and no one being able to communicate, which would rather defeat the purpose of networking.

little (little-endian)

Least significant byte first:

lfe> (binary (16#1234 (size 16) little))
#B(52 18)  ; 0x34 0x12

Called “little-endian” because the little end comes first. Also called “the Intel way” by those who associate it with x86 processors, which is technically correct but glosses over the fact that many processor architectures have used little-endian representation and Intel didn’t invent it, they just popularized it through market dominance.

native

Determined at runtime based on the machine’s native byte order:

lfe> (binary (16#1234 (size 16) native))
; Result depends on your CPU architecture

On Intel processors, this is little-endian. On most network equipment and older RISC processors, it’s big-endian. On some exotic architectures that almost nobody uses, it might be something else entirely, like middle-endian, which exists primarily to make computer architecture textbooks more interesting and to give programming language designers nightmares about portability.

Using native is appropriate when:

1. You’re reading/writing data on the same machine and don’t care about portability
2. You want maximum performance and are willing to deal with potential portability issues
3. You’re implementing something that explicitly needs to respect the machine’s native format

It’s inappropriate when you need to exchange data between machines that might have different architectures, which is approximately 90% of binary data interchange scenarios in modern computing.

Endianness only affects values that span multiple bytes. Single-byte values are immune to byte-order issues, because there’s only one byte and thus only one possible ordering for it. This is one of the arguments for using byte-aligned protocols when possible: fewer opportunities for endianness bugs.

The Unit Specifier

The unit specifier modifies how size is interpreted:

(value (size N) (unit U))

The actual size in bits is N * U. The unit must be from 1 to 256:

lfe> (binary (1 (size 2) (unit 8)))  ; 2 * 8 = 16 bits
#B(0 1)
lfe> (binary (1 (size 16) (unit 1)))  ; 16 * 1 = 16 bits
#B(0 1)

Both produce the same result, because 2 * 8 equals 16 * 1. The unit specifier is primarily useful for:

1. Enforcing alignment: Binary segments default to unit:8, meaning sizes must be multiples of 8 bits
2. Working with specific bit widths: Video codecs often work with pixels or samples of specific bit widths
3. Documentation: (size 2) (unit 8) might be clearer than (size 16) if you’re thinking in terms of “2 bytes”

The default unit values are:

• integer, float, bitstring: unit 1 (arbitrary bit sizes)
• binary, bytes: unit 8 (must be byte-aligned)

If you try to violate alignment constraints (like specifying a binary with a non-multiple-of-8 bit count), the system will object:

lfe> (binary (#B(1 2) (size 15) binary))  ; 15 is not divisible by 8
; Error: the value Binary in size specification of binary segment is expected to be of type integer

The error message is trying to be helpful. It’s not always successful at being helpful, but the attempt is appreciated.

Combining Specifiers

You can combine these specifiers (except where combinations don’t make sense):

(binary (value (size N) (unit U) type sign endian))

For example:

lfe> (binary (1000 (size 16) integer unsigned big))
#B(3 232)  ; 1000 = 0x03E8 in big-endian
lfe> (binary (1000 (size 16) integer unsigned little))
#B(232 3)  ; Same value in little-endian
lfe> (binary (-1000 (size 16) integer signed big))
#B(252 24)  ; -1000 in two's complement, big-endian

The order of specifiers doesn’t matter (the parser sorts it out), but convention is:

1. Size specification
2. Unit (if needed)
3. Type
4. Sign (for integers)
5. Endianness (for multi-byte integers and floats)

This ordering makes the specifications read naturally: “16 bits of unsigned big-endian integer” becomes (size 16) integer unsigned big.

Of course, you can omit defaults:

(binary (value (size 16)))  ; Equivalent to (size 16) integer unsigned big

Special Cases and Edge Conditions

Some combinations are invalid and will generate errors:

• Signed/unsigned with floats (floats have their own sign representation)
• Endianness with single-byte values (there’s no byte order when there’s only one byte)
• Non-multiple-of-8 sizes for binary type
• UTF types with explicit sizes (the encoding determines the size)
• Unit specifications with UTF types

The system will generally catch these at compile time if the values are known, or at runtime if they’re determined by variables. Either way, you’ll be informed of your mistake, which is better than silently producing incorrect results, which is the approach taken by languages that believe in giving programmers enough rope to hang themselves and then selling them more rope.

Summary

Type specifiers transform bit syntax from “here are some bits” to “here are some bits that mean this specific thing in this specific format.” They’re the difference between having a pile of LEGOs and having an instruction manual for assembling them into something recognizable.

Most of the time, you’ll use a subset of these capabilities:

• (size N) integer for fixed-width integer fields
• float for floating-point values
• binary for including other binaries
• bitstring when you need non-byte-aligned data

The full range of possibilities exists for the edge cases, for the protocols designed by committees, and for the situations where you absolutely must handle some exotic binary format that was designed in 1987 and can never be changed because it’s embedded in hardware on satellites that we can’t exactly update remotely.

The good news is that once you understand the system, it handles all these cases uniformly. The bad news is that you have to understand the system. But we’re working on that, which is why you’re reading this chapter instead of directly consulting the specification, which is accurate but reads like it was translated from Vogon through three other languages and then back into English by someone whose native language is actually mathematics.

Size and Unit Specifications

“Space,” remarked Ford Prefect, “is big. Really big. You just won’t believe how vastly, hugely, mind-bogglingly big it is.” Bits, on the other hand, are quite small. Fortunately, LFE lets you specify exactly how many of them you’d like to use at any given moment.

The Fundamental Problem of Bit Accounting

When working with binary data, one quickly encounters what philosophers call “the problem of exactly how many bits we’re talking about here.” Is that integer taking up 8 bits? 16? 42? The default answer, much like the answer to life, the universe, and everything, depends entirely on context—and unlike that famous answer, this one is actually useful.

Default Sizes

LFE, being a sensible language designed by beings who’ve thought about these things, provides reasonable defaults:

• Integers: 8 bits (1 byte), because bytes are the lingua franca of data
• Floats: 64 bits (8 bytes), because IEEE 754 double-precision is the standard, and standards exist for a reason
• Binaries: Whatever size they actually are, which is refreshingly honest

Let’s observe these defaults in their natural habitat:

lfe> (set default-int (binary 42))
#B(42)
lfe> (byte_size default-int)
1

lfe> (set default-float (binary (3.14159 float)))
#B(64 64 9 33 251 84 68 45)
lfe> (byte_size default-float)
8

Explicit Size Specifications

Of course, sometimes the defaults are as inappropriate as a Vogon’s poetry reading at a wedding. When you need to specify exactly how many bits to use, the syntax is delightfully straightforward:

Value:Size

Where Size is the number of bits (not bytes—we’re working at the bit level now, where things get properly interesting).

Size in Action

lfe> (set tiny (binary (42 (size 4))))
#B(10 4)  ; That's 42 crammed into 4 bits, stored as 10100000 in binary
lfe> (bit_size tiny)
4

lfe> (set rgb (binary ((2 (size 5)) (61 (size 6)) (20 (size 5)))))
#B(23 180)  ; RGB color: 5 bits red, 6 bits green, 5 bits blue
lfe> (bit_size rgb)
16

Notice how 42 in 4 bits becomes something entirely different when unpacked as a full byte. This is because we’re only using the lower 4 bits, and the binary syntax helpfully pads or aligns as needed. It’s not quite as dramatic as a matter transference beam accident, but the principle of “things change when you squeeze them” remains valid.

The Unit Specification

Now we arrive at the unit specification, which is where things become delightfully mathematical. The unit specifies a multiplier for the size, allowing you to work in convenient chunks:

Value:Size/TypeSpec-unit:N

The total number of bits used is: Size × Unit

Default Units

• integer, float, bitstring: unit of 1 (counting individual bits)
• binary: unit of 8 (counting bytes, as civilized beings do)

Why Units Matter

Consider the case of reading 4-byte chunks of data:

lfe> (defun read-4byte-chunks (bin)
       (binary-comp
         ((<< (chunk (size 4) binary) >> (binary-gen (<= bin))))
         chunk))

lfe> (read-4byte-chunks #B(1 2 3 4 5 6 7 8 9 10 11 12))
(#B(1 2 3 4) #B(5 6 7 8) #B(9 10 11 12))

Or, more explicitly using unit:

lfe> (set msg #B(72 101 108 108 111))  ; "Hello"
lfe> (binary ((msg (size 5) (unit 8))))  ; 5 units of 8 bits = 40 bits total
#B(72 101 108 108 111)

The Constraint of Divisibility by Eight

Here’s a crucial cosmic law: when working with binaries (as opposed to bitstrings), the total bit count must be evenly divisible by 8. The universe, or at least the Erlang VM, insists on this.

lfe> (binary ((42 (size 3) (unit 8))))
#B(0 0 42)  ; 3 × 8 = 24 bits = 3 bytes ✓

lfe> (binary ((42 (size 3) (unit 7))))
; exception error: badarg  ; 3 × 7 = 21 bits ✗ not divisible by 8

Practical Example: Fixed-Width Records

Imagine you’re encoding a series of sensor readings where each reading consists of:

• 12-bit sensor ID
• 20-bit timestamp (milliseconds)

(defun encode-reading (sensor-id timestamp)
  "Encode a sensor reading into a compact 32-bit (4-byte) binary."
  (binary ((sensor-id (size 12)) (timestamp (size 20)))))

(defun decode-readings (data)
  "Decode a stream of sensor readings."
  (binary-comp
    ((<< (id (size 12)) (ts (size 20)) >> (binary-gen (<= data))))
    (tuple id ts)))

Testing this delightful contraption:

lfe> (set reading (encode-reading 2048 524288))
#B(128 8 0 0)

lfe> (decode-readings reading)
(#(2048 524288))

The Pattern Matching Constraint

In pattern matching, there’s an additional rule that makes perfect sense once you think about it: you can only use bound variables as size specifiers, or literal integers. The pattern matcher needs to know how many bits to extract, and it can’t very well guess.

; This works - N is bound before use:
(defun extract-n-bytes (n data)
  (binary ((header (size n) binary) payload binary) data)
  (tuple 'extracted header payload))

; This won't work - N is unbound:
(defun mysterious-extraction (data)
  (binary ((header (size n) binary) payload binary) data)  ; Error!
  (tuple 'extracted header payload))

A Note on UTF Strings

The UTF-8, UTF-16, and UTF-32 types are special cases that require no unit specification. They know their own sizes based on the character encoding, which is rather like how a towel knows it should be exactly towel-sized:

lfe> (binary ((#"Hello" utf8)))
#B(72 101 108 108 111)

lfe> (binary ((#"你好" utf8)))
#B(228 189 160 229 165 189)  ; Chinese "hello" in UTF-8

Summary

Remember: Total bits = Size × Unit, and for binaries, this product must be divisible by 8. It’s not quite as strict as “always know where your towel is,” but it’s a rule you’ll want to internalize nonetheless.

Endianness

“The problem with defending the purity of the English language is that English is about as pure as a cribhouse whore. We don’t just borrow words; on occasion, English has pursued other languages down alleyways to beat them unconscious and rifle their pockets for new vocabulary.” — James D. Nicoll

The same could be said of byte ordering conventions. Different computer architectures have pursued different strategies for storing multi-byte values, leading to what can only be described as an unnecessarily theological dispute about the correct order of bytes.

The Parable of Big-Endian and Little-Endian

The terms “big-endian” and “little-endian” come from Jonathan Swift’s Gulliver’s Travels, where two fictional kingdoms went to war over which end of a boiled egg should be opened first. Computer engineers, finding this satirical commentary on human pettiness to be mirrored in the constant byte-bickering of the 70s and 80s, decided to apply the same principle.

The Two Schools of Thought

Big-Endian (also called “network byte order”):

• Most significant byte comes first
• Reads naturally from left to right (for humans accustomed to Western number systems)
• Used by network protocols, Java Virtual Machine, and certain RISC processors

Little-Endian:

• Least significant byte comes first
• Reads backwards (from a human perspective)
• Used by Intel x86/x64, most ARM processors, and assorted rebels

A Concrete Example

Consider the 32-bit hexadecimal number 0x12345678. How would different systems store this in memory?

lfe> (set num #x12345678)
305419896

; Big-endian representation (most significant byte first):
lfe> (binary ((num (size 32) big)))
#B(18 52 86 120)  ; 0x12, 0x34, 0x56, 0x78

; Little-endian representation (least significant byte first):
lfe> (binary ((num (size 32) little)))
#B(120 86 52 18)  ; 0x78, 0x56, 0x34, 0x12

Notice how the bytes are in completely opposite order. This is why cross-platform binary protocols require careful specification of byte order—what looks like 0x12345678 on one system might be interpreted as 0x78563412 on another.

The Endianness Specifiers in LFE

LFE provides three endianness specifiers:

big      ; Big-endian (default)
little   ; Little-endian
native   ; Whatever your CPU prefers (determined at runtime)

Default Behavior

If you don’t specify endianness, LFE assumes big-endian. This is because network protocols universally use big-endian ordering, making it the sensible default for a language designed with telecommunications in mind.

lfe> (binary ((12345 (size 16))))
#B(48 57)  ; Default: big-endian

lfe> (binary ((12345 (size 16) big)))
#B(48 57)  ; Explicit big-endian (same result)

lfe> (binary ((12345 (size 16) little)))
#B(57 48)  ; Little-endian (reversed)

When Endianness Matters

Endianness is relevant only for multi-byte data types:

• Integers (when size > 8 bits)
• Floats (which are always multi-byte)
• UTF-16 and UTF-32 (multi-byte character encodings)

It does not matter for:

• Single-byte integers
• UTF-8 (which is byte-oriented)
• Plain binaries (which are just sequences of bytes)

Floating Point and Endianness

IEEE 754 floating-point numbers are also subject to endianness concerns:

lfe> (set pi 3.14159265359)
3.14159265359

lfe> (binary ((pi float)))
#B(64 9 33 251 84 68 45 24)  ; Big-endian double

lfe> (binary ((pi float little)))
#B(24 45 68 84 251 33 9 64)  ; Little-endian double

The Native Option: Runtime Flexibility

The native specifier tells LFE to use whatever byte order your CPU natively prefers. This is determined at runtime, making your code portable while still being efficient:

lfe> (binary ((12345 (size 32) native)))
#B(0 0 48 57)  ; Result depends on your CPU architecture

On an Intel x86-64 system (little-endian):

#B(57 48 0 0)

On a SPARC or PowerPC system (big-endian):

#B(0 0 48 57)

When to Use Native

Use native when:

• You’re storing binary data that will only be read on the same system
• Performance is critical (native operations avoid byte-swapping overhead)
• You’re working with platform-specific file formats

Don’t use native when:

• Exchanging data over networks
• Writing portable file formats
• Any scenario where the data might cross architectural boundaries

Pattern Matching with Endianness

Endianness specifiers work identically in both construction and pattern matching:

(defun decode-network-header (packet)
  "Decode a network packet header with big-endian values."
  (binary ((magic (size 16) big)
           (length (size 32) big)
           (flags (size 16) big)
           payload binary)
          packet)
  `#m(magic ,magic length ,length flags ,flags payload ,payload))

(defun decode-local-data (data)
  "Decode platform-specific data with native endianness."
  (binary ((timestamp (size 64) native)
           (value (size 32) native)
           rest binary)
          data)
  `#m(timestamp ,timestamp value ,value rest ,rest))

Testing these functions:

lfe> (set packet (binary ((#xCAFE (size 16) big)
                          (1024 (size 32) big)
                          (#xFF00 (size 16) big)
                          (#"DATA" binary))))
#B(202 254 0 0 4 0 255 0 68 65 84 65)

lfe> (decode-network-header packet)
#m(magic 51966 length 1024 flags 65280 payload #B(68 65 84 65))

Cross-Platform Binary Protocols

When designing binary protocols for cross-platform use, always explicitly specify big. Network byte order is the universal standard, and failing to use it has caused countless bugs throughout computing history (though admittedly fewer than the confusion caused by time zones).

(defun encode-packet (type-id data)
  "Encode a packet with proper network byte order."
  (let ((data-size (byte_size data)))
    (binary ((#xDEAD (size 16) big)              ; Magic number
             (type-id (size 8))                   ; Type (single byte, no endianness)
             (data-size (size 32) big)            ; Length field
             (data binary)))))                    ; Payload

(defun decode-packet (packet)
  "Decode a packet, expecting network byte order."
  (binary ((#xDEAD (size 16) big)
           (type-id (size 8))
           (data-size (size 32) big)
           (data (size data-size) binary)
           rest binary)
          packet)
  `#m(type ,type-id data ,data rest ,rest))

A Historical Note

The big-endian/little-endian divide is one of those delightful accidents of computing history where multiple incompatible solutions to the same problem all survived. Unlike the VHS/Betamax wars or AC/DC electrical current (the current war, not the band), neither endianness could claim total victory.

The situation is rather like having two equally valid but completely incompatible systems for measuring temperature, except that someone occasionally tries to add Celsius and Fahrenheit together and wonders why their spacecraft crashes into Mars. Which, incidentally, has happened.

Practical Examples

Reading a BMP File Header

Bitmap files use little-endian encoding (because they were designed by Microsoft, which runs on Intel):

(defun read-bmp-header (file-data)
  "Read BMP file header (little-endian format)."
  (binary ((#"BM" binary)                        ; Magic bytes
           (file-size (size 32) little)          ; File size
           (_ (size 16) little)                  ; Reserved
           (_ (size 16) little)                  ; Reserved
           (pixel-offset (size 32) little)       ; Offset to pixel data
           rest binary)
          file-data)
  `#m(file-size ,file-size pixel-offset ,pixel-offset data ,rest))

Reading a PNG File Header

PNG files use big-endian encoding (network byte order):

(defun read-png-header (file-data)
  "Read PNG file header (big-endian format)."
  (binary ((#x89 #x50 #x4E #x47 #x0D #x0A #x1A #x0A)  ; PNG signature
           (chunk-length (size 32) big)                ; Length
           (chunk-type (size 32) binary)               ; Chunk type
           rest binary)
          file-data)
  `#m(length ,chunk-length type ,chunk-type data ,rest))

Summary

Is your data multi-byte? (size > 8)
├─ No → Endianness doesn't matter
└─ Yes →
    ├─ Network/portable format? → Use `big`
    ├─ Local storage only? → Use `native` (for performance)
    └─ Specific platform requirement? → Use whatever that platform uses

Remember: The default is big, which is almost always what you want for interoperability. When in doubt, use big-endian and spare yourself the debugging sessions that come from accidentally mixing byte orders.

Pattern Matching Binaries

“It is a mistake to think you can solve any major problems just with potatoes.” — Douglas Adams

Pattern matching binaries, however, is a different matter entirely. It turns out you can solve major problems by matching bit patterns—a fact that has revolutionized protocol parsing, image processing, and the general manipulation of structured binary data.

The Fundamental Symmetry

One of the more elegant properties of LFE’s binary syntax is its perfect symmetry: the same syntax used to construct a binary can be used to deconstruct it through pattern matching. This is rather like how a proper towel can both dry you off and serve as a makeshift sail, blanket, or weapon, depending on which end you grab.

Construction vs. Deconstruction

; Construction: Building a binary
lfe> (set data (binary ((42 (size 8)) (314159 (size 32)))))
#B(42 0 4 203 47)

; Deconstruction: Pattern matching to extract values
lfe> (binary ((answer (size 8)) (pi-ish (size 32))) data)
#B(42 0 4 203 47)
lfe> answer
42
lfe> pi-ish
314159

The pattern (binary ((answer (size 8)) (pi-ish (size 32))) data) says: “Take the binary in data, extract 8 bits into answer, then extract 32 bits into pi-ish.” The pattern matching will either succeed (binding the variables) or fail (crashing with appropriate alarm), much like attempting to teleport through a device that might not be entirely reliable.

Basic Pattern Matching Examples

Fixed-Size Fields

The simplest case involves extracting fixed-size segments:

(defun parse-rgb (pixel)
  "Extract RGB components from a 24-bit pixel."
  (binary ((r (size 8)) (g (size 8)) (b (size 8))) pixel)
  `#m(red ,r green ,g blue ,b))

Testing this function:

lfe> (parse-rgb #B(255 0 128))
#m(red 255 green 0 blue 128)

Variable-Length Fields

Things become more interesting when the size of one field determines the size of another—a common pattern in length-prefixed data structures:

(defun parse-length-value (data)
  "Parse a length-prefixed value (Pascal string style)."
  (binary ((len (size 8)) (value (size len) binary) rest binary) data)
  `#(,value ,rest))

Note the delightful sequence of events: first we extract len (8 bits), then immediately use that value to determine how many bits to extract into value. This is rather like opening a package that contains instructions for how big the package is—recursive, slightly mind-bending, but ultimately quite useful.

lfe> (set msg (binary ((5 (size 8)) (#"Hello" binary) (#" World" binary))))
#B(5 72 101 108 108 111 32 87 111 114 108 100)

lfe> (parse-length-value msg)
#(#B(72 101 108 108 111) #B(32 87 111 114 108 100))

The Rest of the Binary

Often you want to extract some fields and capture everything remaining in a catch-all variable. This is done by leaving off the size specification:

(defun parse-header (packet)
  "Extract magic number and version, keep the rest."
  (binary ((magic (size 16)) (version (size 8)) payload binary) packet)
  (if (=:= magic #xCAFE)
      `#(ok ,version ,payload)
      '#(error invalid_magic)))

Testing:

lfe> (set pkt (binary ((#xCAFE (size 16)) (42 (size 8)) (#"DATA" binary))))
#B(202 254 42 68 65 84 65)

lfe> (parse-header pkt)
#(ok 42 #B(68 65 84 65))

The payload binary at the end means “bind whatever’s left to payload”—no size specification needed, as the size is implicitly “all remaining bytes.”

Don’t Care Variables

Sometimes you need to skip over fields you don’t care about. Use the anonymous variable _ for this:

(defun extract-timestamp (log-entry)
  "Extract just the timestamp, ignoring other fields."
  (binary ((_ (size 16))              ; Skip magic number
           (timestamp (size 64))       ; Extract timestamp
           (_ binary))                 ; Ignore the rest
          log-entry)
  timestamp)

This is the binary equivalent of the “skip to the good part” button on a remote control, except more reliable and less likely to cause arguments.

Pattern Matching in Function Clauses

Pattern matching binaries truly shines when used in function clauses, allowing you to handle different packet types with elegant dispatch:

(defun handle-message
  ; Ping message
  ([(binary (#x01 rest binary))]
   `#(ping ,rest))

  ; Pong message
  ([(binary (#x02 rest binary))]
   `#(pong ,rest))

  ; Data message with length prefix
  ([(binary (#x03 (len (size 16) big) (payload (size len) binary) rest binary))]
   `#(data ,payload ,rest))

  ; Unknown message type
  ([(binary (type rest binary))]
   `#(unknown ,type ,rest)))

Each clause matches a different binary pattern, and the first matching clause wins. This is protocol parsing at its most readable:

lfe> (handle-message (binary (#x01 #"PING DATA")))
#(ping #B(80 73 78 71 32 68 65 84 65))

lfe> (handle-message (binary (#x03 (5 (size 16) big) #"HELLO" #"EXTRA")))
#(data #B(72 69 76 76 79) #B(69 88 84 82 65))

Bit-Level Pattern Matching

Remember that LFE works at the bit level, not just byte level. You can match on arbitrary bit boundaries:

(defun decode-flags (byte)
  "Decode individual flag bits."
  (binary ((verbose (size 1))
           (debug (size 1))
           (production (size 1))
           (experimental (size 1))
           (_ (size 4)))           ; Skip reserved bits
          (binary (byte)))
  `#m(verbose ,(=/= verbose 0)
      debug ,(=/= debug 0)
      production ,(=/= production 0)
      experimental ,(=/= experimental 0)))

Testing with a flags byte where bits 0, 1, and 3 are set:

lfe> (decode-flags #B(11010000))
#m(verbose true debug true production false experimental true)

The Dual Nature of Size Variables

Here’s a subtle but crucial point: in pattern matching, size specifications can use variables that were previously bound in the pattern:

; This works - 'len' is bound before use:
(binary ((len (size 8)) (data (size len) binary)) input)

; This doesn't work - 'len' not yet bound:
(binary ((data (size len) binary) (len (size 8))) input)

The pattern matcher evaluates left to right, binding variables as it goes. You can only use a variable as a size specifier if it’s been bound earlier in the same pattern. This is less of a limitation and more of a sensible design decision, preventing temporal paradoxes of the sort that plague time travelers and quantum physicists.

Complex Real-World Example: Decoding a TLV Structure

Type-Length-Value (TLV) encoding is ubiquitous in network protocols. Let’s build a proper parser:

(defun parse-tlv (data)
  "Parse TLV records: 1 byte type, 2 bytes length, N bytes value."
  (parse-tlv data '()))

(defun parse-tlv
  ; No more data
  ([('') acc]
   (lists:reverse acc))

  ; Parse one TLV record
  ([(binary ((type (size 8))
             (len (size 16) big)
             (value (size len) binary)
             rest binary))
    acc]
   (let ((record `#m(type ,type value ,value)))
     (parse-tlv rest (cons record acc))))

  ; Invalid data
  ([_ _acc]
   '#(error invalid_tlv)))

Testing with multiple TLV records:

lfe> (set tlv-data
       (binary ((1 (size 8))           ; Type 1
                (5 (size 16) big)      ; Length 5
                (#"HELLO" binary)      ; Value "HELLO"
                (2 (size 8))           ; Type 2
                (5 (size 16) big)      ; Length 5
                (#"WORLD" binary))))   ; Value "WORLD"
#B(1 0 5 72 69 76 76 79 2 0 5 87 79 82 76 68)

lfe> (parse-tlv tlv-data)
(#m(type 1 value #B(72 69 76 76 79))
 #m(type 2 value #B(87 79 82 76 68)))

Pattern Matching with Guards

You can combine binary pattern matching with guards for additional validation:

(defun validate-packet
  ; Valid version 2 packet
  ([(binary ((#xDEAD (size 16) big)
             (2 (size 8))
             rest binary))]
   (when (>= (byte_size rest) 4))
   `#(ok ,rest))

  ; Valid version 1 packet (deprecated)
  ([(binary ((#xDEAD (size 16) big)
             (1 (size 8))
             rest binary))]
   `#(deprecated ,rest))

  ; Invalid packet
  ([_]
   '#(error invalid_packet)))

The guard (when (>= (byte_size rest) 4)) ensures that version 2 packets have at least 4 bytes of payload—a sensible requirement that prevents confusion later.

Common Pitfalls and How to Avoid Them

Pitfall 1: Spacing Around <<

; Wrong - tokenizer reads this as ≤
(set b=<<1,2,3>>)

; Correct - space before <<
(set b (binary (1 2 3)))

In LFE, this particular pitfall is less common due to the explicit binary form, but it’s worth noting if you’re translating Erlang examples.

Pitfall 2: Non-byte-aligned Binary Segments

; This will fail - 7 bits is not byte-aligned
(binary ((x (size 7) binary) rest binary) data)

; This works - 7 bits as an integer or bitstring
(binary ((x (size 7)) rest binary) data)

Remember: segments typed as binary must be byte-aligned (size divisible by 8).

Pitfall 3: Using Arithmetic in Patterns

; This won't work - can't use arithmetic directly
(binary ((value (size (+ n 1)))) data)

; This works - compute size first
(let ((field-size (+ n 1)))
  (binary ((value (size field-size))) data))

Pattern matching expects literal values or bound variables, not arithmetic expressions. Do your math before the pattern, not during it.

Summary

The beauty of binary pattern matching lies in its symmetry with construction. If you can write:

(binary ((type (size 8)) (len (size 16) big) (data (size len) binary)))

to construct a binary, you can use the exact same pattern to deconstruct it. This makes code that handles binary protocols remarkably readable—the pattern itself documents the structure of the data.

This is protocol parsing as it was meant to be: declarative, obvious, and less prone to off-by-one errors than manual bit-shifting in C. It’s one of those rare cases where the elegant solution is also the practical one, rather like discovering that your towel is, improbably, exactly the right size for everything you need it to do.

Binary Comprehensions

“The ships hung in the sky in much the same way that bricks don’t.” — Douglas Adams

Binary comprehensions hang in the syntax of LFE in much the same way that list comprehensions do—which is to say, elegantly and with a certain mathematical inevitability that makes one wonder why all languages don’t have them.

If you’ve grasped list comprehensions (and if you haven’t, may I suggest returning to that section before proceeding, as the concepts build upon each other rather like a Jenga tower, except more stable), then binary comprehensions should slot neatly into your mental model. They are, essentially, list comprehensions that produce binaries instead of lists.

Where list comprehensions transform lists into other lists, binary comprehensions transform:

• Binaries into lists
• Lists into binaries
• Binaries into other binaries

This versatility is rather like having a Swiss Army knife, except that each blade is perfectly suited for cutting bits and bytes rather than wilderness survival tasks.

The Basic Syntax

A binary comprehension has two primary forms, distinguished by what they produce:

Form 1: Producing Lists from Binaries

(list-comp (<qual> ...) (<binary-generator> ...) [<guard> ...])

This generates a list by iterating over binary data. The syntax (<= binary-pattern source-binary) creates a binary generator.

Form 2: Producing Binaries

(binary-comp (<binary-qual> ...) (<binary-generator> ...) [<guard> ...])

This generates a binary by iterating over binary (or list) data. The binary qualifier must be wrapped in << >> (conceptually speaking—LFE uses the binary form).

Binary to List: The Simplest Case

Let’s extract individual bytes from a binary and convert them to a list:

lfe> (list-comp ((<byte> (binary-gen (<= #B(1 2 3 4 5))))) byte)
(1 2 3 4 5)

The pattern (<byte> extracts each byte in sequence. We could be more explicit about the size:

lfe> (list-comp ((<<(byte (size 8))>> (binary-gen (<= #B(72 101 108 108 111)))))
                byte)
(72 101 108 108 111)  ; ASCII codes for "Hello"

Adding Filters

Binary comprehensions support filters just like list comprehensions. Let’s extract only the even bytes:

lfe> (list-comp ((<<byte>> (binary-gen (<= #B(1 2 3 4 5 6 7 8))))
                 (=:= 0 (rem byte 2)))
                byte)
(2 4 6 8)

The guard (=:= 0 (rem byte 2)) filters for even numbers. This is considerably more elegant than manually iterating through the binary with index arithmetic and conditional logic, rather like how using a proper can opener is more elegant than attacking a tin with a rock.

List to Binary: The Reverse Journey

We can just as easily generate binaries from lists:

lfe> (binary-comp ((<<byte>> (list-gen (<- '(1 2 3 4 5))))) byte)
#B(1 2 3 4 5)

Note the subtle but crucial difference: we use list-gen with <- for list generators, and binary-gen with <= for binary generators. The mnemonic device here is that <= looks vaguely like it’s dealing with binary data (or at least, it’s different from <-, which is what matters).

Transforming Values

Binary comprehensions can transform values during generation:

lfe> (binary-comp ((<<n>> (list-gen (<- '(1 2 3 4 5))))) (* n 2))
#B(2 4 6 8 10)

Each number is doubled before being packed into the binary. This is useful for scaling pixel values, applying gain to audio samples, or any number of byte-level transformations that would otherwise require explicit loops.

Binary to Binary: The Power Move

The most powerful use of binary comprehensions is transforming binaries into other binaries. Let’s invert all the bytes:

lfe> (binary-comp ((<<byte>> (binary-gen (<= #B(1 2 3 4 5)))))
                  (- 255 byte))
#B(254 253 252 251 250)

This inverts each byte by subtracting it from 255—a common operation in image processing (creating negatives) and certain encryption schemes (though please don’t use this for actual encryption; that way lies security disasters).

Bit-Level Manipulation

Remember that we’re working with bits, not just bytes. Let’s extract alternating 4-bit nibbles:

lfe> (let ((data #B(#xAB #xCD #xEF)))
       (list-comp ((<<(nibble (size 4))>> (binary-gen (<= data))))
                  nibble))
(10 11 12 13 14 15)  ; 0xA, 0xB, 0xC, 0xD, 0xE, 0xF

Each 4-bit nibble is extracted in sequence. This is particularly useful when dealing with BCD (Binary-Coded Decimal) or similar packed formats.

Practical Example: RGB Color Manipulation

Let’s build a system for manipulating RGB pixels, because color manipulation is both useful and visually verifiable:

(defun parse-pixels (binary-pixels)
  "Convert binary RGB data to a list of RGB tuples."
  (list-comp ((<<(r (size 8)) (g (size 8)) (b (size 8))>>
               (binary-gen (<= binary-pixels))))
             `#(,r ,g ,b)))

(defun pack-pixels (pixel-list)
  "Convert a list of RGB tuples to binary."
  (binary-comp ((<<(r (size 8)) (g (size 8)) (b (size 8))>>
                 (list-gen (<- pixel-list))))
               (tuple r g b)))

(defun grayscale (pixels)
  "Convert RGB pixels to grayscale using luminosity method."
  (binary-comp ((<<(r (size 8)) (g (size 8)) (b (size 8))>>
                 (binary-gen (<= pixels))))
               (let* ((luminosity (trunc (+ (* r 0.299)
                                           (* g 0.587)
                                           (* b 0.114))))
                      (gray (min 255 (max 0 luminosity))))
                 (tuple gray gray gray))))

Testing these functions:

lfe> (set color-data #B(255 0 0  0 255 0  0 0 255))
#B(255 0 0 0 255 0 0 0 255)  ; Red, Green, Blue pixels

lfe> (parse-pixels color-data)
(#(255 0 0) #(0 255 0) #(0 0 255))

lfe> (grayscale color-data)
#B(76 76 76 149 149 149 29 29 29)  ; Grayscale equivalents

The grayscale conversion uses the standard luminosity formula, which weights green more heavily because the human eye is more sensitive to green light. This is one of those delightful facts of human biology that makes absolute sense once you know it, and seems arbitrary until you do.

Multiple Generators: The Cartesian Product

Binary comprehensions can have multiple generators, producing the Cartesian product of their inputs:

lfe> (binary-comp ((<<x>> (binary-gen (<= #B(1 2))))
                   (<<y>> (binary-gen (<= #B(10 20)))))
                  (+ x y))
#B(11 21 12 22)

This generates all combinations: (1+10), (1+20), (2+10), (2+20). While this example is contrived (as examples often are, being designed for clarity rather than immediate utility), the principle is powerful when dealing with structured data like coordinate grids or lookup tables.

Bit String Manipulation

Binary comprehensions work seamlessly with bitstrings (non-byte-aligned binary data):

lfe> (set bitstring #B(#b10101100 #b11110000))  ; Two bytes
#B(172 240)

lfe> (list-comp ((<<(bit (size 1))>> (binary-gen (<= bitstring))))
                bit)
(1 0 1 0 1 1 0 0 1 1 1 1 0 0 0 0)  ; All 16 bits extracted

This extracts each individual bit. We can then flip them:

lfe> (binary-comp ((<<(bit (size 1))>> (binary-gen (<= bitstring))))
                  (- 1 bit))
#B(83 15)  ; Bits flipped: 01010011 00001111

The byte values change from 172,240 to 83,15 because we’ve inverted every bit. This is the binary equivalent of a photographic negative.

Complex Example: Run-Length Encoding

Let’s implement a simple run-length encoder using binary comprehensions. This algorithm compresses sequences of repeated values:

(defun rle-encode (data)
  "Simple run-length encoding for byte data."
  (rle-encode-helper data '()))

(defun rle-encode-helper
  ([(binary ()) acc]
   (pack-rle-pairs (lists:reverse acc)))

  ([(binary ((byte (size 8)) rest binary)) acc]
   (let ((count-and-rest (count-bytes byte rest 1)))
     (rle-encode-helper (element 2 count-and-rest)
                       (cons `#(,byte ,(element 1 count-and-rest)) acc)))))

(defun count-bytes (byte data count)
  "Count consecutive occurrences of byte."
  (case data
    ((binary (byte rest binary))
     (count-bytes byte rest (+ count 1)))
    (_
     `#(,count ,data))))

(defun pack-rle-pairs (pairs)
  "Pack RLE pairs into binary."
  (binary-comp ((<<(byte (size 8)) (count (size 8))>>
                 (list-gen (<- pairs))))
               (tuple byte count)))

Testing this contraption:

lfe> (set input #B(65 65 65 65 66 66 67 68 68 68 68 68))
#B(65 65 65 65 66 66 67 68 68 68 68 68)  ; AAAABBCDDDDD

lfe> (rle-encode input)
#B(65 4 66 2 67 1 68 5)  ; A:4, B:2, C:1, D:5

Each pair of bytes in the output represents (value, count). This is more efficient when you have long runs of repeated values, though admittedly less efficient when you don’t—rather like how a towel is useful when you’re wet but somewhat burdensome when you’re trying to pack light.

Advanced Pattern: Sliding Window

Binary comprehensions can implement sliding window operations, useful for convolution filters or checksum calculations:

(defun sliding-sum (data)
  "Calculate sum of each 3-byte window."
  (sliding-sum-helper data '()))

(defun sliding-sum-helper
  ([(binary ((a (size 8)) (b (size 8)) (c (size 8)) rest binary)) acc]
   (sliding-sum-helper
     (binary (b c rest binary))
     (cons (+ a b c) acc)))
  ([_ acc]
   (lists:reverse acc)))

Testing:

lfe> (sliding-sum #B(1 2 3 4 5))
(6 9 12)  ; 1+2+3, 2+3+4, 3+4+5

This slides a 3-byte window across the data, computing the sum at each position. Real-world applications might compute moving averages, detect patterns, or implement digital signal processing filters.

Performance Considerations

Binary comprehensions compile to efficient code, but there are still considerations:

1. Memory usage: Each intermediate binary allocation takes memory. For very large data, consider streaming approaches.

2. Avoid excessive copying: Binary manipulation in Erlang/LFE is optimized for append operations. Prepending or inserting in the middle is expensive.

3. Use binary comprehensions for transformation, not iteration: If you’re just checking conditions without producing output, use regular recursion or lists:foldl.

Summary

Binary comprehensions represent a peculiar achievement in language design: they make something inherently low-level (bit manipulation) feel high-level (declarative transformation). This is the sort of abstraction that doesn’t hide complexity so much as organize it in a way that makes sense to human minds, rather like how a filing cabinet doesn’t make papers disappear but does make them findable.

They’re particularly valuable when dealing with data formats designed by committees, transmitted over networks designed by different committees, and processed by systems designed by yet other committees who never spoke to the first two. Which is to say, they’re valuable rather often.

Bitstrings

“For instance, on the planet Earth, man had always assumed that he was more intelligent than dolphins because he had achieved so much—the wheel, New York, wars and so on—whilst all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man—for precisely the same reasons.” — Douglas Adams

In the realm of binary data, there has long been an assumption that everything should align on byte boundaries (8 bits), because bytes are civilized units that everyone understands. Bitstrings, however, are the dolphins of the data world—mucking about at arbitrary bit boundaries and being perfectly content with sequences of 7, 13, or 42 bits, having a good time while the byte-aligned world looks on in mild confusion.

What Bitstrings Are

A bitstring is a sequence of bits whose length is not necessarily divisible by 8. It is to a binary what a dolphin is to a submarine—related, existing in the same medium, but operating on fundamentally different principles.

Technically speaking:

• A binary has a bit length that is a multiple of 8 (it’s byte-aligned)
• A bitstring has an arbitrary bit length (not necessarily byte-aligned)
• All binaries are bitstrings, but not all bitstrings are binaries

This is rather like how all squares are rectangles, but not all rectangles are squares—a geometric truth that becomes relevant in unexpected contexts, such as when you’re trying to fit luggage into a vehicle or bits into a protocol.

Identification in the REPL

The Erlang VM (and thus LFE) provides predicates to distinguish these entities:

lfe> (set byte-aligned (binary (1 2 3)))
#B(1 2 3)
lfe> (is_binary byte-aligned)
true
lfe> (is_bitstring byte-aligned)
true  ; All binaries are bitstrings

lfe> (set not-aligned (binary ((1 (size 7)))))
#B(1:7)  ; Only 7 bits
lfe> (is_binary not-aligned)
false  ; Not byte-aligned
lfe> (is_bitstring not-aligned)
true   ; But still a valid bitstring

The notation #B(1:7) means “a bitstring containing the value 1 using 7 bits.” This is how LFE indicates non-byte-aligned data—with a colon and the bit count.

Creating Bitstrings

Creating bitstrings is syntactically identical to creating binaries; you simply specify bit counts that don’t sum to a multiple of 8:

lfe> (binary ((42 (size 6))))
#B(42:6)

lfe> (binary ((1 (size 1)) (0 (size 1)) (1 (size 1))))
#B(5:3)  ; Binary 101 = decimal 5 in 3 bits

lfe> (binary ((15 (size 4)) (7 (size 3))))
#B(127:7)  ; 1111 111 in binary = 127 in 7 bits

Why Non-Alignment Happens

Non-byte-aligned data occurs in the wild for several reasons:

1. Compression schemes often work at bit-level precision
2. Hardware protocols may pack flags and values without regard for byte boundaries
3. Video codecs use variable-length codes that don’t align
4. Huffman encoding produces bit sequences of varying length

These are all perfectly valid reasons to abandon the comfortable world of byte-alignment, rather like how there are valid reasons to swim in the ocean at night despite it being dark and full of things that might nibble.

Size Queries

Two BIFs let you interrogate bitstrings about their dimensions:

lfe> (set bits #B(1 2 3))
#B(1 2 3)
lfe> (bit_size bits)
24
lfe> (byte_size bits)
3

lfe> (set awkward #B(255:7))
#B(127:7)
lfe> (bit_size awkward)
7
lfe> (byte_size awkward)
1  ; Rounds up to nearest byte

Note that byte_size rounds up to the nearest byte. A 7-bit bitstring occupies 1 byte in memory (because memory is byte-addressable), but conceptually it’s still 7 bits. This is similar to how ordering one egg at a restaurant might still result in being charged for a minimum order, despite only wanting one egg.

Pattern Matching Bitstrings

Pattern matching works identically with bitstrings as with binaries, you simply specify non-byte-aligned patterns:

(defun decode-flags (bitstring)
  "Decode a 7-bit flag field."
  (binary ((flag1 (size 1))
           (flag2 (size 1))
           (flag3 (size 1))
           (counter (size 4)))
          bitstring)
  `#m(flag1 ,(=/= flag1 0)
      flag2 ,(=/= flag2 0)
      flag3 ,(=/= flag3 0)
      counter ,counter))

Testing:

lfe> (decode-flags #B(#b1010111:7))
#m(flag1 true flag2 false flag3 true counter 7)

The bit pattern 1010111 is decoded as: flag1=1, flag2=0, flag3=1, counter=0111 (decimal 7).

Appending Bitstrings

One of the more useful operations is appending bitstrings, which works even when neither is byte-aligned:

lfe> (binary ((#B(7:3) bitstring) (#B(15:4) bitstring)))
#B(127:7)  ; 111 + 1111 = 1111111 (7 bits total)

lfe> (binary ((#B(1:1) bitstring) (#B(0:1) bitstring) (#B(1:1) bitstring)))
#B(5:3)  ; 1 + 0 + 1 = 101 (3 bits total)

This is remarkably useful when building up variable-length encoded data, where each component might be any number of bits.

Practical Example: Morse Code

Morse code is naturally represented as a bitstring—dots and dashes of varying length. Let’s build a simple encoder:

(defun encode-morse (char)
  "Encode a single character to Morse (1=dah, 0=dit)."
  (case char
    (#\A (binary ((1 (size 1)) (0 (size 1)))))      ; .-
    (#\B (binary ((0 (size 1)) (1 (size 1))
                  (1 (size 1)) (1 (size 1)))))     ; -...
    (#\C (binary ((0 (size 1)) (1 (size 1))
                  (0 (size 1)) (1 (size 1)))))     ; -.-.
    (#\E (binary ((1 (size 1)))))                   ; .
    (#\T (binary ((0 (size 1)))))                   ; -
    (_ (binary ()))))  ; Unknown character

(defun encode-word (word)
  "Encode a word to Morse bitstring."
  (let ((chars (string:to_upper word)))
    (lists:foldl
      (lambda (char acc)
        (binary ((acc bitstring) ((encode-morse char) bitstring))))
      (binary ())
      chars)))

Testing:

lfe> (encode-word "BEAT")
#B(27:10)  ; -... . .- -

lfe> (bit_size (encode-word "BEAT"))
10  ; Total bits needed

Each letter uses a different number of bits, so the result is naturally a bitstring. This is more space-efficient than using full bytes for each letter, and more faithful to the actual structure of Morse code.

Variable-Length Encoding Example

Consider a hypothetical protocol where integers are encoded with a variable-length scheme:

• 0-15: 4 bits
• 16-255: 8 bits (prefix: 1111)
• 256+: 16 bits (prefix: 11111111)

(defun encode-varint (n)
  "Encode integer with variable-length encoding."
  (cond
    ((=< n 15)
     (binary ((n (size 4)))))
    ((=< n 255)
     (binary ((15 (size 4)) (n (size 8)))))
    ((<= n 65535)
     (binary ((255 (size 8)) (n (size 16)))))
    ('true
     (error 'value_too_large))))

(defun encode-varint-list (nums)
  "Encode a list of integers."
  (lists:foldl
    (lambda (n acc)
      (binary ((acc bitstring) ((encode-varint n) bitstring))))
    (binary ())
    nums))

Testing:

lfe> (encode-varint-list '(5 200 10))
#B(95 200 10 0)  ; 0101 11111111 11001000 1010 (non-aligned!)

lfe> (bit_size (encode-varint-list '(5 200 10)))
28  ; 4 + 12 + 4 + 8 = 28 bits

The result is 28 bits (3.5 bytes), demonstrating how variable-length encoding produces non-aligned bitstrings.

The Alignment Problem

While bitstrings are powerful, there’s a practical constraint: most I/O operations expect byte-aligned data. You can’t write a 7-bit bitstring directly to a file or socket. The data must be padded to a byte boundary first.

Padding to Byte Alignment

(defun pad-to-bytes (bitstring)
  "Pad bitstring to next byte boundary with zeros."
  (let* ((bit-size (bit_size bitstring))
         (padding-needed (rem (- 8 (rem bit-size 8)) 8)))
    (if (=:= padding-needed 0)
        bitstring
        (binary ((bitstring bitstring)
                 ((0 (size padding-needed))))))))

Testing:

lfe> (set bits #B(127:7))
#B(127:7)
lfe> (pad-to-bytes bits)
#B(254)  ; Padded with one 0 bit: 1111111 → 11111110

lfe> (is_binary (pad-to-bytes bits))
true  ; Now it's a proper binary

The 7-bit value 1111111 becomes the 8-bit value 11111110 (decimal 254) when padded.

Extracting Bits from Bytes

A common operation is extracting individual bits from bytes:

(defun byte-to-bits (byte)
  "Convert a byte to a list of 8 bits."
  (list-comp ((<<(bit (size 1))>> (binary-gen (<= (binary (byte))))))
             bit))

(defun bits-to-byte (bits)
  "Convert a list of 8 bits to a byte."
  (binary-comp ((<<bit>> (list-gen (<- bits))))
               bit))

Testing:

lfe> (byte-to-bits 170)  ; 0b10101010
(1 0 1 0 1 0 1 0)

lfe> (bits-to-byte '(1 1 0 0 1 1 0 0))
#B(204)  ; 0b11001100

This is useful for manipulating bitmasks, implementing bit-banging protocols, or simply debugging binary data.

Why Bitstrings Matter

Bitstrings exist because the universe of data is not always byte-aligned. Network protocols pack bits tightly to save bandwidth. Compression algorithms produce variable-length codes. Hardware interfaces have registers with odd bit widths. Video codecs use fractional-bit encoding schemes (not literally fractional bits, but encoding schemes where different symbols use different numbers of bits).

Without bitstrings, you’d be forced to:

1. Manually shift and mask bits (error-prone, unreadable)
2. Waste space padding everything to bytes (inefficient)
3. Give up and accept that some things are just hard (admitting defeat)

LFE’s bitstring support means you can work at whatever granularity the problem demands, rather than being forced into the byte-sized boxes that most languages provide.

Performance Characteristics

Bitstrings have some interesting performance properties:

1. Creation: Fast, similar to binary creation
2. Appending: Efficient when building up bitstrings sequentially
3. Pattern matching: Very efficient, compiled to optimized VM instructions
4. Conversion to bytes: Requires padding, which is a copy operation

In general, if your final output must be byte-aligned (as most I/O is), consider whether working in bitstrings throughout is necessary, or if you can work in bytes and only use bitstrings for specific sub-operations.

Summary

The difference between binaries and bitstrings is not merely technical but philosophical. Binaries represent data as the computer sees it—aligned, addressable, practical. Bitstrings represent data as the problem requires it—precise, minimal, theoretical.

This is rather like the difference between ordering a pint of beer (a sensible, standard unit) and ordering exactly 473.176 milliliters of beer (precise, but likely to annoy the bartender). Both get you beer, but one acknowledges the reality of how beer is actually served, while the other insists on theoretical precision.

For most purposes, binaries are sufficient. But when the problem truly demands bit-level precision—when you’re implementing a compression codec, parsing a video stream, or interfacing with peculiar hardware—bitstrings are not just useful but essential.

They are the dolphins of the data world, and sometimes, being a dolphin is exactly what the situation requires.

Bitwise Operators

“In the beginning the Universe was created. This has made a lot of people very angry and been widely regarded as a bad move.” — Douglas Adams

In a similar vein, someone decided that Boolean algebra should be applied directly to the individual bits of integers, creating bitwise operators. This has made a lot of programmers very confused and been widely regarded as powerful, if occasionally bewildering.

LFE inherits Erlang’s bitwise operators, which operate on integers at the bit level. Unlike the binary syntax (which constructs and deconstructs binary data structures), bitwise operators perform logical and shift operations on integer values interpreted as sequences of bits.

The available operators are:

These operators work on integers of arbitrary size, limited only by available memory—though in practice, very large integers become unwieldy in much the same way that very large towels become difficult to fold.

Bitwise AND

The AND operation returns 1 only when both bits are 1. It’s the pessimist of Boolean operations, assuming the worst and only accepting perfection.

lfe> (band #b1010 #b1100)
8  ; #b1000 in binary

lfe> (band 170 85)
0  ; 10101010 AND 01010101 = 00000000

Practical Use: Masking

AND is commonly used to mask out specific bits:

(defun get-low-nibble (byte)
  "Extract the lower 4 bits of a byte."
  (band byte #b00001111))

(defun is-even? (n)
  "Check if number is even by testing lowest bit."
  (=:= 0 (band n 1)))

Testing:

lfe> (get-low-nibble #xA7)
7  ; Keep only 0111

lfe> (is-even? 42)
true
lfe> (is-even? 43)
false

The even/odd test works because even numbers have a 0 in the least significant bit, while odd numbers have a 1. Masking with (band n 1) extracts just that bit.

Bitwise OR

The OR operation returns 1 if either bit is 1. It’s the optimist, seeing potential everywhere and requiring only minimal commitment.

lfe> (bor #b1010 #b1100)
14  ; #b1110 in binary

lfe> (bor 170 85)
255  ; 10101010 OR 01010101 = 11111111

Practical Use: Setting Bits

OR is used to set specific bits to 1:

(defun set-bit (byte bit-position)
  "Set the bit at bit-position to 1."
  (bor byte (bsl 1 bit-position)))

(defun combine-flags (flag-list)
  "Combine multiple flag bits."
  (lists:foldl (lambda (flag acc) (bor flag acc)) 0 flag-list))

Testing:

lfe> (set-bit #b00000000 3)
8  ; #b00001000

lfe> (combine-flags '(#b0001 #b0100 #b1000))
13  ; #b1101

Bitwise XOR

XOR (exclusive OR) returns 1 when bits differ. It’s the contrarian, celebrating difference and rejecting conformity.

lfe> (bxor #b1010 #b1100)
6  ; #b0110 in binary

lfe> (bxor 170 85)
255  ; 10101010 XOR 01010101 = 11111111

Practical Use: Toggling and Encryption

XOR has unique properties that make it useful for:

• Toggling bits: (bxor x mask) flips bits where mask is 1
• Simple encryption: (bxor data key) encrypts, (bxor encrypted key) decrypts
• Comparison: XOR finds differences between bit patterns

(defun toggle-bit (byte bit-position)
  "Toggle the bit at bit-position."
  (bxor byte (bsl 1 bit-position)))

(defun simple-cipher (data key)
  "XOR cipher (symmetric: same function encrypts and decrypts)."
  (bxor data key))

(defun find-differences (a b)
  "Return bitmask showing where two values differ."
  (bxor a b))

Testing:

lfe> (let ((original #b10101010))
       (toggle-bit (toggle-bit original 0) 0))
170  ; Toggle twice returns to original

lfe> (let* ((plaintext 42)
            (key 128)
            (encrypted (simple-cipher plaintext key))
            (decrypted (simple-cipher encrypted key)))
       `#(encrypted ,encrypted decrypted ,decrypted))
#(encrypted 170 decrypted 42)

lfe> (find-differences #b11110000 #b11001100)
60  ; #b00111100 - bits differ at positions 2,3,4,5

Note: The XOR “cipher” is only secure for one-time pads with truly random keys. Using it with short or predictable keys is approximately as secure as writing your password on a post-it note—which is to say, not at all.

Bitwise NOT

NOT flips all bits. Due to two’s complement representation of signed integers, (bnot x) equals (- (- x) 1).

lfe> (bnot #b00001111)
-16  ; #b11110000 in two's complement

lfe> (bnot 0)
-1

lfe> (bnot (bnot 42))
42  ; Double negation returns original

The result being negative is due to how integers are represented in memory. The leftmost bit indicates sign in two’s complement, so inverting a small positive number produces a negative number.

Practical Use: Creating Masks

(defun create-mask (bit-position)
  "Create a mask with all bits set except bit-position."
  (bnot (bsl 1 bit-position)))

(defun clear-bit (byte bit-position)
  "Clear the bit at bit-position to 0."
  (band byte (create-mask bit-position)))

Testing:

lfe> (create-mask 3)
-9  ; All bits set except bit 3

lfe> (clear-bit #b11111111 3)
247  ; #b11110111

Bit Shift Left

Shifting left by N positions is equivalent to multiplying by 2^N. This is one of the fastest arithmetic operations available.

lfe> (bsl 1 0)
1  ; 1 * 2^0 = 1

lfe> (bsl 1 3)
8  ; 1 * 2^3 = 8

lfe> (bsl 5 4)
80  ; 5 * 16 = 80

lfe> (bsl #b0011 2)
12  ; #b1100

Practical Use: Scaling and Positioning

(defun create-bitmask (n)
  "Create a bitmask with n bits set."
  (- (bsl 1 n) 1))

(defun pack-rgb (r g b)
  "Pack 8-bit RGB values into 24-bit integer."
  (bor (bsl r 16) (bor (bsl g 8) b)))

Testing:

lfe> (create-bitmask 4)
15  ; #b1111

lfe> (pack-rgb 255 128 64)
16744512  ; #xFF8040 in hex

Bit Shift Right

Shifting right by N positions is equivalent to integer division by 2^N, rounding down.

lfe> (bsr 8 1)
4  ; 8 / 2 = 4

lfe> (bsr 10 1)
5  ; 10 / 2 = 5 (rounds down)

lfe> (bsr 255 4)
15  ; 255 / 16 = 15

lfe> (bsr #b1100 2)
3  ; #b0011

Practical Use: Extraction and Division

(defun unpack-rgb (color)
  "Unpack 24-bit color into RGB components."
  `#(,(band (bsr color 16) 255)
     ,(band (bsr color 8) 255)
     ,(band color 255)))

(defun fast-divide-by-power-of-two (n power)
  "Divide by 2^power using bit shift."
  (bsr n power))

Testing:

lfe> (unpack-rgb 16744512)  ; #xFF8040
#(255 128 64)

lfe> (fast-divide-by-power-of-two 1000 3)
125  ; 1000 / 8 = 125

Combining Operators: Real-World Example

Let’s implement a simple permission system using bit flags:

(defun perm-read () 1)      ; #b001
(defun perm-write () 2)     ; #b010
(defun perm-execute () 4)   ; #b100

(defun grant-permission (current perm)
  "Grant a permission."
  (bor current perm))

(defun revoke-permission (current perm)
  "Revoke a permission."
  (band current (bnot perm)))

(defun has-permission? (permissions perm)
  "Check if permission is granted."
  (=:= perm (band permissions perm)))

(defun describe-permissions (perms)
  "Human-readable permission description."
  (let ((r (if (has-permission? perms (perm-read)) "r" "-"))
        (w (if (has-permission? perms (perm-write)) "w" "-"))
        (x (if (has-permission? perms (perm-execute)) "x" "-")))
    (++ r w x)))

Testing:

lfe> (let* ((p0 0)
            (p1 (grant-permission p0 (perm-read)))
            (p2 (grant-permission p1 (perm-write)))
            (p3 (grant-permission p2 (perm-execute)))
            (p4 (revoke-permission p3 (perm-write))))
       (list (describe-permissions p0)
             (describe-permissions p1)
             (describe-permissions p2)
             (describe-permissions p3)
             (describe-permissions p4)))
("---" "r--" "rw-" "rwx" "r-x")

This permission system is both space-efficient (3 bits vs. 3 booleans) and fast (bit operations are among the fastest CPU instructions).

Performance Characteristics

Bitwise operations are:

• Extremely fast: Usually single CPU cycle instructions
• Memory efficient: No allocation needed
• Deterministic: O(1) time complexity

They’re faster than:

• Arithmetic operations (though modern CPUs make these nearly as fast)
• Function calls
• Pattern matching
• Pretty much everything except register moves

This makes them ideal for:

• Hot paths in performance-critical code
• Bit flag manipulation
• Checksums and hash functions
• Low-level protocol parsing

When to Use Bitwise Operators

Use bitwise operators when:

• Implementing bit flags or permission systems
• Optimizing arithmetic by powers of two
• Interfacing with hardware registers
• Implementing checksums (XOR is common)
• Creating bitmasks

Don’t use them when:

• Regular arithmetic would be clearer
• The performance gain is negligible
• Code readability would suffer significantly

Remember: premature optimization is the root of all evil, but knowing when bit twiddling is appropriate is a sign of wisdom. Sometimes shifting left by 3 instead of multiplying by 8 matters. Most times it doesn’t. The art lies in knowing which is which.

A Note on Negative Numbers

Due to two’s complement representation, bitwise operations on negative numbers can produce surprising results:

lfe> (band -1 255)
255  ; -1 is all bits set, so AND with 255 gives 255

lfe> (bsr -16 2)
-4  ; Arithmetic shift preserves sign

In general, bitwise operations are most intuitive when working with unsigned interpretations of integers. If you need to work with negative values, be aware that the sign bit will participate in the operation.

Summary

Bitwise operators are one of those tools that seem esoteric until the moment you need them, at which point they become indispensable. They represent a direct line to the machine’s native operations—a reminder that underneath all the high-level abstractions, computers are still just flipping bits according to Boolean logic, and sometimes it’s useful to speak their language directly.

Serialization

“The history of every major computing civilization tends to move from chaos through complexity to the ultimate simplification of cramming everything into a byte stream and hoping for the best.” — Somewhat Improbable Musings on Data Persistence

There comes a moment in every Erlang (and by extension, LFE) programmer’s life when they must confront an uncomfortable truth: not all data can exist in the beautiful, ephemeral world of process memory. Sometimes—and this is where things get philosophically sticky—data must be serialized.

Serialization is the process of taking a perfectly happy, multi-dimensional data structure living its best life in RAM and squashing it flat into a sequence of bytes. It’s rather like explaining a joke to someone who wasn’t there: technically all the information is preserved, but something ineffable is lost in the translation.

Fortunately, LFE (via Erlang’s runtime) provides term-to-binary and binary-to-term, two functions that perform this existential compression and decompression with remarkable grace.

The Basics

;; In the REPL, observe the miracle of serialization
lfe> (set marvin-state (tuple 'robot 'depressed "brain the size of a planet"))
#(robot depressed "brain the size of a planet")

lfe> (set marvin-binary (term-to-binary marvin-state))
#B(131 104 3 100 0 5 114 111 98 111 116 100 0 9 100 101 112 114 ...)

lfe> (binary-to-term marvin-binary)
#(robot depressed "brain the size of a planet")

What you’re witnessing here is the Erlang External Term Format (ETF), a specification that describes how to represent any Erlang/LFE term as a sequence of bytes. It’s version 131, in case you were wondering, though I’m reliably informed that versions 1 through 130 were “experimental” in the sense that they involved a lot of shouting and the occasional desk-flip.

Why Serialize

There are several compelling reasons to transform your beautiful terms into ugly byte sequences:

1. Network Transmission: You want to send your data to another machine. Networks, being made of copper and light and various other stubbornly physical substances, deal in bytes.

2. Persistent Storage: You want your data to survive the heat death of your process. Files, being even more stubbornly physical than networks, also deal in bytes.

3. Time Travel: You want to save your data now and read it later. Future-you appreciates past-you’s foresight, assuming the format hasn’t changed. (It won’t. Probably.)

4. Distributed Erlang: The distributed Erlang protocol uses this format extensively. Every time you send a message to a process on another node, serialization happens behind the scenes, like a helpful but invisible stage crew.

A More Sophisticated Example: The Deep Space Signal

Let’s imagine we’re building a system for the Sirius Cybernetics Corporation’s new line of spacecraft. We need to serialize complex telemetry data:

(defun create-telemetry (ship-name velocity position)
  "Creates a telemetry packet for a spacecraft.
  Because even paranoid androids need to know where they are."
  (tuple 'telemetry
         (tuple 'timestamp (erlang:system_time 'millisecond))
         (tuple 'ship ship-name)
         (tuple 'velocity velocity)  ; meters per second
         (tuple 'position position)  ; (tuple x y z) in astronomical units
         (tuple 'mood 'vaguely-anxious)))

(defun serialize-telemetry (telemetry)
  "Converts telemetry to a transmissible format.
  Note: Does not include the answer to life, universe, and everything.
  That's a different packet."
  (term-to-binary telemetry))

(defun deserialize-telemetry (binary)
  "Reconstructs telemetry from binary format.
  May contain traces of existential dread."
  (binary-to-term binary))

Let’s use these functions:

lfe> (set heart-of-gold-pos (tuple 42.0 0.0 -100.0))
#(42.0 0.0 -100.0)

lfe> (set telemetry (create-telemetry
...>   "Heart of Gold"
...>   299792.458  ; Nearly the speed of light
...>   heart-of-gold-pos))
#(telemetry
  #(timestamp 1730841234567)
  #(ship "Heart of Gold")
  #(velocity 299792.458)
  #(position #(42.0 0.0 -100.0))
  #(mood vaguely-anxious))

lfe> (set binary-packet (serialize-telemetry telemetry))
#B(131 104 6 100 0 9 116 101 108 101 109 ...)

lfe> (set recovered (deserialize-telemetry binary-packet))
#(telemetry
  #(timestamp 1730841234567)
  #(ship "Heart of Gold")
  #(velocity 299792.458)
  #(position #(42.0 0.0 -100.0))
  #(mood vaguely-anxious))

The round-trip is perfect. Everything that went in comes back out, including the existential anxiety.

The Secret Sauce: What Gets Preserved

The truly remarkable thing about term-to-binary is that it preserves everything:

• Atoms remain atoms
• Numbers (integers, floats, bignums) maintain their precision
• Tuples preserve their structure and arity
• Lists (proper and improper) keep their shape
• Binaries get nested (yes, really—binaries within binaries)
• PIDs, ports, and references get special treatment in distributed contexts

The only things that don’t serialize well are:

1. Functions/lambdas: These are code, not data. You can’t serialize a function and expect it to work on another machine that doesn’t have that code.
2. Open file handles: These are OS resources and don’t travel well.
3. The improbability drive: This was never specified in the original ETF, a notable oversight.

Practical Considerations: Size and Compression

Binary serialization can produce fairly compact representations, but for truly large data structures, you might want compression:

lfe> (set large-data (lists:duplicate 1000 (tuple 'answer 42 'question 'unknown)))
(#(answer 42 question unknown) #(answer 42 question unknown) ...)

lfe> (set uncompressed (term-to-binary large-data))
#B(131 108 0 0 3 232 104 ...)

lfe> (set compressed (term-to-binary large-data (list 'compressed)))
#B(131 80 0 0 0 195 120 ...)

lfe> (tuple (byte-size uncompressed) (byte-size compressed))
#(28004 199)

That’s roughly 140x compression, which would please even the notoriously space-conscious Vogons (though they’d still complain about the inefficiency of storing the meaning of life in the first place).

Storage Patterns: Files and Mnesia

The most common use of serialization is writing to files:

(defun save-state (filename state)
  "Persists application state to disk.
  Returns ok on success, which is more optimistic than most things return."
  (let ((binary (term-to-binary state)))
    (file:write_file filename binary)))

(defun load-state (filename)
  "Loads application state from disk.
  May fail if the universe has ended since the last save."
  (case (file:read_file filename)
    ((tuple 'ok binary)
     (tuple 'ok (binary-to-term binary)))
    ((tuple 'error reason)
     (tuple 'error reason))))

Usage:

lfe> (set game-state
...>   (tuple 'player "Arthur Dent"
...>          'inventory (list 'tea 'towel 'no-tea)
...>          'location "Earth"
...>          'status 'confused))
#(player "Arthur Dent" inventory (tea towel no-tea) ...)

lfe> (save-state "game.sav" game-state)
ok

;; Later, perhaps after Earth's demolition...
lfe> (load-state "game.sav")
#(ok #(player "Arthur Dent" inventory (tea towel no-tea) ...))

Versioning and Evolution: The Heat Death Problem

One challenge with serialization is evolution. Your data format today might not be your data format tomorrow. Consider:

(defun save-versioned-state (filename state version)
  "Saves state with a version tag.
  For when you know the format will change but pretend it won't."
  (let ((versioned-state (tuple 'version version 'data state)))
    (save-state filename versioned-state)))

(defun load-versioned-state (filename expected-version)
  "Loads state and checks version compatibility.
  Returns error if versions don't match, because mixing versions
  is like mixing matter and antimatter: exciting but destructive."
  (case (load-state filename)
    ((tuple 'ok (tuple 'version version 'data state))
     (if (== version expected-version)
         (tuple 'ok state)
         (tuple 'error (tuple 'version-mismatch version expected-version))))
    (error error)))

This pattern helps you evolve your data formats over time without painting yourself into a corner. When you need to upgrade:

(defun migrate-state-v1-to-v2 (v1-state)
  "Migrates from v1 to v2 format.
  v2 adds 'anxiety-level' field because that's progress."
  (let (((tuple 'player name 'inventory items 'location loc 'status status) v1-state))
    (tuple 'player name
           'inventory items
           'location loc
           'status status
           'anxiety-level 'moderate)))

The Distributed Angle: Cross-Node Communication

When you send a message to a process on another Erlang node, serialization happens automatically:

;; On node 'earth@localhost'
lfe> (set msg (tuple 'urgent-message "Vogon fleet incoming" (tuple 'eta 2 'minutes)))
#(urgent-message "Vogon fleet incoming" #(eta 2 minutes))

lfe> (! (tuple 'defense-system 'mars@localhost) msg)
#(urgent-message "Vogon fleet incoming" #(eta 2 minutes))

The runtime serializes msg, sends the bytes over the network, and deserializes on the receiving node. You don’t see this happening—it’s like quantum mechanics but more reliable.

Security Considerations: Trust No Binary

An important warning: binary-to-term will happily reconstruct any valid term, including atoms. If you deserialize untrusted binaries, you could exhaust your atom table (atoms aren’t garbage collected). For untrusted input, use:

lfe> (binary-to-term untrusted-binary (list 'safe))

The 'safe option prevents creation of new atoms, resources, and functions. It’s the equivalent of wearing a towel—not foolproof protection, but sensible nonetheless.

Summary

Serialization is one of those fundamental capabilities that seems almost mundane until you need it, at which point it becomes indispensable. The ability to take any LFE term—no matter how complex, nested, or philosophically questionable—and convert it to a byte stream that can be stored, transmitted, and perfectly reconstructed is a minor miracle of engineering.

The External Term Format has survived decades of use in production systems, processed trillions of messages, and generally proven more reliable than most things in computing (and definitely more reliable than the Sirius Cybernetics Corporation’s products).

Now that we understand how to flatten our data into binary submission, let’s see how these concepts play out in real-world applications, where the data is messier, the constraints are tighter, and the requirements are written by people who’ve clearly never heard of Douglas Adams…

Real-World Applications

“In theory, there is no difference between theory and practice. In practice, there is.” — Attributed to Yogi Berra, though suspiciously similar to things said by engineers throughout the galaxy

We’ve spent considerable time in the comfortable, well-lit corridors of binary syntax theory. The examples have been clean, the data cooperative, and everything has worked on the first try. This is, as anyone who has actually written software knows, completely unlike reality.

Reality is where:

• Protocol specifications were written by committees who couldn’t agree on lunch
• Legacy systems exchange data in formats designed when 640K was a lot of memory
• Network packets arrive malformed because cosmic rays exist and Murphy’s Law has universal jurisdiction
• Documentation says one thing, implementations do another, and the RFC was “clarified” seventeen times

Let’s roll up our sleeves and look at some actual binary protocol handling, starting with a perennial favorite: the Internet Protocol.

Example 1: Dissecting IPv4 Datagrams

The Internet Protocol version 4 (IPv4) has been shuffling packets around the globe since 1981, which in computing terms makes it roughly as ancient as the pyramids. Its header format is a masterclass in bit-level data packing, born from an era when every bit mattered because memory was measured in kilobytes and network bandwidth in… well, let’s just say “optimistically.”

The IPv4 Header: A Map of the Territory

An IPv4 header looks like this (each row is 32 bits):

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version|  IHL  |Type of Service|          Total Length         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Identification        |Flags|      Fragment Offset    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Time to Live |    Protocol   |         Header Checksum       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       Source Address                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Destination Address                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Options (if IHL > 5)                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Notice how nothing lines up nicely on byte boundaries except by accident. The Flags field is 3 bits. THREE BITS. The Fragment Offset is 13 bits. This is what protocols designed by actual engineers look like, as opposed to the sort designed by people who think everything should align on 64-bit boundaries because it’s “cleaner.”

Parsing IPv4 in LFE: One Pattern to Rule Them All

Here’s how we parse this in LFE, using the bit syntax to handle all those awkwardly-sized fields:

;; Constants, because magic numbers are poor documentation
(defun ipv4-version () 4)
(defun ipv4-min-header-length () 5)  ; In 32-bit words

(defun parse-ipv4-datagram (datagram)
  "Parses an IPv4 datagram into its constituent parts.
  Returns a tuple of parsed fields, or 'error if the datagram is malformed.
  Does not validate checksums, because this is an example, not production code.
  (Though in fairness, production code often skips that too.)"
  (let ((dgram-size (byte_size datagram)))
    (case datagram
      ;; The big pattern match that does all the work
      ((binary (version 4) (header-length 4) (service-type 8) (total-length 16)
               (identification 16) (flags 3) (fragment-offset 13)
               (time-to-live 8) (protocol 8) (header-checksum 16)
               (source-ip 32) (dest-ip 32)
               (rest binary))
       ;; Validate header length
       (if (and (>= header-length (ipv4-min-header-length))
                (<= (* 4 header-length) dgram-size))
           ;; Calculate options size and extract
           (let* ((opts-len (* 4 (- header-length (ipv4-min-header-length))))
                  ((binary (options opts-len binary) (payload binary)) rest))
             (tuple 'ok
                    (map 'version version
                         'header-length header-length
                         'service-type service-type
                         'total-length total-length
                         'identification identification
                         'flags (parse-flags flags)
                         'fragment-offset fragment-offset
                         'ttl time-to-live
                         'protocol (protocol-name protocol)
                         'checksum header-checksum
                         'source (ip-to-string source-ip)
                         'destination (ip-to-string dest-ip)
                         'options options
                         'payload payload)))
           'error))
      ;; Anything else is malformed
      (_ 'error))))

(defun parse-flags (flags-bits)
  "Parses the 3-bit flags field.
  Bit 0: Reserved (must be zero, or RFC police will write stern letters)
  Bit 1: Don't Fragment (DF)
  Bit 2: More Fragments (MF)"
  (let (((binary (reserved 1) (dont-fragment 1) (more-fragments 1))
         (binary (flags-bits 3))))
    (map 'reserved (== reserved 1)
         'dont-fragment (== dont-fragment 1)
         'more-fragments (== more-fragments 1))))

(defun protocol-name (protocol-num)
  "Maps protocol numbers to names.
  Only handles the interesting ones. There are 256 possible values,
  and frankly, most of them are boring or deprecated."
  (case protocol-num
    (1 'icmp)   ; For when you want to ping something
    (6 'tcp)    ; The reliable one
    (17 'udp)   ; The fast one
    (41 'ipv6)  ; The future, allegedly
    (58 'ipv6-icmp)
    (n (tuple 'unknown n))))

(defun ip-to-string (ip-int)
  "Converts a 32-bit IP address to dotted-decimal notation.
  Returns a string, because that's how humans like to read IP addresses."
  (let (((binary (a 8) (b 8) (c 8) (d 8)) (binary (ip-int 32))))
    (lists:flatten (io_lib:format "~w.~w.~w.~w" (list a b c d)))))

Let’s see this in action with a real packet (or at least, a facsimile thereof):

lfe> ;; Construct a minimal IPv4 packet
lfe> ;; Source: 192.168.1.42, Dest: 10.0.0.1, TCP packet, TTL=64
lfe> (set test-packet
...>   (binary
...>     (4 4)           ; Version 4, header length 5 words
...>     (0 8)           ; Service type
...>     (60 16)         ; Total length (including payload)
...>     (12345 16)      ; Identification
...>     (0 3) (0 13)    ; Flags and fragment offset
...>     (64 8)          ; TTL
...>     (6 8)           ; Protocol (TCP)
...>     (0 16)          ; Checksum (we're lazy)
...>     ;; Source IP: 192.168.1.42
...>     (192 8) (168 8) (1 8) (42 8)
...>     ;; Dest IP: 10.0.0.1
...>     (10 8) (0 8) (0 8) (1 8)
...>     ;; Minimal TCP payload (not parsed here)
...>     "Hello, packet-switching world!"))

lfe> (parse-ipv4-datagram test-packet)
#(ok
  #M(version 4
     header-length 5
     service-type 0
     total-length 60
     identification 12345
     flags #M(reserved false dont-fragment false more-fragments false)
     fragment-offset 0
     ttl 64
     protocol tcp
     checksum 0
     source "192.168.1.42"
     destination "10.0.0.1"
     options #B()
     payload #B(72 101 108 108 111 ...)))

What This Demonstrates

1. Non-byte-aligned fields: The flags (3 bits) and fragment offset (13 bits) fields don’t align on byte boundaries, but the bit syntax handles this transparently.

2. Conditional parsing: The options field size depends on the header length field that was parsed earlier.

3. Nested patterns: We use pattern matching twice—once for the main header, once for the flags.

4. Real-world messiness: IP headers have evolved over decades, with various fields falling into disuse while remaining in the spec for backwards compatibility.

Example 2: Building a Custom Binary Protocol

Let’s design something more modern: a protocol for communicating with an army of towel-dispensing robots. (Every good spacecraft needs one.)

The Towel Service Protocol (TSP)

Our protocol needs to handle:

• Status queries
• Towel requests (with size preferences)
• Inventory updates
• Emergency towel deployment

We’ll use a simple framed format:

+--------+--------+--------+--------+--------+---...---+
| Magic  | Version| MsgType| Length | MsgID  | Payload |
| (16)   | (8)    | (8)    | (16)   | (32)   | (...)   |
+--------+--------+--------+--------+--------+---...---+

Here’s our LFE implementation:

(defmodule towel-protocol
  (export (encode-message 3)
          (decode-message 1)
          (create-status-query 1)
          (create-towel-request 3)
          (create-inventory-update 2)))

;; Protocol constants
(defun magic-number () 16#FADE)  ; Frood-Approved Data Exchange
(defun protocol-version () 1)

;; Message types
(defun msg-status-query () 1)
(defun msg-towel-request () 2)
(defun msg-inventory-update () 3)
(defun msg-emergency-deploy () 255)

(defun encode-message (msg-type msg-id payload)
  "Encodes a message into TSP binary format.
  Returns a binary ready for transmission.
  Does not include shipping charges or existential considerations."
  (let* ((payload-bin (term-to-binary payload))
         (payload-len (byte_size payload-bin)))
    (binary
      ((magic-number) 16)
      ((protocol-version) 8)
      (msg-type 8)
      (payload-len 16)
      (msg-id 32)
      (payload-bin binary))))

(defun decode-message (binary)
  "Decodes a TSP message.
  Returns (tuple 'ok msg-type msg-id payload) or 'error.
  May return 'error if the magic number is wrong, which suggests
  either transmission errors or you're not talking to a towel robot."
  (case binary
    ((binary
       ((magic-num 16) (version 8) (msg-type 8)
        (payload-len 16) (msg-id 32)
        (payload-bin payload-len binary)))
     (if (and (== magic-num (magic-number))
              (== version (protocol-version)))
         (tuple 'ok msg-type msg-id (binary-to-term payload-bin))
         (tuple 'error 'invalid-magic-or-version)))
    (_
     (tuple 'error 'malformed-message))))

;; Message constructors
(defun create-status-query (msg-id)
  "Creates a status query message.
  Robots love being asked how they're doing. It's only polite."
  (encode-message (msg-status-query) msg-id (tuple 'query 'status)))

(defun create-towel-request (msg-id size color)
  "Creates a towel request message.
  Size: 'small | 'medium | 'large | 'hoopy-frood-sized
  Color: An atom representing a color, preferably one that exists"
  (encode-message (msg-towel-request) msg-id
                  (tuple 'request 'towel
                         (map 'size size 'color color))))

(defun create-inventory-update (msg-id inventory-map)
  "Creates an inventory update message.
  inventory-map should be a map of towel-type to count."
  (encode-message (msg-inventory-update) msg-id
                  (tuple 'update 'inventory inventory-map)))

Usage example:

lfe> ;; Create a towel request
lfe> (set request (towel-protocol:create-towel-request 42 'large 'mostly-harmless-blue))
#B(250 222 1 2 0 42 0 0 0 42 131 104 3 ...)

lfe> ;; Decode it (simulating receiving the message)
lfe> (towel-protocol:decode-message request)
#(ok 2 42
  #(request towel
    #M(size large color mostly-harmless-blue)))

lfe> ;; Create an inventory update
lfe> (set inventory-data
...>   (map 'standard 42
...>        'deluxe 7
...>        'emergency 3
...>        'substandard 0))  ; We don't stock these
#M(standard 42 deluxe 7 emergency 3 substandard 0)

lfe> (set update (towel-protocol:create-inventory-update 43 inventory-data))
#B(250 222 1 3 0 47 0 0 0 43 131 116 ...)

lfe> (towel-protocol:decode-message update)
#(ok 3 43
  #(update inventory
    #M(standard 42 deluxe 7 emergency 3 substandard 0)))

Why This Design Works

1. Magic number: Helps detect when you’re trying to parse the wrong kind of data
2. Version field: Allows protocol evolution
3. Length prefix: Enables streaming protocols where messages arrive in chunks
4. Message ID: Supports request/response correlation
5. Flexible payload: Using term-to-binary means we can send any Erlang term as payload

This pattern—a simple fixed header followed by a length-prefixed payload—is extremely common in protocol design. It’s used by:

• Many RPC systems
• Message queue protocols (like AMQP)
• Database wire protocols
• Custom application protocols

Example 3: Log File Format with Binary Efficiency

Let’s create a binary log file format that’s both efficient and easy to parse:

(defmodule binary-logger
  (export (create-log-file 1)
          (write-log-entry 4)
          (read-log-file 1)))

;; Log file format:
;; Header: <<"BLOG", Version:8, Flags:8, Reserved:16>>
;; Each entry: <<Timestamp:64, Level:8, Length:16, Message/binary>>

(defun file-header-magic () <<"BLOG">>)
(defun file-version () 1)

(defun create-log-file (filename)
  "Creates a new binary log file with appropriate header."
  (let ((header (binary
                  ((file-header-magic) binary)
                  ((file-version) 8)
                  (0 8)   ; Flags (currently unused)
                  (0 16)))) ; Reserved
    (file:write_file filename header)))

(defun write-log-entry (filename level msg-string context)
  "Appends a log entry to the file.
  Level: 'debug | 'info | 'warning | 'error | 'critical
  Returns: ok | error"
  (let* ((timestamp (erlang:system_time 'millisecond))
         (level-num (level-to-number level))
         (msg-bin (term-to-binary (tuple msg-string context)))
         (msg-len (byte_size msg-bin))
         (entry (binary
                  (timestamp 64)
                  (level-num 8)
                  (msg-len 16)
                  (msg-bin binary))))
    (case (file:open filename '(append raw binary))
      ((tuple 'ok device)
       (let ((result (file:write device entry)))
         (file:close device)
         result))
      ((tuple 'error reason)
       (tuple 'error reason)))))

(defun level-to-number (level)
  "Converts log level atom to numeric representation."
  (case level
    ('debug 1)
    ('info 2)
    ('warning 3)
    ('error 4)
    ('critical 5)
    (_ 0)))  ; Unknown

(defun number-to-level (num)
  "Converts numeric level to atom."
  (case num
    (1 'debug)
    (2 'info)
    (3 'warning)
    (4 'error)
    (5 'critical)
    (_ 'unknown)))

(defun read-log-file (filename)
  "Reads all entries from a binary log file.
  Returns list of entries or error."
  (case (file:read_file filename)
    ((tuple 'ok binary)
     (case (parse-log-header binary)
       ((tuple 'ok rest)
        (parse-log-entries rest '()))
       ('error
        (tuple 'error 'invalid-header))))
    ((tuple 'error reason)
     (tuple 'error reason))))

(defun parse-log-header (binary)
  "Validates log file header."
  (case binary
    ((binary
       ((magic 4 binary) (version 8) (_flags 8) (_reserved 16)
        (rest binary)))
     (if (and (== magic (file-header-magic))
              (== version (file-version)))
         (tuple 'ok rest)
         'error))
    (_ 'error)))

(defun parse-log-entries
  ;; Base case: no more data
  (('() acc)
   (tuple 'ok (lists:reverse acc)))
  ;; Parse next entry
  ((binary acc)
   (case binary
     ;; Successfully parse an entry
     ((binary (timestamp 64) (level-num 8) (msg-len 16)
              (msg-bin msg-len binary) (rest binary))
      (let* ((level (number-to-level level-num))
             ((tuple msg-str context) (binary-to-term msg-bin))
             (entry (map 'timestamp timestamp
                        'level level
                        'message msg-str
                        'context context)))
        (parse-log-entries rest (cons entry acc))))
     ;; Not enough data for a complete entry
     (_
      (tuple 'ok (lists:reverse acc))))))

Example usage:

lfe> ;; Create a log file
lfe> (binary-logger:create-log-file "ship.log")
ok

lfe> ;; Write some log entries
lfe> (binary-logger:write-log-entry "ship.log" 'info
...>   "Improbability drive activated"
...>   (map 'drive-factor 'infinite 'status 'improbable))
ok

lfe> (binary-logger:write-log-entry "ship.log" 'warning
...>   "Tea dispenser offline"
...>   (map 'reason 'almost-but-not-quite-entirely-unlike-tea))
ok

lfe> (binary-logger:write-log-entry "ship.log" 'critical
...>   "Vogon poetry detected"
...>   (map 'threat-level 'severe 'evasive-action 'recommended))
ok

lfe> ;; Read the entire log
lfe> (binary-logger:read-log-file "ship.log")
#(ok
  (#M(timestamp 1730841234567
      level info
      message "Improbability drive activated"
      context #M(drive-factor infinite status improbable))
   #M(timestamp 1730841234789
      level warning
      message "Tea dispenser offline"
      context #M(reason almost-but-not-quite-entirely-unlike-tea))
   #M(timestamp 1730841235012
      level critical
      message "Vogon poetry detected"
      context #M(threat-level severe evasive-action recommended))))

Why This Approach Wins

1. Compact: Binary encoding is much denser than JSON or XML
2. Fast: No parsing of text formats, just direct binary unpacking
3. Typed: Using term-to-binary preserves exact types
4. Extensible: The context map can contain anything
5. Sequential: Perfect for log files that are written once, read sequentially

This pattern appears in:

• Database transaction logs
• Event sourcing systems
• Network packet captures
• Embedded system logging

Performance Considerations: The Need for Speed

Binary operations in Erlang/LFE are highly optimized, but there are still pitfalls:

Good Practices

;; DO: Build binaries in one go
(defun build-packet-good (header data checksum)
  (binary (header binary) (data binary) (checksum 32)))

;; DON'T: Build binaries piecemeal in a loop
(defun build-packet-bad (parts)
  (lists:foldl
    (lambda (part acc)
      (binary (acc binary) (part binary)))  ; Creates new binary each iteration
    (binary)
    parts))

;; DO: Use binary comprehensions for transformations
(defun double-bytes (bin)
  (binary ((<< (<< (* b 2) 8) >> || << (b 8) >> <= bin)>>>)))

;; DO: Match on the binary once, extract multiple fields
(defun parse-efficient (bin)
  (let (((binary (a 8) (b 16) (c 32) (rest binary)) bin))
    (tuple a b c rest)))

;; DON'T: Pattern match repeatedly on the same binary
(defun parse-inefficient (bin)
  (let (((binary (a 8) (_r1 binary)) bin)
        ((binary (_8) (b 16) (_r2 binary)) bin)
        ((binary (_8) (_16) (c 32) (rest binary)) bin))
    (tuple a b c rest)))

Binary Memory Management

Erlang has two kinds of binaries:

1. Heap binaries (< 64 bytes): Stored on process heap, garbage collected normally
2. Refc binaries (≥ 64 bytes): Stored in separate area, reference counted

Sub-binaries created by pattern matching share data with the original binary, which is efficient but can prevent garbage collection of large binaries if you keep small references. For this reason:

;; If you only need part of a large binary long-term, copy it:
(defun extract-header (large-binary)
  (let (((binary (header 256 binary) (_rest binary)) large-binary))
    ;; Copy header to new binary so large-binary can be GC'd
    (binary:copy header)))

Debugging Binary Code: When Patterns Don’t Match

Binary patterns can fail silently (well, with exceptions) when they don’t match. Helpful debugging:

(defun debug-parse (binary)
  "Wrapper that shows why parsing failed."
  (try
    (parse-packet binary)
    (catch
      ((tuple 'error 'badmatch value 'stack stacktrace)
       (lfe_io:format "Binary match failed!~n")
       (lfe_io:format "Binary size: ~p bytes~n" (list (byte_size binary)))
       (lfe_io:format "First 20 bytes: ~p~n"
                     (list (binary:part binary 0 (min 20 (byte_size binary)))))
       (tuple 'error 'parse-failed)))))

Use io:format("~p~n", [Binary]) liberally during development. The shell shows binaries in a readable format that helps spot patterns.

Summary

Binary manipulation in LFE/Erlang is one of those rare features that’s both powerful and elegant. The bit syntax lets you write code that:

• Directly expresses protocol specifications
• Performs bit-level operations without manual shifting/masking
• Handles non-byte-aligned data naturally
• Compiles to efficient machine code

These examples demonstrate real-world patterns you’ll encounter:

• Protocol parsing (IPv4)
• Custom protocol design (Towel Service)
• Binary file formats (logging)

The techniques here scale from tiny embedded devices to massive distributed systems. They’ve been battle-tested in telecommunications, embedded systems, network infrastructure, and more.

The key insight is that when you need to manipulate binary data, fighting against byte boundaries and bit positions is unnecessary pain. LFE gives you the tools to express what you mean directly, and the compiler optimizes it into fast machine code. This is how programming should work: express your intent clearly, let the computer handle the tedious details.

Now go forth and manipulate binaries with confidence. Just remember to bring your towel.

Conclusion: The Binary Achievement

“For a moment, nothing happened. Then, after a second or so, nothing continued to happen.”

— Douglas Adams

With binary data in LFE, however, quite a lot happens, and it happens with a precision and elegance that would make a Swiss watchmaker weep with professional jealousy.

What We’ve Learned

We began this journey in a state of innocence—or possibly ignorance—regarding the true nature of binary data. We end it having witnessed the full majesty of LFE’s approach to bits and bytes, which can only be described as “what would happen if pattern matching and binary data had a child that was subsequently raised by telecommunications engineers with exacting standards and a distaste for unnecessary complexity.”

Let us review what we’ve discovered, shall we?

The Fundamental Insights

1. Binaries Are First-Class Citizens
2. The Bit Syntax Is Pattern Matching for Bits
3. Construction and Deconstruction Are Symmetric
4. Bitstrings Liberate Us From Byte Boundaries
5. Binary Comprehensions Extend List Comprehensions
6. Bitwise Operators Complement the Bit Syntax
7. Serialization Provides Persistence

The Conceptual Shift

Perhaps the most important lesson is the conceptual shift that LFE’s binary handling represents. In most languages, binary data manipulation is:

1. Imperative: You write code that explicitly manipulates bytes
2. Error-prone: Off-by-one errors, endianness mistakes, sign confusion
3. Obscure: The code doesn’t resemble the data format specification
4. Performance-focused: Often written in low-level languages for speed

In LFE, binary data manipulation is:

1. Declarative: You describe the structure, the system handles the details
2. Correct: Pattern matching fails cleanly, no buffer overruns
3. Readable: Code documents the data format
4. Still Fast: The BEAM VM optimizes these operations extensively

This is the difference between:

// C version - manual bit manipulation
unsigned int value = (buffer[0] << 24) | (buffer[1] << 16) |
                     (buffer[2] << 8) | buffer[3];

And:

;; LFE version - declarative structure description
(binary ((value (size 32) big)) buffer)

Both extract a 32-bit big-endian integer, but only one makes you double-check the bit shift amounts and worry about whether you got the byte order right.

The Performance Reality

A reasonable concern: doesn’t all this abstraction come with performance costs?

The answer is nuanced. The BEAM VM has spent decades optimizing binary operations because Erlang was designed for telecommunications systems handling millions of messages per second. The bit syntax compiles to specialized VM instructions that are highly optimized.

discipline to not micro-optimize code that runs perfectly adequately.

The Philosophical Takeaway

Binary data in LFE represents something deeper than just “a convenient way to handle bytes.” It represents a philosophy of language design where:

1. Domain problems drive language features: Erlang needed to parse protocols, so it got bit syntax
2. Abstractions should clarify, not obscure: The bit syntax makes structure explicit
3. Similar tasks should have similar syntax: Binary comprehensions mirror list comprehensions
4. Safety and expressiveness aren’t opposed: Pattern matching provides both

This is design by people who had real problems to solve and weren’t satisfied with solutions that merely worked—they wanted solutions that worked well and could be understood six months later at 3 AM when production was down and the logs were unhelpful.

Where to Go From Here

Having absorbed these concepts, you’re equipped to:

1. Parse binary protocols: Network protocols, file formats, data streams
2. Implement codecs: Compression, encryption, encoding schemes
3. Interface with hardware: Bit-level register manipulation, device protocols
4. Build distributed systems: Using serialization for persistence and communication
5. Optimize data structures: Bit packing, custom binary formats

A Final Note

In the grand taxonomy of programming language features, binary handling is rarely exciting. It’s not as philosophically interesting as type systems, not as practically dramatic as garbage collection, and not as immediately gratifying as first-class functions. It’s the plumbing of the programming world—essential, but not glamorous.

Yet LFE’s approach to binary data is worth studying precisely because it takes something mundane and makes it elegant. It takes a domain where most languages say “here’s malloc(), good luck” and provides sophisticated pattern matching, comprehensions, and a syntax that reads like the specifications you’re implementing.

This is what good language design looks like: taking hard problems and making them approachable without sacrificing power or performance. It’s the difference between a language that lets you solve problems and a language that helps you solve problems. The first provides tools. The second provides tools and shows you how to use them effectively.

Tuples: The Universe’s Gift to Structured Chaos

In Which We Discover That Order Matters, But Only Sometimes

Tuples, dear reader, are what happens when the universe decides that complete anarchy isn’t quite working out and perhaps a bit of light organization might be in order—but not too much, mind you. We wouldn’t want to get carried away with all that structure business.

A tuple is, in essence, a fixed-size container for holding a precise number of things that have absolutely no obligation to be related to each other in any meaningful way whatsoever. It’s rather like a very small filing cabinet where one drawer contains your tax returns, another contains a half-eaten sandwich, and a third contains the complete works of Shakespeare rendered in interpretive dance notation. The important bit is that you know exactly how many drawers you have, even if you haven’t the foggiest idea what’s actually in them.

The Anatomy of a Tuple

In LFE, tuples wear curly braces with the kind of casual confidence usually reserved for beings who have transcended the need for matching socks. They look like this:

#(42 zaphod "Don't Panic")

This particular specimen contains three elements: an integer who thinks it knows the answer to everything, an atom who’s probably up to something, and a string offering questionable advice. Note the # prefix, which is LFE’s way of announcing “stand back, we’re doing science.”

The empty tuple #() also exists, though its existential purpose remains a subject of heated philosophical debate. It contains nothing, represents nothing, and yet somehow manages to be something. Rather like a black hole, but with better documentation.

Tagged Tuples: A Convention Posing as Wisdom

Here’s where things get interesting in that particularly Erlangy way that makes you wonder if the original implementers were secretly running a grand social experiment on future programmers.

Consider this: you have a tuple #(182 "joe"). Is it a person’s height and name? Their weight in kg and favorite cartoon character? The number of pages in a book about someone called Joe? The universe, in its infinite inscrutability, refuses to tell you.

The solution, arrived at through centuries of trial, error, and probably some amount of head-scratching, is to stick an atom at the front as a kind of helpful label:

#(person "joe" 182)

Ah! Now it’s obviously a person, followed by their name, followed by… well, some number. Is it their height? Age? The number of times they’ve watched reruns of Red Dwarf? This is where convention enters, stage left, wearing a fake mustache.

The generally accepted approach (and by “generally accepted” we mean “people will look at you funny if you don’t do it this way”) is to make your tuples even more explicit:

#(person #(name "joe") #(height 182) #(shoe-size 42) #(eye-color 'brown))

Notice how we’ve nested tuples like Russian dolls, except these dolls contain data instead of slightly smaller dolls. Each nested tuple starts with an identifying atom, making it blatantly obvious what each field represents. This is called “good style” in some circles and “bleeding obvious common sense” in others.

Creating Tuples: An Exercise in Spontaneous Existence

Unlike certain philosophical positions that require extensive meditation to manifest reality, tuples simply are the moment you declare them. There’s no make-tuple ceremony, no tuple-please-sir-may-I-have-one function call. You simply write:

(set point #(10 45))

And the tuple springs into being with all the enthusiasm of a quantum particle that’s just been observed. The variable point now refers to this tuple, and somewhere in the bowels of the BEAM virtual machine, memory has been allocated, electrons have been shuffled about, and the universe has been ever-so-slightly rearranged to accommodate your data structure.

When tuples become obsolete—when no variable points to them anymore, when they’ve been forgotten by the very code that birthed them—the garbage collector sweeps in like a cosmic janitor and reclaims their memory. You need not worry about this. The garbage collector is surprisingly good at its job, having spent decades perfecting the art of cleaning up after programmers.

Extracting Values: Pattern Matching to the Rescue

Now we arrive at one of Erlang’s (and by extension, LFE’s) most celebrated features, the one that makes functional programmers go all misty-eyed and C programmers reach nervously for their pointer arithmetic: pattern matching.

Suppose you have a tuple and you want to know what’s in it:

(set point #(point 10 45))

In lesser languages, you might extract the X coordinate with something tedious like get-tuple-element-2 or point.x (oh, the mundanity!). But we are not using lesser languages. We are using LFE, where pattern matching reigns supreme:

(let ((#(point x y) point))
  (list x y))

This remarkable bit of syntactic sorcery does several things simultaneously:

1. It checks that point is indeed a three-element tuple
2. It verifies that the first element is the atom point
3. It binds the second element to the variable x
4. It binds the third element to the variable y
5. It evaluates the body of the let, where we can now use x and y with reckless abandon

If the pattern doesn’t match—if, say, our tuple was #(rectangle 10 45) instead—the system will throw an exception with all the diplomatic grace of a bouncer ejecting a philosophy student from a nightclub.

The Anonymous Variable: Underscore of Mystery

Sometimes you don’t care about certain parts of a tuple. Perhaps you only need the X coordinate from a point and the Y can go hang. For such occasions, LFE provides the anonymous variable, written as _ (that’s an underscore, not a very short horizontal line):

(let ((#(point x _) point))
  x)

The underscore is special. Unlike regular variables, which get rather upset if you try to bind them to two different values simultaneously, the anonymous variable accepts whatever you throw at it with the serene indifference of a cat watching television. You can use _ multiple times in the same pattern, and each occurrence can match a different value. It’s the perfect placeholder for data you don’t care about—the “etc.” of the pattern matching world.

Tuple BIFs: The Standard Library Weighs In

For those moments when pattern matching seems like too much work (a rare occurrence, to be sure), Erlang provides several Built-In Functions for tuple manipulation. In LFE, you call these using the module prefix:

;; Get the size of a tuple
(tuple_size #(abc #(def 123) ghi))
;; Returns: 3

;; Extract an element by position (1-indexed, because apparently we're barbarians)
(element 2 #(abc #(def 123) ghi))
;; Returns: #(def 123)

;; Create a new tuple with a replaced element
(setelement 2 #(abc #(def 123) ghi) 'def)
;; Returns: #(abc def ghi)

Notice that element uses 1-based indexing, a decision that historians trace back to the original Erlang implementers’ deep-seated distrust of the number zero. Or possibly it was just Tuesday and they were feeling contrary. History is unclear on this point.

Also note that setelement doesn’t actually set anything—tuples are immutable, after all. Instead, it creates an entirely new tuple with your modification, leaving the original tuple to carry on with its life unbothered. This is functional programming: nothing ever changes, yet somehow everything is different.

Comparing Tuples: In Which Order Matters

Tuples can be compared using the standard comparison operators, and they obey a specific set of rules that would make a librarian weep with joy:

(< #(1 2) #(1 3))     ;; => true (element 2 differs)
(< #(1 2) #(2 1))     ;; => true (element 1 differs)
(=:= #(a b) #(a b))   ;; => true (identical)
(=:= #(1 2) #(1 2 3)) ;; => false (different sizes)

Tuples are compared element by element, from left to right. The first differing element determines the result. If all elements are equal but the tuples are different lengths, the shorter tuple is considered “less than” the longer one, in accordance with the ancient wisdom that suggests brevity is somehow related to smallness.

Nested Tuples: Turtles All the Way Down

Tuples can contain other tuples, which can contain still more tuples, ad infinitum (or at least until your computer’s memory gives up in despair). This allows you to create data structures of Byzantine complexity:

(set person
  #(person
    #(name #(first "Zaphod") #(last "Beeblebrox"))
    #(heads 2)
    #(occupation 'president)
    #(status 'froody)))

To extract deeply nested values, you can use nested pattern matching:

(let ((#(person #(name #(first fname) _) _ _ _) person))
  fname)
;; Returns: "Zaphod"

Those who have studied ancient scrolls will recognize this as remarkably similar to destructuring in other languages, except with more parentheses and a stronger emphasis on pattern correctness.

Tuples vs. Lists: A Brief Philosophical Digression

You might wonder, with the innocent optimism of someone who hasn’t yet been burned by premature optimization, why we need tuples when we have lists. This is an excellent question that deserves a nuanced answer.

Tuples are for when you know exactly how many things you have. They’re fixed-size, stored contiguously in memory, and accessing any element takes constant time. Think of them as arrays that have given up on the whole “mutable” thing and embraced a life of peaceful immutability.

Lists are for when you don’t know how many things you have, or when you might want to add more. They’re variable-size, stored as linked chains of cons cells (more on that later, dear reader), and accessing the Nth element requires traversing N-1 elements first.

Use tuples for records, small fixed collections, and return values from functions. Use lists for sequences, collections of arbitrary size, and whenever you feel like recursing through your data like a functional programming warrior of old.

Pattern Matching Failures: When Things Go Splendidly Wrong

If you attempt to pattern match a tuple and the pattern doesn’t fit—wrong size, wrong values, wrong phase of the moon—LFE will raise an exception:

(let ((#(point x x) #(point 10 45)))  ; x can't be both 10 and 45
  'this-will-never-execute)
;; exception error: no match of right hand side value #(point 10 45)

This is actually a feature, not a bug. It’s how you ensure that your functions receive the data they expect. If someone passes a rectangle to a function expecting a circle, you want to know about it immediately, not three hours later when the calculations have gone mysteriously wrong and you’re debugging at 3 AM with nothing but cold coffee and regret for company.

Practical Examples: Tuples in the Wild

Let’s conclude with some real-world examples of tuple usage, because theory is all well and good until you actually need to store a person’s favorite color.

Example 1: HTTP Responses

;; Representing HTTP responses as tuples
(defun handle-request (url)
  (case (fetch-url url)
    (#(ok body)
     (process-body body))
    (#(error 'not-found)
     (show-404-page))
    (#(error 'timeout)
     (try-again-later))
    (_ (show-generic-error))))

Example 2: Coordinate Systems

;; 2D points
(set origin #(point 0 0))
(set corner #(point 100 100))

;; 3D points with tagged fields
(set spaceship-location
  #(position
    #(x 42.5)
    #(y -15.8)
    #(z 1337.0)
    #(velocity #(30 0 -5))))

Example 3: Return Values

(defun divide (a b)
  (if (=:= b 0)
    #(error 'division-by-zero)
    #(ok (/ a b))))

;; Usage:
(case (divide 10 2)
  (#(ok result)
   (io:format "Result: ~p~n" (list result)))
  (#(error reason)
   (io:format "Error: ~p~n" (list reason))))

Summary: Tuples in Brief

Tuples are:

• Fixed-size collections of elements
• Written with #(...) in LFE
• Often tagged with an atom in the first position
• Immutable (like everything else in LFE)
• Excellent for pattern matching
• Compared element-by-element
• Stored efficiently in contiguous memory
• The go-to choice for fixed-size data structures

They are not:

• Variable-size (that’s what lists are for)
• Mutable (did we mention immutability yet?)
• Named (that would be records, which are basically tuples in a fancy hat)
• Particularly good at explaining themselves without tags

And there you have it: tuples in all their curly-braced glory. Use them wisely, use them well, and for goodness’ sake, remember to tag them properly. Your future self will thank you, probably while muttering about how past-you was surprisingly competent for once.

Now then, shall we move on to lists? They’re like tuples, except completely different and infinitely more recursive.

Proplists: The Elegant Simplicity of Named Chaos

In Which We Discover That Sometimes the Old Ways Are Best (Or At Least Good Enough)

Before maps arrived with their sophisticated ordering semantics and their first-class citizenship, before records donned their tuple disguises and fooled precisely nobody, there existed proplists—property lists—which are to data structures what duct tape is to spacecraft repair: inelegant, occasionally questionable, but remarkably effective when you need something that simply works.

A proplist is, at its heart, a deeply philosophical statement about pragmatism. It’s an ordinary list—nothing fancy, no special VM support, no compiler magic—containing entries that are either tuples (whose first elements serve as keys) or atoms (which work as shorthand for {Atom, true}). Other terms are allowed to lurk in these lists like uninvited guests at a garden party, but the proplists module regards them with the dignified indifference of a butler who has seen everything and judges nothing.

The beauty of proplists lies not in their sophistication but in their simplicity. They are the linguistic equivalent of pointing at things and saying “that one.” Sometimes that’s all you need.

The Fundamental Nature of Proplists

Proplists possess several characteristics that have kept them relevant despite the arrival of more sophisticated alternatives:

• Order matters: The first occurrence of a key takes precedence over later ones, which means you can override defaults simply by consing new options onto the front. This is democracy in action, except with a first-past-the-post system that would make electoral reformers weep.

• Atoms as boolean shorthand: An atom foo in a proplist is semantically equivalent to {foo, true}, which saves typing and looks cleaner when you have many boolean flags. It’s the data structure equivalent of saying “yes” instead of “the answer to your question is affirmative.”

• Inheritance patterns: Proplists excel at representing inherited properties—options passed to functions where user preferences override system defaults, object properties, annotations, and the sort of hierarchical configuration that makes software configurable rather than merely programmable.

• Literal equality: Two keys are considered equal if they match literally (=:=). This means 1 and 1.0 are different keys, which could be either a feature or a trap, depending on how caffeinated you were when you wrote the code.

Creating Proplists

Creating a proplist requires no special syntax, no macro invocations, no appeals to higher powers. You simply construct a list with the appropriate structure:

lfe> '((active true) (buffer_size 1024) read write binary)
((active true) (buffer_size 1024) read write binary)

lfe> (list '(timeout 5000) 'async '(encoding utf8))
((timeout 5000) async (encoding utf8))

lfe> (cons '(priority high) '((retries 3) verbose))
((priority high) (retries 3) verbose)

Notice how boolean options can be expressed as bare atoms (read, write, binary), which will be automatically treated as {read, true}, {write, true}, and {binary, true} when accessed through the proplists module. This is convenience masquerading as syntax.

Options passed to Erlang’s standard library functions demonstrate the proplist philosophy in action:

;; File operations
(file:open "towel.txt" '(read write append (encoding utf8)))

;; Network options
(inet:setopts socket '((active true) binary (packet 4)))

The mixing of tuple pairs and bare atoms creates a pleasant visual heterogeneity—the data structure equivalent of a well-curated bookshelf where some books stand upright while others lie flat.

Accessing Values: The Proplists Module

The proplists module provides a comprehensive suite of functions for working with property lists. Each function approaches the fundamental problem of “find me this thing in this list” with slightly different assumptions about what you want and what should happen if the thing isn’t there.

proplists:get_value/2,3

The workhorse function, get_value/2, retrieves the value associated with a key, or returns undefined if the key doesn’t exist:

lfe> (proplists:get_value 'timeout '((timeout 5000) (retry 3)))
5000

lfe> (proplists:get_value 'nonexistent '((timeout 5000) (retry 3)))
undefined

The three-arity version allows you to specify a default value, which is useful when undefined is either too pessimistic or not sufficiently specific:

lfe> (proplists:get_value 'timeout '((retry 3)) 1000)
1000

lfe> (proplists:get_value 'verbose '((silent true)) 'false)
false

For atoms serving as boolean flags, the value retrieved is simply true:

lfe> (proplists:get_value 'binary '((active true) binary (packet 4)))
true

This is lookup/2 doing its tuple-expansion magic behind the scenes, treating the atom as if it had been {binary, true} all along.

proplists:get_bool/2

When you know you’re dealing with boolean options and want a boolean answer without philosophical complications:

lfe> (proplists:get_bool 'verbose '((verbose true) (timeout 5000)))
true

lfe> (proplists:get_bool 'verbose '((timeout 5000)))
false

lfe> (proplists:get_bool 'verbose '((verbose false)))
false

This function returns true only if lookup/2 would yield {Key, true}. Everything else—absence, explicit false, the key existing with a different value—collapses to false. It’s boolean logic for a universe that doesn’t believe in maybe.

proplists:get_all_values/2

For keys that appear multiple times (because sometimes repetition is emphasis, not error):

lfe> (proplists:get_all_values 'header 
      '((method post) (header "Content-Type: text/plain") 
        (header "Authorization: Bearer token") (timeout 5000)))
("Content-Type: text/plain" "Authorization: Bearer token")

lfe> (proplists:get_all_values 'missing '((a 1) (b 2)))
()

This returns a list of all values associated with the specified key, preserving order. If no entries exist, you get the empty list, which is the Erlang way of saying “nothing to see here, move along.”

proplists:lookup/2

The most semantically honest function in the module, lookup/2 returns exactly what it finds:

lfe> (proplists:lookup 'timeout '((timeout 5000) (retry 3)))
(timeout 5000)

lfe> (proplists:lookup 'binary '((active true) binary))
(binary true)

lfe> (proplists:lookup 'nonexistent '((a 1) (b 2)))
none

For atoms in the list, it synthesizes the tuple {Atom, true}. If nothing is found, it returns the atom none, which is different from undefined in ways that matter only when you’re debugging at 3 AM.

proplists:lookup_all/2

Like get_all_values/2, but returns tuples instead of values:

lfe> (proplists:lookup_all 'cookie 
      '((cookie "session=abc123") (timeout 1000) (cookie "user=arthur")))
((cookie "session=abc123") (cookie "user=arthur"))

lfe> (proplists:lookup_all 'missing '((a 1) (b 2)))
()

Useful when you need both the keys and the values, perhaps because you’ve forgotten which one you’re working with and would like to see the whole thing laid out properly.

Membership and Keys

proplists:is_defined/2

A simple existential check that returns a boolean without drama:

lfe> (proplists:is_defined 'timeout '((timeout 5000) (retry 3)))
true

lfe> (proplists:is_defined 'verbose '((timeout 5000) (retry 3)))
false

lfe> (proplists:is_defined 'binary '(read write binary))
true

This answers the question “does this key exist?” which is simpler than “what is this key’s value?” and thus requires less cosmic contemplation.

proplists:get_keys/1

Returns an unordered list of all keys used in the proplist, without duplicates:

lfe> (proplists:get_keys '((a 1) (b 2) (c 3) (a 99)))
(c b a)

lfe> (proplists:get_keys '(read write (mode binary) read))
(mode write read)

The keys are returned in whatever order the implementation finds convenient, which is to say, don’t count on any particular ordering. If you need ordering, you’ve wandered into map territory.

Modification Operations

proplists:delete/2

Removes all entries associated with a key, because sometimes you need to thoroughly erase evidence:

lfe> (proplists:delete 'timeout '((timeout 5000) (retry 3) (timeout 1000)))
((retry 3))

lfe> (proplists:delete 'verbose '((timeout 5000) (retry 3)))
((timeout 5000) (retry 3))

Note that this removes all occurrences. If you had multiple entries for the same key (perhaps for excellent reasons understood only by past-you), they all vanish with equal efficiency.

Transformation Operations

proplists:append_values/2

A curious function that wraps each value in a list unless it’s already a list, then concatenates everything:

lfe> (proplists:append_values 'include 
      '((include (one two)) (other 42) (include three) (include (four))))
(one two three four)

This is often useful for “incremental” options where you want to accumulate values across multiple sources. It’s the functional programming equivalent of pushing items onto a stack, except the stack is inside-out and upside-down.

proplists:compact/1

Minimizes the representation by converting everything to normal form:

lfe> (proplists:compact '((a true) (b false) (c 42) d (e true)))
(a (b false) (c 42) d e)

This is equivalent to mapping property/1 over the list, which means {Key, true} becomes just Key while everything else stays as it is. The result is aesthetically pleasing in the way that a freshly organized desk is aesthetically pleasing—you’re not entirely sure what changed, but it looks better.

proplists:unfold/1

The inverse of compact/1, expanding all atoms to explicit tuples:

lfe> (proplists:unfold '(read write (mode binary)))
((read true) (write true) (mode binary))

This makes the implicit explicit, which is sometimes necessary when interfacing with systems that lack the sophistication to understand shorthand.

Substitution and Expansion

proplists:substitute_aliases/2

Allows you to rename keys systematically, useful when your API needs to support both British and American spellings:

lfe> (proplists:substitute_aliases 
      '((color colour) (center centre))
      '((color red) (size large) (center middle)))
((colour red) (size large) (centre middle))

The aliases list consists of {OldKey, NewKey} pairs. If the same old key appears multiple times, only the first alias is used—first come, first served, as civilized society demands.

proplists:substitute_negations/2

A delightfully specific function that both renames keys and negates boolean values:

lfe> (proplists:substitute_negations
      '((no_color color) (disable_cache cache))
      '((no_color true) (timeout 5000) (disable_cache false)))
((color false) (timeout 5000) cache)

This handles the common pattern where options can be expressed in negative form (no_foo, disable_bar) and need to be normalized to positive form with inverted values. Note that {disable_cache, false} becomes just cache, because the double negative resolves to an affirmative shorthand.

proplists:expand/2

Expands property macros into their full forms, allowing you to define shorthand notations:

lfe> (proplists:expand
      '((debug ((verbose true) (log_level debug) (trace true))))
      '((timeout 5000) debug (retry 3)))
((timeout 5000) (verbose true) (log_level debug) (trace true) (retry 3))

The expansion list contains {Property, Expansion} pairs. When a property matches, it’s replaced by the expansion terms. If you want the original property preserved, include it in the expansion. Expansions are not recursive—once expanded, the terms are considered done.

proplists:normalize/2

Passes a proplist through a pipeline of transformations, allowing you to build complex normalization strategies:

lfe> (proplists:normalize
      '((timeout 5000) no_verbose (colour red))
      '((negations ((no_verbose verbose)))
        (aliases ((colour color)))
        (expand ((debug ((verbose true) (log_level debug)))))))
((timeout 5000) (verbose false) (color red))

The stages are processed in order, and the result is automatically compacted. Typical usage involves:

1. Negations first: Convert no_foo to {foo, false}
2. Aliases second: Normalize variant spellings
3. Expansions last: Expand shorthand to full options

This is the functional equivalent of data cleaning, where messy user input becomes pristine internal representation.

Partitioning and Splitting

proplists:split/2

Partitions a proplist into sublists based on specified keys:

lfe> (proplists:split 
      '((c 2) (e 1) a (c 3 4) d (b 5) b)
      '(a b c))
(((a) ((b 5) b) ((c 2) (c 3 4))) ((e 1) d))

The result is a tuple {Lists, Rest} where Lists contains one sublist for each key (in the order specified), and Rest contains everything else. Relative order is preserved within each sublist, which means this operation is stable and predictable—rare qualities in a chaotic universe.

Conversion Operations

proplists:from_map/1

Converts a map to a proplist, because sometimes you need to interface with legacy code that hasn’t discovered the joys of first-class key-value structures:

lfe> (proplists:from_map (map 'timeout 5000 'retry 3 'verbose 'true))
((timeout 5000) (retry 3) (verbose true))

The resulting proplist preserves the map’s key-value associations but loses the map’s ordering guarantees. If a value in the map is the atom true and the key is also an atom, you might expect compaction to {Key, true} → Key, but that’s not guaranteed—use compact/1 if you want that.

proplists:to_map/1,2

Converts a proplist to a map, normalizing optionally along the way:

lfe> (proplists:to_map '(a (b 1) (c 2) (c 3)))
#M(a true b 1 c 2)

lfe> (proplists:to_map 
      '((no_verbose true) (colour red) (timeout 5000))
      '((negations ((no_verbose verbose)))
        (aliases ((colour color)))))
#M(verbose false color red timeout 5000)

Shorthand atoms are expanded to Atom => true. If the same key appears multiple times, the first occurrence wins—the value nearest to the head of the list, exactly as get_value/2 would return. The two-arity version applies normalization stages before conversion, allowing you to clean up user input and convert to maps in one operation.

Pattern Matching with Proplists

Because proplists are just lists, you can pattern match on them directly, though the experience is less elegant than with maps:

lfe> (set `(,first . ,rest) '((timeout 5000) (retry 3)))
((timeout 5000) (retry 3))
lfe> first
(timeout 5000)

lfe> (set `((,key ,value) . ,_) '((active true) (buffer 1024)))
((active true) (buffer 1024))
lfe> key
active
lfe> value
true

In practice, you’re more likely to use the proplists module functions rather than direct pattern matching, because proplists lack the structural guarantees that make pattern matching pleasant. They’re fundamentally lists with conventions, not types with contracts.

Proplists vs. Maps: A Philosophical Comparison

The arrival of maps didn’t obsolete proplists, though it did raise uncomfortable questions about their continued existence. Maps offer:

• Native syntax: #M(key value) vs. '((key value))
• Efficient access: O(log n) vs. O(n) for get_value/2
• Pattern matching: Clean destructuring in function heads
• Type safety: The := operator catches typos

Yet proplists retain certain advantages:

• Simplicity: They’re just lists. No special syntax, no compiler involvement.
• Legacy compatibility: Decades of Erlang code uses proplists for options.
• Literal options: [read, write, append] looks cleaner than #{read => true, write => true, append => true}
• Order-based overriding: Cons new options onto the front to override defaults

The general wisdom, as articulated by the ancients (and by the Learn You Some Erlang documentation), is this:

• Use maps for options that are mostly pairs, especially in new code. The maps:merge/2 pattern for defaults is elegant.
• Use proplists for compatibility, when working with existing APIs, or when literal atoms dominate the option list.
• Use records for known, fixed structures where performance matters.

As one sage source notes: “Option lists that are mostly pairs will be replaced by maps. On the other hand, literal options such as read, write, or append will remain much nicer with proplists.”

The Default Values Pattern

One of the most elegant uses of proplists is the default-override pattern:

(defun start-server (user-opts)
  (let ((defaults '((port 8080) 
                    (host "localhost") 
                    (timeout 30000) 
                    (max-connections 100))))
    (let ((opts (++ user-opts defaults)))
      ;; Use opts, knowing all required keys exist
      ;; User options override defaults due to left-to-right precedence
      (listen-on (proplists:get_value 'host opts)
                 (proplists:get_value 'port opts)))))

Because get_value/2 returns the first matching value, prepending user options to defaults creates an elegant override mechanism. Compare this to the map equivalent:

(defun start-server (user-opts)
  (let ((defaults (map 'port 8080 
                       'host "localhost" 
                       'timeout 30000 
                       'max-connections 100)))
    (let ((opts (maps:merge defaults user-opts)))
      ;; Use opts
      (listen-on (mref opts 'host)
                 (mref opts 'port)))))

Both patterns work well. The map version has cleaner access syntax; the proplist version has been doing this job for longer than some programmers have been alive.

Common Patterns and Idioms

Building Options Incrementally

(defun add-auth-headers (opts token)
  (cons `(header ,(++ "Authorization: Bearer " token))
        opts))

(defun add-timeout (opts ms)
  (cons `(timeout ,ms) opts))

;; Usage:
(-> '((method post))
    (add-timeout 5000)
    (add-auth-headers "abc123"))
;; ((header "Authorization: Bearer abc123") (timeout 5000) (method post))

Extracting Multiple Values

(defun get-credentials (opts)
  (let ((username (proplists:get_value 'username opts))
        (password (proplists:get_value 'password opts)))
    (tuple username password)))

Validating Required Options

(defun validate-opts (opts required-keys)
  (lists:all 
    (lambda (key) (proplists:is_defined key opts))
    required-keys))

;; Usage:
(validate-opts '((host "localhost") (port 8080)) 
               '(host port))
;; true

(validate-opts '((host "localhost")) 
               '(host port))
;; false

Converting Between Formats

;; Proplist to keyword list (for external APIs)
(defun to-keyword-list (proplist)
  (lists:flatmap
    (lambda (entry)
      (case entry
        ((tuple k v) (list (tuple ':key (atom_to_binary k 'utf8))
                           (tuple ':value v)))
        (atom (list (tuple ':key (atom_to_binary atom 'utf8))
                    (tuple ':value 'true)))))
    proplist))

Performance Considerations

Proplists are lists, which means their performance characteristics are list characteristics:

• Lookup: O(n) — must traverse from the head until finding the key
• Multiple lookups: O(n*m) where m is the number of lookups
• Deletion: O(n) — must traverse the entire list
• Insertion: O(1) at the head (via cons), but this doesn’t help if you need to update a value

For small option lists (fewer than ~20 entries), this doesn’t matter. For larger collections or when lookup performance is critical, use maps or consider:

• Converting to a map once and using maps functions thereafter
• Using proplists:to_map/1 at the boundary between external and internal representations
• Caching frequently accessed values rather than repeatedly calling get_value/2

The rule of thumb: if you’re calling get_value/2 in a loop or frequently within a hot path, you’re probably using the wrong data structure.

Proplists in the Wild

File Operations

;; Open a file with options
(file:open "guide.txt" '(read binary (read_ahead 4096)))

;; Write with encoding
(file:write_file "data.txt" content '((encoding utf8)))

Network Options

;; Socket options
(inet:setopts socket '((active true) 
                       binary 
                       (packet 4) 
                       (nodelay true)))

;; HTTP client options
(httpc:request 'get 
               (tuple url '()) 
               '((timeout 5000) (connect_timeout 2000)) 
               '())

Process Options

;; Spawn with options
(spawn_opt fun '(link monitor (priority high)))

;; System info with options
(erlang:system_info '(version wordsize))

The Erlang standard library uses proplists extensively for optional arguments, a pattern that has proven robust enough to survive multiple decades and the introduction of competing data structures. This is either a testament to their utility or an example of path dependence, depending on one’s philosophical stance toward legacy code.

When Not to Use Proplists

Proplists are not appropriate for:

• Large collections: Anything over ~50 entries should probably be a map
• Frequent updates: Use maps or records
• Schema-validated data: Use records with well-defined fields
• High-performance lookups: Use maps, ETS tables, or process dictionaries (though the last should be used with the sort of caution normally reserved for handling radioactive materials)
• Data with complex types as keys: Use maps, which support any fully ground term as a key

The JSON Connection

Unlike maps, which map naturally to JSON objects, proplists require translation. The common pattern is:

;; Proplist to JSON (via map)
(defun proplist-to-json (plist)
  (jiffy:encode 
    (proplists:to_map plist)))

;; JSON to proplist (via map)
(defun json-to-proplist (json)
  (proplists:from_map 
    (jiffy:decode json '(return_maps))))

This works, but it’s not seamless. If you’re building web APIs or working with JSON extensively, consider using maps directly rather than converting back and forth.

Proplists in LFE: Syntactic Considerations

LFE’s syntax makes proplists slightly more pleasant than in Erlang:

;; Erlang
[{timeout, 5000}, {retry, 3}, binary]

;; LFE with quoting
'((timeout 5000) (retry 3) binary)

;; LFE with explicit list construction
(list '(timeout 5000) '(retry 3) 'binary)

;; LFE with backquote for computed values
`((timeout ,ms) (retry ,attempts) binary)

The backquote syntax is particularly useful when building options programmatically:

(defun make-opts (timeout retry verbose?)
  `((timeout ,timeout)
    (retry ,retry)
    ,@(if verbose? '(verbose) '())))

The ,@ (unquote-splicing) operator allows conditional inclusion of options without nested conditionals or list concatenation.

Guidelines for Proplist Usage

Based on collective wisdom accumulated over decades of Erlang development:

1. Use proplists for function options unless the options are numerous (>20) or performance-critical
2. Always provide default values using either get_value/3 or the append-defaults pattern
3. Document expected options clearly, especially boolean shorthands
4. Consider maps for new APIs but don’t feel compelled to break compatibility
5. Use normalization when accepting user input that might use variant keys
6. Validate required options early in your function, not deep in the logic
7. Don’t nest proplists deeply—if you find yourself doing get_value on a get_value result, you probably want records or maps
8. Remember that atoms as booleans must be lowercase—'True and 'true are different atoms

The Proplist Testing Pattern

When writing tests for functions that take options, the proplist pattern enables easy test case generation:

(defun test-connect-variants ()
  (let ((base-opts '((host "localhost") (port 8080))))
    (lists:foreach
      (lambda (extra-opts)
        (let ((opts (++ extra-opts base-opts)))
          (test-connect opts)))
      '(((timeout 1000))
        ((timeout 5000) (retry 3))
        ((ssl true) (certfile "/path/to/cert"))
        (verbose)))))

This allows testing multiple option combinations by composing base options with variants—difficult to do cleanly with records, natural with proplists.

In Conclusion: The Utility of Simplicity

Proplists represent a design philosophy that values pragmatism over perfection. They’re not the most efficient data structure. They’re not the most elegant. They don’t have first-class syntax or compiler support. But they work, they’re understood by everyone who’s written Erlang for more than a week, and they solve the “how do I pass optional arguments” problem with admirable directness.

Are they the answer to life, the universe, and everything? No, that’s still 42 (or #M(answer 42) if you’re keeping up with modern times). But they are the answer to “how should I pass these dozen optional parameters to my function without creating a seven-element tuple that nobody can remember the order of?”

And sometimes, that’s answer enough.

Use proplists when simplicity trumps sophistication, when compatibility matters more than performance, when you’re writing options that will be read once at startup rather than accessed in tight loops. Use them with understanding of their limitations, but without apology for their continued relevance.

They may be old, but they’re not obsolete. They’re just lists, after all. Very specialized, conventionally structured, surprisingly useful lists. But lists nonetheless.

The proplist is dead. Long live the proplist.

Maps: The Associative Enlightenment

In Which We Discover That Keys Unlock More Than Doors

Maps arrived in Erlang R17 with the quiet confidence of someone who knows they’re about to revolutionize everything but is too polite to mention it. Think of them as the sophisticated cousin of records and proplists—the one who studied abroad, speaks three languages, and somehow makes ordering at restaurants look effortless.

Unlike records, which are really just tuples wearing a very convincing disguise (the compiler sees right through it, naturally), maps are actual first-class citizens of the Erlang VM. They’re associative collections of key-value pairs that would make any self-respecting dictionary jealous, if dictionaries were capable of jealousy, which in Erlang they decidedly are not, being mere modules and all.

The Fundamental Nature of Maps (Or: What Maps Actually Are When Nobody’s Looking)

Maps possess several properties that distinguish them from the lesser data structures wandering about the BEAM:

• Keys can be any fully ground term. Not mostly ground. Not nearly ground. Not “I’m working on it” ground. Fully ground—meaning no unbound variables lurking in the shadows like existential uncertainties.

• Elements are ordered by their keys, which means maps have opinions about sequence, much like a particularly fastidious librarian.

• Updating a map without changing its keys is space-efficient, a bit like renovating your house without moving to a new address.

• Looking up values is efficient, which is computer science speak for “pleasantly quick without making a fuss about it.”

• Maps have well-defined ordering semantics, meaning you can compare them without philosophical angst.

Creating Maps: The Art of Literal Expression

In LFE, as in life, there are multiple ways to express the same fundamental truth. For maps, the syntax bears a striking resemblance to records, save for the notable absence of a record name and the presence of the distinctly map-like operators => and :=.

;; A simple map, as maps go
(set m1 (map 'a 1 'b 2))
#M(a 1 b 2)

;; Or, if you prefer the Erlang-esque syntax
(set m2 '#M(a 1 b 2))
#M(a 1 b 2)

;; Maps with non-atomic keys, because why limit ourselves?
(set facts '#M((wife fred) "Sue"
               (age fred) 45
               (daughter fred) "Mary"
               (likes jim) ("cooking" "swimming" "reading")))

Note the delightful #M syntax, which is LFE’s way of saying “behold, a map!” without resorting to semaphore flags. The map function provides an alternative construction method for those who prefer their parentheses in different arrangements.

The Two Operators: A Tale of Intention

Maps provide two operators for updates, each with its own philosophical stance on existence:

The => Operator: The Optimist

This operator believes in second chances and new beginnings. It will either:

• Update an existing key with a new value, or
• Add an entirely new key-value pair if the key doesn’t exist

It always succeeds, much like an enthusiastic golden retriever.

(set m3 (mupd m1 'c 'xx))
#M(a 1 b 2 c xx)

The := Operator: The Perfectionist

This operator has standards. It will:

• Update an existing key with a new value, but
• Raise an exception if you try to update a non-existent key

This is precisely the behavior you want when typos could ruin your afternoon.

;; This works because 'a exists
(set m4 (mupd m1 'a 42))
#M(a 42 b 2)

;; This explodes with theatrical precision
(set m5 (mupd m1 'c 3))
;; ** exception error: bad argument
;;    key c does not exist in old map

The wisdom here is to use => when establishing a key for the first time, and := for all subsequent updates. This gives you both flexibility and type safety, which is roughly equivalent to having your cake and eating it too, except without the calories or the cake.

Pattern Matching: The Art of Selective Attention

Maps can be destructured in patterns, allowing you to extract precisely the information you need while ignoring the rest with dignified indifference:

(defun get-age (person)
  (let (((map 'age age) person))
    age))

;; Or using the match-map form
(defun get-age (person)
  (match-map person
    ((map 'age a) a)))

;; Example usage
(set henry8 '#M(class king born 1491 died 1547))
(get-age henry8)
;; 1491

The number of keys in your pattern need not match the number in the map. Maps are generous that way—they don’t insist you acknowledge every detail of their existence.

Function Clauses: Maps as Dispatch Mechanism

Maps in function heads provide a delightfully expressive way to pattern match on specific key-value combinations:

(defun count-characters (str)
  (count-characters str (map)))

(defun count-characters
  (((cons h t) (= (map h n) m))
   (count-characters t (mupd m h (+ n 1))))
  (((cons h t) m)
   (count-characters t (mupd m h 1)))
  (('() m)
   m))

This function counts character occurrences, building a map incrementally. The first clause fires when the character already exists (using := implicitly), the second when it doesn’t, and the third when we’ve run out of characters to count and might as well return what we’ve got.

The Maps Module: A Compendium of Useful Functions

LFE provides access to all of Erlang’s maps module functions. Here are the essential ones, annotated for your convenience:

maps:new/0

Creates an empty map. Philosophically equivalent to a blank canvas or an unwritten novel.

(maps:new)
#M()

maps:size/1

Returns the number of key-value pairs, useful for determining if your map has achieved critical mass.

(maps:size '#M(a 1 b 2 c 3))
3

maps:get/2

Retrieves a value or raises an exception, rather like a particularly strict librarian.

(maps:get 'a '#M(a 1 b 2))
1

(maps:get 'z '#M(a 1 b 2))
;; ** exception error: bad_key

maps:find/2

The polite version of get, returning (tuple 'ok value) or the atom error.

(maps:find 'a '#M(a 1 b 2))
#(ok 1)

(maps:find 'z '#M(a 1 b 2))
error

maps:is_key/2

A simple existential check that returns a boolean without drama.

(maps:is_key 'a '#M(a 1 b 2))
true

(maps:is_key 'z '#M(a 1 b 2))
false

maps:keys/1

Returns all keys in ascending order, neatly arranged like soldiers on parade.

(maps:keys '#M(c 3 a 1 b 2))
(a b c)

maps:values/1

Returns all values in key-sorted order, which is to say, the values corresponding to the sorted keys.

(maps:values '#M(c 3 a 1 b 2))
(1 2 3)

maps:to_list/1

Converts a map to a list of two-tuples, preserving key order.

(maps:to_list '#M(a 1 b 2))
(#(a 1) #(b 2))

maps:from_list/1

The inverse operation, constructing a map from tuple pairs. If duplicate keys exist, the first value wins—first come, first served, as it were.

(maps:from_list '(#(a 1) #(b 2) #(a 99)))
#M(a 1 b 2)

maps:remove/2

Removes a key-value pair, if it exists. Idempotent and unbothered.

(maps:remove 'b '#M(a 1 b 2 c 3))
#M(a 1 c 3)

maps:merge/2

Combines two maps, with the second map’s values taking precedence in key conflicts. Excellent for establishing defaults:

(let ((defaults '#M(timeout 5000 retry 3 verbose false))
      (user-opts '#M(timeout 1000 verbose true)))
  (maps:merge defaults user-opts))
;; #M(retry 3 timeout 1000 verbose true)

maps:map/2

Applies a function to all values while preserving keys, rather like redecorating without changing the floor plan.

(maps:map (lambda (k v) (* v 2)) '#M(a 1 b 2 c 3))
#M(a 2 b 4 c 6)

maps:filter/2

Keeps only the key-value pairs satisfying a predicate, in the finest tradition of selective attention.

(maps:filter (lambda (k v) (> v 2)) '#M(a 1 b 2 c 3 d 4))
#M(c 3 d 4)

Maps vs. Records: A Comparison of Philosophies

Records are compile-time constructs—tuples with aspirations. They’re efficient, yes, but they’re also rigid. Add a field to a record definition, and suddenly every module that uses it needs recompilation. It’s like trying to add a room to your house and discovering you need to rebuild the entire neighborhood.

Maps, on the other hand, are runtime entities. They’re flexible, introspective, and they don’t require a memo to the compiler every time you want to add a key. Use records when you know precisely what fields you need and performance is paramount. Use maps when flexibility matters or when you’re building dynamic data structures.

As one of the source texts wisely notes: “For the core of your process loop, when you know all keys that should exist, a record would be a smart choice, performance-wise.”

Maps vs. Proplists: The Evolution of Options

Proplists have served Erlang well for passing options, but they’re showing their age. Consider:

% Erlang proplist style
[{active, true}, binary, {timeout, 5000}]

% Maps style
#{active => true, binary => true, timeout => 5000}

Maps provide simpler access patterns (Opts#{timeout} rather than proplists:get_value(timeout, Opts, Default)) and the merge/2 function elegantly handles defaults. However, literal options like read, write, and binary look cleaner in proplist form.

The ecosystem is gradually transitioning. For new code, consider using maps for options that are mostly pairs, and reserve proplists for compatibility or when literal atoms dominate.

Maps vs. Dicts and Other Erlang Data Structures

Maps effectively obsolete several older dictionary implementations:

• dict: The original ordered dictionary module. Maps are faster and have native syntax.
• orddict: Ordered dictionary as a sorted list. Maps provide similar ordering with better performance.
• gb_trees: General balanced trees. Use these only if you need minimum/maximum key operations frequently.

For almost all associative data needs, maps are now the default choice, much like how smartphones rendered pocket calculators quaint.

Ordering and Comparison

Maps have well-defined ordering semantics, which means you can sort them without inducing existential crises:

1. Maps with fewer entries are smaller than maps with more entries
2. Maps with equal sizes are compared by converting to sorted key-value lists

(< '#M(a 1) '#M(a 1 b 2))
true

(< '#M(age 23 person "jim") '#M(email "sue@place.com" name "sue"))
true  ; because 'age < 'email

When comparing maps to other terms, maps are considered more complex than lists or tuples but simpler than functions and PIDs, in the grand ontological hierarchy of Erlang terms.

LFE-Specific Conveniences

LFE provides several forms that make working with maps more pleasant than it has any right to be:

The (map) Constructor

(map 'a 1 'b 2 'c 3)
#M(a 1 b 2 c 3)

The (mref Map Key) Function

Safely access values with optional defaults:

(mref '#M(a 1 b 2) 'a)
1

(mref '#M(a 1 b 2) 'z 'default)
default

The (mset Map Key Value) Function

Update or add a key-value pair:

(mset '#M(a 1 b 2) 'c 3)
#M(a 1 b 2 c 3)

The (mupd Map Key Value) Function

Update an existing key (fails if key doesn’t exist):

(mupd '#M(a 1 b 2) 'a 42)
#M(a 42 b 2)

The (map-get Map Key) Macro

Pattern matching extraction in let bindings:

(let (((map-get '#M(a 1 b 2) a val)))
  val)
1

The JSON Connection

Maps map naturally (pun absolutely intended) to JSON objects, making them ideal for web APIs and data interchange:

;; Convert map to JSON
(maps:to_json '#M(name "Arthur" age 42 hobbies ("tea" "towels")))
;; #B("{\"age\":42,\"hobbies\":[\"tea\",\"towels\"],\"name\":\"Arthur\"}")

;; Convert JSON to map
(maps:from_json #B("{\"name\":\"Ford\",\"species\":\"Betelgeusian\"}"))
;; #M(name "Ford" species "Betelgeusian")

The safe_from_json/1 variant protects against atom exhaustion attacks by requiring all atoms to exist before conversion—a sensible precaution when dealing with external data.

A Note on Immutability and Illusions

Unlike JavaScript or Python dictionaries, Erlang maps are immutable. When you “update” a map, you create a new map. The VM optimizes this behind the scenes through structural sharing, so it’s nowhere near as expensive as it sounds.

This immutability prevents the sorts of aliasing bugs that plague mutable languages—you can’t accidentally change a map through a reference held by another part of your program. Each variable maintains its own consistent view of reality, which is rather nice when debugging at 3 AM.

As one text eloquently puts it: “The Erlang system copies only those parts of the internal structures necessary to maintain the illusion that a copy has been created, so creating what appears to be deep copies of an object is an extremely lightweight operation.”

Practical Guidelines

Here’s the accumulated wisdom for working with maps in LFE:

1. Use => for initial key creation, := for updates. This catches typos and signals intent.

2. Use maps for options when they’re mostly pairs. The maps:merge/2 pattern for defaults is elegant.

3. Use records for known, fixed structures where performance matters. Process loop state, for instance.

4. Pattern match freely. You needn’t match all keys—maps don’t mind selective attention.

5. Remember that maps are ordered. This makes their behavior predictable and their output deterministic.

6. Consider maps for any associative data unless you have specific reasons (like needing min/max operations frequently) to use something else.

In Conclusion

Maps represent the culmination of Erlang’s journey toward a native, efficient, flexible associative data structure. They’re not perfect for every situation—records still have their place, and proplists maintain certain advantages for literal options—but they’re remarkably close to being the universal solution for “I need to associate keys with values.”

Use them with confidence, but not recklessness. Use them with understanding, but not anxiety. And remember: in the end, they’re just data structures. Very sophisticated, beautifully implemented, immutable data structures. But data structures nonetheless.

The answer to the ultimate question of maps, the universe, and everything is not 42. It’s #M(answer 42). Subtle difference, that.

Arrays

In the grand tapestry of Erlang data structures, arrays occupy a peculiar position—rather like finding a perfectly functional bicycle in a showroom full of hovercrafts. They work, certainly. They do their job with admirable efficiency. But you can’t help wondering if perhaps you’ve wandered into the wrong universe, one where zero-based indexing is considered normal and mutable-looking structures somehow maintain their functional purity through sheer force of will.

Arrays in Erlang (and by extension, LFE) are functional, extendible data structures that provide O(log n) access to elements via numeric indices. They were designed for situations where you absolutely, positively need to access element 42 without first traversing through elements 0 through 41, and where the list’s charming linked-list aesthetic just won’t cut it. Think of them as lists that went to business school and came back wearing sensible shoes.

A Brief Philosophical Interlude

Before we dive into the mechanics, it’s worth pondering why arrays exist in a language so thoroughly committed to lists that it named itself after one. The answer, as with most things in life, comes down to pragmatism wrestling with ideology in a dimly lit car park behind the theoretical computer science building.

Lists are magnificent for sequential processing, for building data structures one cons cell at a time like a particularly methodical coral reef. But if you need random access—if you need to leap directly to element 1,000 without the tedious business of hopping through 999 predecessors—lists begin to look rather less magnificent and rather more like a very long queue at the Department of Motor Vehicles.

Arrays solve this. They provide indexed access with logarithmic complexity, which in practical terms means you can access any element in a 100,000-element array as quickly as you can access the tenth element. This is achieved through a clever tree structure that would make a botanist weep with joy, though naturally, the implementation details are hidden away like a magician’s card tricks—best appreciated without too much scrutiny.

Creating Arrays

The most straightforward way to birth an array into existence is to simply call it into being with array:new:

lfe> (array:new)
#(array 10 0 undefined 10)

This creates an extendible array with initial size zero, though the representation you see is an implementation detail subject to change without notice, like the venue for a surprise party. The important bit is that you now have an array, ready and waiting to store things at numeric indices.

You can also create an array with a specific initial size:

lfe> (array:new 100)
#(array 10 100 undefined 10)
lfe> (array:size (array:new 100))
100

This creates a fixed-size array with 100 slots, all initially set to undefined—the default value that serves as the placeholder for “nothing here yet, move along.”

If undefined doesn’t suit your philosophical outlook, you can specify your own default value:

lfe> (array:new '(#(default 0)))
#(array 10 0 0 10)
lfe> (array:new 50 '(#(default "vacant")))
#(array 10 50 "vacant" 10)

You can also build arrays from lists, which is a bit like converting a pleasant country walk into a precisely mapped coordinate system:

lfe> (array:from_list '(1 2 3 4 5))
#(array 10 5 undefined 10)
lfe> (array:get 0 (array:from_list '(arthur ford trillian)))
arthur
lfe> (array:get 2 (array:from_list '(arthur ford trillian)))
trillian

Or from ordered dictionaries (which are lists of index-value tuples), for those occasions when you’ve already done half the work:

lfe> (array:from_orddict '(#(0 "first") #(5 "sixth") #(10 "eleventh")))
#(array 10 11 undefined 10)

Note that this creates an array large enough to hold index 10, with undefined values filling the gaps like polite placeholders at a dinner party.

The Zero-Based Heresy

Here’s where arrays commit their most controversial act: they use zero-based indexing. In a language where tuples start at 1, where lists:nth counts from 1, where the very notion of “first” implies the number 1, arrays boldly declare that the first element lives at index 0.

This is not an accident. This is not a bug. This is a deliberate design choice, presumably made by someone who spent too much time programming in C and not enough time at Erlang philosophy seminars. The official documentation acknowledges this heresy with remarkable sangfroid: “Arrays uses zero-based indexing. This is a deliberate design choice and differs from other Erlang data structures.”

lfe> (let ((arr (array:from_list '(a b c d e))))
       (array:get 0 arr))
a
lfe> (let ((arr (array:from_list '(a b c d e))))
       (array:get 4 arr))
e

Element 0 is a. Element 4 is e. The fifth element is at index 4. Try not to think about this too hard after consuming any amount of Pan Galactic Gargle Blasters.

Basic Operations

Setting values:

The primary way to modify an array is with array:set, which doesn’t actually modify anything (this is still functional programming, after all) but rather returns a new array with the specified element changed:

lfe> (let ((arr (array:new 10)))
       (array:set 5 "hello" arr))
#(array 10 10 undefined 10)
lfe> (let* ((arr1 (array:new 10))
            (arr2 (array:set 5 "hello" arr1)))
       (array:get 5 arr2))
"hello"

If you set an index beyond the current size in an extendible array, it grows automatically, like a particularly accommodating houseplant:

lfe> (let* ((arr1 (array:new))
            (arr2 (array:set 100 "far away" arr1)))
       (array:size arr2))
101

However, if the array has fixed size (more on this shortly), attempting to set beyond the boundary will result in the sort of error message that makes you reconsider your life choices:

lfe> (let* ((arr1 (array:new 10))
            (arr2 (array:fix arr1)))
       (array:set 10 "beyond the pale" arr2))
exception error: badarg

Getting values:

Retrieving values is straightforward enough, though you must remember the zero-based indexing lest you find yourself off by one in that peculiar way that suggests you’re from a parallel universe where counting starts at one:

lfe> (let ((arr (array:from_list '(red green blue yellow))))
       (list (array:get 0 arr)
             (array:get 2 arr)))
(red blue)

Accessing an unset entry returns the default value:

lfe> (let ((arr (array:new 10)))
       (array:get 7 arr))
undefined
lfe> (let ((arr (array:new 10 '(#(default "empty")))))
       (array:get 7 arr))
"empty"

And accessing beyond the end of an extendible array also returns the default:

lfe> (let* ((arr1 (array:new))
            (arr2 (array:set 5 "here" arr1)))
       (array:get 100 arr2))
undefined

Checking the size:

The array:size function returns the number of entries, which is also one more than the highest accessible index (thanks to zero-based indexing providing us with these delightful arithmetic gymnastics):

lfe> (array:size (array:new 42))
42
lfe> (let* ((arr1 (array:new))
            (arr2 (array:set 17 "surprise" arr1)))
       (array:size arr2))
18

Resetting values:

You can reset an entry back to the default value without changing the array’s size:

lfe> (let* ((arr1 (array:from_list '(1 2 3 4 5)))
            (arr2 (array:reset 2 arr1)))
       (list (array:get 2 arr2)
             (array:size arr2)))
(undefined 5)

This is distinct from setting the value to undefined—a philosophical distinction that matters only if your default value happens to be something other than undefined, in which case it matters quite a lot.

Getting the sparse size:

For arrays that have many default-valued entries, array:sparse_size returns the position after the last non-default value:

lfe> (let* ((arr1 (array:new 100))
            (arr2 (array:set 50 "last one" arr1)))
       (list (array:size arr2)
             (array:sparse_size arr2)))
(100 51)

This can be useful for understanding how much of your array is actually being used versus how much is just cosmic background radiation.

Fixed vs. Extendible Arrays

Arrays come in two philosophical varieties: fixed and extendible. This is the difference between a house and a TARDIS—one has immutable boundaries, the other expands as needed.

By default, arrays are extendible:

lfe> (let ((arr (array:new)))
       (array:is_fix arr))
false

You can create a fixed-size array explicitly:

lfe> (array:is_fix (array:new 50))
true
lfe> (array:is_fix (array:new '(fixed)))
true
lfe> (array:is_fix (array:new 50 '(fixed)))
true

Or you can fix an existing array in place (metaphorically speaking):

lfe> (let* ((arr1 (array:new))
            (arr2 (array:set 10 "something" arr1))
            (arr3 (array:fix arr2)))
       (array:is_fix arr3))
true

Once fixed, an array refuses to grow, which means attempting to set beyond its boundaries results in the program’s equivalent of a stern look and a finger wag:

lfe> (let* ((arr1 (array:new 10))
            (arr2 (array:fix arr1)))
       (array:set 10 "too far" arr2))
exception error: badarg

You can unfix an array with array:relax, which is either a function name or lifestyle advice:

lfe> (let* ((arr1 (array:new 10))
            (arr2 (array:relax arr1)))
       (array:is_fix arr2))
false

Conversion Operations

To and from lists:

Arrays and lists can be converted back and forth, though this is an O(n) operation that involves visiting every element like an overly thorough census taker:

lfe> (array:to_list (array:from_list '(1 2 3 4 5)))
(1 2 3 4 5)
lfe> (let ((arr (array:new 5)))
       (array:to_list arr))
(undefined undefined undefined undefined undefined)

For sparse arrays, you can use array:sparse_to_list to skip the default values:

lfe> (let* ((arr1 (array:new 100))
            (arr2 (array:set 10 "here" arr1))
            (arr3 (array:set 50 "there" arr2)))
       (array:sparse_to_list arr3))
("here" "there")

To and from ordered dictionaries:

Ordered dictionaries (lists of #(index value) tuples) can be converted to and from arrays:

lfe> (array:to_orddict (array:from_list '(a b c)))
(#(0 a) #(1 b) #(2 c))
lfe> (let* ((arr1 (array:new 10))
            (arr2 (array:set 3 "three" arr1))
            (arr3 (array:set 7 "seven" arr2)))
       (array:sparse_to_orddict arr3))
(#(3 "three") #(7 "seven"))

Higher-Order Functions

Arrays support the traditional triumvirate of functional programming operations: map, fold, and their various cousins who show up uninvited to family gatherings.

Mapping:

array:map applies a function to each element, returning a new array:

lfe> (let ((arr (array:from_list '(1 2 3 4 5))))
       (array:to_list
         (array:map
           (lambda (i x) (* x x))
           arr)))
(1 4 9 16 25)

Note that the mapping function receives both the index and the value, which is useful for operations that care about position. If you don’t need the index, simply ignore it with underscore-based nonchalance:

lfe> (let ((arr (array:from_list '("hello" "world" "goodbye"))))
       (array:to_list
         (array:map
           (lambda (_ s) (string:uppercase s))
           arr)))
("HELLO" "WORLD" "GOODBYE")

For sparse mapping (skipping default values), use array:sparse_map:

lfe> (let* ((arr1 (array:new 100))
            (arr2 (array:set 10 5 arr1))
            (arr3 (array:set 50 10 arr2)))
       (array:sparse_to_orddict
         (array:sparse_map
           (lambda (i x) (* x 2))
           arr3)))
(#(10 10) #(50 20))

Folding (left):

array:foldl processes the array from index 0 upward, accumulating a result:

lfe> (let ((arr (array:from_list '(1 2 3 4 5))))
       (array:foldl
         (lambda (i x acc) (+ x acc))
         0
         arr))
15
lfe> (let ((arr (array:from_list '("the" "answer" "is" "42"))))
       (array:foldl
         (lambda (i word acc)
           (++ acc " " word))
         ""
         arr))
" the answer is 42"

The fold function receives the index, the value, and the accumulator, in that order. This can be slightly disorienting if you’re accustomed to lists, where the fold function typically takes just the value and accumulator, but arrays like to remind you that they know where they are at all times.

Folding (right):

array:foldr processes from the highest index downward:

lfe> (let ((arr (array:from_list '(1 2 3 4 5))))
       (array:foldr
         (lambda (i x acc) (cons x acc))
         '()
         arr))
(1 2 3 4 5)
lfe> (let ((arr (array:from_list '(1 2 3 4 5))))
       (array:foldl
         (lambda (i x acc) (cons x acc))
         '()
         arr))
(5 4 3 2 1)

Sparse folding:

Both array:sparse_foldl and array:sparse_foldr skip default-valued entries:

lfe> (let* ((arr1 (array:new 100))
            (arr2 (array:set 10 "alpha" arr1))
            (arr3 (array:set 50 "omega" arr2)))
       (array:sparse_foldl
         (lambda (i x acc) (cons x acc))
         '()
         arr3))
("omega" "alpha")

Default Values and Philosophy

The default value of an array is set at creation time and cannot be changed afterward—it’s baked into the array’s very essence like the speed of light into spacetime. This default serves as the placeholder for uninitialized entries:

lfe> (let ((arr (array:new 5)))
       (array:default arr))
undefined
lfe> (let ((arr (array:new 5 '(#(default 0)))))
       (array:default arr))
0

This creates an interesting philosophical question: if you set an entry to the default value, is it different from leaving it unset? The answer, according to arrays, is “no”:

lfe> (let* ((arr1 (array:new 10))
            (arr2 (array:set 5 undefined arr1)))
       (array:sparse_to_orddict arr2))
()

The sparse operations treat explicitly-set-to-default-value entries exactly the same as never-touched entries. This is either profound wisdom about the nature of existence or a minor implementation detail, depending on your philosophical orientation.

Performance Characteristics

Arrays in Erlang are implemented using a tree structure that provides the following complexity characteristics:

• Access by index: O(log n) — better than lists, worse than actual arrays in languages with mutable memory
• Setting a value: O(log n) — creates a new array with structural sharing
• Size queries: O(1) — constant time
• Conversion to/from lists: O(n) — must visit every element

This makes arrays most suitable for scenarios where:

1. You need better-than-linear access to arbitrary indices
2. You’re not doing constant-time updates (this is functional programming, after all)
3. The numeric indices are meaningful in your problem domain
4. You’re not obsessing about the zero-based indexing

When to Use Arrays

The crucial question remains: when should you actually use arrays instead of the more idiomatic lists, tuples, or maps?

Use arrays when:

• You need indexed access to a large collection with changing size
• You’re implementing algorithms that naturally think in terms of arrays
• You need to grow a fixed-index structure dynamically
• You’re translating code from imperative languages and arrays make the translation clearer

Avoid arrays when:

• You’re primarily doing sequential processing (use lists)
• You have a fixed number of elements (use tuples)
• You need key-value storage with non-numeric keys (use maps)
• You need actual constant-time indexed access (consider port drivers or NIFs)

In practice, arrays are rarely seen in idiomatic Erlang/LFE code. They occupy a niche for specific use cases where the alternatives are genuinely worse, rather than just different. Think of them as the specialized tool you keep in the back of the toolbox—you’re glad it’s there when you need it, but most days you reach for something else.

Working Examples

Building an array incrementally:

(defun build-array (n)
  (build-array-helper 0 n (array:new)))

(defun build-array-helper (i n acc)
  (if (>= i n)
      acc
      (build-array-helper
        (+ i 1)
        n
        (array:set i (* i i) acc))))

lfe> (array:to_list (build-array 10))
(0 1 4 9 16 25 36 49 64 81)

Finding elements by predicate:

(defun array-find (pred arr)
  (array-find-helper pred arr 0 (array:size arr)))

(defun array-find-helper (pred arr i size)
  (cond
    ((>= i size) 'not-found)
    ((funcall pred (array:get i arr)) i)
    ('true (array-find-helper pred arr (+ i 1) size))))

lfe> (let ((arr (array:from_list '(1 3 5 8 9 12))))
       (array-find (lambda (x) (== 0 (rem x 2))) arr))
3

Filtering an array:

(defun array-filter (pred arr)
  (array:sparse_foldr
    (lambda (i x acc)
      (if (funcall pred x)
          (cons x acc)
          acc))
    '()
    arr))

lfe> (let ((arr (array:from_list '(1 2 3 4 5 6 7 8 9 10))))
       (array-filter (lambda (x) (> x 5)) arr))
(6 7 8 9 10)

Common Pitfalls

Off-by-one errors from zero-based indexing:

;; Wrong: trying to access the "fifth" element
lfe> (array:get 5 (array:from_list '(a b c d e)))
exception error: badarg

;; Right: the fifth element is at index 4
lfe> (array:get 4 (array:from_list '(a b c d e)))
e

Forgetting that set returns a new array:

;; Wrong: the original array is unchanged
lfe> (let ((arr (array:new 10)))
       (array:set 5 "hello" arr)
       (array:get 5 arr))
undefined

;; Right: capture the returned array
lfe> (let* ((arr1 (array:new 10))
            (arr2 (array:set 5 "hello" arr1)))
       (array:get 5 arr2))
"hello"

Attempting to set beyond a fixed array’s bounds:

;; Wrong: fixed arrays don't grow
lfe> (let* ((arr1 (array:new 10))
            (arr2 (array:set 15 "too far" arr1)))
       arr2)
exception error: badarg

;; Right: use an extendible array or relax first
lfe> (let* ((arr1 (array:new 10))
            (arr2 (array:relax arr1))
            (arr3 (array:set 15 "just right" arr2)))
       (array:get 15 arr3))
"just right"

Integration with Other LFE Patterns

Arrays can be pattern-matched in function definitions, though the match is structural and depends on the internal representation:

;; This works but is fragile and not recommended
(defun process-array
  (((tuple 'array _ size _ _))
   (list 'array-of-size size)))

Instead, use the array module functions:

(defun process-array (arr)
  (if (array:is_array arr)
      (list 'array-of-size (array:size arr))
      'not-an-array))

Arrays integrate naturally with higher-order functions through their fold and map operations, allowing you to compose array operations with other functional patterns:

(defun sum-of-squares (arr)
  (array:foldl
    (lambda (_ x acc)
      (+ acc (* x x)))
    0
    arr))

A Final Note on Pragmatism

Arrays exist at the intersection of functional purity and practical necessity. They’re not the most elegant solution—that honor belongs to lists and their recursive majesty. They’re not the most efficient—native arrays in imperative languages still reign supreme in raw performance. But they occupy a useful middle ground where indexed access meets functional immutability, where zero-based counting meets Erlang’s conventions, where theory meets the real world and they shake hands somewhat awkwardly before getting on with the job.

Use them when they make sense. Avoid them when they don’t. And try not to lose too much sleep over the zero-based indexing. After all, in the grand scheme of things, whether you start counting at zero or one is merely a question of where you happened to begin—and in a universe as bewilderingly vast as ours, starting from zero seems as reasonable a choice as any.

The Ancient Key-Value Triumvirate: A Historical Retrospective

In Which We Examine The Data Structures That Came Before Maps (And Occasionally Still Matter)

Long before maps arrived with their sophisticated syntax and first-class citizenship, Erlang programmers navigated the murky waters of associative data storage using three venerable modules: dict, orddict, and gb_trees. Think of them as the founding members of a particularly earnest folk trio—each with their own distinctive sound, each convinced they were doing it the right way, and each absolutely correct within their specific context.

These modules represent different philosophical approaches to the eternal problem of “I have a key, I have a value, now what?” The orddict believes in transparency and ordered simplicity. The dict believes in pragmatic efficiency through mysterious internal structures. And gb_trees believes in balanced hierarchies and giving you enough rope to either hang yourself or construct a magnificent suspension bridge—the choice, as always, is yours.

The Historical Context (Or: Why These Still Exist When Maps Are So Nice)

Maps were introduced in Erlang R17 with considerable fanfare and immediately rendered several of these modules somewhat obsolete, rather like how the automobile made the horse-drawn carriage industry reconsider its investment strategies. However, unlike horse-drawn carriages, these modules haven’t disappeared into museums. They persist for three excellent reasons:

1. Legacy Code: Vast swathes of Erlang code were written before maps existed, and rewriting working code simply to chase fashion is the sort of thing that leads to regret and missing deadlines.

2. Specific Use Cases: Each module has particular strengths that occasionally make it the optimal choice, even in a post-maps world.

3. API Stability: These modules have been stable for decades. When you need absolutely zero surprises, vintage sometimes beats shiny.

The key philosophical difference between these modules and maps is that maps are a native data type—the BEAM understands them intrinsically. These modules, on the other hand, are implementations—they’re libraries that construct their associative arrays using the primitive building blocks of tuples and lists, then wrap them in opaque types to prevent you from peeking at the machinery.

Comparing The Triumvirate: A Buyer’s Guide

Before we dive into the details of each, let’s establish when you might actually want to use these modules instead of maps:

orddict: The Transparent Pragmatist

What it is: A sorted list of {Key, Value} tuples. That’s it. That’s the whole mystery.

When to use it:

• You have fewer than ~75 key-value pairs
• You want a human-readable data structure you can examine without tools
• You need deterministic ordering by keys
• You’re debugging and want to see exactly what’s in your data structure
• You’re interoperating with code that expects lists

When to avoid it:

• You have more than 75 pairs (performance degrades linearly)
• You need fast lookups on large datasets
• Someone might judge you for using something so delightfully simple

dict: The Pragmatic Workhorse (Now Mostly Retired)

What it is: A hash-based dictionary with an opaque internal structure.

When to use it:

• You have legacy code using it already
• You’re working in a pre-R17 codebase (bless your heart)
• You have 75+ key-value pairs and can’t use maps for some reason

When to avoid it:

• You’re writing new code (use maps instead)
• You need any feature maps provide
• You value your future self’s time

Historical note: The dict module was the go-to for large key-value stores before maps arrived. It’s now in the awkward position of being “perfectly functional but superseded,” rather like a flip phone in the smartphone era—it makes calls just fine, but nobody’s impressed.

gb_trees: The Structured Perfectionist

What it is: General Balanced Trees implementing Prof. Arne Andersson’s balanced binary tree algorithm.

When to use it:

• You need to efficiently find minimum/maximum keys
• You want to iterate through keys in order
• You need the specific performance characteristics of balanced trees
• You’re implementing an algorithm that specifically requires tree operations
• You enjoy the aesthetic of logarithmic complexity

When to avoid it:

• You just need basic key-value storage (use maps)
• The learning curve for its dual-mode API seems excessive
• You prefer the simplicity of hash-based lookups

Performance Characteristics: A Rough Guide

The performance of these structures varies based on size and operation type:

For small datasets (< 75 elements):

• orddict: Perfectly acceptable, human-readable, simple
• dict: Overkill, slower than orddict for small sizes
• gb_trees: Also overkill, but faster than dict
• maps: Optimal in almost all cases

For medium datasets (75-10,000 elements):

• orddict: Unacceptably slow
• dict: Good performance, but maps are better
• gb_trees: Excellent performance, competitive with maps
• maps: Generally optimal

For large datasets (10,000+ elements):

• orddict: Don’t even think about it
• dict: Acceptable but deprecated
• gb_trees: Excellent, sometimes faster than maps for ordered operations
• maps: Generally optimal

The actual performance depends on your read/write ratio, whether you need ordered access, and whether you’re running a benchmark or actual production code (these can differ by surprising margins).

orddict: The Ordered Dictionary

In Which Simplicity Becomes A Virtue

The orddict module is remarkably straightforward: it’s a list of {Key, Value} tuples kept in ascending order by key. That’s the entire data structure. There’s no hidden magic, no opaque internals, no clever hashing schemes. It’s just a sorted list, and it makes no apologies for this fact.

This transparency is both its greatest strength and its fundamental limitation. Because it’s just a list, you can pattern-match on it, print it, inspect it visually, and generally treat it like the simple structure it is. Because it’s just a list, operations like lookup require scanning through elements one by one until you find what you’re looking for (or don’t), which makes it unsuitable for large datasets.

Creating Orddicts: The Art of Sorted Pairs

In LFE, you create orddicts using the orddict module functions:

;; An empty orddict - the beginning of all things
(orddict:new)
()

;; Building one up from nothing
(set d1 (orddict:store 'b 2 (orddict:new)))
((b 2))

(set d2 (orddict:store 'a 1 d1))
((a 1) (b 2))

;; Note the automatic ordering - 'a' comes before 'b'
(set d3 (orddict:store 'c 3 d2))
((a 1) (b 2) (c 3))

;; Or create from a list in one go
(set d4 (orddict:from_list '(#(z 26) #(a 1) #(m 13))))
((a 1) (m 13) (z 26))

Notice how the keys are automatically sorted regardless of insertion order. This is orddict’s defining characteristic—it maintains sorted order at all times, trading some insertion performance for predictable, ordered output.

The Core Operations: CRUD Without The Drama

The orddict module provides the standard suite of dictionary operations, with function names that are pleasantly self-explanatory:

orddict:store/3

Store a key-value pair, updating if the key exists, inserting if it doesn’t:

(set d1 (orddict:store 'name "Arthur" (orddict:new)))
((name "Arthur"))

(set d2 (orddict:store 'age 42 d1))
((age 42) (name "Arthur"))

;; Updating an existing key
(set d3 (orddict:store 'age 43 d2))
((age 43) (name "Arthur"))

orddict:find/2

Search for a key, returning {ok, Value} or error:

(orddict:find 'name '#((age 42) (name "Arthur")))
#(ok "Arthur")

(orddict:find 'species '#((age 42) (name "Arthur")))
error

This is the polite version of lookup—it never crashes, just admits defeat gracefully when the key isn’t present.

orddict:fetch/2

The impolite version that assumes the key exists:

(orddict:fetch 'name '#((age 42) (name "Arthur")))
"Arthur"

(orddict:fetch 'species '#((age 42) (name "Arthur")))
;; ** exception error: {badkey,species}

Use fetch when the key’s absence would indicate a programming error. Use find when you’re genuinely uncertain.

orddict:is_key/2

The existential inquiry:

(orddict:is_key 'name '#((age 42) (name "Arthur")))
true

(orddict:is_key 'species '#((age 42) (name "Arthur")))
false

orddict:erase/2

Remove a key-value pair:

(orddict:erase 'age '#((age 42) (name "Arthur")))
((name "Arthur"))

Idempotent—erasing a non-existent key is perfectly fine and changes nothing.

Iteration and Transformation

orddict:fold/3

The workhorse of orddict processing. Fold a function over all key-value pairs:

(orddict:fold 
  (lambda (key val acc) 
    (cons (tuple key (* val 2)) acc))
  '()
  '#((a 1) (b 2) (c 3)))
((c 6) (b 4) (a 2))

Note that fold processes elements in key order, but cons-ing them into an accumulator reverses the order. If you want to maintain order, you’ll need to reverse at the end or use a different accumulation strategy.

orddict:map/2

Transform all values while preserving keys and structure:

(orddict:map 
  (lambda (key val) (+ val 100))
  '#((a 1) (b 2) (c 3)))
((a 101) (b 102) (c 103))

The mapping function receives both key and value, but only the returned value is used. The key remains unchanged.

orddict:filter/2

Keep only the key-value pairs that satisfy a predicate:

(orddict:filter 
  (lambda (key val) (> val 5))
  '#((a 3) (b 7) (c 2) (d 9)))
((b 7) (d 9))

The Special Append Operations

Orddicts provide two specialized functions for accumulating values in lists:

orddict:append/3

Append a value to a list associated with a key:

(let* ((d1 (orddict:store 'files '() (orddict:new)))
       (d2 (orddict:append 'files "file1.txt" d1))
       (d3 (orddict:append 'files "file2.txt" d2)))
  (orddict:fetch 'files d3))
("file1.txt" "file2.txt")

This assumes the current value (if any) is a list. If the key doesn’t exist, it creates a new list with the value. If the key exists but the value isn’t a list, prepare for an exception.

orddict:append_list/3

Append an entire list of values:

(let* ((d1 (orddict:store 'files '("a.txt") (orddict:new)))
       (d2 (orddict:append_list 'files '("b.txt" "c.txt") d1)))
  (orddict:fetch 'files d2))
("a.txt" "b.txt" "c.txt")

These functions save you from the ceremony of fetching, appending, and storing manually. They’re particularly useful when building up collections of values keyed by some attribute.

Merging and Combining

orddict:merge/3

Merge two orddicts, using a function to resolve conflicts:

(let ((d1 '#((a 1) (b 2) (c 3)))
      (d2 '#((b 20) (c 30) (d 4))))
  (orddict:merge 
    (lambda (key v1 v2) (+ v1 v2))
    d1 
    d2))
((a 1) (b 22) (c 33) (d 4))

The merge function is called only for keys that exist in both dictionaries. Keys unique to either dictionary are included as-is.

Utility Functions

orddict:size/1

Count the key-value pairs:

(orddict:size '#((a 1) (b 2) (c 3)))
3

orddict:is_empty/1

Check for emptiness:

(orddict:is_empty (orddict:new))
true

(orddict:is_empty '#((a 1)))
false

orddict:fetch_keys/1

Get all keys:

(orddict:fetch_keys '#((a 1) (b 2) (c 3)))
(a b c)

orddict:to_list/1 and orddict:from_list/1

Convert to and from list representation:

(orddict:to_list '#((a 1) (b 2)))
(#(a 1) #(b 2))

(orddict:from_list '(#(z 26) #(a 1) #(m 13)))
((a 1) (m 13) (z 26))

The from_list function sorts the input and removes duplicates (keeping the first occurrence of each key).

Update Operations: The Functional Approach

orddict:update/3

Update an existing key’s value using a function:

(orddict:update 
  'count 
  (lambda (n) (+ n 1))
  '#((count 5) (name "test")))
((count 6) (name "test"))

If the key doesn’t exist, this crashes. It’s for when you know the key must be present.

orddict:update/4

Update with a default if the key doesn’t exist:

(orddict:update 
  'count 
  (lambda (n) (+ n 1))
  1
  '#((name "test")))
((count 1) (name "test"))

This version uses the default value (1 in this case) when the key is missing, then applies the function to that default.

orddict:update_counter/3

Specialized for incrementing numeric values:

(orddict:update_counter 'score 10 '#((score 5)))
((score 15))

(orddict:update_counter 'score 10 '#((name "test")))
((name "test") (score 10))

Adds the increment to the existing value, or uses the increment as the initial value if the key doesn’t exist.

Pattern Matching on Orddicts

Because orddicts are just lists, you can pattern-match on them directly in function clauses and case expressions:

(defun get-name 
  (('()) 'no-name)
  (((cons (tuple 'name n) _rest)) n)
  (((cons _ rest)) (get-name rest)))

(get-name '#((age 42) (name "Zaphod")))
"Zaphod"

However, this breaks the abstraction and ties your code to the internal representation. It’s generally better to use the orddict module functions, which remain valid even if the internal representation changes (though given orddict’s age and simplicity, such a change is approximately as likely as discovering that pi has been wrong all along).

When Orddict Is The Right Choice

Despite being the simplest of the key-value stores, orddict has its place:

1. Small datasets: For fewer than ~75 elements, orddict performs acceptably and provides crystal-clear inspection.

2. Debugging: When you need to see exactly what’s in your data structure without ceremony, orddict obliges.

3. Ordered output: If you need keys in sorted order and you’re using the structure itself (not just calling fetch_keys), orddict maintains order naturally.

4. Simple configuration: For configuration data or small option sets, orddict’s transparency is a feature.

5. Teaching: When explaining associative data structures, orddict is mercifully simple.

When To Graduate Beyond Orddict

Move to dict or maps when:

• You have more than 75 key-value pairs
• Lookup performance becomes measurable
• You’re building a long-lived production system
• You need better algorithmic complexity
• Your benchmarks show orddict as a bottleneck

The transition is usually straightforward since the API is very similar across these modules.

dict: The Pragmatic Workhorse (Historical Division)

In Which We Acknowledge A Module’s Graceful Obsolescence

The dict module occupies an interesting position in Erlang’s history. For many years, it was the solution for large key-value stores—a hash-based dictionary with good performance characteristics and a clean API. Then maps arrived in R17, providing similar functionality with better syntax and native support, and dict found itself in the position of a veteran general suddenly outranked by a younger officer with shinier medals.

Here’s the essential truth: if you’re writing new code and you can use maps, use maps. The dict module isn’t deprecated, isn’t broken, and isn’t going away—but it’s no longer the first choice for new development. It remains relevant primarily for maintaining existing codebases and for those rare situations where you’re constrained to pre-R17 Erlang.

The Dict Data Structure: Opaque By Design

Unlike orddicts, which are transparently lists, dicts are opaque structures. You cannot pattern-match on them, you cannot inspect them without dict module functions, and you shouldn’t try to construct them manually. This opacity was intentional—it allows the implementation to use sophisticated internal structures (hash tables with dynamic resizing) without exposing those details to user code.

The practical implication is that all dict operations must go through the module’s API. There’s no cheating by accessing the internal structure directly.

Creating Dicts: The Standard Ceremony

;; Empty dict
(set d1 (dict:new))
#(dict ...)  ; Opaque structure - don't examine too closely

;; Building up
(set d2 (dict:store 'a 1 d1))
(set d3 (dict:store 'b 2 d2))
(set d4 (dict:store 'c 3 d3))

;; From a list
(set d5 (dict:from_list '(#(a 1) #(b 2) #(c 3))))

The API should look familiar—it mirrors orddict quite deliberately, making transitions between the two relatively painless.

Core Operations: The Same But Different

dict:store/3

Store or update a key-value pair:

(dict:store 'name "Ford" (dict:new))
;; Returns a dict containing name -> "Ford"

(let ((d (dict:store 'age 200 (dict:new))))
  (dict:store 'age 201 d))
;; Updates age to 201

dict:find/2 and dict:fetch/2

The familiar search operations:

(let ((d (dict:from_list '(#(a 1) #(b 2)))))
  (dict:find 'a d))
#(ok 1)

(let ((d (dict:from_list '(#(a 1) #(b 2)))))
  (dict:find 'z d))
error

(let ((d (dict:from_list '(#(a 1) #(b 2)))))
  (dict:fetch 'a d))
1

(let ((d (dict:from_list '(#(a 1) #(b 2)))))
  (dict:fetch 'z d))
;; ** exception error: bad_key

dict:is_key/2

Check for key existence:

(let ((d (dict:from_list '(#(a 1) #(b 2)))))
  (dict:is_key 'a d))
true

dict:erase/2

Remove a key:

(let ((d (dict:from_list '(#(a 1) #(b 2) #(c 3)))))
  (dict:erase 'b d))
;; Returns dict with only a and c

Iteration and Transformation

dict:fold/3

Fold a function over all key-value pairs:

(let ((d (dict:from_list '(#(a 1) #(b 2) #(c 3)))))
  (dict:fold 
    (lambda (k v acc) (+ acc v))
    0
    d))
6

Note that unlike orddict, the iteration order is undefined. You cannot rely on keys being processed in any particular sequence.

dict:map/2

Transform values:

(let ((d (dict:from_list '(#(a 1) #(b 2) #(c 3)))))
  (dict:map 
    (lambda (k v) (* v v))
    d))
;; Returns dict with squared values: a->1, b->4, c->9

dict:filter/2

Keep only matching pairs:

(let ((d (dict:from_list '(#(a 1) #(b 2) #(c 3) #(d 4)))))
  (dict:filter 
    (lambda (k v) (=:= 0 (rem v 2)))
    d))
;; Returns dict with only even values: b->2, d->4

Append Operations

Like orddict, dict provides specialized functions for list accumulation:

dict:append/3

(let* ((d1 (dict:store 'items '() (dict:new)))
       (d2 (dict:append 'items "first" d1))
       (d3 (dict:append 'items "second" d2)))
  (dict:fetch 'items d3))
("first" "second")

dict:append_list/3

(let* ((d1 (dict:store 'items '("a") (dict:new)))
       (d2 (dict:append_list 'items '("b" "c") d1)))
  (dict:fetch 'items d2))
("a" "b" "c")

Merging and Updating

dict:merge/3

Merge two dicts with a conflict resolution function:

(let ((d1 (dict:from_list '(#(a 1) #(b 2))))
      (d2 (dict:from_list '(#(b 3) #(c 4)))))
  (dict:merge 
    (lambda (k v1 v2) (max v1 v2))
    d1 
    d2))
;; Returns dict: a->1, b->3, c->4

dict:update/3 and dict:update/4

Update with function:

;; Requires key to exist
(let ((d (dict:from_list '(#(count 5)))))
  (dict:update 'count (lambda (n) (+ n 1)) d))
;; Returns dict with count->6

;; Provides default if key missing
(let ((d (dict:new)))
  (dict:update 'count (lambda (n) (+ n 1)) 0 d))
;; Returns dict with count->0 (default was used, then function applied)

dict:update_counter/3

Increment numeric values:

(let ((d (dict:from_list '(#(score 10)))))
  (dict:update_counter 'score 5 d))
;; Returns dict with score->15

Utility Functions

dict:size/1

(dict:size (dict:from_list '(#(a 1) #(b 2) #(c 3))))
3

dict:is_empty/1

(dict:is_empty (dict:new))
true

dict:fetch_keys/1

(dict:fetch_keys (dict:from_list '(#(a 1) #(b 2) #(c 3))))
;; Returns list of keys: (a b c) - but order is undefined

dict:to_list/1

(dict:to_list (dict:from_list '(#(a 1) #(b 2))))
;; Returns list: (#(a 1) #(b 2)) - order undefined

Key Differences from Orddict

1. Ordering: Dict provides no ordering guarantees. Keys come out in arbitrary order.

2. Performance: Dict uses hash tables internally, providing O(log n) or better average-case performance for most operations, compared to orddict’s O(n).

3. Opacity: You cannot pattern-match on dict structures or inspect them without using module functions.

4. Size: Dict excels with larger datasets (75+ elements) where orddict becomes sluggish.

Why Dict Still Exists (And Why You Probably Shouldn’t Use It)

The dict module remains in the standard library for three reasons:

1. Legacy code: Enormous amounts of Erlang code was written before maps existed. Removing or deprecating dict would break that code.

2. Compatibility: Some projects must remain compatible with older Erlang versions that predate maps.

3. It’s not actually broken: Dict works fine. It’s just superseded by something better.

For new code, use maps instead. Maps provide:

• Better syntax (native literal support)
• Pattern matching in function heads
• Better performance in many cases
• First-class support from the VM
• More modern APIs

The only reason to choose dict over maps for new code is if you’re working in a pre-R17 environment, in which case you have my sympathy and my respect for maintaining legacy systems.

Migrating from Dict to Maps

The transition is usually straightforward:

;; Dict style
(let ((d (dict:from_list '(#(a 1) #(b 2)))))
  (dict:fetch 'a d))

;; Maps style
(let ((m (maps:from_list '(#(a 1) #(b 2)))))
  (maps:get 'a m))

;; Or even simpler with map literals
(mref '#M(a 1 b 2) 'a)

Most dict operations have direct maps equivalents. The main challenges arise with code that relies on dict:fold behavior or specific iteration characteristics, but these are usually manageable.

gb_trees: The Balanced Perfectionist

In Which We Discover That Trees Solve Problems You Didn’t Know You Had

General Balanced Trees represent a different philosophy from both orddict and dict. Where orddict prioritizes transparency and dict prioritizes pragmatic hash-based lookup, gb_trees prioritize structural elegance and the specific capabilities that balanced binary trees provide. They implement Prof. Arne Andersson’s algorithm for maintaining balance, which sounds impressive at parties and actually is impressive when you need it.

The key insight of gb_trees is that balanced binary trees offer logarithmic lookup, insertion, and deletion, combined with efficient ordered traversal and the ability to quickly find minimum and maximum elements. If you need these specific capabilities—particularly the min/max operations—gb_trees might be your solution even in the age of maps.

The Nature of General Balanced Trees

A balanced binary tree maintains itself in a configuration where the height of the tree is proportional to log(n) of the number of elements. This means that as your dataset grows, the number of steps required to find any element grows very slowly. A tree with 1,000 elements requires about 10 steps to find anything. A tree with 1,000,000 elements requires about 20 steps.

The “general balanced” part means the algorithm makes certain trade-offs: deletions don’t trigger rebalancing (which is fine since deletions don’t increase height), and the balance condition has been slightly relaxed from the theoretical ideal to simplify implementation and improve practical performance.

Creating GB Trees: The Foundation

;; Empty tree
(set t1 (gb_trees:empty))
#(0 nil)  ; Size 0, no nodes

;; Insert elements
(set t2 (gb_trees:insert 'b 2 t1))
(set t3 (gb_trees:insert 'a 1 t2))
(set t4 (gb_trees:insert 'c 3 t3))

;; From a list
(set t5 (gb_trees:from_orddict '(#(a 1) #(b 2) #(c 3))))

Note the from_orddict function—it requires an ordered list of key-value tuples. If your list isn’t ordered, use orddict:from_list first to sort it.

The Dual-Mode API: Smart vs. Naive

GB Trees provide two sets of functions: “smart” functions that assume you know what you’re doing, and “naive” functions that check everything carefully. The smart functions are faster but will crash if your assumptions are wrong. The naive functions are safer but do extra work.

Smart Mode: For When You’re Certain

gb_trees:insert/3 - Insert a new key (crashes if key exists):

(let ((t (gb_trees:empty)))
  (gb_trees:insert 'a 1 t))
;; Success - key was new

(let ((t (gb_trees:insert 'a 1 (gb_trees:empty))))
  (gb_trees:insert 'a 2 t))
;; ** exception error: key exists

gb_trees:get/2 - Get a value (crashes if key missing):

(let ((t (gb_trees:from_orddict '(#(a 1) #(b 2)))))
  (gb_trees:get 'a t))
1

(let ((t (gb_trees:from_orddict '(#(a 1) #(b 2)))))
  (gb_trees:get 'z t))
;; ** exception error: key not found

gb_trees:update/3 - Update existing key (crashes if key missing):

(let ((t (gb_trees:from_orddict '(#(a 1) #(b 2)))))
  (gb_trees:update 'a 42 t))
;; Success - returns tree with a->42

(let ((t (gb_trees:from_orddict '(#(a 1) #(b 2)))))
  (gb_trees:update 'z 42 t))
;; ** exception error: key not found

gb_trees:delete/2 - Delete existing key (crashes if key missing):

(let ((t (gb_trees:from_orddict '(#(a 1) #(b 2)))))
  (gb_trees:delete 'a t))
;; Success - returns tree without 'a

(let ((t (gb_trees:from_orddict '(#(a 1) #(b 2)))))
  (gb_trees:delete 'z t))
;; ** exception error: key not found

Naive Mode: For When You’re Prudent

gb_trees:enter/3 - Insert or update:

(let ((t (gb_trees:empty)))
  (gb_trees:enter 'a 1 t))
;; Creates new entry

(let ((t (gb_trees:from_orddict '(#(a 1)))))
  (gb_trees:enter 'a 2 t))
;; Updates existing entry to a->2

gb_trees:lookup/2 - Safe lookup:

(let ((t (gb_trees:from_orddict '(#(a 1) #(b 2)))))
  (gb_trees:lookup 'a t))
#(value 1)

(let ((t (gb_trees:from_orddict '(#(a 1) #(b 2)))))
  (gb_trees:lookup 'z t))
none

gb_trees:delete_any/2 - Delete if present:

(let ((t (gb_trees:from_orddict '(#(a 1) #(b 2)))))
  (gb_trees:delete_any 'a t))
;; Deletes 'a

(let ((t (gb_trees:from_orddict '(#(a 1) #(b 2)))))
  (gb_trees:delete_any 'z t))
;; Does nothing, returns original tree

Which Mode Should You Use?

Use smart mode when:

• You’re in a hot path where performance matters
• You’ve just checked for the key’s existence
• The key’s presence/absence is guaranteed by program logic
• You want crashes to indicate programming errors

Use naive mode when:

• You’re unsure whether a key exists
• You’re prototyping or debugging
• Correctness trumps performance
• You prefer handling errors as data rather than exceptions

Ordered Operations: The Tree’s Special Powers

This is where gb_trees shine. Because they maintain a balanced tree structure, they can efficiently provide ordered access:

gb_trees:smallest/1 and gb_trees:largest/1

Get the minimum or maximum key-value pair:

(let ((t (gb_trees:from_orddict '(#(a 1) #(m 13) #(z 26)))))
  (gb_trees:smallest t))
#(a 1)

(let ((t (gb_trees:from_orddict '(#(a 1) #(m 13) #(z 26)))))
  (gb_trees:largest t))
#(z 26)

These are O(log n) operations—remarkably efficient even for large trees.

gb_trees:take_smallest/1 and gb_trees:take_largest/1

Get and remove the extreme elements:

(let ((t (gb_trees:from_orddict '(#(a 1) #(b 2) #(c 3)))))
  (gb_trees:take_smallest t))
#(a 1 #(2 #(b 2 nil nil) #(c 3 nil nil)))
;; Returns: {Key, Value, TreeWithoutThatKey}

(let ((t (gb_trees:from_orddict '(#(a 1) #(b 2) #(c 3)))))
  (gb_trees:take_largest t))
#(c 3 #(2 #(a 1 nil nil) #(b 2 nil nil)))

Perfect for implementing priority queues or processing elements in order.

gb_trees:smaller/2 and gb_trees:larger/2

Find the next smaller or larger key:

(let ((t (gb_trees:from_orddict '(#(a 1) #(m 13) #(z 26)))))
  (gb_trees:smaller 'm t))
#(a 1)

(let ((t (gb_trees:from_orddict '(#(a 1) #(m 13) #(z 26)))))
  (gb_trees:larger 'm t))
#(z 26)

;; Returns 'none' if no such key exists
(let ((t (gb_trees:from_orddict '(#(a 1) #(m 13) #(z 26)))))
  (gb_trees:smaller 'a t))
none

These operations are useful for range queries and ordered traversal.

Iteration: The Iterator Pattern

Unlike dict and orddict which provide fold, gb_trees use an iterator pattern:

(defun print-all (tree)
  (let ((iter (gb_trees:iterator tree)))
    (print-iter iter)))

(defun print-iter (iter)
  (case (gb_trees:next iter)
    ('none 'done)
    ((tuple key val next-iter)
     (progn
       (lfe_io:format "~p -> ~p~n" (list key val))
       (print-iter next-iter)))))

;; Usage
(let ((t (gb_trees:from_orddict '(#(a 1) #(b 2) #(c 3)))))
  (print-all t))
;; Prints:
;; a -> 1
;; b -> 2
;; c -> 3

The iterator approach gives you more control than fold, at the cost of requiring manual recursion. You can also create iterators that start at specific positions:

;; Start iteration from a specific key
(let* ((t (gb_trees:from_orddict '(#(a 1) #(b 2) #(c 3) #(d 4))))
       (iter (gb_trees:iterator_from 'b t)))
  ;; This iterator will produce b, c, d (but not a)
  (gb_trees:next iter))
#(b 2 #(iterator ...))

You can also specify iteration order (ascending or descending):

;; Iterate in reverse order
(let* ((t (gb_trees:from_orddict '(#(a 1) #(b 2) #(c 3))))
       (iter (gb_trees:iterator t 'reversed)))
  (gb_trees:next iter))
#(c 3 #(iterator ...))

Transformation

gb_trees:map/2

Transform all values:

(let ((t (gb_trees:from_orddict '(#(a 1) #(b 2) #(c 3)))))
  (gb_trees:map 
    (lambda (k v) (* v v))
    t))
;; Returns tree with squared values

Unfortunately, there’s no built-in filter or fold function. If you need those, you’ll need to implement them using iterators or convert to a list, process it, and convert back.

Utility Functions

gb_trees:size/1

(gb_trees:size (gb_trees:from_orddict '(#(a 1) #(b 2) #(c 3))))
3

This is O(1)—the size is stored in the tree structure.

gb_trees:is_empty/1

(gb_trees:is_empty (gb_trees:empty))
true

gb_trees:keys/1 and gb_trees:values/1

(let ((t (gb_trees:from_orddict '(#(a 1) #(b 2) #(c 3)))))
  (gb_trees:keys t))
(a b c)

(let ((t (gb_trees:from_orddict '(#(a 1) #(b 2) #(c 3)))))
  (gb_trees:values t))
(1 2 3)

Both return lists in ascending key order.

gb_trees:to_list/1

(gb_trees:to_list (gb_trees:from_orddict '(#(a 1) #(b 2) #(c 3))))
(#(a 1) #(b 2) #(c 3))

Returns the key-value pairs as a list of tuples, in ascending order.

gb_trees:balance/1

Rebalance a tree:

(let ((t (gb_trees:from_orddict '(#(a 1) #(b 2) #(c 3)))))
  ;; After many deletions...
  (gb_trees:balance t))

You rarely need this—trees maintain reasonable balance automatically. Use it only if you’ve done many deletions and want to optimize the tree structure.

When GB Trees Are The Right Choice

Choose gb_trees when:

1. You need min/max operations: The smallest and largest functions are killer features.

2. You need ordered iteration: Iterating through elements in key order is efficient.

3. You need range queries: The smaller and larger functions enable efficient range operations.

4. You know your access patterns: The smart/naive dichotomy lets you optimize hot paths.

5. You’re implementing tree-specific algorithms: Priority queues, ordered sets, and certain scheduling algorithms benefit from tree structure.

6. You need proven algorithmic guarantees: O(log n) operations across the board.

When To Choose Something Else

Avoid gb_trees when:

• You just need basic key-value storage (use maps)
• You need the simplest possible solution (use orddict for small datasets)
• Fold operations are central to your code (dict or maps are better)
• You don’t need ordered access
• The learning curve seems excessive for your use case

GB Trees vs. Maps: A Nuanced Comparison

Maps have largely superseded both dict and gb_trees for general key-value storage. However, gb_trees retain advantages in specific scenarios:

GB Trees win when you need:

• Efficient min/max key operations
• Ordered iteration without sorting
• Range queries
• Tree-specific algorithms

Maps win when you need:

• Simple, readable code
• Pattern matching in function heads
• Native syntax support
• Better integration with modern Erlang code
• General-purpose key-value storage

The performance characteristics are similar enough that algorithmic requirements, not raw speed, should drive the choice.

Choosing Your Key-Value Store: A Decision Framework

In Which We Synthesize Our Options Into Actionable Wisdom

Having examined three different approaches to the “I have keys, I have values, now what?” problem, let’s construct a decision tree:

For Small Datasets (< 75 pairs)

Use orddict if:

• You want transparency
• You’re debugging
• Human readability matters
• You need ordered output
• Simplicity trumps everything

Use maps if:

• You want modern syntax
• Pattern matching would be useful
• You’re writing new code
• Integration with other modern code matters

For Medium Datasets (75-10,000 pairs)

Use maps unless:

• You’re maintaining legacy code (then use what’s there)
• You specifically need gb_trees features

Use gb_trees if:

• You need min/max operations
• Ordered iteration is central to your algorithm
• You’re implementing a tree-based algorithm

Avoid:

• orddict (too slow)
• dict (deprecated by maps)

For Large Datasets (10,000+ pairs)

Use maps unless:

• You have specific needs only gb_trees fulfill
• You’re maintaining legacy dict code

Use gb_trees if:

• Min/max operations are critical
• Your algorithm fundamentally requires tree operations
• You’ve benchmarked and gb_trees is measurably better

Never use:

• orddict (will ruin your performance)
• dict (unless forced to by legacy constraints)

Special Considerations

Use dict only if:

• You’re maintaining pre-R17 code
• You must support Erlang versions without maps
• You’re gradually migrating to maps and need interim compatibility

Use gb_trees even in preference to maps when:

• Finding minimum/maximum keys is a frequent operation
• You need ordered traversal without separate sorting
• Your algorithm is naturally tree-based
• You’ve measured and gb_trees performs better for your specific use case

A Note On Proplists

The document you provided mentions proplists, which are worth a brief note here. A proplist is simply a list of tuples: '(#(key1 val1) #(key2 val2)). They’re not a formal data structure so much as a common pattern, particularly for passing options to functions.

Proplists are appropriate for:

• Function arguments with optional parameters
• Configuration that humans will write
• Very small datasets where the loose structure is acceptable
• Compatibility with code expecting this pattern

For anything more structured, use one of the modules discussed above.

Performance Isn’t Everything

A final philosophical point: the fastest data structure is not always the best choice. Consider:

• Maintainability: Code using orddict is often clearer than code using gb_trees
• Debuggability: Transparent structures are easier to inspect
• Integration: Matching your team’s conventions matters
• Future-proofing: Maps are the present and future; choose them when practical

Premature optimization remains, as ever, a source of evil. Choose the simplest structure that meets your needs, measure if performance matters, and optimize only when measurement proves it necessary.

In Conclusion: The Legacy Lives On

The three modules we’ve examined—orddict, dict, and gb_trees—represent different eras and philosophies in Erlang’s evolution. Orddict championed transparency. Dict championed pragmatic performance. GB_trees championed algorithmic elegance. Maps, the newcomer, have usurped most of their use cases but haven’t eliminated the need for any of them.

Understanding these modules gives you:

1. The ability to maintain legacy code
2. Options when maps don’t quite fit
3. Appreciation for how Erlang’s abstractions have evolved
4. Tools for those rare cases where their specific strengths matter

Use maps for new code, but know that orddict, dict, and gb_trees stand ready should you need them. They’re not deprecated, not obsolete, and not broken—merely specialized. In software as in life, having options is valuable even when you rarely exercise them.

The answer to the ultimate question of legacy key-value stores, the universe, and everything is not 42. It’s “it depends on your requirements, dataset size, and whether you need ordered access.”

Which is considerably less pithy, but more useful.

Graphs: Drawing Lines Between Things

In Which We Discover That Some Relationships Refuse to Be Flattened Into a List

“The History of every major Galactic Civilization tends to pass through three distinct and recognizable phases, those of Survival, Inquiry and Sophistication, otherwise known as the How, Why, and Where phases. For instance, the first phase is characterized by the question ‘How can we eat?’, the second by the question ‘Why do we eat?’ and the third by the question ‘Where shall we have lunch?’” — Douglas Adams

Sooner or later, dear reader, you will meet a problem that a list simply refuses to model honestly. You have a pile of things, and between those things run relationships — this module depends on that one, this airport has flights to those three, this task must finish before that task may begin. You could try to cram all of this into nested lists and tuples, and for a little while it would even work, in the way that balancing your accounts on the back of a napkin works right up until the moment it spectacularly doesn’t.

What you actually want is a graph: a set of vertices (the things) and a set of directed edges (the relationships between them). In the formal telling, a directed graph — a “digraph”, which is the word we shall use henceforth because typing “directed graph” several hundred times is its own small tragedy — is a pair (V, E) where V is a finite set of vertices and E is a set of edges, each edge being an ordered pair of vertices drawn from V. An edge (v, w) emanates from v and is incident on w, which is mathematician for “there’s an arrow pointing from v to w.”

In LFE, you do not build this machinery yourself. Erlang’s standard library hands you two modules, fully assembled and quietly battle-tested:

• digraph — constructs and modifies graphs: adding and deleting vertices and edges, attaching labels, finding paths and cycles, asking how many of each thing you have.
• digraph_utils — the power tools: topological sorting, reachability, connected components, cycle detection, and the other algorithms you half-remember from a course you have spent the intervening years doing your level best to forget.

Before we go a single step further, one fact deserves to be stated plainly, in good light, while everyone is paying attention: a digraph is mutable. It is the one data structure in this entire corner of the standard library that does not play by the immutable rules every other chapter has so lovingly drilled into you. A digraph is implemented with ETS tables, which means the thing you hold is a reference to shared, mutable state rather than a value you can pass around with a clear conscience. We will devote an entire, slightly scandalised section to this (The Scandalous Truth: Graphs Mutate); for now, simply file it away somewhere you will not lose it.

The rest of the chapter proceeds in the natural order: how to conjure a graph, how to populate it with vertices and edges, how to interrogate it for paths and cycles, and how to reach for the power tools when you need a real algorithm. We finish with a worked example — working out build order from a tangle of dependencies, which is the single most common reason a working programmer ever touches this module — and some honest counsel on when a graph is worth the trouble and when a humble map would have done.

Right, then. Shall we begin?

Conjuring a Graph

Every graph begins life empty, which is to say as the empty digraph — a perfectly respectable graph with no vertices and no edges, much like a guest list before anyone has been invited. You summon one with digraph:new/0:

lfe> (set g (digraph:new))
#(digraph #Ref<0.1.2.3> #Ref<0.1.2.4> #Ref<0.1.2.5> true)

That alarming-looking return value is the graph itself — or rather, your handle to it. Those #Ref<...> entries are the identifiers of the ETS tables in which your graph quietly lives. You are not meant to read this, poke at it, or take it apart; it is an opaque term, and the polite thing to do is to bind it to a name (here, g) and pass that name to every other digraph function. Treat the internals as you would the wiring behind a wall socket: present, essential, and absolutely none of your business.

Choosing the Graph’s Character: new/1

If you want a graph with opinions, hand digraph:new/1 a list of option atoms. There are two independent choices to make, each a pair of mutually exclusive moods:

Cyclicity — may the graph contain cycles?

• cyclic — cycles are permitted. This is the default.
• acyclic — the graph is to be kept free of cycles. Any edge you try to add that would close a loop is refused at the door (more on that refusal in Edges).

Protection — who is allowed to touch it?

• protected — the creating process may modify the graph; other processes may read it. This is the default.
• private — only the creating process may read or modify the graph; everyone else is locked out entirely.

So a graph meant to model build dependencies — where a cycle is not a quirk but a catastrophe — is best conjured acyclic:

lfe> (set deps (digraph:new '(acyclic)))
#(digraph #Ref<0.1.2.6> #Ref<0.1.2.7> #Ref<0.1.2.8> false)

You may combine one option from each pair; order does not matter:

lfe> (set g (digraph:new '(acyclic private)))

Hand new/1 an option it does not recognise, or something that is not a proper list, and it will raise a badarg exception rather than guess at what you meant — a small kindness, since a graph silently created with the wrong cyclicity is exactly the sort of thing that ruins a Tuesday.

Taking the Graph’s Pulse: info/1

To ask a graph about itself, use digraph:info/1, which returns a list of #(Tag Value) pairs:

lfe> (digraph:info deps)
(#(cyclicity acyclic) #(memory 950) #(protection protected))

You get back its cyclicity (cyclic or acyclic), its protection (protected or private), and its memory — the number of words allocated to its underlying ETS tables. That last figure is a genuine, load-bearing detail: a digraph occupies real memory in real tables, and unlike the immutable values elsewhere in this book, it will not quietly vanish when you stop looking at it.

The Most Important Function in This Chapter: delete/1

Which brings us, with indecent haste for a section titled “creating things”, to destruction. When you are finished with a graph, you must dispose of it yourself:

lfe> (digraph:delete g)
true

digraph:delete/1 frees the ETS tables backing the graph. This is not optional housekeeping you may get around to eventually; it is the counterpart to new, and forgetting it is the canonical way to leak memory with this module. There is no garbage collector coming to tidy up after you — the graph is not an immutable term that becomes unreachable and gets swept away by the cosmic janitor. The one mercy is that the tables are also released automatically if the process that created the graph terminates, so a short-lived worker process that builds a graph, uses it, and dies will not haunt you. A long-lived process that builds graphs in a loop and forgets to delete them, on the other hand, absolutely will.

We will return to why all of this is true — the ETS tables, the ownership, the conspicuous absence of garbage collection — in The Scandalous Truth: Graphs Mutate. For now: conjure with new, inspect with info, and always, always remember to delete.

Vertices: The Things Themselves

A graph without vertices is a philosophical curiosity; a graph with vertices is a tool. Vertices are the things your graph is about — the modules, the airports, the tasks, the characters in a tediously complicated space opera. In a digraph, a vertex can be any Erlang term: an atom, a tuple, a binary, an integer, whatever names your thing most naturally. The graph asks only that each vertex be unique.

Adding Vertices: add_vertex/1,2,3

The everyday form is digraph:add_vertex/2, which adds a vertex you name yourself and returns it:

lfe> (set g (digraph:new))
lfe> (digraph:add_vertex g 'zaphod)
zaphod
lfe> (digraph:add_vertex g 'ford)
ford
lfe> (digraph:add_vertex g 'arthur)
arthur

Each vertex here is an atom, which is the most common and most readable choice. The return value is simply the vertex you handed in, which feels redundant until you meet the labelled and anonymous forms below.

You may also attach a label — an arbitrary term carrying ancillary information about the vertex — with add_vertex/3:

lfe> (digraph:add_vertex g 'marvin "Paranoid Android, second-hand")
marvin

The label is not the vertex’s identity. The vertex is marvin; the string is merely a note pinned to it. Two crucial consequences follow. First, you still refer to the vertex by 'marvin, never by its label. Second — and this surprises people — add_vertex/3 on an existing vertex does not create a duplicate; it modifies the label of the vertex already there. Adding 'marvin twice gives you one Marvin, the second call quietly overwriting the first’s label. This is your first taste of mutation, served early so it doesn’t ambush you later.

The third form, add_vertex/1, takes no name at all and invents one for you, returning an opaque auto-generated identifier — internally a cons of a reserved atom and an integer, which the REPL prints in all its unlovely glory:

lfe> (digraph:add_vertex g)
(|$v| . 0)

These anonymous vertices have their uses, but in practice you almost always want to name your own — a graph of 'zaphod and 'ford is infinitely easier to debug six months later than a graph of machine-generated gensyms. We mention add_vertex/1 chiefly so you recognise it when it turns up in someone else’s code, presumably written under deadline.

Looking Vertices Up: vertex/2 and vertices/1

To retrieve a single vertex and its label, use digraph:vertex/2, which returns a #(Vertex Label) tuple — or the atom false if no such vertex exists:

lfe> (digraph:vertex g 'marvin)
#(marvin "Paranoid Android, second-hand")
lfe> (digraph:vertex g 'zaphod)
#(zaphod ())
lfe> (digraph:vertex g 'trillian)
false

Note that 'zaphod, added without a label, reports the empty list () as its label — the default. Note also that a missing vertex yields false rather than an exception, which makes vertex/2 a perfectly civil way to test for membership.

To get all the vertices, use digraph:vertices/1:

lfe> (digraph:vertices g)
(arthur zaphod marvin ford)

A word of caution that applies throughout this module: the order is unspecified. The list above comes back in an order that is neither insertion order nor anything else you could usefully predict, and the documentation promises nothing of the sort, so you must never write code that depends on the ordering of vertices/1. If you need an order, impose one yourself — or reach for topological sorting, which exists precisely to hand you a meaningful order when the graph’s structure implies one.

To count vertices without dragging the whole list out, use digraph:no_vertices/1:

lfe> (digraph:no_vertices g)
4

Deleting Vertices: del_vertex/2 and del_vertices/2

Removing a vertex is the work of digraph:del_vertex/2:

lfe> (digraph:del_vertex g 'arthur)
true

This does more than it might appear. Deleting a vertex also deletes every edge emanating from or incident on it — which is exactly what you want, since an edge to a vertex that no longer exists would be a dangling reference and a small lie. Poor Arthur departs, and every flight to and from Arthur departs with him.

To remove several vertices at once, hand digraph:del_vertices/2 a list:

lfe> (digraph:del_vertices g '(ford marvin))
true

Counting the Neighbourhood: Degrees and Neighbours

Once edges enter the picture (the next section), each vertex acquires a neighbourhood, and the digraph module offers a small family of functions for surveying it. Suppose a fresh graph in which zaphod points at both ford and trillian, and ford in turn points at trillian. The out-degree of a vertex is the number of edges emanating from it; the in-degree is the number incident on it:

lfe> (digraph:out_degree g 'zaphod)
2
lfe> (digraph:in_degree g 'zaphod)
0

Zaphod points at two things and is pointed at by none — the out-degree is two, the in-degree zero. To get the neighbours themselves rather than merely count them, there are out_neighbours/2 (the vertices your vertex points at) and in_neighbours/2 (the vertices that point at your vertex):

lfe> (digraph:out_neighbours g 'zaphod)
(trillian ford)
lfe> (digraph:in_neighbours g 'trillian)
(ford zaphod)

These, too, come back in unspecified order, as is by now the house custom. With the things themselves accounted for, we may turn to the lines between them.

Edges: The Lines Between

If vertices are the nouns of a graph, edges are the verbs. An edge is a directed connection from one vertex to another — it emanates from V1 and is incident on V2, which is to say it is an arrow with a tail at V1 and a head at V2. Direction matters: an edge from 'zaphod to 'ford is emphatically not the same as an edge from 'ford to 'zaphod, a distinction that anyone who has confused “depends on” with “is depended on by” will have learned the hard way.

Adding Edges: add_edge/3,4,5

The workhorse is digraph:add_edge/3. You give it the graph and two vertices — tail first, then head — and it returns an opaque edge identifier:

lfe> (set g (digraph:new))
lfe> (digraph:add_vertex g 'zaphod)
lfe> (digraph:add_vertex g 'ford)
lfe> (digraph:add_edge g 'zaphod 'ford)
(|$e| . 0)

That (|$e| . 0) is the edge’s identity — its name — handed back so you can refer to the edge later. Like the auto-generated vertex ids of the previous section, it is opaque and not for human consumption; bind it to a name if you need it and otherwise let it be.

As with vertices, an edge may carry a label — arbitrary ancillary information — via add_edge/4:

lfe> (digraph:add_edge g 'zaphod 'ford "owes drinks")
(|$e| . 1)

It is worth being precise here, because the documentation is precise: an edge’s identifier and an edge’s label are different things. The identifier names the edge; the label describes it. You can have two distinct edges between the same pair of vertices, each with its own identifier — multiple edges between vertices are explicitly allowed, which is what makes a digraph a non-proper directed graph in the formal sense, and what makes it able to model “two flights from Heathrow to Betelgeuse” without complaint.

If you want to name the edge yourself, or modify an existing edge, reach for the five-argument add_edge/5, which takes the graph, the edge identifier, the two vertices, and the label:

lfe> (digraph:add_edge g 'flight-42 'zaphod 'ford "first class")
flight-42

Calling add_edge/5 with an identifier that already exists modifies that edge rather than creating a second one — the same mutate-in-place behaviour we met with vertex labels — provided you keep the same endpoints. Reuse an existing edge identifier to connect a different pair of vertices and digraph refuses, returning #(error #(bad_edge (V1 V2))) rather than silently rewiring the edge under you. An edge id may have its label changed, in other words, but not its destination.

When an Edge Is Refused

add_edge does not always succeed, and its failures are informative rather than merely annoying. Instead of an edge identifier it may return an error tuple:

• #(error #(bad_vertex V)) — you named a vertex V (either the tail or the head) that is not in the graph. The digraph module will not invent vertices for you; add them first.
• #(error #(bad_edge Path)) — adding this edge would create a cycle in an acyclic graph. Recall from Conjuring a Graph that a graph made with the acyclic option refuses to close a loop, and here is where it makes good on that promise. The Path it hands back is the cycle your edge would have completed, which is enormously helpful for working out which dependency you got backwards.

lfe> (set dag (digraph:new '(acyclic)))
lfe> (digraph:add_vertex dag 'a)
lfe> (digraph:add_vertex dag 'b)
lfe> (digraph:add_edge dag 'a 'b)
(|$e| . 0)
lfe> (digraph:add_edge dag 'b 'a)
#(error #(bad_edge (a b)))

The acyclic graph has just saved you from a circular dependency at the moment you tried to introduce it, rather than three hours later when a topological sort mysteriously returns false. This is the kind of error you should be grateful for.

Inspecting Edges: edge/2, edges/1, edges/2

To look up a single edge by its identifier, use digraph:edge/2, which returns a four-tuple #(E V1 V2 Label) — the edge id, its tail, its head, and its label — or false if no such edge exists:

lfe> (digraph:edge g 'flight-42)
#(flight-42 zaphod ford "first class")

That four-tuple is the canonical, complete description of an edge: this is edge flight-42, it runs from zaphod to ford, and it is labelled "first class". Commit its shape to memory; you will pattern-match on it constantly.

For all the edges in the graph, use digraph:edges/1; for just the edges touching a particular vertex (emanating from or incident on it), use digraph:edges/2:

lfe> (digraph:edges g)
((|$e| . 0) (|$e| . 1) flight-42)
lfe> (digraph:edges g 'zaphod)
((|$e| . 0) (|$e| . 1) flight-42)

Both return edge identifiers, in the usual unspecified order, leaving you to call edge/2 on each if you want the full picture. When you care about direction, the finer-grained out_edges/2 (edges emanating from a vertex) and in_edges/2 (edges incident on it) let you ask precisely which way the arrows point.

To count edges without materialising the list, there is digraph:no_edges/1:

lfe> (digraph:no_edges g)
3

Deleting Edges: del_edge/2 and del_edges/2

Removing an edge is the work of digraph:del_edge/2, which takes the edge’s identifier:

lfe> (digraph:del_edge g 'flight-42)
true

Deleting an edge removes only the edge — the vertices it connected carry on, unbothered, possibly relieved. (Contrast this with deleting a vertex, which takes all its edges down with it.) To remove several edges at once, hand digraph:del_edges/2 a list of identifiers:

lfe> (digraph:del_edges g (digraph:edges g 'zaphod))
true

With vertices placed and edges drawn between them, you have a genuine graph — a structure with shape, direction, and the capacity to be asked interesting questions. The most interesting of those questions are about paths, and that is where we turn next.

Paths, Cycles, and Getting From Here to There

A path is the graph-theoretic answer to the question every traveller eventually asks: can I get there from here, and if so, how? Formally, a path from v[1] to v[k] is a non-empty sequence of vertices in which each consecutive pair is joined by an edge pointing the right way. The path’s length is the number of edges it uses — one fewer than the number of vertices, rather like the way a row of five fence posts has only four gaps between them, a discrepancy that has tripped up generations of programmers and at least one of the authors. A path is simple if no vertex repeats (except that the first and last may coincide), and a path that begins and ends at the same vertex with non-zero length is a cycle. A cycle of length one — a vertex with an edge to itself — is a loop.

The digraph module will find these things for you, so you need never write a depth-first search by hand again unless you genuinely want to, which some evenings one does.

Finding a Path: get_path/3

digraph:get_path/3 tries to find a simple path from V1 to V2. It returns the path as a list (V1 ... V2) of vertices, or false if no such path of length one or more exists. Consider a small map of the galaxy:

lfe> (set g (digraph:new))
lfe> (lc ((<- v '(earth magrathea betelgeuse ursa-minor)))
       (digraph:add_vertex g v))
(earth magrathea betelgeuse ursa-minor)
lfe> (digraph:add_edge g 'earth 'betelgeuse)
lfe> (digraph:add_edge g 'betelgeuse 'magrathea)
lfe> (digraph:add_edge g 'magrathea 'ursa-minor)
lfe> (digraph:get_path g 'earth 'ursa-minor)
(earth betelgeuse magrathea ursa-minor)

There is your itinerary: Earth to Betelgeuse to Magrathea to Ursa Minor, every leg an edge in the right direction. Ask for a journey that the edges do not permit, and you get an honest false:

lfe> (digraph:get_path g 'ursa-minor 'earth)
false

The arrows point one way; there is no flight home. A small caution worth internalising: get_path/3 traverses the graph depth-first and returns the first path it finds, which is not promised to be the shortest. If you simply need to know whether a route exists, get_path/3 is your function. If you need the best route, read on.

Finding the Shortest Path: get_short_path/3

When length matters, digraph:get_short_path/3 traverses the graph breadth-first and returns an as-short-as-possible simple path:

lfe> (digraph:add_edge g 'earth 'magrathea)
lfe> (digraph:get_short_path g 'earth 'ursa-minor)
(earth magrathea ursa-minor)

With a direct Earth-to-Magrathea edge now in place, the shortest route skips Betelgeuse entirely. Same start, same destination, fewer stops — the difference between get_path and get_short_path is precisely the difference between “a way” and “the short way”, and choosing the wrong one is how you end up routing a packet across three extra hops for no reason anyone can later reconstruct.

Finding Cycles: get_cycle/2 and get_short_cycle/2

To discover whether a vertex sits on a cycle, use digraph:get_cycle/2. If a simple cycle of length two or more passes through V, you get it back as a list (V ... V); if V has a loop (an edge to itself), you get (V); and if no cycle through V exists, you get false:

lfe> (set h (digraph:new))
lfe> (lc ((<- v '(a b c)))
       (digraph:add_vertex h v))
(a b c)
lfe> (digraph:add_edge h 'a 'b)
lfe> (digraph:add_edge h 'b 'c)
lfe> (digraph:add_edge h 'c 'a)
lfe> (digraph:get_cycle h 'a)
(a b c a)

The cycle a → b → c → a is reported beginning and ending at the vertex you asked about. Its companion digraph:get_short_cycle/2 does the same job but tries to find as short a cycle as possible — and note one small wrinkle the documentation is careful to spell out: a loop through V comes back from get_short_cycle/2 as the two-element list (V V), not the single-element (V) that get_cycle/2 would give. A pedantic distinction, perhaps, but the kind that matters when you are pattern-matching the result.

Severing Connections: del_path/3

Finally, the demolition crew. digraph:del_path/3 deletes edges from the graph until no path remains from V1 to V2:

lfe> (digraph:del_path g 'earth 'ursa-minor)
true
lfe> (digraph:get_path g 'earth 'ursa-minor)
false

This is more thorough than deleting a single edge. The procedure finds a path, removes every edge along it, and repeats until the two vertices are well and truly disconnected — useful when you need to guarantee that one part of a system can no longer reach another, and merrily destructive in exactly the way that demands you remember a digraph mutates in place. Which, as ever, brings us back to the central scandal of this chapter — but first, let us meet the power tools.

The Power Tools: digraph_utils

The digraph module builds and modifies graphs; the digraph_utils module reasons about them. Where digraph lets you add an edge, digraph_utils lets you ask whether the whole tangled mess can be put in order, which parts of it are mutually reachable, and whether it secretly contains a cycle you would very much rather it didn’t. Everything here is built on depth-first traversal, and everything here takes a graph you have already built with digraph and hands back an answer — usually a plain list of vertices, which is a refreshing change from the opaque handles of the previous sections.

Topological Sorting: topsort/1

This is the function that justifies the entire module for most working programmers. A topological sort of a graph is a linear ordering of its vertices in which, for every vertex, none of its out-neighbours appears earlier in the list. Put less formally: if a points to b, then a comes before b. This is exactly the order in which you must perform tasks when some tasks depend on others — build order, load order, the order in which to wake up the galaxy’s subsystems so that nothing boots before the thing it relies on.

lfe> (set g (digraph:new '(acyclic)))
lfe> (lc ((<- v '(power core drive hull ship)))
       (digraph:add_vertex g v))
(power core drive hull ship)
lfe> (digraph:add_edge g 'power 'core)
lfe> (digraph:add_edge g 'core 'drive)
lfe> (digraph:add_edge g 'drive 'ship)
lfe> (digraph:add_edge g 'hull 'ship)
lfe> (digraph_utils:topsort g)
(hull power core drive ship)

Every prerequisite precedes the thing that needs it: power before core, core before drive, everything before ship. Note that the ordering is not unique — hull and power have no dependency between them, so a different valid run could just as legitimately swap them. topsort/1 promises a valid order, not the valid order; never write code that depends on which one you got.

And the crucial failure case: if the graph contains a cycle, no topological ordering can exist, and topsort/1 returns false:

lfe> (set bad (digraph:new))
lfe> (lc ((<- v '(chicken egg)))
       (digraph:add_vertex bad v))
(chicken egg)
lfe> (digraph:add_edge bad 'chicken 'egg)
lfe> (digraph:add_edge bad 'egg 'chicken)
lfe> (digraph_utils:topsort bad)
false

A false from topsort/1 is the standard library’s polite way of telling you that you have a circular dependency. (Notice we had to build bad as a cyclic graph for the cycle to get in at all — an acyclic graph would have refused the second edge outright, per Edges. The two mechanisms are complementary: build acyclic to prevent cycles, or build cyclic and topsort to detect them.)

Detecting Cycles: is_acyclic/1

If all you want is a yes-or-no answer, digraph_utils:is_acyclic/1 returns true if and only if the graph has no cycles:

lfe> (digraph_utils:is_acyclic g)
true
lfe> (digraph_utils:is_acyclic bad)
false

Reachability: reachable/2 and reaching/2

The reachability functions answer “what can I get to?” and its mirror image “what can get to me?”. digraph_utils:reachable/2 takes a list of starting vertices and a graph, and returns every vertex reachable from any of them by a path of length zero or more — which, the “length zero” being the operative phrase, means the starting vertices are themselves always included:

lfe> (digraph_utils:reachable '(core) g)
(ship drive core)

From core you can reach drive, then ship, and core counts as reaching itself. Its mirror, digraph_utils:reaching/2, returns every vertex that can reach the given ones — following the arrows backwards:

lfe> (digraph_utils:reaching '(ship) g)
(power core drive hull ship)

Everything that must exist before ship can: the entire ancestry. These two functions are the backbone of impact analysis — “if I change this module, what is affected?” is reaching; “what does this module pull in?” is reachable.

Each has a _neighbours variant — reachable_neighbours/2 and reaching_neighbours/2 — that demands a path of length one or more, thereby excluding any starting vertex that cannot get back to itself. The practical effect is that a starting vertex only appears in the result if it sits on a cycle, which makes the _neighbours forms a slightly oblique way of asking “is this vertex part of a loop?”

Connected Components: components/1 and strong_components/1

A connected component is a maximal clump of vertices all reachable from one another if you ignore edge direction. digraph_utils:components/1 carves the graph into these clumps, each represented by its list of vertices, every vertex appearing in exactly one:

lfe> (digraph_utils:components g)
((hull ship drive core power))

Our ship is all one piece, so it forms a single component. A strongly connected component, by contrast, respects direction: it is a maximal set of vertices in which there is a directed path from each to every other. digraph_utils:strong_components/1 finds these, and digraph_utils:cyclic_strong_components/1 returns only the ones that actually contain a cycle (discarding the lonely single-vertex components that are technically “strongly connected” only in the trivial sense of a vertex reaching itself):

lfe> (digraph_utils:strong_components bad)
((egg chicken))
lfe> (digraph_utils:cyclic_strong_components bad)
((egg chicken))

The chicken and the egg, correctly identified as inseparable.

Traversal Orders: preorder/1 and postorder/1

For when you want the vertices in depth-first-traversal order rather than topological order, digraph_utils:preorder/1 returns them as collected on the way down (a vertex before its descendants), and digraph_utils:postorder/1 as collected on the way back up (a vertex after its descendants). Both visit every vertex exactly once.

The Specialist Tools

The remainder of the module is a drawer of specialist instruments you will reach for rarely but be glad exist when you do:

• is_tree/1 — true if the graph is a tree (acyclic, with a unique undirected path between every pair of vertices).
• is_arborescence/1 and arborescence_root/1 — the former tests whether the graph is an arborescence (an acyclic graph with a root from which there is a unique directed path to every other vertex); the latter returns #(yes Root) or no. An arborescence is, essentially, a tree that has made up its mind about which way is down.
• loop_vertices/1 — the list of all vertices that sit on some loop.
• subgraph/2,3 — builds a brand-new graph containing only the vertices you name and the edges among them. Note well: this creates a new digraph, with its own ETS tables, which means it is your responsibility to delete when you are done. The power tools are not exempt from the chapter’s central obligation.
• condensation/1 — collapses each strongly connected component down to a single vertex, yielding a new (and necessarily acyclic) graph that shows the large-scale shape of a cyclic one. Also a fresh digraph; also yours to delete.

These last two are the first hint of the recurring caveat we keep deferring — that these graphs are mutable, ETS-backed, reference-typed things that allocate real tables and demand real cleanup. We have deferred it long enough. It is time to face the scandal directly.

The Scandalous Truth: Graphs Mutate

Here digraph does something faintly scandalous for an Erlang data structure: it mutates. Where tuples and maps cling to immutability like a cherished principle, a digraph is secretly a cluster of ETS tables wearing a trench coat — a reference to shared, mutable state rather than a value you can pass around with a clear conscience. Operations change the graph in place, which is enormously convenient and exactly the sort of thing that will surprise you at 3 a.m. if you forget it. Pass a digraph to two processes expecting two independent copies and you will get one graph and two confused processes.

Everything else in this Part has trained you to expect a particular bargain: a function takes a value and returns a new value, the original sitting serenely unchanged, immutable and eternal. setelement doesn’t set; maps:put doesn’t mutate; the old version is always still there if you kept a name bound to it. This bargain is the bedrock of Erlang’s concurrency story, and digraph quietly breaks every clause of it. Understanding how it breaks them is the difference between using graphs confidently and being ambushed by them.

A Digraph Is a Reference, Not a Value

When you write (set g (digraph:new)), the variable g does not hold a graph. It holds a handle — a small tuple of references to ETS tables where the actual vertices and edges live. Every digraph function you call reaches through that handle and modifies the shared tables behind it. There is no copy. There is no new version. There is one graph, and g is a finger pointing at it.

The consequences ripple outward immediately. Consider:

lfe> (set g (digraph:new))
lfe> (digraph:add_vertex g 'marvin)
marvin
lfe> (set h g)
lfe> (digraph:add_vertex h 'zaphod)
zaphod
lfe> (digraph:vertices g)
(marvin zaphod)

Binding h to g did not copy the graph; it made a second finger pointing at the same graph. Adding a vertex “through h” adds it to the one and only graph, and g sees it too, because g and h were always the same graph wearing two names. Every other data structure in this book would have given you two independent values here. The digraph gives you one, and a lesson.

Only the Owner May Write

Because the tables are ETS tables, the ordinary ETS ownership rules apply, and they are stricter than you might guess. Only the process that created the digraph is allowed to update it. Another process may read a protected graph (the default) — call vertices, get_path, topsort to its heart’s content — but any attempt to modify it from a process other than its creator will fail. A private graph (see Conjuring a Graph) goes further and forbids even reading from the outside.

This has a sharp practical edge: a digraph is not a thing you casually send in a message to another process and expect to keep working with. You can send the handle — it is just a term — but the receiving process cannot modify the graph, and if the creating process dies, the graph dies with it (more on that below), leaving the recipient holding a handle to a graph that has ceased to be. If you need a graph shared across processes, you build it in one long-lived owner process and have that process mediate all changes. The graph does not parallelise the way an immutable value does, and pretending otherwise is a fine recipe for a genuinely baffling bug.

No Garbage Collection — You Must delete

Here is the part that catches people. Every immutable value you have ever created in Erlang gets cleaned up automatically: when the last reference goes away, the cosmic janitor that is the garbage collector sweeps in and reclaims the memory. ETS tables are different. Digraphs are not garbage collected. The tables backing a graph persist until one of exactly two things happens:

1. You call digraph:delete/1 on the graph, explicitly freeing its tables; or
2. The process that created the graph terminates, at which point its ETS tables are released along with everything else it owned.

lfe> (set g (digraph:new))
lfe> ;; ... build, query, use the graph ...
lfe> (digraph:delete g)
true

In a short-lived worker — a process spawned to do one job, build a graph along the way, and then exit — point (2) saves you: the graph is cleaned up when the worker dies, and you can be a little careless. But in a long-lived process — a gen_server that handles request after request, building a throwaway graph for each — point (2) never arrives, and a forgotten delete/1 means every request leaks a fistful of ETS tables. Do this a few hundred thousand times and you will meet the bug at 3 a.m., cold coffee in hand, watching memory climb for reasons that the immutable rest of your codebase has left you wholly unprepared to suspect.

The rule, then, is simple and unforgiving: if you created it, you delete it — unless you are certain the owning process is about to die anyway. Treat digraph:new and digraph:delete the way a careful C programmer treats malloc and free, because under the trench coat that is very nearly what they are.

Making Peace With It

None of this is a defect. The mutability is why digraphs are fast and convenient for the graph algorithms in the power tools: building a ten-thousand-vertex graph by threading an immutable structure through ten thousand function calls would be a misery of allocation, whereas mutating ETS tables in place is brisk and pleasant. The standard library made a deliberate, sensible trade — performance and convenience in exchange for surrendering immutability — and it was the right call for the job. It simply asks one thing of you in return: that you remember the trade exists. Hold the handle, respect the ownership, delete when you’re done, and the digraph will serve you well. Forget, and the trench coat comes off at the worst possible moment.

A Worked Example: Build Order from Dependencies

Theory is all very well, but let us now do the single most common thing a working programmer ever uses a graph for: working out what order to build things in when some of them depend on others. Compilers do this with modules, package managers do it with libraries, build tools do it with targets, and at least one spacecraft does it with subsystems. The underlying question is always the same — given a tangle of “X needs Y first” relationships, in what order may I proceed? — and the answer is always the same: a topological sort.

The Problem

Suppose we are assembling the Heart of Gold from a parts list, where each component declares the components it depends on:

• ship needs drive and hull
• drive needs core
• core needs power
• hull needs nothing
• power needs nothing

We want a build order in which nothing is assembled before the things it depends on. By eye, on a list this small, you can squint and work it out. With four hundred components and a deadline, you want the machine to do it — and you want the machine to notice if someone has accidentally declared a circular dependency, because a build order for a circular dependency does not exist and no amount of squinting will conjure one.

The Module

(defmodule build-order
  (export (resolve 1)))

;; A spec is #(component (dependency ...)).
;; resolve/1 returns #(ok Order), with every dependency appearing
;; before the component that needs it, or #(error cyclic) if the
;; specs contain a circular dependency that no build order can satisfy.
(defun resolve (specs)
  (let ((g (digraph:new)))
    (try
      (progn
        ;; Model each "component needs dependency" as an edge pointing
        ;; FROM the dependency TO the component — "dependency first".
        ;; add-edge needs both endpoints to exist, so we add them too;
        ;; add_vertex is idempotent for our label-free purposes.
        (lc ((<- spec specs))
          (let ((component (element 1 spec))
                (deps (element 2 spec)))
            (digraph:add_vertex g component)
            (lc ((<- dep deps))
              (progn
                (digraph:add_vertex g dep)
                (digraph:add_edge g dep component)))))
        ;; topsort hands back a valid order, or 'false for a cycle.
        (case (digraph_utils:topsort g)
          ('false #(error cyclic))
          (order (tuple 'ok order))))
      ;; Whatever happens — success, cycle, or exception — free the
      ;; ETS tables. A digraph is not garbage collected; see the
      ;; Scandalous Truth. The `after` clause is how we keep that promise.
      (after
        (digraph:delete g)))))

Two details earn their keep here. The first is the edge direction: an edge runs from a dependency to the component that needs it, so that topsort/1 — which guarantees no out-neighbour appears earlier than the vertex it points from — places every dependency before its dependent. Get this backwards and you will get a perfectly valid topological sort of a perfectly backwards graph, which is the kind of bug that survives code review because everything looks right.

The second is the try ... (after ...) wrapper. Because a digraph mutates and lives in ETS tables that no garbage collector will reclaim (The Scandalous Truth), the after clause guarantees we call digraph:delete/1 on every exit path — normal return, cycle, or unexpected exception. This is the idiomatic shape for any function that builds a temporary graph: create, use, and unconditionally delete.

Running It

lfe> (c "build-order.lfe")
#(module build-order)
lfe> (build-order:resolve
       '(#(ship  (drive hull))
         #(drive (core))
         #(core  (power))
         #(hull  ())
         #(power ())))
#(ok (hull power core drive ship))

There is our build order: hull and power first (they depend on nothing), then core (which needed power), then drive (which needed core), and finally ship (which needed both drive and hull). Every part arrives after its prerequisites. As ever, the order is a valid one rather than the valid one — power could just as correctly have come first, or hull later — so resist the urge to write a test asserting this exact list.

And now the part that makes it worth automating. Suppose a weary engineer declares, late in the day, that the chicken depends on the egg and the egg depends on the chicken:

lfe> (build-order:resolve
       '(#(chicken (egg))
         #(egg     (chicken))))
#(error cyclic)

No build order exists, and resolve/1 says so plainly rather than looping forever or returning nonsense. topsort/1 returned false, we turned that into a tidy #(error cyclic), and the after clause cleaned up the graph regardless. Forty lines of LFE, and you have a dependency resolver that a great many real build systems would recognise as a close cousin — which is rather the point of having digraph in the standard library at all.

When to Reach for a Graph

Every chapter in this Part has ended by comparing its subject to the alternatives, and graphs are no exception — though the comparison here is unusual, because the chief alternative to a digraph is not another library. It is a map. A great many problems that look like they need a graph are served perfectly well by a humble #M(...) of vertex-to-neighbours, with none of the ETS tables, none of the mutability, and none of the obligation to remember delete/1. The honest question is therefore not “is this a graph problem?” — almost everything with relationships is, technically — but “is this a graph problem big and algorithmic enough to be worth the trench coat?”

The Hand-Rolled Alternative

Before reaching for digraph, consider what a map already gives you. An adjacency map is just a map from each vertex to its list of out-neighbours:

lfe> (set graph #M(earth (betelgeuse magrathea)
                   betelgeuse (magrathea)
                   magrathea (ursa-minor)))
#M(betelgeuse (magrathea) earth (betelgeuse magrathea) magrathea (ursa-minor))

This is an immutable value. You can pass it between processes freely, store old versions, pattern-match on it, and let the garbage collector tidy it away when you are done — every comfort the rest of this book has taught you to expect, and every one of which digraph quietly revokes. For representing a graph, for traversing it with your own recursion, for storing it in a gen_server’s state, the map is very often the better-behaved choice. It is the data-structure equivalent of taking the stairs: unglamorous, but you always know exactly where you stand.

Use a digraph when:

• You need the algorithms, not just the structure. Topological sort, strongly connected components, shortest paths, cycle detection — these are real, fiddly algorithms, and digraph_utils has already written, debugged, and optimised them for you. Re-implementing topological sort over an adjacency map to avoid one ETS table is a poor trade; this is precisely the work the module exists to spare you.
• The graph is large and you build it incrementally. Threading an immutable map through ten thousand insertions allocates ten thousand intermediate maps; mutating ETS tables in place does not. For big graphs assembled piece by piece, the mutable representation is genuinely faster, and that speed is the whole reason it surrendered immutability.
• You need labelled edges, multiple edges between vertices, or rich edge identity. Modelling “two distinct flights from Heathrow to Betelgeuse, each with its own details” is awkward in a plain adjacency map and native to a digraph.
• The graph lives in one owning process for its lifetime. A long-lived service that builds a graph once and answers many queries against it plays to every one of digraph’s strengths and none of its weaknesses.

Avoid a digraph when:

• The graph is small and you only need to traverse it. If a few lines of recursion over a map will do, they will do it with less ceremony and no cleanup obligation.
• You want to share the graph across processes, or send it in messages. Only the creating process may modify a digraph, and it evaporates when that process dies. An immutable map travels between processes without complaint and without surprises.
• You want to keep old versions, or treat the graph as a value in a larger immutable structure. A digraph has exactly one version — the current one — because it mutates. If history matters, the immutable representation is not a preference but a requirement.
• You cannot guarantee cleanup. A digraph built in a long-lived process and never deleted is a memory leak with a slow fuse. If the lifecycle is murky and the owning process outlives the graph’s usefulness, a map sidesteps the entire hazard — there is nothing to forget to free, and no one will judge you for choosing the structure that cannot leak.

The Short Version

Reach for a digraph when the algorithms in digraph_utils are doing the heavy lifting — when you genuinely need topological order, component analysis, or path-finding over a graph large enough that hand-rolling would hurt. Reach for a map when you mostly need to represent relationships and walk them yourself. The graph module is a specialist’s tool: superb at the job it was built for, slightly dangerous left lying around, and — like a good towel — far better to have and not need than to need and not have.

Summary

Graphs are how you model things and the relationships between them when a list or a map can no longer keep the relationships straight. In LFE, you get them from two standard-library modules — digraph for building and querying, digraph_utils for the real algorithms — and the whole chapter reduces to a handful of moves worth carrying away:

• Conjure and dispose. digraph:new/0,1 creates a graph (default cyclic and protected; pass acyclic to forbid cycles at insertion time, private to lock other processes out). digraph:delete/1 frees it. The second is not optional.
• Vertices are the things. add_vertex, vertex, vertices, del_vertex, no_vertices, plus the degree and neighbour functions. A vertex is any term; a label is a note pinned to it, not its identity.
• Edges are the directed lines between. add_edge, edge (returning the canonical #(E V1 V2 Label)), edges, del_edge, no_edges. Direction matters; multiple edges and labels are allowed; an acyclic graph refuses an edge that would close a loop, handing you the offending path.
• Paths and cycles. get_path (depth-first, a path) versus get_short_path (breadth-first, the short path); get_cycle and get_short_cycle for loops; del_path for severance.
• The power tools. topsort for build order (false on a cycle), is_acyclic for a quick verdict, reachable/reaching for impact analysis, components/strong_components for structure, and a drawer of specialists for the rarer days.

And above all, the one fact that makes graphs different from everything else in this Part: a digraph mutates. It is a reference to shared, ETS-backed state, not an immutable value — owned by one process, invisible to the garbage collector, and yours to delete. Hold the handle, respect the ownership, clean up when you’re done, and digraph is a superb tool. Forget, and it will wait for you at 3 a.m. with the patience of a thing that knows it has all the time in the world.

The other data structures in this book ask nothing of you but that you understand them. The graph asks that, and one thing more: that you remember to draw the line, walk it, and then — like any good guest — turn off the lights on your way out. Don’t Panic. Just don’t forget to delete.

Queues

Records in LFE: A Field Guide to Tagged Tuples

Or: How I Learned to Stop Worrying and Love the Syntactic Sugar

The Fundamental Problem (And Why Records Are a Beautiful Hack)

Picture, if you will, a universe where you’ve built an entire application around three-element tuples representing people: #("Joe" 21 "999-999"). It works splendidly until the day someone casually mentions that perhaps—just perhaps—you should also track email addresses.

Suddenly you’re faced with a choice that makes choosing between the red pill and the blue pill look like deciding between tea flavours: either update every single occurrence of these tuples throughout your codebase (all seven thousand of them, naturally), or pretend email doesn’t exist and hope nobody notices.

This is not, as they say in more civilised programming circles, ideal.

Records solve this problem with what can only be described as a masterful compiler trick—the kind of elegant deception that would make a confidence artist weep with professional admiration. They give you named fields while maintaining the speed and memory efficiency of tuples underneath. It’s tuple access by name rather than position, which is approximately 42 times more maintainable than remembering whether age is element 2 or element 3.

Declaring Records: The Art of Structured Procrastination

Record declarations in LFE use the defrecord form, which looks refreshingly similar to how you’d explain the concept to a particularly bright parrot:

(defrecord person
  name
  (age 0)           ; default value
  (phone ""))       ; another default

This declares a record type called person with three fields. The age field defaults to 0 (presumably for immortal beings or those who refuse to acknowledge the passage of time), and phone defaults to an empty string (for those ahead-of-their-time individuals who hadn’t invented the telephone yet).

You can also write this in the more explicit form, should you feel the need to be unnecessarily verbose:

(defrecord person
  (name 'undefined)
  (age 0)
  (phone ""))

The atom 'undefined is Erlang’s way of saying “I have no idea what goes here, but I’m sure it’ll come to me eventually.”

Creating Records: Instantiation Without the Existential Crisis

Once you’ve declared a record, creating instances is pleasantly straightforward:

; Creating a new person with all fields specified
(make-person name "Joe" age 21 phone "999-999")

; Creating a person and letting defaults do their thing
(make-person name "Fred")  ; age=0, phone=""

; The order doesn't matter (take that, positional parameters!)
(make-person phone "999-999" name "Joe" age 21)

The make-person function is generated automatically by the defrecord declaration. It’s one of those delightful compiler conveniences that makes you wonder why you ever bothered with manual tuple construction in the first place.

Note that you can specify fields in any order you fancy. The compiler will sort it out, much like a particularly organized filing clerk who insists on alphabetizing everything despite your protests that chaos is more authentic.

Accessing Fields: The Dot Syntax Situation

To extract values from records, LFE provides several syntactic options, ranging from the elegant to the “why would you do this to yourself.”

The Match-Extraction Method (Recommended)

Pattern matching remains the most idiomatic way to extract field values:

(defun birthday
  (((match-person age a) p)
   (set-person-age p (+ a 1))))

This function pattern-matches on a person record, extracting the age field into variable a, while keeping the entire record bound to p. Then it returns a new record with the age incremented. Note the critical philosophical point: we’re not changing the record (variables are immutable, remember?), we’re creating a slightly modified copy—the Platonic ideal of the original, but one year older.

The Accessor Function Method (Also Fine)

Each field gets its own accessor function automatically:

(person-name joe)    ; returns "Joe"
(person-age joe)     ; returns 21
(person-phone joe)   ; returns "999-999"

These generated functions have names following the pattern recordname-fieldname. Predictable, if not terribly exciting.

The Field-Pattern-Match Method (For Specific Scenarios)

You can also match against specific field values:

(defun joes-birthday
  (((match-person name "Joe" age a) p)
   (set-person-age p (+ a 1)))
  ((p) p))  ; everyone else stays the same age

This only increments Joe’s age. Everyone else remains ageless, which seems unfair but is presumably what they signed up for.

Updating Records: The Philosophy of Selective Modification

Records are immutable (as everything in functional programming must be, by ancient decree), so “updating” means creating a new record with some fields changed:

(defun update-contact (person new-phone)
  (set-person-phone person new-phone))

The set-person-phone function is another compiler-generated convenience. It creates a new record copying all fields from the original except phone, which gets the new value.

You can update multiple fields at once using the set macro:

(set-person person
  age 37
  phone "999-999")

This is particularly useful when you need to update several fields and want to avoid a chain of nested set-person-fieldname calls that would make even a Lisp enthusiast’s eyes water.

Records in Practice: A Complete Example

Let’s build something vaguely useful—a system for managing robots, because robots are inherently more interesting than people and less likely to file complaints with Human Resources:

(defmodule robots
  (export all))

(defrecord robot
  name
  (type 'industrial)
  hobbies
  (details '()))

(defun crusher ()
  (make-robot 
    name "Crusher"
    hobbies '("Crushing people" "Petting cats")))

(defun car-factory (corp-name)
  (make-robot 
    name corp-name
    hobbies "Building cars"))

(defun repairman (rob)
  (let ((details (robot-details rob)))
    (tuple 'repaired
           (set-robot-details rob 
             (cons "Repaired by repairman" details)))))

(defun show-robot
  (((match-robot name n hobbies h details d))
   (io:format "Robot: ~p~nHobbies: ~p~nDetails: ~p~n" 
              (list n h d))))

Using this module would look approximately like this:

lfe> (set crusher (robots:crusher))
#(robot "Crusher" industrial ("Crushing people" "Petting cats") ())

lfe> (robots:show-robot crusher)
Robot: "Crusher"
Hobbies: ("Crushing people" "Petting cats")
Details: ()
ok

lfe> (set fixed-crusher (robots:repairman crusher))
#(repaired #(robot "Crusher" industrial ...))

The Shell and Records: A Slightly Awkward Relationship

Records are a compile-time feature—they’re entirely transformed into tuples before your code runs. This creates an interesting situation in the REPL, where there is no compile time, only an eternal present tense of evaluation.

To use records in the shell, you must first import the record definitions from a module. In LFE, you do this by loading the module and then using the (record-name ...) syntax directly, as the record definitions come along for the ride:

lfe> (c "robots.lfe")
#(module robots)

lfe> (make-robot name "Mechatron" type 'handmade)
#(robot "Mechatron" handmade undefined ())

The shell understands record syntax once the module is loaded, which is convenient if not particularly magical.

The Terrible Truth: Records Are Just Tuples

We might as well tell you now, before you discover it yourself in a debugging session at 3 AM: records are not real. They’re tuples. The compiler transforms all your lovely named-field record syntax into positional tuple operations, and the BEAM VM has no idea records ever existed.

lfe> (set joe (make-person name "Joe" age 21 phone "999-999"))
#(person "Joe" 21 "999-999")  ; It's a tuple. It was always a tuple.

The first element is the record name (as an atom) acting as a “tag,” and the remaining elements are the field values in the order they appeared in the record declaration. This means:

1. A person record is actually a 4-tuple: #(person Name Age Phone)
2. The shell knows about this and tries to pretty-print it as a record when possible
3. If you forget the record definition, you’ll just see tuples
4. Pattern matching on records is just pattern matching on tagged tuples

CRITICAL WARNING: Never, under any circumstances, use the tuple representation directly in your code. Don’t write #(person "Fred" 0 "") when you mean (make-person name "Fred"). The authors of this guide will disavow all knowledge of you, your code, and your existence if you do this. We have a reputation to maintain.

Why? Because using tuple representations breaks data abstraction. If you change the record definition (add a field, reorder fields), any tuple-based code will:

• Fail to compile (if you’re lucky)
• Create badmatch errors (if you’re moderately lucky)
• Silently do the wrong thing (if the universe hates you)

Records let you change the field order or add new fields without touching the code that uses them. Tuples don’t. Choose wisely.

Record Metadata: Asking Questions About Records

LFE inherits Erlang’s record-info mechanism for querying record metadata:

lfe> (record-info 'fields 'person)
(name age phone)

lfe> (record-info 'size 'person)
4  ; three fields plus the tag atom

Note that these must be literal atoms—you can’t use variables. The compiler needs to know at compile time what you’re asking about, because records don’t exist at runtime. It’s a bit like asking someone to describe a dream: the description exists, but the dream itself is merely a neural firing pattern that’s long since dissipated.

Guards and Records: Testing for Type

The BIF is_record/2 lets you check if something is a record of a particular type:

(defun process-person (p)
  (if (is_record p 'person)
    (do-person-things p)
    (io:format "That's not a person!~n")))

Or in a guard:

(defun process-person (p) (when (is_record p 'person))
  (do-person-things p))

This checks three things:

1. Is p a tuple?
2. Is the first element the atom person?
3. Is the tuple the right size?

If all three conditions hold, congratulations: you probably have a person record. Or an extremely coincidental tuple. The universe tends not to make such distinctions.

Nested Records: Here Be Dragons

Sometimes you need records within records, like when a person’s name should itself be a structured thing:

(defrecord name
  first
  surname)

(defrecord person
  (name (make-name first "Anonymous" surname "Coward"))
  (age 0)
  (phone ""))

; Creating a person with a structured name
(set p (make-person 
         name (make-name first "Robert" surname "Virding")))

; Accessing nested fields requires some gymnastics
(let (((match-person name n) p))
  (name-first n))  ; returns "Robert"

Modern Erlang (R14A+) allows chained record access, but in LFE you’ll typically use pattern matching or let bindings to navigate nested structures. It’s slightly more verbose but arguably clearer about what’s happening.

Header Files: Sharing Records Between Modules

When multiple modules need the same record definition, you use header files (with a .lfe extension by convention):

;;; person.lfe - record definitions
(defrecord person
  name
  (age 0)
  (phone ""))

Then include it in your modules:

(defmodule contacts
  (export all))

(include-file "person.lfe")

(defun make-contact (n a p)
  (make-person name n age a phone p))

Important Philosophical Point: While sharing record definitions via header files is common in the wild, it’s often better to keep record definitions private to a single module and provide accessor functions for other modules to use. This prevents tight coupling, makes refactoring easier, and generally keeps your code from turning into the kind of tangled mess that causes programmers to take up gardening instead.

Maps vs Records: The Eternal Question

You might be wondering: “If records are just syntactic sugar over tuples, and we have maps, why use records at all?”

Excellent question. Here’s the philosophical breakdown:

Use Records When:

• You know all possible fields at compile time
• The structure is relatively small (< 20 fields)
• Performance matters (records are slightly faster than maps)
• You want compile-time field checking
• The fields are always atoms

Use Maps When:

• Fields are dynamic or unknown at compile time
• You need arbitrary keys (not just atoms)
• You’re working with JSON or external data
• You want more flexibility at the cost of less compile-time checking
• You’re building something where the data structure evolves frequently

Records are like strongly-typed structs. Maps are like flexible dictionaries. Choose according to your needs and tolerance for compiler complaints.

Practical Patterns: Working With Records Idiomatically

The Builder Pattern

(defun make-person-from-data (data)
  (make-person 
    name (proplists:get_value 'name data "Unknown")
    age (proplists:get_value 'age data 0)
    phone (proplists:get_value 'phone data "")))

The Updater Pattern

(defun update-if-valid (person new-age)
  (if (>= new-age 0)
    (set-person-age person new-age)
    person))

The Transformer Pattern

(defun person-to-proplist
  (((match-person name n age a phone p))
   `(#(name ,n) #(age ,a) #(phone ,p))))

Final Wisdom

Records are a pragmatic compromise between the efficiency of tuples and the maintainability of named access. They’re not perfect—the syntax can be verbose, they’re just tuples in disguise, and nested records get awkward quickly—but they’re genuinely useful for building maintainable systems.

Think of them as a kind of lightweight, compile-time database schema for your data structures. They won’t solve all your problems (nothing does, despite what the marketing materials claim), but they’ll make the problems you do have significantly more tractable.

And remember: the universe is vast, consciousness is mysterious, and sometimes the best we can do is give our tuples names and pretend they understand us.

See Also

• LFE Style Guide (for conventions on when to use records)
• Maps documentation (for the more flexible alternative)
• The BEAM VM internals (if you want to see how deep the tuple-rabbit-hole goes)
• “The Restaurant at the End of the Universe” by Douglas Adams (for existential comfort when debugging nested records)

This guide compiled at compile-time, which is the only time records actually exist. Any runtime resemblance to tuples is entirely deliberate.

Pattern Matching

Generic Sequence Functions

Part III - Data as Code

Where Part II explored how LFE represents information through data structures, Part III reveals how those same structures spring to life as executable programs. In languages such as Lisp, data structures can be directly entered and evaluated as code, and LFE exemplifies this principle through its evaluation model and execution semantics.

The transition from data to code occurs through evaluation—the process by which static data structures become dynamic computations. Part III begins with expressions and functions, showing how lists like (factorial 5) transform from mere data into function calls that produce results. We explore how LFE’s evaluator breathes life into these structures, turning symbols into function references and nested lists into complex program flows.

The concept of treating code as data and the manipulation and evaluation thereof becomes particularly powerful when we examine processes and message passing. In LFE, not only can you construct programs as data, but you can also send code between processes, enabling sophisticated distributed computing patterns. A process might receive a data structure that represents a function, evaluate it locally, and send the results elsewhere in the system.

The later chapters of Part III demonstrate how data-as-code principles scale to larger programming constructs. Modules and packages organize collections of functions and data, while still maintaining the fundamental property that program structure can be analyzed and manipulated as data. This makes metaprogramming easier than in a language without this property, as you can write LFE programs that generate, modify, or analyze other LFE programs using the same data manipulation techniques you use for ordinary programming tasks.

By Part III’s conclusion, you will understand not just how to write LFE programs, but how LFE programs can write themselves—the ultimate expression of the data-as-code philosophy that makes Lisp family languages uniquely powerful for both practical programming and theoretical computer science.RetryClaude can make mistakes. Please double-check cited sources.

Expressions

Functions

Closures

Evaluation

Flow of Control

Processes

Messages and Their Passing

Objects and Flavors

I/O

Accessing Files

Modules

Packages

Scripting with LFE

Creating LFE Projects

Using rebar3

Project Layout Conventions

Part IV - Advanced Topics

Having firmly established in our minds the fundamental data structures, evaluation semantics, and basic programming constructs of LFE, Part IV ventures into the advanced territories where LFE truly demonstrates its power as both a Lisp and a dialect of Erlang. These chapters explore the sophisticated techniques and system-level capabilities that distinguish expert LFE programmers from casual users, covering everything from metaprogramming to distributed computing.

The journey begins with the practical realities of software development: errors and debugging. In LFE, error handling is not an afterthought but a fundamental design principle inherited from Erlang’s “let it crash” philosophy. Lisp pioneered many ideas in computer science, including automatic storage management and error handling, and LFE extends these concepts with Erlang’s supervision trees and process isolation. Understanding how to debug LFE programs means understanding both the introspective capabilities of Lisp and the fault-tolerance mechanisms of OTP.

Testing in LFE spans multiple paradigms, from traditional unit testing to property-based testing with Propr. The latter represents a particularly powerful approach where you specify the properties your functions should satisfy, and the testing framework generates thousands of test cases automatically. This mathematical approach to testing aligns naturally with LFE’s functional programming roots while leveraging Erlang’s QuickCheck heritage.

The chapter on macros reveals LFE’s true metaprogramming capabilities. This property allows code to be treated as data and vice versa, enabling powerful metaprogramming techniques, and LFE macros exemplify this principle. Unlike preprocessor macros in other languages, LFE macros operate on the parsed abstract syntax tree, allowing you to extend the language itself. You can create new control structures, embed domain-specific languages, and transform code in ways that would be impossible in non-homoiconic languages.

Distributed LFE programming showcases one of the platform’s greatest strengths. The actor model that underlies Erlang extends seamlessly across network boundaries, allowing LFE processes to communicate transparently whether they’re running on the same core or on opposite sides of the globe. This capability, combined with OTP’s battle-tested distribution protocols, enables building systems that scale horizontally while maintaining the same programming model.

Ports and port drivers represent LFE’s gateway to the broader computing ecosystem. While the BEAM virtual machine provides excellent isolation and fault tolerance, real systems often need to interface with external programs, databases, or hardware. Ports allow you to communicate safely with external processes, while port drivers provide high-performance integration for systems programming tasks.

The final chapters on servers and clients bring together all these advanced concepts in the context of building production systems. LFE servers can handle thousands of concurrent connections while maintaining state consistency and fault tolerance. Client architectures demonstrate how to build resilient distributed applications that gracefully handle network partitions and service failures.

Part IV represents the transition from learning LFE to mastering it. These advanced topics require not just technical knowledge but a deep understanding of the design philosophies that make LFE effective: the principle that complexity should be managed through composition rather than monolithic design, that failure should be expected and planned for rather than avoided, and that the most powerful abstractions are often the simplest ones consistently applied. By the end of Part IV, you’ll have the tools and knowledge to build LFE systems that are not just functional, but elegant, robust, and scalable.

Errors and Debugging

Writing Unit Tests

The Common Test Framework

The Propr Test Framework

The Compiler

Macros

Distributed LFE

Ports and Port Drivers

Servers

Clients

Part V - OTP

Open Telecom Platform (OTP) represents one of the most sophisticated and battle-tested frameworks for building distributed, fault-tolerant systems ever created. Originally developed by Ericsson for telecommunications equipment that must run continuously for years without failure, OTP has evolved into a general-purpose platform for building any system that demands high availability, scalability, and reliability. Part V explores how LFE leverages this remarkable foundation to create applications that embody the famous “nine nines” reliability (99.9999999% uptime) that has made Erlang legendary in industries where downtime is measured in millions of dollars per minute.

OTP is built around a fundamental insight: most distributed systems follow predictable patterns, and these patterns can be abstracted into reusable components called behaviours. Rather than reimplementing the complex logic of process supervision, state management, or event handling from scratch, OTP provides tested implementations that handle the difficult edge cases while allowing you to focus on your application’s unique business logic. The behaviours chapter introduces these powerful abstractions - gen_server for stateful services, gen_statem for complex state machines, supervisor for fault tolerance, and others that form the building blocks of robust systems.

Applications in OTP are more than just programs - they are complete, self-contained units that can be started, stopped, and supervised as atomic entities. An OTP application encapsulates not just code, but also configuration, dependencies, and metadata about how the application should behave in various circumstances. This packaging model enables sophisticated deployment strategies and makes it possible to upgrade running systems without stopping them - a capability that has allowed some Erlang systems to achieve continuous operation measured in decades.

The progression from applications to releases represents the scaling of OTP concepts from individual components to complete systems. A release bundles multiple applications together with the Erlang runtime itself, creating a self-contained deployment artifact that can run on any compatible platform. OTP’s release handling tools support hot code swapping, allowing you to upgrade production systems while they continue serving requests - a capability that seems almost magical until you understand the careful engineering that makes it possible.

Tables and databases in the OTP context extend far beyond simple data storage. ETS (Erlang Term Storage) and DETS (Disk ETS) provide high-performance, concurrent access to in-memory and persistent data structures, while Mnesia offers a distributed database designed specifically for telecom applications. These storage systems are deeply integrated with OTP’s supervision and distribution mechanisms, ensuring that data remains consistent and available even in the face of node failures or network partitions.

The example OTP project serves as the capstone of Part V, demonstrating how all these concepts integrate into a complete, production-ready system. You’ll see how behaviours coordinate to handle complex workflows, how applications compose to create larger systems, and how OTP’s supervision trees ensure that failures in one component don’t cascade throughout the system. Lisp pioneered many ideas in computer science, including automatic storage management and recursion, and OTP extends these concepts with process supervision and automatic restart capabilities that make systems truly self-healing.

What makes OTP particularly powerful in the context of LFE is how it preserves the interactive, exploratory nature of Lisp development while providing the industrial-strength reliability guarantees needed for production systems. You can experiment with OTP behaviours at the REPL, hot-swap code in running applications, and inspect system state using the same tools you use for everyday development. This combination of power and accessibility has made OTP the secret weapon behind some of the world’s most reliable distributed systems.

By the end of Part V, you’ll understand not just how to use OTP, but why it represents such a significant advancement in systems programming. You’ll have the knowledge to build applications that gracefully handle the complexities of distributed computing - network failures, partial updates, cascading errors, and load balancing - using patterns that have been refined through decades of real-world deployment in some of the most demanding environments on Earth.

OTP Behaviours

Applications

Releases

Tables and Databases

Example OTP Project

Part VI - Tooling

The most elegant programming language in the world is only as powerful as the tools that support it. Part VI explores the ecosystem that transforms LFE from an interesting academic exercise into a practical platform for building real software. At the heart of this ecosystem lies rebar3, the de facto standard build tool for Erlang and LFE projects, extended with a comprehensive plugin that brings first-class LFE support to every aspect of the development lifecycle.

Rebar3 represents more than just a build system - it embodies the collective wisdom of the Erlang community about how software projects should be structured, built, tested, and deployed. Originally created to manage the complex dependencies and compilation requirements of Erlang applications, rebar3 has evolved into a sophisticated project management platform that handles everything from generating initial project scaffolding to creating production releases. The LFE plugin seamlessly integrates into this ecosystem, ensuring that LFE developers can leverage the same battle-tested workflows that have made Erlang development so productive.

The Quick Start section demonstrates how modern tooling can dramatically reduce the friction of getting started with a new language. What once required understanding complex compilation pipelines and dependency management can now be accomplished with a few simple commands. You’ll go from an empty directory to a running LFE application in minutes, with a project structure that follows community conventions and includes everything needed for testing, documentation, and deployment.

The plugin reference reveals the depth of integration between LFE and the broader Erlang ecosystem. The rebar3 LFE plugin doesn’t just compile LFE code - it understands LFE’s unique characteristics and provides specialized support for features like macros, packages, and interactive development. Whether you’re running unit tests with eunit, exploring code at the REPL, or building complex multi-application releases, the plugin ensures that LFE feels like a first-class citizen in the Erlang world.

Project creation commands showcase the flexibility of modern LFE development. Different project types - libraries, standalone applications, escripts, OTP applications, and full releases - each come with appropriate scaffolding and configuration. This isn’t just convenience; it’s the embodiment of best practices accumulated over years of real-world development. When you generate a new LFE project, you’re not starting from scratch - you’re beginning with a foundation that incorporates decades of lessons learned about how to structure maintainable, scalable software.

The testing integration deserves particular attention, as it demonstrates how tooling can elevate testing from an afterthought to a central part of the development process. The plugin supports multiple testing frameworks - eunit for traditional unit testing, ltest for LFE-specific testing patterns, and Common Test for integration and system testing. This multi-layered approach reflects the reality that different types of testing require different tools, and good tooling should make it easy to use the right tool for each job. Running and deployment commands bridge the gap between development and production. The ability to run code directly from the build system, execute escripts, or start full releases provides a seamless transition from experimentation to deployment. Hot code loading and release management - capabilities that seem almost magical in other languages - become routine operations supported by simple commands.

Perhaps most importantly, Part VI demonstrates how good tooling preserves the interactive, exploratory nature of Lisp development while providing the structure needed for serious software engineering. The REPL integration ensures that you can always drop into an interactive session to explore ideas, test hypotheses, or debug problems. This combination of structure and flexibility exemplifies the LFE philosophy: powerful abstractions that don’t constrain creativity.

By the end of Part VI, you’ll have mastered not just the mechanics of LFE development, but the workflows that make LFE development productive and enjoyable. You’ll understand how to leverage community conventions while maintaining the flexibility to adapt tools to your specific needs. Most importantly, you’ll have the confidence that comes from working with mature, well-designed tools that scale from quick experiments to large, complex systems.

rebar3

Quick Start

Introduction

This is a version of the LFE quick start that has been re-imagined with the LFE rebar3 plugin in mind.

We will cover the following:

• How to get started quickly with LFE using just rebar3 and your local installation of Erlang
• Creating a new LFE project
• Looking at LFE code in the REPL and in modules
• Provide a little insight on how this works
• Leave you with resources for jumping into LFE in more detail

About rebar3

Rebar3 is an Erlang tool that makes it easy to create, develop, and release Erlang libraries, applications, and systems in a repeatable manner.

Rebar3 will:

• respect and enforce standard Erlang/OTP conventions for project structure so they are easily reusable by the community;
• manage source dependencies and Erlang packages while ensuring repeatable builds;
• handle build artefacts, paths, and libraries such that standard development tools can be used without a headache;
• adapt to projects of all sizes on almost any platform;
• treat documentation as a feature, and errors or lack of documentation as a bug.

Rebar3 is also a self-contained Erlang script. It is easy to distribute or embed directly in a project. Tasks or behaviours can be modified or expanded with a plugin system flexible enough that even other languages on the Erlang VM will use it as a build tool.

The rebar3 site provides some nice instructions on installing rebar3.

Going Plaid

Remember: this is a quick-start; it’s going to be a blur of colour! You’re not going to get very many details here, but you will get to jump into the LFE REPL and see a little code in action.¹

The rest of this quick-start assumes that you’ve followed the links in the previous section and have installed both Erlang as well as rebar3, but to take things further, you’ll need to do one more thing: set up the LFE plugin.

Each project you create with the LFE rebar3 plugin will generate a rebar.config file that automatically includes the plugin dependency, but that’s only inside an LFE project. You need to bootstrap the LFE plugin by setting it up in your global rebar.config.

The rebar3 docs tell you this file is located at ~/.config/rebar3/rebar.config. To set this up, you can safely execute the following in a terminal, even if the file already exists:

mkdir -p ~/.config/rebar3/
touch ~/.config/rebar3/rebar.config

Then, in your preferred editor, open that file and add the entry for LFE rebar3 plugin. If that file is empty when you open it, then you can paste this whole thing in there:

{plugins, [
  {rebar3_lfe,
    {git, "https://github.com/lfe-rebar3/rebar3_lfe.git", {branch, "main"}}}
]}.

If you want to pin your project to a specific release of the plugin, you can view the list of released versions here:

• https://github.com/lfe-rebar3/rebar3_lfe/tags

And then use tag (with the version) instead of branch:

{plugins, [
  {rebar3_lfe,
    {git, "https://github.com/lfe-rebar3/rebar3_lfe.git", {tag, "x.y.z"}}}
]}.

If your global rebar3 config file already has one or more plugins in it, then simply add a comma after the last one and paste the {rebar3_lfe ...} line from above (with no trailing comma!).

Next Stop

Ready for some LFE? Next you’ll learn how to create a new LFE project with just one command …

Creating a New Project

A project? Already?! It sounds daunting, but it’s easier than you might think. Open up a terminal window and do this in a directory of your choosing:

rebar3 new lfe-lib my-test-lib

It might take a minute or two to finish; here’s what’s happening:

• rebar3 downloads the LFE plugin
• Finds its dependencies, downloads those too
• Compiles all of them (plugins and dependencies)
• rebar3 then executes the new command, searches for (and finds!) the lfe-lib template
• Creates the project files for the given template

As that last step executes, you will see the following output:

===> Writing my-test-lib/README.md
===> Writing my-test-lib/LICENSE
===> Writing my-test-lib/rebar.config
===> Writing my-test-lib/.gitignore
===> Writing my-test-lib/src/my-test-lib.lfe
===> Writing my-test-lib/src/my-test-lib.app.src

It’s as simple as that! Your new project is ready to go :-)

Next Stop

You can taste it, can’t you? That LFE flavour coming your way? Yup, you’re right. You’re going to be looking at LFE code next …

Hitting the Code

It may not seem like it, but we’re off to a pretty fast start. If you had to do everything we’ve done, manually, you’d have given up by now. Seriously.

(Okay, maybe not.)

Time to put the brakes on, though, ’cause you’re gonna want to see this next part in slow motion.

REPL Me Up!

Make sure you’ve cded into your new LFE project directory, and then do this:

$ rebar3 lfe repl

On windows first enter Erlang’s repl then run lfe_shell:start().

D:\Lfe\my-test-lib
λ rebar3 lfe repl
===> Verifying dependencies...
===> Compiling my-test-lib
Eshell V10.3  (abort with ^G)
1> lfe_shell:start().

This should give you something that looks like the following:

Erlang/OTP 23 [erts-11.0.2] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [hipe] [dtrace]

   ..-~.~_~---..
  (      \\     )    |   A Lisp-2+ on the Erlang VM
  |`-.._/_\\_.-':    |   Type (help) for usage info.
  |         g |_ \   |
  |        n    | |  |   Docs: http://docs.lfe.io/
  |       a    / /   |   Source: http://github.com/rvirding/lfe
   \     l    |_/    |
    \   r     /      |   LFE v1.3-dev (abort with ^G)
     `-E___.-'

lfe>

Try setting a variable:

> (set my-list (lists:seq 1 6))
(1 2 3 4 5 6)

Here are some operations using more functions from the built-in Erlang lists module:

> (* 2 (lists:sum my-list))
42
> (* 2 (lists:foldl (lambda (n acc) (+ n acc)) 0 my-list))
42

Let’s turn that into a function:

> (defun my-sum (start stop)
    (let ((my-list (lists:seq start stop)))
      (* 2 (lists:foldl
             (lambda (n acc)
               (+ n acc))
             0 my-list))))
my-sum

And try it out!

> (my-sum 1 6)
42

Enough with the fancy REPL-play … What about some real code? What does a project look like?

Sample Code

Well, you’ve already seen some! But here is the full, if minimal, module generated by the LFE rebar3 plugin:

(defmodule my-test-lib
  (export (my-fun 0)))

;;; -----------
;;; library API
;;; -----------

(defun my-fun ()
  'hello-world)

You’ll note that the function we define has been exported via the export form in the module definition. The number after the function is the arity of that function (Erlang views functions of the same name but different arity as different functions, and LFE does the same).

In the REPL you will have access to this module and its one function. Try it out:

lfe> (my-test-lib:my-fun)
hello-world

Let’s add to this module our new my-sum function from the REPL jam session in the previous section. In another terminal window (or text editor pane) open up the src/my-test-lib.lfe file and paste the my-sum function at the bottom. Afterwards, add (my-sum 2) to the export section of defmodule at the top.

When you’re done, the entire file should look like this:

(defmodule my-test-lib
  (export (my-fun 0)
          (my-sum 2)))

;;; -----------
;;; library API
;;; -----------

(defun my-fun ()
  'hello-world)

(defun my-sum (start stop)
  (let ((my-list (lists:seq start stop)))
    (* 2 (lists:foldl
            (lambda (n acc)
              (+ n acc))
            0 my-list))))

Then come back to the REPL sessions and compile the module with its new addition:

> (c "src/my-test-lib.lfe")
#(module my-test-lib)

And call the module functions:

> (my-test-lib:my-sum 1 6)
42
> (my-test-lib:my-sum 1 60)
3660
>

Here’s something a little more involved you may enjoy, from the examples in the LFE source code:

(defun print-result ()
  (receive
    ((tuple pid msg)
      (io:format "Received message: '~s'~n" (list msg))
      (io:format "Sending message to process ~p ...~n" (list pid))
      (! pid (tuple msg))
      (print-result))))

(defun send-message (calling-pid msg)
  (let ((spawned-pid (spawn 'my-test-lib 'print-result ())))
    (! spawned-pid (tuple calling-pid msg))))

That bit of code demonstrates one of Erlang’s core features in lovely Lisp syntax: message passing. When loaded into the REPL, that code can demonstrate bidirectional message passing between the LFE shell and a spawned process.

Want to give it a try? Add those two new functions to your module, and don’t forget to update the export section, too! (note that one function has an arity of 0 and the other and arity of 2).

When you’re done, your project module should look like this:

(defmodule my-test-lib
  (export (my-fun 0)
          (my-sum 2)
          (print-result 0)
          (send-message 2)))

;;; -----------
;;; library API
;;; -----------

(defun my-fun ()
  'hello-world)

(defun my-sum (start stop)
  (let ((my-list (lists:seq start stop)))
    (* 2 (lists:foldl
            (lambda (n acc)
              (+ n acc))
            0 my-list))))

(defun print-result ()
  (receive
    ((tuple pid msg)
      (io:format "Received message: '~s'~n" (list msg))
      (io:format "Sending message to process ~p ...~n" (list pid))
      (! pid (tuple msg))
      (print-result))))

(defun send-message (calling-pid msg)
  (let ((spawned-pid (spawn 'my-test-lib 'print-result ())))
    (! spawned-pid (tuple calling-pid msg))))

Message-Passing

We glossed over this in the previous section, but in LFE (and Erlang) you can compile on-the-fly in a REPL session. This is super-convenient when prototyping functionality for a new project where you want to use the REPL, but you also want the benefits of using a file, so you don’t lose your work.

We made some changes to the sample code in the last section; let’s compile it and take it for a spin:

> (c "src/my-test-lib.lfe")
#(module my-test-lib)

Next, let’s send two messages to another Erlang process (in this case, we’ll send it to our REPL process, (self):

> (my-test-lib:send-message (self) "And what does it say now?")
#(<0.26.0> "And what does it say now?")
Received message: 'And what does it say now?'
Sending message to process <0.26.0> ...
> (my-test-lib:send-message (self) "Mostly harmless.")
#(<0.26.0> "Mostly harmless.")
Received message: 'Mostly harmless.'
Sending message to process <0.26.0> ...

In the above calls, for each message sent we got a reply acknowledging the message (because the example was coded like that). But what about the receiver itself? What did it, our REPL process, see? We can flush the message queue in the REPL to find out.

What, what? Does each …

Yup, every process in LFE (and Erlang, of course) has an inbox. You can see how many messages a given process has by looking at the process’ info:

lfe> (erlang:process_info (self) 'message_queue_len)
#(message_queue_len 2)

And there you can see that our REPL process has two messages queued up in its inbox. Let’s take a look!

> (c:flush)
Shell got {"And what does it say now?"}
Shell got {"Mostly harmless."}
ok

If you found this last bit interesting and want to step through a tutorial on Erlang’s light-weight threads in more detail, you may enjoy this tutorial.

Next Stop

We did promise a bit more information, so we’re going to do that next and then wrap up the quick start and point you in some directions for your next LFE adventures …