LFE MACHINE MANUAL

Adatped from multiple sources
by Duncan McGreggor and Robert Virding

Published by Cowboys 'N' Beans Books

https://github.com/cnbbooks ◈ http://cnbb.pub/ ◈ info@cnbb.pub

First electronic edition published: 2020

Portions © 1974, David Moon

Portions © 1978-1981, Daniel Weinreb and David Moon

Portions © 1979-1984, Douglas Adams

Portions © 1983, Kent Pitman

Portions © 1992-1993, Peter Norvig and Kent Pitman

Portions © 2003-2020, Ericsson AB

Portions © 2008-2012, Robert Virding

Portions © 2010-2020, Steve Klabnik and Carol Nichols

Portions © 2013-2023, Robert Virding and Duncan McGreggor

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License

About the Cover

The LFE "Chinenual" cover is based upon the Lisp Machine Manual covers of the early 80s. The Lisp Machine Manual editions we have seen from 1977 and 1979 had only hand-typed title pages, no covers, so we're not sure if the famous graphic/typographic design occurred any earlier than 1982. We've also been unable to discover who the original designer was, but would love to give them credit, should we find out.

The Original

The Software Preservation Group has this image on their site:

Bitsavers has a 3rd edition of the Chinenual with the full cover

The LFE Edition

Whole Cover

Back Cover

The Spine

Dedication

To all LFE Community members, Lispers, programmers as well as all our friends and families.

Preface

The original Lisp Machine Manual, the direct spiritiaul ancestor of the LFE Machine Manual, described both the language and the "operating system" of the Lisp Machine. The language was a dialect of Lisp called Zetalisp. Zetalisp was a direct descendant of MACLISP, created as a systems programming language for the MIT Lisp machines. This is of special note since Erlang was created as a systems programming language too. One of it's co-creators, Robert Virding, created Lisp Flavoured Erlang (LFE) based upon his expereinces with Franz Lisp (which based largely upon MACLISP), Portable Standard Lisp (itself an experiment in systems programming), and ultimately in an implementation he made of Lisp Machine Flavors on top of VAX/VMS where he extensively utilized the Lisp Machine Manual.

As such, LFE has a very strong inheritance of systems programming from both parents, as it were. First and foremost, it is a BEAM language written on top of the Erlang VM and from which it strays very little. Secondly, it is a Lisp dialect. It is, however, entirely a systems programming language.

Which brings us back to Zetalisp and the Lisp Machine Manual. It seemed only fitting to base the LFE manual upon the fantastic work and docuentation that was done on Lisp systems programming in the 70s and 80s, work that so many of us treasure and adore and to which we still defer. Thus the machine that is OTP in the context and syntax of the LFE Lisp dialect is extensively documented in the LFE MACHINE MANUAL.

Forward

The original Lisp programming language was implemented nearly 65 years ago¹ at MIT (Massachusetts Institute of Technology), as part of the work done by John McCarthy and Marvin Minsky at the then-nascent AI (Artificial Intelligence) Lab. In the intervening half-century, the original Lisp evolved and experienced significant transformation in a technological diaspora originally fuelled by an explosion of research in the field of artificial intelligence. Through this, the industry witnessed 40 years of development where countless independent Lisp implementations were created. A small sampling of these number such favourites as Lisp 1.5, MacLisp, ZetaLisp, Scheme, Common Lisp, and ISLISP. However, the early 1990s saw the beginning of what would become the AI winter, and Lisp declined into obscurity – even notoriety – as graduating computer scientists were ushered into the “enterprise” world of Java, never to look back.

Except some did. By the early to mid-2000s, groundwork was being laid for what is now being recognized as a “Lisp renaissance.” The rediscovery of Lisp in the new millennium has led to the creation of a whole new collection of dialects, many implemented on top of other languages: Clojure on Java; LFE (Lisp Flavoured Erlang) and Joxa on Erlang; Hy on Python; Gherkin on Bash. New languages such as Go and Rust have several Lisp implementations, while JavaScript seems to gain a new Lisp every few years. While all of these ultimately owe their existence to the first Lisp, conceived in 1956² and defined in 1958³, they represent a new breed and a new techno-ecological niche for Lisp: bringing the power of meta-programming and the clarity of near syntaxlessness⁴ to established language platforms. Whereas in the past Lisp has been an either-or choice, the new era of Lisps represents a symbiotic relationship between Lisp and the language platforms or VMs (virtual machines) upon which they are implemented; you now can have both, without leaving behind the accumulated experiences and comfort of your preferred platform.

Just as the Lisp implementations of the 1960s were greatly impacted by the advent of time-sharing computer systems, the new Lisps mentioned above like Clojure and LFE have been deeply influenced not only by their underlying virtual machines, but – more importantly – by the world view which the creators and maintainers of those VMs espoused. For the Erlang ecosystem of BEAM (Bogdan/Björn's Erlang Abstract Machine) languages, the dominant world view is the primacy of highly-concurrent, fault-tolerant, soft real-time, distributed systems. Erlang was created with these requirements in mind, and LFE inherits this in full. As such, LFE is more than a new Lisp; it is a language of power designed for creating robust services and systems. This point bears some discussion in order to properly prepare the intrepid programming language enthusiast who wishes to travel through the dimensions of LFE.

Unlike languages whose prototypical users were developers working in an interactive shell engaged in such tasks as solving math problems, Erlang's prototypical “user” was a telecommunications device in a world where downtime was simply unacceptable. As such, Erlang's requirements and constraints were very unusual when compared to most other programming languages of its generation.⁵ Erlang, and thus its constellation of dialects, was designed from the start to be a programming language for building distributed systems, one where applications created with it could survive network and systems catastrophes, millions of processes could be sustained on a single machine, where tens and hundreds of thousands of simultaneous network connections could be supported, where even a live, production deployment could have its code updated without downtime. Such is the backdrop against which the Erlang side of the LFE story unfolds – not only in the teaching and learning of it, but in its day-to-day use, and over time, in the minds of its developers.

When bringing new developers up to speed, this perspective is often overlooked or set aside for later. This is often done intentionally, since one doesn't want to overwhelm or discourage a newcomer by throwing them into the “deep end” of distributed systems theory and the critical minutia of reliability. However, if we ignore the strengths of LFE when teaching it, we do our members as well as ourselves a disservice that leads to much misunderstanding and frustration: “Why is LFE so different? You can do X so much more simply in language Y”. It should be stated quite clearly in all introductory materials that the BEAM languages are not like other programming languages; in many ways, the less you rely upon your previous experiences with C, Java, Python, Ruby, etc., the better off you will be.

When compared to mainstream programming languages, Erlang's development is akin to the divergent evolution of animals on a continent which has been completely isolated for hundreds of millions of years. For instance, programming languages have their “Hello, world”s and their “first project”s. These are like lap dogs for newcomers to the programming world, a distant and domesticated version of their far more powerful ancestors. Though each language has its own species of puppy to help new users, they are all loyal and faithful companions which share a high percentage of common genetic history: this is how you print a line, this is how you create a new project. Erlang – the Down Under of programming languages – has its “Hello, world”s and “first project”s, too. But in this case, the lapdog does not count the wolf in its ancestral line. It's not even a canid.⁶ It's a thylacine⁷ with terrifying jaws and and unfamiliar behaviours. Its “hello world” is sending messages to thousands of distributed peers and to nodes in supervised, monitored hierarchies. It has just enough familiarity to leave one feeling mystified by the differences and with the understanding that one is in the presence of something mostly alien.

That is the proper context for learning Erlang when coming from another programming language.

In LFE, we take that a step further by adding Lisp to the mix, supplementing a distributed programming language platform with the innovation laboratory that is Lisp. Far from making the learning process more difficult, this algebraic, time-honoured syntax provides an anchoring point, a home base for future exploration: it is pervasive and familiar, with very few syntactical rules to remember. We have even had reports of developers more easily learning Erlang via LFE.

In summary, by the end of this book we hope to have opened the reader's eyes to a new world of promise: distributed systems programming with a distinctly 1950s flavour – and a capacity to create applications that will thrive for another 50 years in our continually growing, decentralized technological world.

Notes

1. The first draft of this forward which was written in 2015 said "almost 60 years ago" but was never published. Today, at the end of 2023, this content is finally seeing the light of day, with the origins of Lisp receding further into the past ... ↩

2. See McCarthy's 1979 paper History of Lisp, in particular the section entitled “LISP prehistory - Summer 1956 through Summer 1958”. ↩

3. Ibid., section “The implementation of LISP”. ↩

4. As you learn and then take advantage of Lisp's power, you will find yourself regularly creating new favourite features that LFE lacks. The author has gotten so used to this capability that he has applied this freedom to other areas of life. He hopes that you can forgive the occasional English language hack. ↩

5. In fact, in the 1980s when Erlang was born, these features were completely unheard of in mainstream languages. Even today, the combination of features Erlang/OTP (Open Telecom Platform) provides is rare; an argument can be made that Erlang (including its dialects) is still the only language which provides them all. ↩

6. The family of mammals that includes dogs, wolves, foxes, and jackals, among others. ↩

7. An extinct apex predator and marsupial also known as the Tasmanian tiger or Tasmanian wolf. ↩

Acknowledgments

Thank you Robert Virding for all that you have done to make Lisp Flavoured Erlang a reality.

Part I - Getting Started

Every programming journey begins with a single step, and in the Lisp tradition, that step is often taken at the REPL—the Read-Eval-Print Loop that transforms programming from a batch process into an interactive conversation with the computer. Part I of this manual guides you through your first steps in LFE (Lisp Flavored Erlang), from understanding what makes this language unique to building your first complete application.

LFE occupies a fascinating position in the programming language landscape. It brings the elegant syntax and powerful abstractions of Lisp to the robust, fault-tolerant world of the Erlang Virtual Machine (BEAM). This combination offers something remarkable: a language that supports both the reflective, code-as-data philosophy of Lisp and the "let it crash" resilience of actor-model concurrency. Whether you're drawn to LFE from the Lisp side or the Erlang side, you'll discover that this synthesis creates new possibilities for building distributed, fault-tolerant systems.

We begin with the traditional "Hello, World!" programs, but in three distinct flavors that showcase LFE's versatility. The REPL version demonstrates immediate interactivity—type an expression, see the result instantly. The main function approach shows how to create standalone programs, while the LFE/OTP version introduces you to the powerful supervision trees and process management that make Erlang applications famously reliable.

The heart of Part I is the guessing game walk-through, a complete project that evolves from simple procedural code to a proper (if very simple) OTP application. This progression mirrors the journey many LFE programs take: starting as interactive explorations at the REPL, growing into functions and modules, and ultimately becoming supervised processes that can handle failures gracefully and scale across multiple machines.

Finally, we dive deep into the REPL itself—not just as a place to test code fragments, but as a powerful development environment. The REPL is where LFE's dual nature as both Lisp and Erlang becomes most apparent. You can manipulate code as data using traditional Lisp techniques, while simultaneously spawning processes, sending messages, and exploring the vast ecosystem of Erlang and OTP libraries.

By the end of Part I, you'll have not just written your first LFE programs, but developed an intuition for the LFE way of thinking—interactive, incremental, and designed for systems that never stop running. You'll understand how to leverage both the immediate feedback of Lisp development and the industrial-strength reliability of the BEAM platform, setting the foundation for the deeper explorations that follow in the subsequent parts of this manual.

Introduction

Far out in the uncharted backwaters of the unfashionable end of computer science known as "distributed systems programming" lies a small red e. Orbitting this at a distance roughly proportional to the inverse of the likelihood of it being noticed is an utterly insignificant little green mug filled with the morning beverage stimulant equivalent of That Old Janx Spirit. Upon that liquid floats a little yellow 𝛌 whose adherents are so amazingly primitive that they still think cons, car, and cdr are pretty neat ideas.

This is their book.

Their language, Lisp Flavoured Erlang (henceforth "LFE"), lets you use the archaic and much-beloved S-expressions to write some of the most advanced software on the planet. LFE is a general-purpose, concurrent, functional Lisp whose underlying virtual machine (Erlang) was designed to create distributed, fault-tolerant, soft-realtime, highly-availale, always-up, hot-swappable appliances, applications, and services. In addition to fashionable digital watches, LFE sports immutable data, pattern-matching, eager evaluation, and dynamic typing.

This manual will not only teach you what all of that means and why you want it in your breakfast cereal, but also: how to create LFE programs; what exactly are the basic elements of the language; the ins-and-outs of extremely efficient and beautiful clients and servers; and much, much more.

Note, however, that the first chapter is a little different than most other books, and is in fact different from the rest of the chapters in this manual. We wrote this chapter with two guiding thoughts: firstly and foremost, we wanted to provide some practical examples of code-in-action as a context in which a programmer new to LFE could continuously refer -- from the very beginning through to the successful end -- while learning the general principles of that language; secondly, most programming language manuals are dry references to individual pieces of a language or system, not representatives of the whole, and we wanted to provide a starting place for those who learn well via examples, who would benefit from a view toward that whole. For those who have already seen plenty of LFE examples and would rather get to down to the nitty-gritty, rest assured we desire your experience to be guilt-free and thus encourage you to jump into next chapter immediately!

This book is divided into 6 parts with the following foci:

• Introductory material
• Core data types and capabilities
• The basics of LFE code and projects
• Advanced language features
• The machine that is OTP
• Concluding thoughts and summaries

There is a lot of material in this book, so just take it section by section. If at any time you feel overwhelmed, simply set down the book, take a deep breath, fix yourself a cuppa, and don't panic.

Welcome to the LFE MACHINE MANUAL, the definitive LFE reference.

About LFE

LFE is a modern programming language of two lineages, as indicated by the expansion of its acronym: Lisp Flavoured Erlang. In this book we aim to provide the reader with a comprehensive reference for LFE and will therefore explore both parental lines. Though the two language branches which ultimately merged in LFE are separated by nearly 30 years, and while LFE was created another 20 years after that, our story of their origins reveals that age simply doesn't matter. More significantly, LFE has unified Lisp and Erlang with a grace that is both simple and natural. This chapter is a historical overview on how Lisp, Erlang, and ultimately LFE came to be, thus providing a broad context for a complete learning experience.

What Is LFE?

LFE is a Lisp dialect which is heavily flavoured by the programming language virtual machine upon which it rests, the Erlang VM.¹ Lisps are members of a programming language family whose typical distinguishing visual characteristic is the heavy use of parentheses and what is called prefix notation.² To give you a visual sense of the LFE language, here is some example code:

(defun remsp
  (('())
   '())
  (((cons #\  tail))
   (remsp tail))
  (((cons head tail))
   (cons head (remsp tail))))

This function removes spaces from a string that it passed to it. We will postpone explanation and analysis of this code, but in a few chapters you will have the knowledge necessary to understand this bit of LFE.

Besides the parentheses and prefix notation, other substantial features which LFE shares with Lisp languages include the interchangeability of data with code and the ability to write code which generates new code using the same syntax as the rest of the language. Examples of other Lisps include Common Lisp, Scheme, and Clojure.

Erlang, on the other hand, is a language inspired most significantly by Prolog and whose virtual machine supports not only Erlang and LFE, but also newer BEAM languages including Joxa, Elixir, and Erlog. BEAM languages tend to focus on the key characteristics of their shared VM: fault-tolerance, massive scalability, and the ability to easily build soft real-time systems.

One way of describing LFE is as a programming language which unites these two. LFE is a happy mix of the serious, fault-tolerant philosophy of Erlang combined with the flexibility and extensibility offered by Lisp dialects. It is a homoiconic distributed systems programming language with Lisp macros which you will soon come to know in much greater detail.

A Brief History

To more fully understand the nature of LFE, we need to know more about Lisp and Erlang – how they came to be and even more importantly, how they are used today. To do this we will cast our view back in time: first we'll look at Lisp, then we'll review the circumstances of Erlang's genesis, and finally conclude the section with LFE's beginnings.

The Origins of Lisp

Lisp, originally spelled LISP, is an acronym for "LISt Processor". The language's creator, John McCarthy, was inspired by the work of his colleagues who in 1956 created IPL (Information Processing Language), an assembly programming language based upon the idea of manipulating lists. Though initially intrigued with their creation, McCarthy's interests in artificial intelligence required a higher-level language than IPL with a more general application of its list-manipulating features. So after much experimentation with FORTRAN, IPL, and heavy inspiration from Alonzo Church's lambda calculus,³ McCarthy created the first version of Lisp in 1958.

In the 1950s and 1960s programming languages were actually created on paper, due to limited computational resources. Volunteers, grad students, and even the children of language creators were used to simulate registers and operations of the language. This is how the first version of Lisp was “run”.⁴ Furthermore, there were two kinds of Lisp: students in the AI lab wrote a form called S-expressions, which was eventually input into an actual computer. These instructions had the form of nested lists, the syntax that eventually became synonymous with Lisp. The other form, called M-expressions, was used when McCarthy gave lectures or presented papers.⁵ These had a syntax which more closely resembles what programmers expect. This separation was natural at the time: for over a decade programmers had entered instructions using binary or machine language while describing these efforts in papers using natural language or pseudocode. McCarthy's students programmed entirely in S-expressions and as their use grew in popularity, the fate of M-expressions was sealed: they were never implemented.⁶

The Lisp 1.5 programmer's manual, first published in 1962, used M-expressions extensively to introduce and explain the language. Here is an example function⁷ for removing spaces from a string input defined using M-expressions:

remsp[string] = [null[string]→F;
              eq[car[string];" "]→member[cdr[string]];
              T→cons[car[string];remsp[cdr[string]]]]

The corresponding S-expression is what the Lisp programmer would actually enter into the IBM 704 machine that was used by the AI lab at MIT:⁸

DEFINE ((
  (REMSP (LAMBDA (STRING)
    (COND ((NULL STRING)
           F)
          ((EQ (CAR STRING) " ")
           (REMSP (CDR STRING)))
          (T
           (CONS (CAR STRING)
                 (REMSP (CDR STRING)))))))))

The period from 1958 to 1962, when Lisp 1.5 was released, marked the beginning of a new era in computer science. Since then Lisp dialects have made an extraordinary impact on the design and theory of other programming languages, changing the face of computing history perhaps more than any other language group. Language features that Lisp pioneered include such significant examples as: homoiconicity, conditional expressions, recursion, meta-programming, meta-circular evaluation, automatic garbage collection, and first class functions. A classic synopsis of these accomplishments was made by the computer scientist of great renown, Edsger Dijkstra in his 1972 Turing Award lecture, where he said the following about Lisp:

“With a few very basic principles at its foundation, [Lisp] has shown a remarkable stability. Besides that, Lisp has been the carrier for a considerable number of, in a sense, our most sophisticated computer applications. Lisp has jokingly been described as ‘the most intelligent way to misuse a computer’. I think that description a great compliment because it transmits the full flavour of liberation: it has assisted a number of our most gifted fellow humans in thinking previously impossible thoughts.”

Lisp usage is generally described as peaking in the 80s and early 90s, experiencing an adoption setback with the widespread view that problems in artificial intelligence were far more difficult to solve than originally anticipated.⁹ Another problem which faced Lisp was related hardware requirements: specialized architectures were developed in order to provide sufficient computational power to its users. These were expensive with few vendors, and a slow product cycle.

In the midst of this Lisp cold-spell, two seminal Lisp books were published: On Lisp, and ANSI Common Lisp, both by famed entrepreneur Paul Graham.¹⁰ Despite a decade of decline, these events helped catalyse a new appreciation for the language by a younger generation of programmers and within a few short years, the number of Lisp books and Lisp-based languages began growing, giving the world the likes of Practical Common Lisp and Let Over Lambda in the case of the former, and Clojure and LFE, in the case of the latter.

Constructing Erlang

Erlang was born in the heart of Ericsson's Computer Science Laboratory,¹¹ just outside of Stockholm, Sweden.¹² The lab had the general aim “to make Ericsson software activities as efficient as possible through the purposeful exploitation of modern technology.” The Erlang programming language was the lab's crowning achievement, but the effort leading up to this was extensive with a great many people engaged in the creation of many prototypes and the use of numerous of programming languages.

One example of this is the work that Nabiel Elshiewy and Robert Virding did in 1986 with Parlog, a concurrent logic programming language based on Prolog. Though this work was eventually abandoned, you can read the paper The Phoning Philosopher's Problem or Logic Programming for Telecommunications Applications and see the impact its features had on the future development of Erlang. The paper provides some examples of Parlog usage; using that for inspiration we can envision what our space-removing program would looking like:¹³

remsp([]) :-
  [].

remsp([$ |Tail]) :-
  remsp(Tail).

remsp([Head|Tail]) :-
  [Head|remsp(Tail)].

Another language that was part of this department-wide experimentation was Smalltalk. Joe Armstrong started experimenting with it in 1985 to model a telephone exchanges and used this to develop a telephony algebra with it. A year later, when his colleague Roger Skagervall showed him the equivalence between this and logic programming, Prolog began its rise to prominence, and the first steps were made towards the syntax of Erlang as the world now knows it. In modern Erlang, our program has the following form:

remsp([]) ->
  [];
remsp([$ |Tail]) ->
  remsp(Tail);
remsp([Head|Tail]) ->
  [Head|remsp(Tail)].

The members of the Ericsson lab who were tasked with building the next generation telephone exchange system, and thus involved with the various language experiments made over the course of a few years, came to the following conclusions:

• Small languages seemed better at succinctly addressing the problem space.
• The functional programming paradigm was appreciated, if sometimes viewed as awkward.
• Logic programming provided the most elegant solutions in the given problem space.
• Support for concurrency was viewed as essential.

If these were the initial guideposts for Erlang development, its guiding principles were the following:

• To handle high-concurrency
• To handle soft real-time constraints
• To support non-local, distributed computing
• To enable hardware interaction
• To support very large scale software systems
• To support complex interactions
• To provide non-stop operation
• To allow for system updates without downtime
• To allow engineers to create systems with only seconds of down-time per year
• To easily adapt to faults in both hardware and software

These were accomplished using such features as immutable data structures, light weight processes, no shared memory, message passing, supervision trees, and heartbeats. Furthermore, having adopted message-passing as the means of providing high-concurrency, Erlang slowly evolved into the exemplar of a programming paradigm that it essentially invented and even today, dominates: concurrent functional programming.

After four years of development from the late 80s into the early 90s, Erlang matured to the point where it was adopted for large projects inside Ericsson. In 1998 it was released as open source software, and has since seen growing adoption in the wider world of network- and systems-oriented programming.

The Birth of LFE

One of the co-inventors of Erlang, and part of the Lab's early efforts in language experimentation was Robert Virding. Virding first encountered Lisp in 1980 when he started his PhD in theoretical physics at Stockholm University. His exposure to the language came as a result of the physics department's use in performing symbolic algebraic computations. Despite this, he spent more time working on micro-processor programming and didn't dive into it until a few years later when he was working at Ericsson's Computer Science Laboratory. One of the languages evaluated for use in building telephony software was Lisp, but to do so properly required getting to know it in-depth – both a the language level as well as the operating system level.¹⁴ It was in this work that Virding's passion for Lisp blossomed and he came to appreciate deeply its functional nature, macros, and homoiconicity – all excellent and time-saving tools for building complicated systems.

Though the work on Lisps did not become the focus of Erlang development, the seeds of LFE were planted even before Erlang itself had come to be. After 20 years of contributions to the Erlang programming language, these began to bear fruit. In 2007 Virding decided to do something fun in his down time: to see what a Lisp would look like if written on top of the Prolog-inspired Erlang VM. After several months of hacking, he announced a first version of LFE to the Erlang mail list in early 2008.

A few years latter, when asked about the origins of LFE and the motivating elements behind his decision to start the project, Virding shared the following on the LFE mail list:

• It had always been a goal of Robert's to make a Lisp which could fully interact with Erlang/OTP, to see what it would look like and how it would run.
• He was looking for some interesting programming projects that were not too large to do in his spare time.
• He thought it would be a fun, open-ended problem to solve with many interesting parts.
• He likes implementing languages.

We showed an example of LFE at the beginning of this chapter; in keeping with our theme for each language subsection, we present it here again, though in a slightly altered form:

(defun remsp
  (('())
   '())
  ((`(32 . ,tail))
   (remsp tail))
  ((`(,head . ,tail))
   (cons head (remsp tail))))

What is LFE Good For?

Very few languages have the powerful capabilities which Erlang offers – both in its standard library as well as the set of Erlang libraries, frameworks, and patterns that are provided in OTP. This covers everything from fault-tolerance, scalability, soft real time capacity, and high-availability to proper design, component assembly, and deployment in distributed environments.

Similarly, despite the impact that Lisp has had on so many programming languages, its full suite of features is still essentially limited to Lisp dialects. This includes the features we have already mentioned: the ability to treat code as data, easily generate new code from data, as well as the interrelated power of writing macros – the last allows developers to modify the language to suit their needs. These rare features from two different language branches are unified in LFE and there is no well-established language that provides the union of these.

As such, LFE gives developers everything they need to envision, prototype, and then build distributed applications – ones with unique requirements that no platform provides and which can be delivered thanks to LFE's language-building capabilities.

To paraphrase and augment the opening of Chapter 1 in Designing for Scalability with Erlang/OTP:

“You need to implement a fault tolerant, scalable soft real time system with requirements for high availability. It has to be event driven and react to external stimulus, load and failure. It must always be responsive. You also need language-level features that don't exist yet. You would like to encode your domain's best practices and design patterns seamlessly into your chosen platform.”

LFE has everything you need to realize this dream ... and so much more.

In Summary

What LFE Is

Here's what you can expect of LFE:

• A proper Lisp-2, based on the features and limitations of the Erlang VM
• Compatibility with vanilla Erlang and OTP
• It runs on the standard Erlang VM

Furthermore, as a result of Erlang's influence (and LFE's compatibility with it), the following hold:

• there is no global data
• data is not mutable
• only the standard Erlang data types are used
• you get pattern matching and guards
• you have access to Erlang functions and modules
• LFE has a compiler/interpreter
• functions with declared arity and fixed number of arguments
• Lisp macros

What LFE Isn't

Just to clear the air and set some expectations, we'll go a step further. Here's what you're not going to find in LFE:

• An implementation of Scheme
• An implementation of Common Lisp
• An implementation of Clojure

As such, you will not find the following:

• A Scheme-like single namespace
• CL packages or munged names faking packages
• Access to Java libraries

Notes

1. Robert Virding, the creator of LFE and one of the co-creators of the Erlang programming language, has previously stated that, were he to start again, he would name his Lisp dialect EFL, since it truly is a Lisp with an Erlang flavour, rather than the other way round. ↩

2. We will be covering prefix notation when we cover symbolic expressions later in the book. ↩

3. Alonzo Church was one of McCarthy's professors at Princeton. McCarthy did not use all of the lambda calculus when creating Lisp, as there were many esoteric aspects for which he had no practical need. ↩

4. In the case of Lisp, university students were the primary computer hardware ... and sometimes even high school students (see REPL footnote below). ↩

5. This approach was not uncommon at the time: the ALGOL 58 specification defined a syntax for the language reference, one for publications, and a third for implementation. ↩

6. The single greatest contributor to the ascendance of the S-expression is probably the invention of the REPL by L Peter Deutsch, which allowed for interactive Lisp programming. This was almost trivial in S-expressions, whereas a great deal of effort would have been required to support a similar functionality for M-expressions. ↩

7. The function we use in this chapter to demonstrate various syntaxes and dialects was copied from the cover of Byte Magazine's August 1979 issue which focused on Lisp and had part of a Lisp 1.5 program on its cover. ↩

8. The formatting applied to the S-expression version of the function is a modern convention, added here for improved readability. There was originally no formatting, since there was no display – a keypunch was used to enter text on punchcards, 80 characters at a time. As such, a more historically accurate representation would perhaps be: DEFINE (((REMSP (LAMBDA (STRING) (COND ((NULL STRING) F) ((EQ (CAR STRING) " ") (REMSP (CDR STRING))) (T (CONS (CAR STRING) (REMSP (CDR STRING))))))))) ↩

9. This time period is commonly referred to as the “AI winter”. ↩

10. Paul Graham sold his Lisp-based e-commerce startup to Yahoo! In 1998. ↩

11. The majority of this section's content was adapted from Joe Armstrong's paper “A History of Erlang” by, written for the HOPL III conference in 2007. ↩

12. The Computer Science Laboratory operated from 1982 to 2002 in Älvsjö, Stockholm. ↩

13. We've taken the liberty of envisioning the Parlog of 1986 as one that supported pattern matching on characters. ↩

14. One of Virding's project aims was to gain a deeper understanding of Lisp internals. As part of this, he ported the Lisp Machine Lisp object framework Flavors to Portable Standard Lisp running on UNIX. His work on this project contributed to his decision to use Flavour as part of the name for LFE (spelling divergence intentional). ↩

Prerequisites

Anyone coming to LFE should have experience programming in another language, ideally a systems programming language, especially if that language was found lacking. If the corageous reader is attmping to use LFE as a means of entering the study of computer science, we might offer several other paths of study which may bear fruit more quickly and with less pain.

No prior Lisp experience is required, but that would certinaly be helpful. The same goes for Erlang/OTP (or any of the suite of BEAM languages). The reader with experience writing concurrent applications, wrestling with fault-tolerance, or maintaining highly-available applications and services does receive bonus points for preparedness. Such well-prepared readers landing here may have, in fact, done so due to a quest for a distributed Lisp. For those whom this does apply, your quest has found its happy end.

This book assumes the reader has the following installed upon their system:

• a package manager for easily installing software (in particular, development tools and supporting libraries)
• git, make, and other core open source software development tools
• a modern version of Erlang (as of the writing of this book, that would include versions 19 through 23); the rebar3 documentation has great suggestions on what to use here, depending upon your need
• the rebar3 build tool for Erlang (and other BEAM languages); see its docs for installation instructions

Conventions

Typography

Key Entry

We use the angle bracket convention to indicate typing actual key on the keyboard. For instance, when the reader sees <ENTER> they should interpret this as an actual key they should type. Note that all keys are given in upper-case. If the reader is expected to use an upper-case "C" instead of a lower-case "c", they will be presented with the key combination <SHIFT><C>.

Code

Color syntax highlighting is used in this text to display blocks of code. The formatting of this display is done in such a way as to invoke in the mind of the reader the feeling of a terminal, thus making an obvious visual distinction in the text. For instance:

(defun fib
  ((0) 0)
  ((1) 1)
  ((n)
    (+ (fib (- n 1))
       (fib (- n 2)))))

Examples such as this one are readily copied and may be pasted without edit into a file or even the LFE REPL itself.

For interactive code, we display the default LFE prompt the reader will see when in the REPL:

lfe> (integer_to_list 42 2)
;; "101010"

We also distinguish the output from the entered LFE code using code comments displayed afer the command.

For shell commands, the commands to enter at the prompt are prefixed by a $ for the prompt. Input and any relevant output are provided as comment strings:

$ echo "I am excited to learn LFE"
# I am excited to learn LFE

LiffyBot

This is a severly hoopy frood. With an attitude. He pops up from time to time, generally with good advice. Or simply as a marker for something the authors hope you will pay special note.

Messages of Note

From time to time you will see call-out boxes, aimed at drawing your attention to something of note. There are four differnt types of these:

• ones that share useful info (blue)
• ones that highlight something of a momentus nature (green)
• ones that offer warnings to tred carefully (orange)
• ones that beg you not to follow a particular path (red)

These messages will take the following forms:

Development Setup

rebar3 Configuration

Having followed the notes and linked instructions in the Prerequisites section, you are ready to add global support for the LFE rebar3 plugin.

First, unless you have configured other rebar3 plugins on your system, you will need to create the configuration directory and the configuration file:

$ mkdir ~/.config/rebar3
$ touch ~/.config/rebar3/rebar.config

Next, open up that file in your favourite editor, and give it these contents:

{plugins, [
  {rebar3_lfe, "0.4.8"}
]}.

If you already have a rebar.config file with a plugins entry, then simply add a comma after the last plugin listed and paste the {rebar3_lfe, ...} line from above (with no trailing comma!). When a new version of rebar3_lfe is released, you can follow the instructions in the rebar3_lfe repo to upgrade.

A Quick Test with the REPL

With the LFE rebar3 plugin successfully configured, you should be able to start up the LFE REPL anywhere on your system with the following:

$ rebar3 lfe repl

Erlang/OTP 23 [erts-11.0] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [hipe]

   ..-~.~_~---..
  (      \\     )    |   A Lisp-2+ on the Erlang VM
  |`-.._/_\\_.-':    |   Type (help) for usage info.
  |         g |_ \   |
  |        n    | |  |   Docs: http://docs.lfe.io/
  |       a    / /   |   Source: http://github.com/rvirding/lfe
   \     l    |_/    |
    \   r     /      |   LFE v1.3-dev (abort with ^G)
     `-E___.-'

lfe>

Exit out of the REPL for now by typing <CTRL><G> and then <Q>.

'Hello, World!'

Hello-World style introductory programs are intended to give the prospective programmer for that language a sense of what it is like to write a minimalist piece of software with the language in question. In particular, it should show off the minimum capabilitiues of the language. Practically, this type of program should sigify to the curious coder what they could be in for, should they decide upon this particular path.

In the case of LFE/OTP, a standard Hello-World program (essentially a "print" statement) is extremely misleading; more on that in the OTP version of the Hello-World program. Regardless, we concede to conventional practice and produce a minimal Hello-World that does what many other languages' Hello-World programs do. We do, however, go further afterwards ...

From the REPL

As previously demonstrated, it is possible to start up the LFE 'read-eval-print loop' (REPL) using rebar3:

$ rebar3 lfe repl

Once you are at the LFE prompt, you may write a simple LFE "program" like the following:

lfe> (io:format "~p~n" (list "Hello, World!"))

Or, for the terminally lazy:

lfe> (io:format "~p~n" '("Hello, World!"))

While technically a program, it is not a very interesting one: we didn't create a function of our own, nor did we run it from outside the LFE interactive programming environment.

Let's address one of those points right now. Try this:

lfe> (defun hello-world ()
lfe>   (io:format "~p~n" '("Hello, World!")))

This is a simple function definition in LFE.

We can run it by calling it:

lfe> (hello-world)
;; "Hello, World!"
;; ok

When we execute our hello-world function, it prints our message to standard-output and then lets us know everything really quite fine with a friendly ok.

Now let's address the other point: running a Hello-World programming from outside LFE.

Hit <CTRL-G><Q> to exit the REPL and get back to your terminal.

From the Command Line

From your system shell prompt, run the following to create a new project that will let us run a Hello-World program from the command line:

$ rebar3 new lfe-main hello-world
$ cd ./hello-world

Once in the project directory, you can actually just do this:

$ rebar3 lfe run

You will see code getting downloaded and compiled, and then your script will run, generating the following output:

Running script '/usr/local/bin/rebar3' with args [] ...
'hello-world

When you created a new LFE project of type 'main', a Hello-World function was automatically generated for you, one that's even simpler than what we created in the previous section:

(defun my-fun ()
  'hello-world)

The other code that was created when we executed rebar3 new lfe-main hello-world was a script meant to be used by LFE with LFE acting as a shell interpreter:

#!/usr/bin/env lfescript

(defun main (args)
  (let ((script-name (escript:script_name)))
    (io:format "Running script '~s' with args ~p ...~n" `(,script-name ,args))
    (io:format "~p~n" `(,(hello-world:my-fun)))))

You may be wondering about the args argument to the main function, and the fact that the printed output for the args when we ran this was []. Let's try something:

$ rebar3 lfe run -- Fenchurch 42

Running script '/usr/local/bin/rebar3' with args [<<"Fenchurch">>,<<"42">>] ...
'hello-world'

We have to provide the two dashes to let rebar3 know that we're done with it, that the subsequent argsuments are not for it, but rather for the program we want it to start for us. Using it causes everything after the -- to be passed as arguments to our script.

As for the code itself, it's tiny. But there is a lot going on just with these two files. Have no fear, though: the remainder of this book will explore all of that and more. For now, know that the main function in the executable is calling the hello-world module's my-fun function, which takes no arguments. To put another way, what we really have here is a tiny, trivial library project with the addition of a script that calls a function from that library.

For now just know that an executable file which starts with #!/usr/bin/env lfescript and contains a main function accepting one argument is an LFE script capable of being executed from the command line -- as we have shown!

LFE/OTP 'Hello, World!'

What have been demonstrated so far are fairly vanilla Hello-World examples; there's nothing particularly interesting about them, which puts them solidly iin the company of the millions of other Hello-World programs. As mentioned before, this approach is particularly vexing in the case of LFE/OTP, since it lures the prospective developer into the preconception that BEAM languages are just like other programming languages. They most decidedly are not.

What makes them, and in this particular case LFE, special is OTP. There's nothing quite like it, certainly not another language with OTP's feature set baked into its heart and soul. Most useful applications you will write in LFE/OTP will be composed of some sort of long-running service or server, something that manages that server and restarts it in the event of errors, and lastly, a context that contains both -- usually referred to as the "application" itself.

As such, a real Hello-World in LFE would be honest and let the prospective developer know what they are in for (and what power will be placed at their fingertips). That is what we will show now, an LFE OTP Hello-World example.

If you are still in the directory of the previous Hello-World project, let's get out of that:

cd ../

Now we're going to create a new project, one utilising the some very basic OTP patterns:

rebar3 new lfe-app hello-otp-world
cd ./hello-otp-world

We won't look at the code for this right now, since there are chapters dedicated to that in the second half of the book. But let's brush the surface with a quick run in the REPL:

rebar3 lfe repl

To start your new hello-world application, use the OTP application module:

lfe> (application:ensure_all_started 'hello-otp-world)
;; #(ok (hello-otp-world))

That message lets you know that not only was the hello-otp-word application and server started without issue, any applications upon which it depends were also started. Furthermore, there is a supervisor for our server, and it has started as well. Should our Hello-World server crash for any reason, the supervisor will restart it.

To finish the demonstration, and display the clichéd if classic message:

(hello-otp-world:echo "Hello, OTP World!")
;; "Hello, OTP World!"

And that, dear reader, is a true LFE/OTP Hello-World program, complete with message-passing and pattern-matching!

Feel free to poke around in the code that was generated for you, but know that eventually all its mysteries will be revealed, and by the end of this book, that program's magic will just seem like ordinary code to you, ordinary, dependable, fault-tolerant, highly-availble, massively-concurrent code.

Walk-through: An LFE Guessing Game

Now that you've seen some LFE in action, let's do something completely insane: write a whole game before we even know the language!

We will follow the same patterns established in the Hello-World examples, so if you are still in one of the Hello-World projects, change directory and then create a new LFE project:

$ cd ../
$ rebar3 new lfe-app guessing-game
$ cd ./guessing-game

We will create this game by exploring functions in the REPL and then saving the results in a file. Open up your generated project in your favourite code-editing application, and then open up a terminal from your new project directory, and start the REPL:

$ rebar3 lfe repl

Planning the Game

We've created our new project, but before we write even a single atom of code, let's take a moment to think about the problem and come up with a nice solution. By doing this, we increase our chances of making something both useful and elegant. As long as what we write remains legible and meets our needs, the less we write the better. This sort of practice elegance will make the code easier to maintain and reduce the chance for bugs (by the simple merrit of there being less code in which a bug may arise; the more code, the greater opportunities for bugs).

Our first step will be making sure we understand the problem and devising some minimal abstractions. Next, we'll think about what actually need to happen in the game. With that in hand, we will know what state we need to track. Then, we're off to the races: all the code will fall right into place and we'll get to play our game.

Key Abstractions

In a guessing game, there are two players: one who has the answer, and one who seeks the answer. Our code and data should clearly model these two players.

Actions

The player with the answer needs to peform the following actions:

1. At the beginning of the game, state the problem and tell the other player to start guessing
2. Receive the other player's guess
3. Check the guess against the answer
4. Report back to the other player on the provided guess
5. End the game if the guess was correct

The guessing player needs to take only one action:

1. guess!

State

We need to track the state of the game. Based upon the actions we've examined, the overall state is very simple. Through the course of the game, will only need to preserve the answer that will be guessed.

Code Explore

Now that we've thought through our problem space clearly and cleanly, let's do some code exploration and start defining some functions we think we'll need.

We've already generated an OTP application using the LFE rebar3 plugin, and once we've got our collection of functions that address the needed game features, we can plug those into the application.

We'll make those changes in the code editor you've opened, and we'll explore a small set of possible functions to use for this using the project REPL session you've just started.

Getting User Input

How do we get user input in LFE? Like this!

lfe> (io:fread "Guess number: " "~d")

This will print the prompt Guess number: and then await your input and the press of the <ENTER> key. The input you provide needs to match the format type given in the second argument. In this case, the ~d tells us that this needs to be a decimal (base 10) integer.

fe> (io:fread "Guess number: " "~d")
;; Guess number: 42
;; #(ok "*")

If we try typing something that is not a base 10 integer, we get an error:

lfe> (io:fread "Guess number: " "~d")
;; Guess number: forty-two
;; #(error #(fread integer))

With correct usage, how do we capture the value in a variable? The standard way to do this in LFE is destructuring via pattern matching. The following snippet extracts the value and then prints the extracted value in the REPL:

lfe> (let ((`#(ok (,value)) (io:fread "Guess number: " "~d")))
lfe>   (io:format "Got: ~p~n" `(,value)))
;; Guess number: 42
;; Got: 42
;; ok

We'll talk a lot more about pattern matching in the future, as well as the meaning of the backtick and commas. For now,let's keep pottering in the REPL with these explorations, and make a function for this:

lfe> (defun guess ()
lfe>   (let ((`#(ok (,value)) (io:fread "Guess number: " "~d")))
lfe>     (io:format "You guessed: ~p~n" `(,value))))

And call it:

lfe> (guess)
;; Guess number: 42
;; You guessed: 42
;; ok

Checking the Input

In LFE there are several ways in which you can perform checks on values:

• the if form
• the cond form
• the case form
• pattern-matching and/or guards in function heads

The last one is commonly used in LFE when passing messages / data between functions. Our initial, generated project code is already doing this, and given the game state data we will be working with, this feels like a good fit what we need to implement.

Normally records are used for application data, but since we just care about the value of two integers (the number selected for the answer and the number guessed by the player), we'll keep things simple in this game:

(set answer 42)

Let's create a function with a guard:

lfe> (defun check
lfe>   ((guess) (when (< guess answer))
lfe>    (io:format "Guess is too low~n")))

The extra parenthesis around the function's arguments is due to the use of the pattern-matching form of function definition we're using here. We need this form, since we're going to use a guard. The when after the function args is called a "guard" in LFE. As you might imagine, we could use any number of these.

lfe> (check 10)
;; Guess is too low
;; ok

Let's add some more guards for the other checks we want to perform:

lfe> (defun check
lfe>   ((guess) (when (< guess answer))
lfe>    (io:format "Guess is too low~n"))
lfe>   ((guess) (when (> guess answer))
lfe>    (io:format "Guess is too high~n"))
lfe>   ((guess) (when (== guess answer))
lfe>    (io:format "Correct!~n")))

lfe> (check 10)
;; Guess is too low
;; ok

lfe> (check 100)
;; Guess is too high
;; ok

lfe> (check 42)
;; Correct!
;; ok

This should give a very general sense of what is possible.

Integrating into an Application

We're only going to touch one of the files that was generated when you created the guessing-game project: ./src/guessing-game.lfe. You can ignore all the others. Once we've made all the changes summarized below, we will walk through this file at a high level, discussing the changes and how those contribute to the completion of the game.

First though, we need to reflect on the planning we just did, remembering the actions and states that we want to support. There's also another thing to consider, since we're writing this as is an always-up OTP app. With some adjustments for state magagement, it could easily be turned into something that literally millions of users could be accessing simultaneouslyi. So: how does a game that is usually implemented as a quick CLI toy get transformed in LFE/OTP such that it can be run as a server?

In short, we'll use OTP's gen_server capability ("behaviour") and the usual message-passing practices. As such, the server will need to be able to process the following messages:

• #(start-game true) (create a record to track game state)
• #(stop-game true) (clear the game state)
• #(guess n)
  • check for guess equal to the answer
  • greater than the answer, and
  • less than the answer

We could have just used atoms for the first two, and done away with the complexity of using tuples for those, but symmetry is nice :-)

To create the game, we're going to need to perform the following integration tasks:

• Update the handle_cast function to process the commands and guards we listed above
• Create API functions that cast the appropriate messages
• Update the export form in the module definition
• Set the random seed so that the answers are different every time you start the application

handle_cast

The biggest chunk of code that needs to be changed is the handle_cast function. Since our game doesn't return values, we'll be using handle_cast. (If we needed to have data or results returned to us in the REPL, we would have used handle_call instead. Note that both are standard OTP gen_server callback functions.)

The generated project barely populates this function and the function isn't of the form that supports patten-matching (which we need here) so we will essentially be replacing what was generated. In the file ./src/guessing-game.lfe, change this:

(defun handle_cast (_msg state)
  `#(noreply ,state))

to this:

(defun handle_cast
  ((`#(start-game true) _state)
   (io:format "Guess the number I have chosen, between 1 and 10.~n")
   `#(noreply ,(random:uniform 10)))
  ((`#(stop-game true) _state)
   (io:format "Game over~n")
   '#(noreply undefined))
  ((`#(guess ,n) answer) (when (== n answer))
   (io:format "Well-guessed!!~n")
   (stop-game)
   '#(noreply undefined))
  ((`#(guess ,n) answer) (when (> n answer))
   (io:format "Your guess is too high.~n")
   `#(noreply ,answer))
  ((`#(guess ,n) answer) (when (< n answer))
   (io:format "Your guess is too low.~n")
   `#(noreply ,answer))
  ((_msg state)
   `#(noreply ,state)))

That is a single function in LFE, since for every match the arity of the function remains the same. It is, however, a function with six different and separate arguement-body forms: one for each pattern and/or guard.

These patterns are matched:

1. start
2. stop
3. guess (three times)
4. any

For the three guess patterns (well, one pattern, really) since there are three different guards we want placed on them:

1. guess is equal
2. guess is greater
3. guess is less

Note that the pattern for the function argument in these last three didn't change, only the guard is different between them.

Finally, there's the original "pass-through" or "match-any" pattern (this is used to prevent an error in the event of an unexpected message type).

Game API

In order to send a message to a running OTP server, you use special OTP functions for the type of server you are running. Our game is running a gen_server so we'll be using that OTP module to send messages, in particular we'll be calling gen_server:cast. However, creating messages and sending them via the appropriate gen_server function can get tedious quickly, so it is common practice to create API functions that do these things for you.

In our case, we want to go to the section with the heading ;;; our server API and add the following:

(defun start-game ()
  (gen_server:cast (SERVER) '#(start-game true)))

(defun stop-game ()
  (gen_server:cast (SERVER) '#(stop-game true)))

(defun guess (n)
  (gen_server:cast (SERVER) `#(guess ,n)))

Functions in LFE are private by default, so simply adding these functions doesn't make them publicly accessible. As things now stand these will not be usable outside their module; if we want to use them, e.g., from the REPL, we need to export them.

Go to the top of the guessing-game module and update the "server API" sectopm of the exports, chaning this:

    ;; server API
    (pid 0)
    (echo 1)))

to this:

    ;; server API
    (pid 0)
    (echo 1)
    (start-game 0)
    (stop-game 0)
    (guess 1)))

The final form of your module definition should look like this:

(defmodule guessing-game
  (behaviour gen_server)
  (export
    ;; gen_server implementation
    (start_link 0)
    (stop 0)
    ;; callback implementation
    (init 1)
    (handle_call 3)
    (handle_cast 2)
    (handle_info 2)
    (terminate 2)
    (code_change 3)
    ;; server API
    (pid 0)
    (echo 1)
    (start-game 0)
    (stop-game 0)
    (guess 1)))

Now our game functions are public, and we'll be able to use them from the REPL.

Finishing Touches

There is one last thing we can do to make our game more interesting. Right now, the game will work. But every time we start up the REPL and kick off a new game, the same "random" number will be selected for the answer. In order to make things interesting, we need to generate a random seed when we initialize our server.

We want to only do this once, though -- not every time the game starts, and certainly not every time a user guesses! When the LFE server supervisor starts our game server, one functions is called and called only once: init/1. That's where we want to make the change to support a better-than-default random seed.

Let's change that function:

(defun init (state)
  `#(ok ,state))

to this:

(defun init (state)
  (random:seed (erlang:phash2 `(,(node)))
               (erlang:monotonic_time)
               (erlang:unique_integer))
  `#(ok ,state))

Now we're ready to play!

Playing the Game

If you are still in the REPL, quit out of it so that rebar3 can rebuild our changed module. Then start it up again:

$ rebar3 lfe repl

Once at the LFE propmpt, start up the application:

lfe> (application:ensure_all_started 'guessing-game)

With the application and all of its dependencies started, we're ready to start the game and play it through:

lfe> (guessing-game:start-game)
;; ok
;; Guess the number I have chosen, between 1 and 10.

lfe> (guessing-game:guess 10)
;; ok
;; Your guess is too high.

lfe> (guessing-game:guess 1)
;; ok
;; Your guess is too low.

lfe> (guessing-game:guess 5)
;; ok
;; Your guess is too low.

lfe> (guessing-game:guess 7)
;; ok
;; Your guess is too low.

lfe> (guessing-game:guess 8)
;; ok
;; Well-guessed!!
;; Game over

Success! You've just done something pretty amazing, if still mysterious: you've not only created your first OTP application running a generic server, you've successully run it through to completion!

Until we can dive into all the details of what you've seen in this walkthrough, much of what you've just written will seem strange and maybe even overkill. For now, though, we'll mark a placeholder for those concepts: the next section will briefly review what you've done and indicate which parts of this book will provide the remaining missing pieces.

Review

We've got the whole rest of the book ahead of us to cover much of what you've seen in the sample project we've just created with our guessing game. In the coming pages, you will revisit every aspect of what you've seen so far in lots of detail with correspondingly useful instructions on these matters.

That being said, it would be unfair to not at least read through the code together and mention the high-level concepts involved. Since we only touched the code in one file, that will be the one that gets the most of our attention for this short review, but let's touch on the others here, too.

Project Files

rebar.config

This is the file you need in every LFE project you will write in order to take advantage of the features (and time-savings!) that rebar3 provides. For this project, the two important parts are:

1. the entry for dependencies (only LFE in this case), and
2. the plugins entry for the LFE rebar3 plugin.

Project setup will be covered in Chapter XXX, section XXX.

Source Files

The source files for our sample program in this walkthrough are for an OTP application. OTP-based projects will be covered in Chapter XXX, section XXX.

.app.src

This file is mostly used for application metadata. Most of what our app uses in this file is pretty self-explanatory. Every LFE application will have one of these in the project source code. Every LFE library and application needs this file.

guessing-game-app.lfe

This is the top-level file for our game, an OTP application. It only exports two functions: one to start the app and the other to stop it. The application is responsible for starting up whatever supervisors all your services/servers need. For this sample application, only one supervisor is needed (with a very simple supervision tree).

guessing-game-sup.lfe

This module is a little more invloved and has all the configuration and code necessary to properly set up a supervisor for our server. When something goes wrong with our server, the restart strategy defined by our supervisor will kick in and get things back up and running again. This is one of the key secrets to OTP's wizardry, and we will be covering this in great detail later.

src/guessing-game.lfe

This is the last file we'll look at, and is the one we'll cover in the most detail right now. Here's the entire content of what we created for our game:

(defmodule guessing-game
  (behaviour gen_server)
  (export
   ;; gen_server implementation
   (start_link 0)
   (stop 0)
   ;; callback implementation
   (init 1)
   (handle_call 3)
   (handle_cast 2)
   (handle_info 2)
   (terminate 2)
   (code_change 3)
   ;; server API
   (pid 0)
   (echo 1)
   (start-game 0)
   (stop-game 0)
   (guess 1)))

;;; ----------------
;;; config functions
;;; ----------------

(defun SERVER () (MODULE))
(defun initial-state () '#())
(defun genserver-opts () '())
(defun unknown-command () #(error "Unknown command."))

;;; -------------------------
;;; gen_server implementation
;;; -------------------------

(defun start_link ()
  (gen_server:start_link `#(local ,(SERVER))
                         (MODULE)
                         (initial-state)
                         (genserver-opts)))

(defun stop ()
  (gen_server:call (SERVER) 'stop))

;;; -----------------------
;;; callback implementation
;;; -----------------------

(defun init (state)
  (random:seed (erlang:phash2 `(,(node)))
               (erlang:monotonic_time)
               (erlang:unique_integer))
  `#(ok ,state))

(defun handle_cast
  ((`#(start-game true) _state)
   (io:format "Guess the number I have chosen, between 1 and 10.~n")
   `#(noreply ,(random:uniform 10)))
  ((`#(stop-game true) _state)
   (io:format "Game over~n")
   '#(noreply undefined))
  ((`#(guess ,n) answer) (when (== n answer))
   (io:format "Well-guessed!!~n")
   (stop-game)
   '#(noreply undefined))
  ((`#(guess ,n) answer) (when (> n answer))
   (io:format "Your guess is too high.~n")
   `#(noreply ,answer))
  ((`#(guess ,n) answer) (when (< n answer))
   (io:format "Your guess is too low.~n")
   `#(noreply ,answer))
  ((_msg state)
   `#(noreply ,state)))

(defun handle_call
  (('stop _from state)
   `#(stop shutdown ok state))
  ((`#(echo ,msg) _from state)
   `#(reply ,msg state))
  ((message _from state)
   `#(reply ,(unknown-command) ,state)))

(defun handle_info
  ((`#(EXIT ,_from normal) state)
   `#(noreply ,state))
  ((`#(EXIT ,pid ,reason) state)
   (io:format "Process ~p exited! (Reason: ~p)~n" `(,pid ,reason))
   `#(noreply ,state))
  ((_msg state)
   `#(noreply ,state)))

(defun terminate (_reason _state)
  'ok)

(defun code_change (_old-version state _extra)
  `#(ok ,state))

;;; --------------
;;; our server API
;;; --------------

(defun pid ()
  (erlang:whereis (SERVER)))

(defun echo (msg)
  (gen_server:call (SERVER) `#(echo ,msg)))

(defun start-game ()
  (gen_server:cast (SERVER) '#(start-game true)))

(defun stop-game ()
  (gen_server:cast (SERVER) '#(stop-game true)))

(defun guess (n)
  (gen_server:cast (SERVER) `#(guess ,n)))

The beginning of the file opens with a declaration of the module: not only its name, but the public functions we want to expose as part of our API. This will be covered in Chapter XXX, section XXX.

Next, we have a few constant functions. Functions are necessary here due to the fact that LFE does not have global variables. This will be covered in Chapter XXX, section XXX.

Then we define the functions that will be used as this module's implementation of a generic OTP server. There is some boilerplate here that will be discussed when we dive into LFE/OTP. This will be covered in Chapter XXX, section XXX.

After that, we define the functions that are used by the OTP machinery that will run our server. Here you see several examples of pattern matching function heads in LFE, a very powerful feature that lends itself nicely to consise and expressive code. This will be covered in Chapter XXX, section XXX.

Lastly, we define our own API. Most of these functions simply send messages to our running server. More on this in Chapter XXX, section XXX.

The LFE REPL

We briefly introduced the REPL in the first version of the Hello-World example we wrote, stating that it was an acronym for 'read-eval-print loop' and how to start it with rebar3. As an LFE developer, this is one of the primnary tools -- arguably the most powerful -- at your disposal, so we're going to do a more thorough job of introducing its capabilities in this section.

A basic REPL can be implemented with just four functions; such an implementation could be started with the following:

(LOOP (PRINT (EVAL (READ))))

LFE has implemented most these functions for us already (and quite robustly), but we could create our own very limited REPL (single lines with no execution context or environment) within the LFE REPL using the following convenience wrappers:

(defun read ()
  (case (io:get_line "myrepl> ")
    ("quit\n" "quit")
    (str (let ((`#(ok ,expr) (lfe_io:read_string str)))
           expr))))

(defun print (result)
  (lfe_io:format "~p~n" `(,result))
  result)

(defun loop
  (("quit")
   'good-bye)
  ((code)
   (loop (print (eval (read))))))

Now we can start our custom REPL inside the LFE REPL:

lfe> (loop (print (eval (read))))

This gives us a new prompt:

myrepl>

At this prompt we can evaluate basic LFE expressions:

myrepl> (+ 1 2 3)
;; 6
myrepl> (* 2 (lists:foldl #'+/2 0 '(1 2 3 4 5 6)))
;; 42
myrepl> quit
;; good-bye
lfe>

Note that writing an evaluator is the hard part, and we've simply re-used the LFE evaluator for this demonstration.

Now that we've explored some of the background of REPLs and Lisp interpreters, let's look more deeply into the LFE REPL and how to best take advantage of its power when using the machine that is LFE and OTP.

Core Features

In keeping with the overall herritage of LFE, its REPL is both a Lisp REPL as well as an Erlang shell. In fact, when support was added for the LFE REPL to the rebar3_lfe plugin, it utilised all of the plumbing for the Erlang shell support in rebar3.

For the developer, though, this means that the LFE REPL holds a dual set of features, multiplying the set of features available to just the Erlang shell. These features include the following support:

• Evaluation of Lisp S-expressions
• Definition of functions, completely with LFE tail-recursion support
• Definition of records and use of record-specific support functions/macros
• Creation of LFE macros via standard Lisp-2 syntax
• Macro examination and debugging with various expansion macros
• The ability to start the LFE REPL in distribution mode, complete with Erlang cookie, and thus to not only access remote LFE and Erlang nodes, but to be accessed as a remote node itself (for nodes that have been granted access
• Access to the Erlang JCL and the ability to start separate, LFE shells running concurrently

Unsupported

The following capabilities are not supported in the LFE REPL:

• Module definitions; these are a file-based feature in LFE, just as with Erlang.

Starting LFE

The lfe executable

While this book focuses upon the use of rebar3 and its LFE plugin -- due entirely to the amount of time it saves through various features it supports -- LFE may be used quite easily without it.

To use LFE and its REPL without rebar3, you'll need to clone the repo, e.g.:

cd ~/lab
git clone https://github.com/lfe/lfe.git
cd lfe

Since you have read the earlier section on dependencies, you already have Erlang, make, and your system build tools installed. As such, all you have to do is run the following to build LFE:

make

This will generate an executable in ./bin and you can start the LFE REPL by calling it:

./bin/lfe

Erlang/OTP 28 [erts-16.0] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit] [dtrace]

   ..-~.~_~---..
  (      \\     )    |   A Lisp-2+ on the Erlang VM
  |`-.._/_\\_.-':    |   Type (help) for usage info.
  |         g |_ \   |
  |        n    | |  |   Docs: http://docs.lfe.io/
  |       a    / /   |   Source: http://github.com/lfe/lfe
   \     l    |_/    |
    \   r     /      |   LFE v2.2.0 (abort with ^G)
     `-E___.-'

lfe>

If you opt to install LFE system-wide with make install, then you can start the REPL from anywhere by simply executing lfe.

Via rebar3 lfe repl

As demonstrated earlier on several occasions, you can start the LFE REPL with the rebar3 LFE plugin (and this is what we'll do in the rest of this manual):

rebar3 lfe repl

Since you have updated your global rebar3 settings (in the "Prerequisites" section, after following the instructions on the rebar3 site), you may also start the LFE REPL from anywhere on your machine using the rebar3 command.

readline Support

The LFE REPL, being built atop the Erlang shell, can benefit from GNU Readline support to provide enhanced line editing capabilities including command history, keyboard shortcuts, and tab completion. This section covers how to enable and configure these features for optimal development experience.

Built-in History Support (Erlang/OTP 20+)

Starting with Erlang/OTP 20, persistent shell history is available out of the box, though it's disabled by default. This feature provides basic command history persistence across sessions.

Enabling Shell History

To enable persistent history for all Erlang-based shells (including LFE), add the following to your shell's configuration file (~/.bashrc, ~/.zshrc, etc.):

# Enable Erlang shell history
export ERL_AFLAGS="-kernel shell_history enabled"

Advanced History Configuration

You can customize the history behavior using additional kernel parameters:

# Complete history configuration
export ERL_AFLAGS="+pc unicode"
ERL_AFLAGS="$ERL_AFLAGS -kernel shell_history enabled"
ERL_AFLAGS="$ERL_AFLAGS -kernel shell_history_path '\"$HOME/.erl_history\"'"
ERL_AFLAGS="$ERL_AFLAGS -kernel shell_history_file_bytes 1048576"
ERL_AFLAGS="$ERL_AFLAGS -kernel shell_history_drop '[\"q().\",\"init:stop().\",\"halt().\"]'"
export ERL_AFLAGS

Where:

• shell_history_path - Custom path for history file (default: ~/.erlang-history.*))
• shell_history_file_bytes - Maximum history file size in bytes (default: 512KB, minimum: 50KB)
• shell_history_drop - Commands to exclude from history (e.g., exit commands)

Built-in Tab Completion

The Erlang shell (and by extension, LFE) includes sophisticated built-in tab completion functionality through the edlin and edlin_expand modules. LFE extends this with its own lfe_edlin_expand module to provide Lisp-aware completion capabilities. Starting with Erlang/OTP 26, the auto-completion feature has been vastly improved, supporting auto-completion of variables, record names, record field names, map keys, function parameter types, and file names.

LFE-Specific Tab Completion

LFE provides enhanced tab completion that understands Lisp syntax and LFE-specific constructs. When you type in the LFE REPL, tab completion works seamlessly with:

• Module names: Type (code and press TAB to see code and code_server
• Function names with arities: Type (code: and press TAB to see all available functions with their arities displayed in a paged format
• LFE syntax: Completion works properly with LFE's S-expression syntax
• Standard Erlang modules: Full access to Erlang's standard library with completion

What Tab Completion Covers

The standard expansion function is able to expand strings to valid Erlang terms, including module names, and automatically add commas or closing parentheses when no other valid expansion is possible. Specifically, tab completion works for:

• Module names: Type li and press TAB to complete to lists:
• Function names: Type lists: and press TAB to see all available functions
• Variables: Complete previously defined shell variables by typing the first letter(s) and pressing TAB
• Record names: Type # and TAB to see available records
• Record fields: Complete field names within record syntax
• Map keys: Complete keys when working with maps
• Built-in functions (BIFs): Complete standard Erlang functions
• File names: When working with file operations

Using Tab Completion

• Single TAB: Autocomplete the current word, or show 5 lines of possible completions
• Double TAB: Output all possible tab completions
• Navigation: Use move_expand_up and move_expand_down to navigate through completion lists in the expand area

LFE Tab Completion Examples

Here are some examples of LFE's tab completion in action:

;; Module completion
lfe> (code<TAB>
code           code_server

;; Function completion with paged display
lfe> (code:<TAB>
add_path/1                 add_path/2                 add_patha/1
add_patha/2                add_paths/1                add_paths/2
add_pathsa/1               add_pathsa/2               add_pathsz/1
add_pathsz/2               add_pathz/1                add_pathz/2
...
rows 1 to 7 of 8

;; LFE syntax-aware completion
lfe> (defun my-func<TAB>
;; Will complete user-defined functions and LFE constructs

The completion system understands LFE's parenthesized syntax and provides contextually appropriate suggestions.

Customizing Tab Completion

You can customize tab completion by setting an expand function using io:setopts/1,2 with the {expand_fun, expand_fun()} option:

;; In LFE, you would set this via Erlang interop
(: io setopts `(#(expand_fun ,(lambda (input)
                               ;; Custom completion logic here
                               '#(yes "completion" ("option1" "option2"))))))

Full Readline Integration

For a more complete readline experience with features like reverse search (Ctrl-R), customizable key bindings, and improved line editing beyond the built-in capabilities, you have several options.

Option 1: Using rlwrap

rlwrap (readline wrapper) is a utility that adds GNU Readline capabilities to any command-line program. It's the most straightforward way to add full readline support to LFE.

Installing rlwrap

On most Unix-like systems:

# Ubuntu/Debian
sudo apt-get install rlwrap

# macOS (Homebrew)
brew install rlwrap

# CentOS/RHEL/Fedora
sudo yum install rlwrap  # or dnf install rlwrap

Basic Usage

# Start LFE with readline support
rlwrap rebar3 lfe repl

# Or if using the lfe executable directly
rlwrap lfe

Advanced rlwrap Configuration

For optimal LFE/Lisp experience, you can customize rlwrap with specific options:

# Enhanced configuration for Lisp-like languages
rlwrap -r -m -q '"' -b "(){}[],^%#@\\;:'" rebar3 lfe repl

Where:

• -r puts all words seen on input and output on the completion list
• -m enables multi-line editing
• -q '"' handles double quotes specially
• -b "(){}[],^%#@\\;:'" defines characters that break words for completion

Creating a Wrapper Script

For convenience, create a wrapper script in your $PATH (e.g., ~/bin/lfe-repl):

#!/bin/bash
# LFE REPL with readline support

# Basic version
rlwrap rebar3 lfe repl

# Or enhanced version with better Lisp support
rlwrap -r -m -q '"' -b "(){}[],^%#@\\;:'" rebar3 lfe repl "$@"

Make it executable:

chmod +x ~/bin/lfe-repl

Tab Completion vs rlwrap

Important Note: Using rlwrap with the -a option (always use readline) will disable the Erlang shell's built-in tab completion functionality. This is a trade-off between rlwrap's enhanced features (like persistent history and colored prompts) and the shell's native completion capabilities.

If you prioritize tab completion:

• Use the built-in shell history (ERL_AFLAGS="-kernel shell_history enabled")
• Use rlwrap without the -a flag, though this may provide a less integrated experience

If you prioritize rlwrap's features:

• Accept that built-in tab completion will be disabled
• Consider creating custom completion files for rlwrap (advanced)

Option 2: Terminal-Specific Configuration

Many modern terminal emulators provide their own line editing features that can complement or replace readline functionality.

Using .inputrc Configuration

The ~/.inputrc file configures GNU Readline behavior system-wide. Here's a useful configuration for Lisp development:

# ~/.inputrc - GNU Readline configuration

# Use vi or emacs editing mode
set editing-mode emacs

# Enable case-insensitive completion
set completion-ignore-case on

# Show completion matches immediately
set show-all-if-ambiguous on
set show-all-if-unmodified on

# Enable colored completion
set colored-stats on
set colored-completion-prefix on

# Better history search
"\e[A": history-search-backward
"\e[B": history-search-forward

# Alternative: use Ctrl-P/Ctrl-N for history search
# "\C-p": history-search-backward
# "\C-n": history-search-forward

# Disable terminal bell
set bell-style none

# Enable bracketed paste mode (safe pasting)
set enable-bracketed-paste on

# Show mode indicator for vi mode (if using vi editing-mode)
set show-mode-in-prompt on

After editing ~/.inputrc, you can reload it with:

bind -f ~/.inputrc

Environment Variable Summary

For a complete readline-enabled LFE environment, add these to your shell configuration:

# ~/.bashrc, ~/.zshrc, etc.

# Enable Erlang shell history
export ERL_AFLAGS="-kernel shell_history enabled"

# Optional: customize history location and behavior
# export ERL_AFLAGS="$ERL_AFLAGS -kernel shell_history_path '\"$HOME/.lfe_history\"'"
# export ERL_AFLAGS="$ERL_AFLAGS -kernel shell_history_file_bytes 1048576"

# Create an alias for LFE with readline
alias lfe-repl='rlwrap rebar3 lfe repl'

# Or for the lfe executable
alias lfe='rlwrap lfe'

Troubleshooting

History Not Persisting

If command history isn't being saved between sessions:

1. Ensure shell_history is enabled in ERL_AFLAGS
2. Check that the history directory is writable
3. Wait a moment before exiting the shell (history is written asynchronously)
4. Try setting a custom history path with proper permissions

rlwrap Issues

If rlwrap isn't working correctly:

1. Verify rlwrap is installed: which rlwrap
2. Check that your terminal supports readline properly
3. Try with minimal options first: rlwrap rebar3 lfe repl
4. On some systems, you may need to omit the -q option

Tab Completion Problems

If tab completion behaves unexpectedly:

1. Try adjusting the word-breaking characters in rlwrap's -b option
2. Check your ~/.inputrc configuration
3. Some terminal emulators may require specific settings for completion

Performance Issues

If the REPL feels sluggish with readline enabled:

1. Reduce history file size with shell_history_file_bytes
2. Use fewer rlwrap options for minimal overhead
3. Consider using built-in history only without rlwrap for simple use cases

(help)

As you gain familiarity with the LFE REPL, one of the most useful and convenient references will be the summary of functions, commands, and variablese that come with the LFE REPL.

To see these, simple call the help or h function:

lfe> (help)

That will result in the following being displyed to your terminal:

LFE shell built-in functions

(c file)       -- compile and load code in <file>
(cd dir)       -- change working directory to <dir>
(clear)        -- clear the REPL output
(doc mod)      -- documentation of a module
(doc mod:mac)  -- documentation of a macro
(doc m:f/a)    -- documentation of a function
(ec file)      -- compile and load code in erlang <file>
(ep expr)      -- print a term in erlang form
(epp expr)     -- pretty print a term in erlang form
(exit)         -- quit - an alias for (q)
(flush)        -- flush any messages sent to the shell
(h)            -- an alias for (help)
(help)         -- help info
(i)            -- information about the system
(i pids)       -- information about a list of pids
(l module)     -- load or reload <module>
(ls)           -- list files in the current directory
(ls dir)       -- list files in directory <dir>
(m)            -- which modules are loaded
(m mod)        -- information about module <mod>
(p expr)       -- print a term
(pp expr)      -- pretty print a term
(pid x y z)    -- convert <x>, <y> and <z> to a pid
(pwd)          -- print working directory
(q)            -- quit - shorthand for init:stop/0
(regs)         -- information about registered processes

LFE shell built-in commands

(reset-environment)             -- reset the environment to its initial state
(run file)                      -- execute all the shell commands in a <file>
(set pattern expr)
(set pattern (when guard) expr) -- evaluate <expr> and match the result with
                                   pattern binding
(slurp file)                    -- slurp in a LFE source <file> and makes
                                   everything available in the shell
(unslurp)                       -- revert back to the state before the last
                                   slurp

LFE shell built-in variables

+/++/+++      -- the tree previous expressions
*/**/***      -- the values of the previous expressions
-             -- the current expression output
$ENV          -- the current LFE environment

ok

Most of those are documented in stdlib reference for their Erlang counterparts, so be sure to reference that information for details on many of the above.

Those not covered in that Erlang reference manual, or those that are different in their LFE versionsm, include:

• Built-in Functions
  • Compilation functions
  • LFE code documentation
  • Printing and pretty-printing
• Built-in commands
• Built-in variables

REPL Functions

Most of the LFE REPL functions are documented in stdlib reference for their Erlang counterparts. This section documents where the LFE REPL help diverges from the Erlang Shell help.

Compilation

If you view the Erlang reference manual documentation for compiling files in the shell, you will see differences from what is show in the LFE help text. In particular, (c) is for compiling LFE modules and (ec) needs to be used for compiling Erlang source files.

In both cases, the resulting .beam files are compiled to the current working directory and not to an ebin directory. These .beam files will be found by LFE, since the current working directory is included in the path, but you'll likely want to perform some cleanup afterward.

Documentation

You may access the documentation for LFE modules, macros, and functions in the REPL via the doc function. For instance, the Common Lisp compatibility module's documentation:

lfe> (doc cl)
;; ____________________________________________________________
;; cl
;;
;; LFE Common Lisp interface library.
;;
;; ok

That module's cond macro documentation:

lfe> (doc cl:cond)
;; ____________________________________________________________
;; cond
;; args
;; CL compatible cond macro.
;;
;; ok

That module's pairlis/2 function documentation:

lfe> (doc cl:pairlis/2)
;; ____________________________________________________________
;; pairlis/2
;; keys values
;; Make an alist from pairs of keys values.
;;
;; ok

Documentation for Erlang modules and fucntions is available via the Command Interface

Printing Data

LFE Formatting

LFE provides some nice convenience functions for displaying data structions in the REPL. Let's say we had a data structure defined thusly:

lfe> (set data `(#(foo bar baz) #(quux quuz) #(corge grault garply)
lfe> #(plugh xyzzy) #(flurb nirf) #(zobod zirfid)))

We can print our data with the following:

lfe> (p data)
;; (#(foo bar baz) #(quux quuz) #(corge grault garply) #(plugh xyzzy) #(flurb nirf) #(zobod zirfid))
;; ok

Or we can pretty-print it:

lfe> (pp data)
;;(#(foo bar baz)
;; #(quux quuz)
;; #(corge grault garply)
;; #(plugh xyzzy)
;; #(flurb nirf)
;; #(zobod zirfid))
;; ok

Erlang Formatting

The same may be done for displaying data in the Erlang format:

lfe> (ep data)
;; [{foo,bar,baz},{quux,quuz},{corge,grault,garply},{plugh,xyzzy},{flurb,nirf},{zobod,zirfid}]
;; ok

lfe> (epp data)
;; [{foo,bar,baz},
;;  {quux,quuz},
;;  {corge,grault,garply},
;;  {plugh,xyzzy},
;;  {flurb,nirf},
;;  {zobod,zirfid}]
;; ok

REPL Commands

The LFE REPL provides several useful commands users:

(reset-environment)             -- reset the environment to its initial state
(run file)                      -- execute all the shell commands in a <file>
(set pattern expr)
(set pattern (when guard) expr) -- evaluate <expr> and match the result with
                                   pattern binding
(slurp file)                    -- slurp in a LFE source <file> and makes
                                   everything available in the shell
(unslurp)                       -- revert back to the state before the last
                                   slurp

These are fairly self-explanatory, with the possible exception of clarifying how run and slurp differ:

• Calling (run "some/file.lfe") will cause the LFE REPL to read the contents of that file and then execute every line in that file as if they had been typed at the terminal. This is a convenient way of duplicating REPL state between sessions. (If you haven't kept track of your entries, you can always open up your BEAM history file and create an .lfe file with all the required commands!)
• Calling (slurp "some/other/file.lfe") will place all functions, records, and macros defined in that file into the LFE environment, allowing you to call them without a module: prefix. Note that the code is not executed, but is instead placed into the current environment, ready for use.

Special Variables

LFE shell built-in variables

+/++/+++      -- the tree previous expressions
*/**/***      -- the values of the previous expressions
-             -- the current expression output
$ENV          -- the current LFE environment

Most of these variables are taken directly from Common Lisp and have the same exact meaning. From the Common Lisp HyperSpec for +,++,+++:

The variables +, ++, and +++ are maintained by the Lisp read-eval-print loop to save forms that were recently evaluated.

The value of + is the last form that was evaluated, the value of ++ is the previous value of +, and the value of +++ is the previous value of ++.

And for *,**,***:

The variables *, **, and *** are maintained by the Lisp read-eval-print loop to save the values of results that are printed each time through the loop.

The value of * is the most recent primary value that was printed, the value of ** is the previous value of *, and the value of *** is the previous value of **.

Lastly, for -:

The value of - is the form that is currently being evaluated by the Lisp read-eval-print loop.

The $ENV variable in the LFE REPL is a critical tool for debugging particularly tricky issues in the REPL (especially useful when creating complex macros).

Command Interface

While many of the functions listed in the LFE (help) have their documentation in the Erlang Command Interface module (CIM), not everything in the CIM has been provided in the LFE REPL, some of which can be useful at times.

Here are some of the more useful functions you may with to be aware of from that Erlang module:

• (c:bt pid) - Stack backtrace for a process. Equivalent to (erlang:process_display pid 'backtrace).
• (c:h mod) - Print the documentation for mod
• (c:h mod fun) - Print the documentation for all mod:fun
• (c:h mod fun arity) - Print the documentation for mod:fun/arity
• (c:lm) - Reloads all currently loaded modules that have changed on disk (see (c:mm)). Returns the list of results from calling (l mod) for each such loaded module.
• (c:memory) - Memory allocation information. Equivalent to (erlang:memory).
• (c:mm) -
• (c:ni) - Display system information, listing information about all nodes on the network
• (c:nl mod) - Loads Module on all nodes
• (c:nregs) - Displays information about all registered processes for all nodes in the network.
• (c:uptime) - Prints the node uptime (as specified by (erlang:statistics 'wall_clock)) in human-readable form.
• (c:xm mod) - Finds undefined functions, unused functions, and calls to deprecated functions in a module by calling xref:m/1.

Job Control

Thanks to Erlang, when working in the LFE REPL you have access to a powerful job control system that allows you to manage multiple shell sessions, interrupt running processes, and switch between different contexts. This system is accessed through Job Control Language (JCL) mode, which becomes available when you press <CTRL-G>.

Entering JCL Mode

When in the LFE REPL, typing <CTRL-G> will detach the current job and activate JCL mode:

lfe> (set ltuae (* 2 (+ 1 2 3 4 5 6)))
42
lfe>
User switch command (enter 'h' for help)
 -->

At the JCL --> prompt, you may get help text by typing ? or h:

 --> h
  c [nn]            - connect to job
  i [nn]            - interrupt job
  k [nn]            - kill job
  j                 - list all jobs
  s [shell]         - start local shell
  r [node [shell]]  - start remote shell
  q                 - quit erlang
  ? | h             - this message
 -->

Understanding Jobs

In the context of the Erlang/LFE shell, a job refers to a single evaluator process along with any local processes it spawns. When you start the LFE REPL, you begin with one job that acts as your interactive shell session.

Each job maintains its own:

• Variable bindings
• Process dictionary
• Record definitions
• Evaluation context

Only the currently connected job can perform operations with standard I/O, while detached jobs are blocked from using standard I/O (though they can continue running background computations).

JCL Commands

Each JCL command serves a specific purpose in managing your shell sessions:

• c [nn] - Connect to job: Connects to job number nn or the current job if no number is specified. This resumes the standard shell and allows you to interact with that job's evaluation context.

• i [nn] - Interrupt job: Stops the evaluator process for job number nn or the current job, but preserves the shell process. Variable bindings and the process dictionary are maintained, so you can reconnect to the job later. This is particularly useful for interrupting infinite loops or long-running computations.

• k [nn] - Kill job: Completely terminates job number nn or the current job. All spawned processes in the job are killed (provided they haven't changed their group leader and are on the local machine). This permanently destroys the job's context.

• j - List all jobs: Displays all current jobs with their numbers and descriptions. The currently connected job is marked with an asterisk (*).

• s [shell] - Start local shell: Creates a new job with a fresh shell environment. If no shell module is specified, it starts the default shell (LFE shell in our case).

• r [node [shell]] - Start remote shell: Starts a remote job on the specified node. This is used in distributed Erlang to control applications running on other nodes in the network.

• q - Quit Erlang: Completely exits the Erlang runtime system. Note that this command may be disabled if Erlang was started with the +Bi system flag.

Running Multiple Shells: A Practical Example

Here's a practical demonstration of using multiple jobs to maintain separate evaluation contexts:

$ lfe
lfe> (set ltuae (* 2 (+ 1 2 3 4 5 6)))
42
lfe> ^G
User switch command (enter 'h' for help)
 --> j
   1* {lfe_shell,start,[]}
 --> s lfe_shell
 --> j
   1  {lfe_shell,start,[]}
   2* {lfe_shell,start,[]}
 --> c 2
lfe> ltuae
** exception error: symbol ltuae is unbound

lfe> (set arch (erlang:system_info 'system_architecture))
"aarch64-apple-darwin24.4.0"
lfe> ^G
User switch command (enter 'h' for help)
 --> j
   1  {lfe_shell,start,[]}
   2* {lfe_shell,start,[]}
 --> c 1
lfe> ltuae
42
lfe> arch
** exception error: symbol arch is unbound

lfe> ^G
User switch command (enter 'h' for help)
 --> c 2
lfe> arch
"aarch64-apple-darwin24.4.0"

This example demonstrates how each job maintains its own variable bindings:

• Job 1 has ltuae defined but not arch
• Job 2 has arch defined but not ltuae
• Each job represents a completely separate evaluation environment

Common Use Cases

1. Interrupting Infinite Loops

If your code gets stuck in an infinite loop:

lfe> (defun infinite-loop () (infinite-loop))
infinite-loop
lfe> (infinite-loop)
;; Process gets stuck here - press Ctrl-G
User switch command
 --> i
 --> c
lfe> ;; Back to a responsive shell

2. Experimental Development

Use separate jobs for different experiments:

;; Job 1: Working on feature A
lfe> (set feature-a-data '(1 2 3 4))

;; Switch to Job 2 for feature B
User switch command
 --> s lfe_shell
 --> c 2
lfe> (set feature-b-data '(a b c d))

;; Switch back to continue feature A work
User switch command
 --> c 1
lfe> feature-a-data
(1 2 3 4)

3. Remote Development

Connect to remote nodes in distributed systems:

User switch command
 --> r other_node@other_host
;; Now connected to a shell on the remote node

Important Notes

• Variable Isolation: Each job maintains completely separate variable bindings and evaluation contexts
• Process Preservation: Using i (interrupt) preserves the job's state, while k (kill) destroys it permanently
• I/O Blocking: Only the connected job can use standard I/O; detached jobs will block if they attempt I/O operations
• Record Definitions: Each job has its own set of record definitions, though some default records may be loaded automatically

Customizing JCL Behavior

The behavior of the shell escape key (Ctrl-G) can be modified using the STDLIB application variable shell_esc:

• jcl (default): Activates JCL mode
• abort: Terminates the current shell and starts a new one instead of entering JCL mode

This is set with: erl -stdlib shell_esc abort

Job Control in LFE provides a powerful way to manage multiple development contexts, handle problematic code execution, and work with distributed systems, making it an essential tool for effective LFE development.

Files

So far, everything we've looked at in the REPL involves typing (or pasting) code. When wanting to use the REPL to experiment with more complicated code, there's a better, time-honoured way: files. There are several ways you can use file-based code in the REPL:

• evaluation
• compilation
• including
• loading

Each of these is covered in more detail in the following sub-sections.

Evaluation: : slurp and unslurp

LFE provides a unique and powerful way to evaluate file contents directly in the REPL through the slurp and unslurp commands. This feature allows you to temporarily import function and macro definitions from files into your current REPL session.

Using slurp

The slurp function reads an LFE source file and makes all functions and macros defined in that file available directly in the shell, without requiring module prefixes:

lfe> (slurp "examples/my-functions.lfe")
#(ok -no-mod-)
lfe> $ENV
;; Shows all the new function and macro definitions from the file

Key characteristics of slurp:

• Only one file can be slurped at a time
• Slurping a new file automatically removes all data from the previously slurped file
• Functions and macros become available without module prefixes
• The code is evaluated in the current REPL environment
• Variable bindings from the file are added to your current session

Using unslurp

The unslurp command reverts the REPL back to the state before the last slurp, removing all function and macro definitions that were imported:

lfe> (unslurp)
ok
lfe> $ENV
;; Back to the original environment state

This is particularly useful when experimenting with different versions of functions or when you want to clean up your REPL environment.

Practical Example

Let's say you have a file called math-utils.lfe:

(defun double (x)
  (* x 2))

(defun square (x)
  (* x x))

(defmacro when-positive (x body)
  `(if (> ,x 0) ,body 'not-positive))

After slurping this file:

lfe> (slurp "math-utils.lfe")
#(ok -no-mod-)
lfe> (double 21)
42
lfe> (square 6)
36
lfe> (when-positive 5 'yes)
yes
lfe> (when-positive -1 'yes)
not-positive

Compilation

The LFE REPL provides several functions for compiling source files and loading the resulting modules.

Compiling LFE Files

Use the c function to compile and load LFE modules:

lfe> (c "my-module.lfe")
#(module my-module)

This function:

• Compiles the LFE source file
• Loads the resulting .beam file into the current session
• Makes the module's exported functions available for use
• Places the compiled .beam file in the current working directory

You can also provide compilation options:

lfe> (c "my-module.lfe" '(debug_info export_all))
#(module my-module)

Common compilation options include:

• debug_info - Include debugging information
• export_all - Export all functions (useful for development)
• warn_unused_vars - Warn about unused variables

Compiling Erlang Files

To compile Erlang source files from the LFE REPL, use the ec function:

lfe> (ec "utility.erl")
#(module utility)

This allows you to work with existing Erlang code from within your LFE development session.

Working with Compiled Modules

Once a module is compiled and loaded, you can call its functions using the standard module:function syntax:

lfe> (my-module:some-function arg1 arg2)
result

You can also check which modules are currently loaded:

lfe> (m)
;; Lists all loaded modules
lfe> (m 'my-module)
;; Shows information about a specific module

Cleanup

If you compile a module in the REPL and forget to clean up the resulting .beam file, you may encounter mysterious issues later, naming continuing to see old behaviour from the module compiled in your current working directory instead of the .beam file most recently updated.

Inclusion

LFE supports including header files and library files, which is essential for larger projects and when working with Erlang/OTP libraries.

include-lib

The include-lib directive allows you to include files from installed OTP applications or other libraries:

(include-lib "kernel/include/file.hrl")
(include-lib "stdlib/include/qlc.hrl")

This searches for the include file in the standard library locations and makes the definitions available in your code.

include-file

The include-file directive includes files using relative or absolute paths:

(include-file "local-definitions.lfe")
(include-file "../shared/common.lfe")
(include-file "/absolute/path/to/file.lfe")

Include File Content

Include files typically contain:

• Record definitions
• Macro definitions
• Constant definitions
• Type specifications

Example include file (records.lfe):

(defrecord person
  name
  age
  email)

(defmacro debug (msg)
  `(io:format "DEBUG: ~p~n" (list ,msg)))

After including this file, you can use the record and macro definitions in your code:

lfe> (include-file "records.lfe")
debug

;; In LFE, when including a file in the REPL, the last
;; function defined in the file is printed to stdout.

lfe> (make-person name "Robert" age 54 email "robert@lfe.io")
#(person "Robert" 54 "robert@lfe.io")
lfe>

lfe> (debug "oops")
DEBUG: "oops"
ok

Loading

LFE provides several ways to load pre-compiled modules into your REPL session.

Using l (Load Module)

The l function loads or reloads a specific module:

lfe> (l 'my-module)
#(module my-module)

This is useful when:

• You've recompiled a module and want to reload it
• You want to load a module that exists but isn't currently loaded
• You're working with hot code reloading during development

Using code:ensure_loaded

For more control over the loading process, you can use Erlang's code server directly:

lfe> (code:ensure_loaded 'my-module)
#(module my-module)

This function:

• Loads the module if it's not already loaded
• Does nothing if the module is already loaded
• Returns error information if the module can't be found

Checking Module Status

You can check what modules are currently loaded and get information about them:

;; List all loaded modules
lfe> (m)

;; Get information about a specific module
lfe> (m 'lists)

;; Check if a module is loaded
lfe> (code:is_loaded 'my-module)
#(file "/path/to/my-module.beam")

Hot Code Reloading

During development, you can reload modules without restarting the REPL:

;; Edit and recompile your module file
lfe> (c "my-module.lfe")
#(module my-module)

;; The module is automatically reloaded with new code
lfe> (my-module:updated-function)

Part II - Code as Data

In the Lisp tradition, all program code is written as s-expressions, or parenthesized lists Lisp, making the primary representation of programs also a data structure in a primitive type of the language itself. This fundamental principle—code as data—means that before we can understand how LFE programs execute, we must first master the data structures from which they are constructed.

Part II explores the foundational data types and structures that form the vocabulary of LFE programming. Every LFE program, from the simplest function call to the most complex macro, is ultimately composed of the primitive types and data structures covered in these chapters. We begin with the atomic elements—integers, floats, atoms, and characters—and progress through increasingly sophisticated composite structures like lists, tuples, maps, and records.

The interchangeability of code and data gives Lisp its instantly recognizable syntax, and LFE inherits this property. When you write (+ 1 2 3) in LFE, you are simultaneously creating a list data structure and expressing a computation. The list contains the atom + followed by three integers, but it also represents the addition of those numbers. This duality is not accidental—it is the essence of homoiconicity that makes LFE programs both readable as data and executable as code.

Understanding these data structures deeply is crucial because data representing code can be passed between the meta and base layer of the program. Pattern matching, one of LFE's most powerful features, allows you to decompose and analyze these structures with precision. By the end of Part II, you will have the tools to construct, manipulate, and understand any LFE data structure—knowledge that directly translates to understanding LFE code itself.

Variables

In LFE, variables are implemented with atoms. Atoms are used in the language for such things as naming functions and macros, use as keywords and in data structures. As you'll find out when reading about atoms, they are evaluated as simply themselves. However, when they are used as variable names, they evaulate to the value of which they were assigned.

There are two contexts for setting variables:

• in the LFE REPL
• in LFE modules

This distinction is important, not only because the contexts use different forms, but because like Erlang, LFE does not support global variables.

This chpater will also introduce the reader to pattern-matching in LFE, as it applies to variable assignment; a fuller discussion is presented in a dedicated chapter as well in the chapters that cover forms which support pattern-matching (i.e., compound data types, functions, etc.).

Lastly we'll talk more about some of the LFE nuances around global variables.

Bindings

In the REPL

To set a variable in the LFE REPL, use the set macro:

lfe> (set answer 42)
42

In the language itself, LFE doesn't support global variables -- a valuable feature inherited from Erlang. However, in order for a REPL experience to be useful, an environment must be maintained in which the user may write expressions and then refer to them later. This environment is essentially a mechanism for global state in the context of a single user running a single Erlang VM. If we set a variable called answer, that variable will be available to us as long as the REPL process continues or until we reset the environment.

Setting another value with the same variable name is allowed: it merely replaces the assignment in the current REPL environment:

lfe> (set answer "forty-two")
"forty-two"

With a variable assigned with set it may be used at any time in the REPL environment where it was defined:

lfe> (++ "The answer is " answer)
"The answer is forty-two"

Attempting to use a variable that has not been defined results in an error:

lfe> (++ "The question was " question)
** exception error: symbol question is unbound
  in lfe_eval:eval_error/1 (src/lfe_eval.erl, line 1292)
  in lists:map/2 (lists.erl, line 1243)
  in lists:map/2 (lists.erl, line 1243)

If you don't have any need to update the environment with data that you only need for a specific calculation, you may use the let form:

lfe> (let ((short-lived-value (* 2 (+ 1 2 3 4 5 6))))
lfe>   (io:format "The answer is ~p~n" `(,short-lived-value)))
The answer is 42
ok

Let's make sure that variable wasn't saved to our environment:

lfe> short-lived-value
** exception error: symbol short-lived-value is unbound

The lexical scope for the short-lived-value is within the let only and is not available outside of that scope.

In Functions and Macros

Within functions, variables are lexically scoped and bound with let and let*. One may also define lexically scoped functions inside other fucntions, and this is done with flet and fletrec (the latter required for defining recursive functions inside another function). These will be covered in detail later in the book.

We've seen let used above in the REPL; the same applies inside functions:

(defun display-answer ()
  (let ((answer (* 2 (+ 1 2 3 4 5 6))))
    (io:format "The answer is ~p~n" `(,answer))))

This is a straight-forward case of assignment; but what if we needed to assign a varaible that depended upon another variable. Using let, you'd have to do this:

(defun display-answer ()
  (let ((data '(1 2 3 4 5 6)))
    (let ((answer (* 2 (lists:sum data))))
      (io:format "The answer is ~p~n" `(,answer)))))

However, as with other Lisps, LFE provides a convenience macro for this: let*. Here's how it's used:

(defun display-answer ()
  (let* ((data '(1 2 3 4 5 6))
         (answer (* 2 (lists:sum data))))
    (io:format "The answer is ~p~n" `(,answer))))

Lexical scoping helps one isolate unrelated data or calculations, even in the same function: multiple let or let* blocks may be declared in a function and none of the bound variables in one block will be available to another block. Attempting to do so will result in an error.

In Modules

In LFE, one cannot bind variables at the module-level, only functions and macros. This is part of the "no global variables" philosophy (and practice!) of Erlang and LFE. Module-level bindings are done with defun for functions and defmacro for macros. The creation of modules, functions, and macros will all be covered in detail later in the book.

Shadowing

One shadows a variable in one scope when, at a higher scope, that variable was also defined. Here's an annotated example:

(defun shadow-demo ()
  (let ((a 5))
    (io:format "~p~n" `(,a))    ; prints 5
    (let ((a 'foo))             ; 'a' here shadows 'a' in the previous scope
      (io:format "~p~n" `(,a))) ; prints foo
    (io:format "~p~n" `(,a)))   ; prints 5; the shadow binding is out of scope
  (let ((a 42))
    (io:format "~p~n" `(,a))))  ; prints 42 - new scope, no shadowing

Shadowing also may occur at the module-level with the definition of functions, and the shadowing could be of functions at one of several levels. Here's a run-down on function shadowing in modules, from the highest or "outermost" to the lowest or "innermost":

• Predefined Erlang built-in functions (BIFs) may be shadowed by any of the following
• Predefined LFE BIFs may be shadowed by any of the following
• Module imports may shadow any of the above via aliasing
• Functions defined in a module may shadow any of the above
• Functions defined inside a function (e.g., via flet or fletrec) may shadow any of the above

Note that to shadow functions in LFE, functions must match both in name as well as arity (number of arguments).

The hd Erlang BIF returns the "head" of a list (the first item). Here's an example of shadowing it in the REPL. Here's the BIF at work:

lfe> (hd '(a b c d e))
a

Next, paste this into the REPL:

(defun hd (_)
  ;; part of the pun here is that the same function in Lisp is called 'car'
  "My other car is The Heart of Gold.")

The hd function takes one argument (a list), so our function also needs to take one. However, since we don't do anything with that, we use the "don't care" variable _.

Now let's call hd again:

lfe> (hd '(a b c d e))
"My other car is The Heart of Gold."

Shadowed!

Note that, like many other Lisps, LFE has the car function, but since this is a core form, it can't be shadowed (see the next section).

The Unshadowable

Shadowing does not apply to the supported LFE core forms. It may appear that your code is shadowing those forms, but the compiler will always use the core meaning and never an alternative. It does this silently, without warning -- so take care and do not be surprised!

Pattern-matching Preview

Pattern matching is deeply integral to Erlang, and thus LFE. Great swaths of the language touch upon it, so therefore it is a farily large subject to cover in LFE. We will be revisiting pattern matching throughout the rest of the book, but for now we will provide a brief preview in the context of variables and binding values to them.

In the previous section, we saw in-REPL binding like this:

lfe> (set answer 42)
42

But what if it wasn't a simple integer we wanted to assign? What if we wanted to bind something in a data structure, like the tuple #(answer 42)? This is one of the ways in which pattern matching is used in Erlang:

lfe> (set (tuple 'answer answer) `#(answer 42))
#(answer 42)
lfe> answer
42

If we want to capture the key name, too?

lfe> (set (tuple key answer) `#(answer 42))
#(answer 42)
lfe> answer
42
lfe> key
answer

And if we don't care about the key at all?

lfe> (set (tuple _ answer) `#(answer 42))
#(answer 42)
lfe> answer
42

We can do the same thing in many other LFE forms, but here's a quick example of variable assignment with pattern matching in a let form, again borrowing from the previous section:

(defun display-answer ()
  (let (((tuple _ answer) `#(answer 42)))
    (io:format "The answer is ~p~n" `(,answer))))

Pattern matching can be used in may places in LFE, but things really start getting interesting when you define function heads to extract specific values! We will cover examples of that later in the book, in addition to many others.

Global Variables Revisited

In traditional programming languages, global variables are widely considered dangerous and bug-prone. They introduce several critical problems:

• Race conditions: Multiple threads can modify global variables simultaneously, leading to unpredictable state and difficult-to-reproduce bugs
• Hidden side effects: Functions that modify globals create non-obvious dependencies, making code harder to understand and maintain
• Debugging difficulties: When any function can modify a global variable, tracing the source of bugs becomes extremely challenging
• Testing challenges: Global state creates hidden dependencies that make isolated unit testing nearly impossible

Erlang fundamentally solves these problems through immutable data. When you create a data structure in Erlang, its contents cannot be modified in-place. Operations create entirely new data structures, preserving the original. For example, updating a tuple element creates a new tuple rather than modifying the existing one. This eliminates race conditions, makes data flow explicit and predictable, and enables safer concurrent programming without the pitfalls of shared mutable state.

However, real-world applications still need to maintain state across function calls or share data between processes. Erlang provides several safe alternatives, each with its own trade-offs and appropriate use cases.

The Process Dictionary

The process dictionary is a key-value storage mechanism unique to each Erlang process. It provides simple functions like put/2, get/1, and erase/1 for storing and retrieving values within a process's lifetime.

Benefits: The process dictionary offers quick access to process-local data without passing it explicitly through function parameters. Each process has its own isolated dictionary, preventing interference between processes. It's particularly useful for cross-cutting concerns like request IDs in logging or maintaining parse state in complex recursive operations.

Weaknesses: Using the process dictionary breaks functional programming principles by introducing hidden mutable state. This makes code less transparent, harder to test, and more difficult to debug, as data flow becomes implicit rather than explicit. Most Erlang style guides recommend avoiding it except in specific cases where its convenience outweighs these concerns, such as instrumenting third-party libraries or maintaining debug context.

ETS Tables

ETS (Erlang Term Storage) provides in-memory database tables that can be shared across multiple processes. ETS supports several table types: set (unique keys), ordered_set (keys in sorted order), bag (multiple objects per key), and duplicate_bag (allowing duplicate objects).

Benefits: ETS tables provide extremely fast access—constant time O(1) for set tables and logarithmic time O(log N) for ordered_set tables. They support concurrent access with atomic and isolated operations, making them ideal for caching, session storage, counters, and high-performance in-memory data structures. Tables can store large amounts of data efficiently within the Erlang runtime, and access patterns don't require message passing between processes.

Weaknesses: ETS tables have no automatic garbage collection—they persist until explicitly deleted or their owner process terminates. Select and match operations can be expensive as they typically scan the entire table unless properly indexed. Memory management requires careful attention, as tables exist outside normal process memory and don't benefit from Erlang's generational garbage collector. Overly complex match specifications can also become difficult to maintain.

State and OTP Servers

While huge chunks of this book are dedicated to OTP servers and state management, here's the essential concept: gen_server is Erlang's standard behavior for implementing stateful server processes.

A gen_server initializes state through an init/1 callback and maintains that state by passing it through callback functions like handle_call/3 and handle_cast/2. State updates work by creating new copies of data structures with the desired changes, then returning that new immutable data as the current state. This approach combines the benefits of persistent, mutable-seeming state with the safety guarantees of immutable data.

Benefits: OTP servers provide structured, predictable state management with built-in support for supervision, debugging, and code upgrades. State changes are centralized in a single process, making the system easier to reason about. The gen_server behavior handles message queuing, timeouts, and system messages automatically.

Weaknesses: All requests must pass through a single process, which can become a bottleneck under high load. Each state update creates new data structures, which can impact performance for very large or frequently-updated state. Care must be taken to avoid blocking operations that could make the server unresponsive.

External Databases

Erlang applications often connect to external databases for persistent storage. Mnesia, Erlang's built-in distributed database, provides a native solution with strong integration into the Erlang ecosystem. It offers ACID transactions, table replication across nodes, and the ability to store tables in RAM, on disk, or both.

For other databases, Erlang has drivers and libraries for PostgreSQL, MySQL, MongoDB, Redis, and more. These typically use connection pools and message-passing patterns to maintain safety, with processes dedicated to managing database connections and handling queries asynchronously.

Benefits: External databases provide durable persistence across system restarts, ACID transaction guarantees, and the ability to handle datasets larger than available RAM. They enable data sharing across multiple applications and programming languages, and often include sophisticated query capabilities, indexes, and analytical tools. Databases like PostgreSQL offer mature backup, replication, and disaster recovery solutions.

Weaknesses: Database operations introduce network latency and I/O overhead, becoming a potential bottleneck in high-throughput systems. They create external dependencies that can affect system reliability and require careful connection management to avoid resource leaks. The impedance mismatch between Erlang's term-based data model and SQL's relational model can complicate data mapping. Database connections are limited resources that must be pooled and managed carefully.

Choosing the Right Approach

Each alternative serves different needs:

• Process Dictionary: Use sparingly for cross-cutting concerns or when integrating with third-party code
• ETS Tables: Ideal for shared, high-performance in-memory data that multiple processes need to access
• OTP Servers: The default choice for managing application state with clear ownership and structure
• External Databases: Essential for persistent data, large datasets, or sharing data across systems

The key insight is that Erlang doesn't prevent you from maintaining state—it just ensures you do so safely and explicitly, avoiding the pitfalls of traditional global variables.

Primitive Types

This chapter covers the basic types of data available to LFE, upon which primatives rest all LFE libraries and applications.

• Integers
• Floats
• Atoms
• Booleans
• Characters

Each of these types has an LFE test function of the form TYPENAMEp (which wrap the respective Erlang is_TYPENAME function). These are used to perform type checks (especially common in guard expressions). These predicate functions will be covered in their respective type sections.

Integers

Integers in LFE may be either positive or negative and are by default base 10. Like Erlang, LFE does not have a maximum size of integer (in contrast to languages like C). This is accomplished via automatic conversion to larger (or smaller) internal representations (including the use of bignums).

lfe> 42
42
lfe> -42
-42
lfe> 1764
1764
lfe> 150130937545296561928688012959677941476701514734130607701636390322176
150130937545296561928688012959677941476701514734130607701636390322176

Bases

Several bases are supported via special syntax:

lfe> #b101010 ; binary
42
lfe> #o52 ; octal
42
lfe> #d42 ; decimal (explicit base 10)
42)
lfe> #x2a ; hexadecimal
42

Generic bases are supported, too:

lfe> #36r16
42

The number after the hash # is the base and may be any positive integer from 2 through 36. The number after the radix r is the actual value and must only bve comprised of integers allowed for the given base.

Converting between bases in LFE is most easily done via the integer_to_list Erlang function. For example:

lfe> (integer_to_list 1000 2)
"1111101000"
lfe> (integer_to_list 1000 8)
"1750"
lfe> (integer_to_list 1000 10)
"1000"
lfe> (integer_to_list 1000 16)
"3E8"
lfe> (integer_to_list 1000 36)
"RS"

If, for whatever reason, you want your base 10 integrer as a list (Erlang/LFE string), you can do that with this:

lfe> (integer_to_list 1000)
"1000"

Conversion to the binary type is also supported:

lfe> (integer_to_binary 1000)
#"1000"
lfe> (integer_to_binary 1000 2)
#"1111101000"
lfe> (integer_to_binary 1000 8)
#"1750"

The results above show LFE's literal representations of binary data; this will be covered in the chapter on "Bytes and Binaries".

Arithmetic Operators

Integers may be operated upon with the following:

lfe> (+ 1)
1
lfe> (+ 1 2)
3
lfe> (+ 1 2 3)
6
lfe> (- 1 2 3)
-4
lfe> (* 1 2 3)
6
lfe> (/ 1 2 3)
0.16666666666666666

Note that the division operator returns a float; floats will be covered in the next section.

Integer division is supported with a 2-arity function:

lfe> (div 1 2)
0
lfe> (div 10 2)
5

LFE also supports the remainder operation:

lfe> (rem 10 3)
1

As with any functional programming language, these operations may be composed (to any depth):

lfe> (div (* 12 (+ 1 2 3 4 5 6)) 6)
42

Mathematical Functions

The auto-loaded erlang module has several mathematical functions and is accessible in LFE without having to type the erlang: module prefix in the function calls. These include the following:

lfe> (abs -42)
42
lfe> (min 1 2)
1
lfe> (max 1 2)
2

Additional maths functions are provided via the math module. Since this module is not auto-loaded, in order to auto-complete it in the REPL you will need to load it:

lfe> (code:ensure_loaded 'math)
#(module math)

Now you can hit <TAB> after typing the following:

lfe> (math:

Which gives:

acos/1         acosh/1        asin/1         asinh/1        atan/1
atan2/2        atanh/1        ceil/1         cos/1          cosh/1
erf/1          erfc/1         exp/1          floor/1        fmod/2
log/1          log10/1        log2/1         module_info/0  module_info/1
pi/0           pow/2          sin/1          sinh/1         sqrt/1
tan/1          tanh/1

lfe> (round (math:pow 42 42))
150130937545296561928688012959677941476701514734130607701636390322176

The documentation for these functions is limited (available here) due in part to the fact that these are C library wrappers. Those that aren't documented should be self-explanatory for anyone who has used simular mathematical functions in other programming language libraries.

Predicates

To test if a value is an integer, we will first include some code:

lfe> (include-lib "lfe/include/cl.lfe")

That adds Common Lisp inspired functions and macros to our REPL session.

lfe> (integerp 42)
true
lfe> (integerp 42.24)
false
lfe> (integerp "forty-two")
false

If you prefer the Clojure-style of predicates:

lfe> (include-lib "lfe/include/clj.lfe")
lfe> (integer? 42)
true
lfe> (integer? "forty-two")
false

Of course there is always the Erlang predicate, usable without having to do any includes:

lfe> (is_integer 42)
true
lfe> (is_integer "forty-two")
false

Floats

Real numbers in LFE are represented using the stadnard floating point numbers.

lfe> 42.42
42.42
lfe> (/ 10 3)
3.3333333333333335
lfe> (math:pow 42 42)
1.5013093754529656e68

Note that the ~1.5e68 above is the floating point equivalent of scientific notation, namely 1.5 x 1068. LFE follows the 64-bit standard for float representation given by IEEE 754-1985.

Converting

An integer may be converted to a float explicitly:

lfe> (float 42)
42.0

Or, as with integers, to binaries:

lfe> (float_to_binary 42.42)
#"4.24200000000000017053e+01"
lfe> (float_to_binary 42.42 '(#(scientific 10)))
#"4.2420000000e+01"
lfe> (float_to_binary 42.42 '(#(scientific 20)))
#"4.24200000000000017053e+01"
lfe> (float_to_binary 42.42 '(#(decimals 10)))
#"42.4200000000"
lfe> (float_to_binary 42.42 '(#(decimals 10) compact))
#"42.42"

Or to lists (LFE and Erlang strings):

lfe> (float_to_list 42.42 '(#(scientific 10)))
"4.2420000000e+01"
lfe> (float_to_list 42.42 '(#(scientific 20)))
"4.24200000000000017053e+01"
lfe> (float_to_list 42.42 '(#(decimals 10)))
"42.4200000000"
lfe> (float_to_list 42.42 '(#(decimals 10) compact))
"42.42"

Formatting

If you need to round floating point numbers to a specific precision, you'll want to use the format function from either the io, io_lib, or lfe_io modules. If just want to print a value using Erlang syntax and formatting, the io module is what you want. If you prefer LFE syntax and formatting for your output, you'll want to use the lfe_io module. If you want to use the data or store it in a variable, you'll need the io_lib library.

For default precision:

lfe> (io_lib:format "~f" `(,(math:pow 42 42)))
"150130937545296561929000000000000000000000000000000000000000000000000.000000"

Two decimal places:

lfe> (io_lib:format "~.2f" `(,(math:pow 42 42)))
"150130937545296561929000000000000000000000000000000000000000000000000.00"

20 decimal places:

lfe> (io_lib:format "~.20f" `(,(math:pow 42 42)))
"150130937545296561929000000000000000000000000000000000000000000000000.00000000000000000000"

Arithmetic Operators & Mathematical Functions

Floats use most of the same operators and functions as integers, so be sure to review these subsections in the "Integers" section.

Others include:

lfe> (abs -42)
42
lfe> (ceil 42.1)
43
lfe> (floor 42.1)
42
lfe> (round 42.4)
42
lfe> (round 42.5)
43
lfe> (min 1 2)
1
lfe> (max 1 2)
2

Predicates

To test if a value is an integer, we will first include some code:

lfe> (include-lib "lfe/include/cl.lfe")

That adds Common Lisp inspired functions and macros to our REPL session.

lfe> (floatp 42.42)
true
lfe> (floatp 42)
false
lfe> (floatp "forty-two.forty-two")
false

If you prefer the Clojure-style of predicates:

lfe> (include-lib "lfe/include/clj.lfe")
lfe> (float? 42.42)
true
lfe> (float? "forty-two.forty-two")
false

Of course there is always the Erlang predicate, usable without having to do any includes:

lfe> (is_float 42.42)
true
lfe> (is_float "forty-two.forty-two")
false

Atoms

The cloest analog in LFE to what most Lisp dialects call symbols is the Erlang atom. Just as with Lisps do with symbols, LFE uses atoms for its variable and function names. Atoms are literals and constants, which means that their value is the same as their name and once created, cannot be changed.

Some basic examples of atoms:

lfe> 'zark
zark
lfe> 'zarking-amazing
zarking-amazing

Slightly less straight-forward examples which start with non-alphanumeric characters:

lfe> ':answer
:answer
lfe> '42answer
42answer
lfe> '42.0e42answer
42.0e42answer
lfe> '42°C
42°C

Standard LFE atom names may be comprised of all the latin-1 character set except the following:

• control character
• whitespace
• the various brackets
• double quotes
• semicolon

Of these, only |, \, ', ,, and # may not be the first character of the symbol's name (but they are allowed as subsequent letters).

Non-standard atom names may be created using atom quotes:

lfe> '|symbol name with spaces|
|symbol name with spaces|
lfe> '|'with staring quote!|
|'with staring quote!|
lfe> '|| ; <-- empty atoms are supported too
||
lfe> '|really weird atom: '#[]{}()<>";\||
|really weird atom: '#[]{}()<>";\||

In this case the name can contain any character of in the range from 0 to 255, and even no character at all.

In the case of atoms, it is important to understand a little something about their internals. In particular, Erlang and LFE atoms are global for each instance of a running Erlang virtual machine. Atoms are maintained by the VM in a table and are not garbage collected. By default, the Erlang atom table allows for a maximum of 1,048,576 entries.

As Symbols

The following code shows LFE's use of atoms in variable names. First, let's use a function for a slighly different purpose than designed: calling list_to_existing_atom on a string will only return a result if an atom of the same name already exists in the atom table. Otherwise, it will return an error:

lfe> (list_to_existing_atom "zaphod")
** exception error: bad argument
  in (erlang : list_to_existing_atom "zaphod")

This confirms that there is no zaphod atom in the atom table.

Now let's create a variable, assigning a value to it, and then use our indirect means of checkihg the atom table:

lfe> (set zaphod "frood")
"frood"
lfe> (list_to_existing_atom "zaphod")
zaphod

And here's an example showing LFE's use of atoms in function names using the same approach as above:

lfe> (list_to_existing_atom "beez")
** exception error: bad argument
  in (erlang : list_to_existing_atom "beez")

lfe> (defun beez (x) x)
beez
lfe> (list_to_existing_atom "beez")
beez

Converting

Atoms may be converted to strings and bitstrings, and vice versa.

(atom_to_binary 'trisha)
#"trisha"
(atom_to_binary 'mcmillan 'latin1)
#"mcmillan"
lfe> (atom_to_list 'trillian)
"trillian"

Note that the first one above is only available in Erlang 23.0 and above.

Some more examples for encoding:

lfe> (atom_to_binary '42°C 'latin1)
#B(52 50 176 67)
lfe> (atom_to_binary '42°C 'utf8)
#"42°C"

Functions that convert atoms only if they already exist in the atom table:

lfe> (binary_to_existing_atom #"trisha")
trisha
lfe> (binary_to_existing_atom #"trisha" 'latin1)
trisha
lfe> (list_to_existing_atom "trisha")
trisha

Operators

The only operators you may use on atoms are the comparison operators, e.g.:

lfe> (> 'a 'b)
false
lfe> (< 'a 'b)
true
lfe> (=:= 'a 'a)
true

Predicates

To test if a value is an atom, we will first include some code:

lfe> (include-lib "lfe/include/cl.lfe")

That adds Common Lisp inspired functions and macros to our REPL session.

lfe> (atomp 'arthur)
true
lfe> (atomp 42)
false
lfe> (atomp "forty-two.forty-two")
false

If the atom in question has been used in a function name definition:

If you prefer the Clojure-style of predicates:

lfe> (include-lib "lfe/include/clj.lfe")
lfe> (atom? 'dent)
true
lfe> (atom? "Ford")
false

Of course there is always the Erlang predicate, usable without having to do any includes:

lfe> (is_atom 'arthur)
true
lfe> (is_atom "forty-two.forty-two")
false

Caveats

As mentioned above (and documented), one needs to take care when creating atoms. By default, the maximum number of atoms that the Erlang VM will allow is 1,048,576; any more than that, and the VM will crash.

The first rule of thumb is not to write any code that generates large numbers of atoms. More explicitly useful, there are some handy functions for keeping track of the atom table, should you have the need.

lfe> (erlang:memory 'atom_used)
244562
lfe> (erlang:system_info 'atom_count)
9570
lfe> (erlang:system_info 'atom_limit)
1048576

Note that support for easily extracting the current atom data from system_info -- as demonstrated by the last two function calls above -- were added in Erlang 20; should you be running an older version of Erlang, you will need to parse the (system_info 'info) bitstring.

The default atom table size may be overridden during startup by passing a value with the +t options:

$ lfe +t 200000001

lfe> (erlang:system_info 'atom_limit)
200000001

Booleans

Strictly speaking, LFE has no Boolean type, just like Erlang. Instead, the atoms true and false are treated as Booleans

lfe> (== 42 "the question")
false
lfe> (== 42 42)
true
lfe> (> 1 2)
false
lfe> (< 1 2)
true

Operators

The standard logical operators are available to LFE Booleans:

lfe> (not 'true)
false
lfe> (not 'false)
true
lfe> (and 'true 'false)
false
lfe> (or 'true 'false)
true
lfe> (and 'true 'true 'true)
true
lfe> (and 'true 'true 'false)
false
lfe> (or 'false 'false 'false)
false
lfe> (or 'false 'false 'true)
true
lfe> (xor 'true 'true)
false
lfe> (xor 'true 'false)
true
lfe> (xor 'false 'false)
false
lfe> (xor 'false 'true)
true

With the and and or Boolean oprators, every argument is evaluated. To accomodate situations where complex and possible expensive logical expressions comprise the arguments to Boolean operators, short-circuit versions of these functions are also provided:

• with andalso, returns as soon as the first 'false is encountered;
• with orelse, returns as soon as the first 'true is encountered.

To demonstrate this, we'll define a boolean function that prints to standard out when it is evaluated:

(defun hey (x)
  (io:format "Made it here!~n") x)

Short-circuit demonstration of andalso:

lfe> (andalso 'true 'true 'false (hey 'true))
false
lfe> (andalso 'false 'true 'true (hey 'true))
false
lfe> (andalso 'true 'true 'true (hey 'true))
Made it here!
true

Short-circuit demonstration of orelse:

lfe> (orelse 'false 'false 'true (hey 'true))
true
lfe> (orelse 'true 'false 'false (hey 'true))
true
lfe> (orelse 'false 'false 'false (hey 'true))
Made it here!
true

Predicates

To test if a value is a Boolean, we will first include some code:

lfe> (include-lib "lfe/include/cl.lfe")

That adds Common Lisp inspired functions and macros to our REPL session.

lfe> (booleanp 'true)
true
lfe> (booleanp 'false)
true
lfe> (booleanp 'arthur)
false
lfe> (booleanp 42)
false

If the atom in question has been used in a function name definition:

If you prefer the Clojure-style of predicates:

lfe> (include-lib "lfe/include/clj.lfe")
lfe> (boolean? 'true)
true
lfe> (boolean? 'false)
true
lfe> (boolean? 'arthur)
false

Of course there is always the Erlang predicate, usable without having to do any includes:

lfe> (is_boolean 'true)
true
lfe> (is_boolean 'false)
true
lfe> (is_boolean 'arthur)
false

Note that, since LFE Booleans are also atoms, these are valid as well:

lfe> (atomp 'true)
true
lfe> (atom? 'false)
true
lfe> (is_atom 'true)
true

Characters

Characters in LFE Are represented internally by integers, however a literal syntax is offered for convenience:

lfe> #\a
97
lfe> #\A
65
lfe> #\ü
252
lfe> #\Æ
198

Converting

Since a character literal and integer are the same thing as far as LFE is concerned, there is no such thing as converting between a "char" and "ord" like there is in some other languages.

However, one can format an integer as a string by telling the class of format functions that the input is "character" type:

lfe> (io_lib:format "~c" `(198))
"Æ"

For merely printing to standard out instead of returning a value, one may use:

lfe> (lfe_io:format "~c~n" `(198))
Æ
ok
lfe> (io:format "~c~n" `(198))
Æ
ok

Operators

All operations that are valid for integers are valid for characters.

Predicates

All predicates that are valid for integers are valid for characters.

Cons Cells

Cons cells are the fundamental building blocks of lists in both Lisp and Erlang, and by extension, LFE. Understanding cons cells is essential to mastering list manipulation and pattern matching in LFE. While the concept originates from Lisp, Erlang (and thus LFE) adapts it in ways that make it particularly powerful for functional programming and recursive operations.

At first encounter, cons cells might seem like an unnecessarily complicated way to think about lists—after all, why not just have an array-like structure that holds multiple values? The answer lies in the elegance and efficiency they provide for functional programming. Think of a cons cell as a simple container with exactly two compartments: one holds a value (the "head" or "first element"), and the other holds a pointer to the rest of the list (the "tail"). This structure creates a chain—like a linked list in other languages—but with an important difference: in functional programming, these chains are immutable. When you "add" an element to the front of a list, you're not modifying the original list; you're creating a new cons cell that points to the existing list. This makes operations like prepending elements incredibly fast (O(1) constant time) and allows multiple "versions" of a list to safely share structure in memory.

The real power of cons cells emerges when combined with pattern matching. Instead of writing imperative code that indexes into lists or checks for empty conditions, you can write function definitions that naturally express "here's what to do with an empty list, and here's what to do with a list that has at least one element." The cons cell structure makes this pattern matching both intuitive and efficient: destructuring a list into its head and tail is a natural operation that matches how the data is actually stored in memory. As you work through the following sections, you'll see how this simple two-slot structure enables elegant recursive algorithms, efficient list manipulation, and a programming style that feels natural once the initial conceptual hurdle is overcome.

Lisp Cons Cells

In Lisp dialects, a cons cell is a fundamental data structure that holds two values or pointers to values. These two slots are traditionally called the CAR (Contents of the Address Register) and the CDR (Contents of the Decrement Register), names that derive from the original Lisp implementation on IBM 704 hardware.

The cons function constructs these memory objects, and the expression "to cons x onto y" means to construct a new cons cell with x in the car slot and y in the cdr slot.

Structure and Notation

A simple cons cell holding two values can be represented in dotted pair notation:

(cons 'a 'b)  ; Creates a cons cell
=> (a . b)    ; Dotted pair notation

Lists in Lisp are built by having the car slot contain an element and the cdr slot point to another cons cell or to nil (the empty list). This creates a singly-linked list structure:

(cons 1 (cons 2 (cons 3 nil)))
=> (1 2 3)

Visual Representation

Here's a diagram showing how the list (1 2 3) is constructed from cons cells:

graph LR
    A["cons cell 1"] -->|car| V1["1"]
    A -->|cdr| B["cons cell 2"]
    B -->|car| V2["2"]
    B -->|cdr| C["cons cell 3"]
    C -->|car| V3["3"]
    C -->|cdr| N["nil"]

    style A fill:#e1f5ff
    style B fill:#e1f5ff
    style C fill:#e1f5ff
    style V1 fill:#fff4e1
    style V2 fill:#fff4e1
    style V3 fill:#fff4e1
    style N fill:#ffe1e1

Each cons cell contains two pointers: the car points to the element value, and the cdr points to the next cons cell (or nil for the last cell).

Erlang Cons Cells

Erlang lists are built as sequences of cons cells, with each cell composed of a value and a pointer to another cons cell or empty list. While structurally similar to Lisp cons cells, Erlang's implementation and usage patterns differ in important ways.

The Pipe Operator

In Erlang, the cons operator is represented by the pipe symbol (|), which separates the head of a list from its tail. The syntax [Head | Tail] constructs or pattern-matches a cons cell.

% Building a list with cons
[1 | [2, 3]]      % => [1, 2, 3]
[1 | [2 | [3]]]   % => [1, 2, 3]
[1 | [2 | [3 | []]]]  % => [1, 2, 3]

Pattern Matching and Head/Tail

Every function operating on lists in Erlang is defined in terms of two primitives: head and tail, which return the first element and the rest of the list respectively. Pattern matching with cons cells provides an elegant idiom for recursive list operations, where you can extract the head and tail in function definitions.

% Pattern matching to extract head and tail
[Head | Tail] = [1, 2, 3].
% Head => 1
% Tail => [2, 3]

This makes recursive list processing natural and efficient:

length([]) -> 0;
length([_Head | Tail]) -> 1 + length(Tail).

Proper vs Improper Lists

A proper list in Erlang ends with an empty list as its last cell. When the tail of the last cons cell contains something other than another cons cell or the empty list, you have an improper list:

[1 | 2]      % Improper list - tail is not a list
[1 | [2]]    % Proper list

While improper lists are valid Erlang terms, most standard list functions expect proper lists.

LFE Cons Cells

LFE provides Lisp-style syntax for working with Erlang's cons cell implementation, giving you the best of both worlds: familiar Lisp notation with Erlang's powerful pattern matching.

The cons Function

In LFE, the cons function creates a new list by prepending an element to an existing list:

lfe> (cons 1 '())
(1)
lfe> (cons 1 '(2 3))
(1 2 3)
lfe> (cons 1 (cons 2 (cons 3 '())))
(1 2 3)

Pattern Matching with Cons

LFE allows pattern matching on cons cells in function definitions, making recursive list processing elegant and readable:

lfe> (defun my-length
  (('()) 0)
  (((cons _ tail)) (+ 1 (my-length tail))))
my-length
lfe> (my-length '(a b c d))
4

In this example:

• The first clause matches the empty list () and returns 0
• The second clause uses (cons _ tail) to extract the tail, ignoring the head with _
• The function recursively processes the tail

Common Patterns

Here are some common patterns for working with cons cells in LFE:

Accessing the head and tail:

lfe> (set (cons head tail) '(1 2 3 4))
(1 2 3 4)
lfe> head
1
lfe> tail
(2 3 4)

Building lists incrementally:

lfe> (defun build-list (n)
  (build-list n '()))

lfe> (defun build-list
  ((0 acc) acc)
  ((n acc) (build-list (- n 1) (cons n acc))))

lfe> (build-list 5)
(1 2 3 4 5)

Pattern matching in let bindings:

lfe> (let (((cons first (cons second rest)) '(a b c d e)))
  (list first second rest))
(a b (c d e))

Recursive list transformation:

lfe> (defun double-all
  (('()) '())
  (((cons h t)) (cons (* 2 h) (double-all t))))

lfe> (double-all '(1 2 3 4 5))
(2 4 6 8 10)

Using Backtick Syntax

LFE also supports backtick (quasiquote) syntax for pattern matching, which can be more concise:

lfe> (set `(,first ,second . ,rest) '(1 2 3 4 5))
(1 2 3 4 5)
lfe> first
1
lfe> second
2
lfe> rest
(3 4 5)

The dot (.) in the pattern (,first ,second . ,rest) represents the cons operator, separating the explicitly matched elements from the remaining tail.

List Construction vs Traversal

One crucial performance consideration: prepending to a list with cons is O(1), but appending to the end requires traversing the entire list and is O(n):

; Fast - O(1)
lfe> (cons 0 '(1 2 3))
(0 1 2 3)

; Slow for large lists - O(n)
lfe> (++ '(1 2 3) '(4))
(1 2 3 4)

This is why many recursive functions in LFE build lists in reverse order using an accumulator, then reverse the final result:

(defun map-helper (f lst acc)
  (case lst
    ('() (lists:reverse acc))
    ((cons h t) (map-helper f t (cons (funcall f h) acc)))))

(defun my-map (f lst)
  (map-helper f lst '()))

Predicates

To check if a value is a list (a chain of cons cells ending in []), you can use the standard predicates:

lfe> (is_list '(1 2 3))
true
lfe> (is_list '())
true
lfe> (is_list 42)
false

With Common Lisp-style predicates:

lfe> (include-lib "lfe/include/cl.lfe")
lfe> (listp '(1 2 3))
true
lfe> (listp '())
true

Or Clojure-style:

lfe> (include-lib "lfe/include/clj.lfe")
lfe> (list? '(1 2 3))
true

Summary

Cons cells are the foundation of list processing in LFE. Understanding how they work—as pairs of values forming linked structures—is essential for effective functional programming. The ability to pattern match on cons cells makes LFE code both elegant and efficient, allowing you to express complex list operations with clarity and precision.

The key insights to remember:

• Lists are chains of cons cells terminating in the empty list ()
• Pattern matching with (cons head tail) is the idiomatic way to destructure lists
• Prepending with cons is fast; building lists in reverse and then reversing is a common pattern
• Proper lists always end in [], while improper lists end in other values

Lists and Strings

In most programming languages, lists and strings are distinct creatures—separate types with their own operations, behaviors, and internal representations. In LFE, however, these two concepts share a rather intimate relationship: a string is a list. More precisely, a string is simply a list of integers, where each integer represents the character code of a printable character. This might seem peculiar at first, perhaps even a bit unsettling if you're accustomed to languages where strings are special, privileged types with their own dedicated machinery.

But consider the elegance of this approach: if strings are just lists, then all the wonderful things you can do with lists—pattern matching, recursive processing, higher-order functions—work equally well on strings. Want to reverse a string? Use the same function you'd use to reverse any list. Want to filter characters? Use the same filter function you'd use on a list of numbers or atoms. This unification means fewer concepts to learn, more code reuse, and a consistency that permeates the entire language.

The practical implication is straightforward: when you see "hello" in LFE, you're looking at syntactic sugar for (104 101 108 108 111)—a list of five integers corresponding to the ASCII values of those letters. The REPL helpfully displays lists of printable character codes as strings, but underneath, it's lists all the way down. This chapter explores both lists and strings, first separately to understand their individual characteristics, then together to appreciate how this design choice shapes the way you write LFE code.

Lists

In the beginning, there was the cons cell. And from the cons cell came forth lists, which were chains of cons cells stretching out across memory like improbability particles across the fabric of spacetime—each pointing to the next with what can only be described as dogged determination. And lo, these lists were good, for they were immutable, they were elegant, and most importantly, they made recursion look less like wizardry and more like afternoon tea.

Lists are the fundamental sequential data structure in LFE, inherited from both Erlang's practical heritage and Lisp's theoretical elegance. They are singly-linked lists built from cons cells (as explored in the previous chapter), making them ideally suited for functional programming patterns where you process data from front to back, building new structures as you go rather than mutating old ones.

Creating Lists

The most straightforward way to create a list is with the quote:

lfe> '(1 2 3 4 5)
(1 2 3 4 5)
lfe> '(arthur ford trillian zaphod)
(arthur ford trillian zaphod)
lfe> '()
()

You can also use the list function:

lfe> (list 1 2 3 4 5)
(1 2 3 4 5)
lfe> (list 'arthur 'ford 'trillian 'zaphod)
(arthur ford trillian zaphod)
lfe> (list)
()

Lists can contain elements of mixed types:

lfe> '(42 "meaning" of life 42.0)
(42 "meaning" of life 42.0)
lfe> (list 'atom 42 3.14 "string" #b101010 '(nested list))
(atom 42 3.14 "string" 42 (nested list))

Building lists with cons:

lfe> (cons 1 '())
(1)
lfe> (cons 1 (cons 2 (cons 3 '())))
(1 2 3)
lfe> (cons 'first '(second third))
(first second third)

List Operations

Accessing elements:

lfe> (car '(1 2 3 4 5))
1
lfe> (cdr '(1 2 3 4 5))
(2 3 4 5)
lfe> (cadr '(1 2 3 4 5))  ; equivalent to (car (cdr ...))
2
lfe> (cddr '(1 2 3 4 5))  ; equivalent to (cdr (cdr ...))
(3 4 5)

LFE provides Common Lisp-style accessor combinations up to four levels deep:

lfe> (caddr '(1 2 3 4))   ; third element
3
lfe> (cadddr '(1 2 3 4))  ; fourth element
4

Finding length:

lfe> (length '(1 2 3 4 5))
5
lfe> (length '())
0
lfe> (length '(a (nested (list structure)) here))
3

Appending lists:

lfe> (++ '(1 2 3) '(4 5 6))
(1 2 3 4 5 6)
lfe> (++ '(a) '(b) '(c) '(d))
(a b c d)
lfe> (append '(1 2) '(3 4) '(5 6))
(1 2 3 4 5 6)

Note that ++ is the operator form while append is the function form. Both are O(n) operations where n is the total length of all lists except the last one.

Reversing lists:

lfe> (lists:reverse '(1 2 3 4 5))
(5 4 3 2 1)
lfe> (lists:reverse '())
()

Taking and dropping elements:

lfe> (lists:sublist '(1 2 3 4 5) 3)
(1 2 3)
lfe> (lists:sublist '(1 2 3 4 5) 2 3)
(2 3 4)
lfe> (lists:nthtail 2 '(1 2 3 4 5))
(3 4 5)

Membership testing:

lfe> (lists:member 3 '(1 2 3 4 5))
(3 4 5)
lfe> (lists:member 6 '(1 2 3 4 5))
false

Note that lists:member returns the tail of the list starting with the found element, or false if not found.

Finding elements:

lfe> (lists:nth 1 '(a b c d))
a
lfe> (lists:nth 3 '(a b c d))
c

Be aware that lists:nth uses 1-based indexing (Erlang convention), unlike 0-based indexing common in many languages.

Sorting:

lfe> (lists:sort '(5 2 8 1 9 3))
(1 2 3 5 8 9)
lfe> (lists:sort '(zaphod arthur ford trillian))
(arthur ford trillian zaphod)
lfe> (lists:sort (lambda (a b) (> a b)) '(5 2 8 1 9 3))
(9 8 5 3 2 1)

Higher-Order List Functions

Mapping:

lfe> (lists:map (lambda (x) (* x 2)) '(1 2 3 4 5))
(2 4 6 8 10)
lfe> (lists:map #'atom_to_list/1 '(arthur ford trillian))
("arthur" "ford" "trillian")

Filtering:

lfe> (lists:filter (lambda (x) (> x 3)) '(1 2 3 4 5 6))
(4 5 6)
lfe> (lists:filter #'is_atom/1 '(1 arthur 2 ford 3))
(arthur ford)

Folding (reducing):

lfe> (lists:foldl #'+/2 0 '(1 2 3 4 5))
15
lfe> (lists:foldl (lambda (x acc) (cons x acc)) '() '(1 2 3))
(3 2 1)
lfe> (lists:foldr (lambda (x acc) (cons x acc)) '() '(1 2 3))
(1 2 3)

The difference between foldl (fold left) and foldr (fold right) is the direction of traversal. foldl is tail-recursive and generally more efficient, while foldr processes from right to left.

Flatmapping:

lfe> (lists:flatmap (lambda (x) (list x (* x 2))) '(1 2 3))
(1 2 2 4 3 6)

Zipping:

lfe> (lists:zip '(1 2 3) '(a b c))
((1 a) (2 b) (3 c))
lfe> (lists:zip3 '(1 2 3) '(a b c) '(x y z))
((1 a x) (2 b y) (3 c z))

List Comprehensions

LFE supports powerful list comprehensions that combine filtering, mapping, and Cartesian products:

lfe> (lc ((<- x '(1 2 3 4 5))) (* x 2))
(2 4 6 8 10)

lfe> (lc ((<- x '(1 2 3 4 5 6)) (> x 3)) (* x x))
(16 25 36)

lfe> (lc ((<- x '(1 2 3)) (<- y '(a b))) (tuple x y))
(#(1 a) #(1 b) #(2 a) #(2 b) #(3 a) #(3 b))

The <- operator draws elements from a list, and optional guard conditions filter elements before the body expression is evaluated.

Pattern Matching with Lists

List pattern matching is one of the most elegant features in LFE:

lfe> (set (list first second) '(1 2))
(1 2)
lfe> first
1
lfe> second
2

lfe> (set (cons head tail) '(1 2 3 4))
(1 2 3 4)
lfe> head
1
lfe> tail
(2 3 4)

lfe> (set `(,a ,b . ,rest) '(1 2 3 4 5))
(1 2 3 4 5)
lfe> a
1
lfe> b
2
lfe> rest
(3 4 5)

Predicates

To test if a value is a list, first include the Common Lisp compatibility library:

lfe> (include-lib "lfe/include/cl.lfe")
lfe> (listp '(1 2 3))
true
lfe> (listp '())
true
lfe> (listp 42)
false

Clojure-style predicates:

lfe> (include-lib "lfe/include/clj.lfe")
lfe> (list? '(1 2 3))
true
lfe> (list? "string")
false

Standard Erlang predicate:

lfe> (is_list '(1 2 3))
true
lfe> (is_list '())
true
lfe> (is_list 42)
false

Performance Considerations

Understanding the performance characteristics of list operations is crucial for writing efficient LFE code:

• Prepending (cons) is O(1) — constant time, very fast
• Appending (++, append) is O(n) — requires traversing the entire first list
• Length is O(n) — must traverse the entire list
• Accessing by index is O(n) — must traverse to the position
• Reversing is O(n) — must traverse the entire list

For these reasons, idiomatic LFE code often:

• Builds lists in reverse order then reverses once at the end
• Uses cons instead of append when possible
• Avoids repeated length calculations
• Uses pattern matching instead of index access

Example of the reverse-and-accumulate pattern:

(defun process-list (lst)
  (process-list-helper lst '()))

(defun process-list-helper
  (('() acc) (lists:reverse acc))
  (((cons h t) acc)
   (let ((processed (* h 2)))
     (process-list-helper t (cons processed acc)))))

Common Patterns

Collecting results:

(defun collect-evens
  (('()) '())
  (((cons h t)) (when (== (rem h 2) 0))
   (cons h (collect-evens t)))
  (((cons _ t))
   (collect-evens t)))

Processing pairs:

(defun process-pairs
  (('()) '())
  (((list x)) (list x))
  (((list x y . rest))
   (cons (+ x y) (process-pairs rest))))

Searching:

(defun find-first
  ((_ '()) 'not-found)
  ((pred (cons h t))
   (case (funcall pred h)
     ('true h)
     ('false (find-first pred t)))))

Strings

In LFE, as in Erlang, strings are simply lists of integers representing character codes. This might seem unusual if you're coming from languages with dedicated string types, but it's a perfectly sensible representation that fits naturally with list processing. A string is just a list where each element happens to be an integer in the valid character range.

Creating Strings

String literals are enclosed in double quotes:

lfe> "Hello, World!"
"Hello, World!"
lfe> "The answer is 42"
"The answer is 42"
lfe> ""
""

Under the hood, these are lists of integers:

lfe> (== "ABC" '(65 66 67))
true
lfe> (== "ABC" (list 65 66 67))
true

You can see this more clearly by forcing LFE to show the underlying representation:

lfe> (io:format "~w~n" '("ABC"))
[65,66,67]
ok

Character literals make this relationship explicit:

lfe> (list #\A #\B #\C)
"ABC"
lfe> #\A
65

String Operations

Since strings are lists, all list operations work on strings:

lfe> (++ "Hello, " "World!")
"Hello, World!"
lfe> (lists:reverse "stressed")
"desserts"
lfe> (length "Hello")
5

Accessing characters:

lfe> (car "Hello")
72
lfe> (cdr "Hello")
"ello"
lfe> (lists:nth 2 "Hello")
101

Case conversion:

lfe> (string:uppercase "hello world")
"HELLO WORLD"
lfe> (string:lowercase "HELLO WORLD")
"hello world"
lfe> (string:titlecase "hello world")
"Hello World"

Trimming whitespace:

lfe> (string:trim "  hello  ")
"hello"
lfe> (string:trim "  hello  " 'leading)
"hello  "
lfe> (string:trim "  hello  " 'trailing)
"  hello"

Splitting and joining:

lfe> (string:split "one,two,three" "," 'all)
("one" "two" "three")
lfe> (string:split "a:b:c:d" ":")
("a" "b:c:d")
lfe> (string:split "a:b:c:d" ":" 'all)
("a" "b" "c" "d")

lfe> (string:join '("one" "two" "three") ", ")
"one, two, three"

Searching:

lfe> (string:find "hello world" "world")
"world"
lfe> (string:find "hello world" "universe")
'nomatch

lfe> (string:str "hello world" "world")
7
lfe> (string:str "hello world" "universe")
0

Note that string:str returns 1-based position or 0 if not found.

Replacing:

lfe> (string:replace "hello world" "world" "universe")
"hello universe"
lfe> (string:replace "a,b,c,d" "," ":" 'all)
"a:b:c:d"

Checking prefixes and suffixes:

lfe> (string:prefix "hello world" "hello")
"world"
lfe> (string:prefix "hello world" "goodbye")
'nomatch

lfe> (lists:prefix "hello" "hello world")
true
lfe> (lists:prefix "goodbye" "hello world")
false

lfe> (lists:suffix "world" "hello world")
true

String Predicates

Testing for empty strings:

lfe> (string:is_empty "")
true
lfe> (string:is_empty "hello")
false

Testing if all characters match a predicate:

lfe> (lists:all (lambda (c) (and (>= c #\a) (=< c #\z))) "hello")
true
lfe> (lists:all (lambda (c) (and (>= c #\a) (=< c #\z))) "Hello")
false

Formatting Strings

The format family of functions provides powerful string interpolation:

lfe> (io_lib:format "The answer is ~p" '(42))
"The answer is 42"

lfe> (io_lib:format "~s ~s ~p" '("Hello" "world" 123))
"Hello world 123"

lfe> (io_lib:format "~.2f" '(3.14159))
"3.14"

lfe> (io_lib:format "~10.2.0f" '(3.14159))
"      3.14"

Common format specifiers:

• ~p — pretty-print any term
• ~s — string
• ~w — write in Erlang term format
• ~c — character
• ~f — float
• ~e — exponential notation
• ~.Nf — float with N decimal places
• ~n — newline

For console output, use lfe_io:format or io:format:

lfe> (lfe_io:format "Hello, ~s!~n" '("World"))
Hello, World!
ok

lfe> (io:format "The answer is ~p~n" '(42))
The answer is 42
ok

Converting Between Types

Atoms to strings:

lfe> (atom_to_list 'hello)
"hello"
lfe> (atom_to_list 'hello-world)
"hello-world"

Strings to atoms:

lfe> (list_to_atom "hello")
hello
lfe> (list_to_existing_atom "hello")
hello

Numbers to strings:

lfe> (integer_to_list 42)
"42"
lfe> (integer_to_list 42 16)
"2A"

lfe> (float_to_list 3.14159)
"3.14159000000000009237e+00"
lfe> (float_to_list 3.14159 '(#(decimals 2)))
"3.14"

Strings to numbers:

lfe> (list_to_integer "42")
42
lfe> (list_to_integer "2A" 16)
42

lfe> (list_to_float "3.14159")
3.14159

Binaries to strings:

lfe> (binary_to_list #"Hello")
"Hello"
lfe> (list_to_binary "Hello")
#"Hello"

Unicode Support

LFE strings can contain Unicode characters, though the integer representation may be surprising:

lfe> "Hello, 世界"
"Hello, 世界"

lfe> (length "Hello, 世界")
9

lfe> (lists:nth 8 "Hello, 世界")
19990

For proper Unicode string handling, especially with grapheme clusters, use the Unicode-aware functions in the string module:

lfe> (string:length "Hello, 世界")
9

lfe> (string:slice "Hello, 世界" 7 2)
"世界"

The modern string module (Erlang 20+) handles Unicode correctly and should be preferred over older list-based string functions when working with international text.

Common Patterns

Building strings incrementally:

(defun build-message (name age)
  (++ "Name: " name ", Age: " (integer_to_list age)))

Processing each character:

(defun count-vowels
  (("") 0)
  (((cons c rest))
   (let ((is-vowel (lists:member c "aeiouAEIOU")))
     (if is-vowel
         (+ 1 (count-vowels rest))
         (count-vowels rest)))))

Filtering characters:

(defun remove-spaces
  (("") "")
  (((cons #\space rest)) (remove-spaces rest))
  (((cons c rest)) (cons c (remove-spaces rest))))

Using list comprehensions on strings:

lfe> (lc ((<- c "hello")) (- c 32))
"HELLO"

lfe> (lc ((<- c "hello world") (=/= c #\space)) c)
"helloworld"

Performance Notes

String operations have the same performance characteristics as list operations because they are list operations:

• Concatenation with ++ is O(n) in the length of the first string
• Repeated concatenation in a loop can be O(n²)
• For building large strings, accumulate in a list and call lists:flatten once
• Consider using binaries for large text or when performance is critical

Example of efficient string building:

(defun build-csv-row (items)
  (lists:flatten
    (lists:join ","
      (lists:map #'format-item/1 items))))

Summary

Strings in LFE are lists of character codes—simple, elegant, and slightly peculiar if you're new to this representation. This design choice means:

• All list operations work on strings
• Strings naturally integrate with pattern matching
• String manipulation is intuitive if you think in terms of list processing
• Performance characteristics match those of lists
• Unicode requires some care but is fully supported

For most string operations in modern code, prefer the string module's Unicode-aware functions. For performance-critical text processing with large strings, consider using binaries (covered in a later chapter). But for everyday string manipulation, list-based strings remain a perfectly reasonable and often elegant choice.

Remember: in LFE, as in life, it's not about having a separate type for everything—it's about recognizing that sometimes, a list of integers masquerading as text is exactly what you need.

Bits, Bytes, and Binaries in LFE

Or: How I Learned to Stop Worrying and Love the Bit Stream

Introduction

In most programming languages, manipulating binary data at the bit level is rather like performing surgery with a pickaxe. It's technically possible, and if you squint hard enough while wielding your bit shifts and masking operations, you might even get something resembling the desired result. But it won't be pretty, and there's a non-zero chance you'll need therapy afterward.

LFE, inheriting Erlang's particularly sophisticated relationship with binary data, takes a different approach. Instead of making you grovel through arcane incantations of bitwise operations (though those are available if you're feeling nostalgic), it provides what can only be described as pattern matching on steroids. Or perhaps pattern matching that's been to finishing school and learned proper table manners. The result is that tasks which would normally require careful bit-shuffling and a comprehensive insurance policy can be accomplished with code that looks suspiciously like the specification of the data format you're working with.

This is not an accident. It's the kind of elegant design that makes you wonder what other programming languages have been doing with their time.

Why Binaries Matter

Let us consider, for a moment, the humble byte. Eight bits, representing values from 0 to 255 (or -128 to 127 if you're feeling signed about it). Unremarkable in isolation, perhaps, but stack enough of them together and you have:

• Network protocol packets pretending to be well-behaved structured data
• Image files storing millions of pixels that somehow form pictures of cats
• Audio streams carrying the complete works of humanity's musical genius (and also whatever's popular on the charts)
• Video codecs performing computational miracles that would have made our ancestors believe in magic
• Encrypted data that looks like someone spilled alphabet soup into a blender

In the world of telecommunications, distributed systems, and general data wrangling—which is to say, the world Erlang and LFE were designed for—the ability to efficiently pack, unpack, and pattern-match binary data isn't merely convenient. It's essential. It's the difference between "this will work" and "this will work while handling millions of messages per second without breaking a sweat."

The LFE Advantage

What makes LFE's approach to binaries particularly noteworthy is that it treats binary data as a first-class citizen with its own syntax and pattern-matching capabilities. You don't convert things to binaries as an afterthought; you work with them directly, naturally, and—dare we say it—elegantly.

Consider this: in many languages, if you wanted to parse an MPEG audio frame header (which consists of 11 bits of sync word, 2 bits of version ID, 2 bits of layer description, and so on), you'd be looking at code that resembles a particularly obtuse mathematical proof. In LFE, you write something that looks like:

(binary (11 (sync bits)) (2 (version bits)) (2 (layer bits)) ...)

And that's it. You've just pattern-matched against the bit-level structure of the data. The code reads like the specification. This is the kind of thing that makes programmers weep with joy, or at least nod approvingly while adjusting their glasses.

What You'll Learn

In the chapters that follow, we'll explore:

• The fundamentals: What binaries are, how they're represented in memory, and why the BEAM VM is particularly good at handling them
• Basic construction and manipulation: Creating binaries with the #B(...) syntax and using built-in functions to dissect them
• The bit syntax: The full glory of size specifications, type qualifiers, and pattern matching at arbitrary bit boundaries
• Pattern matching: Because if you can pattern-match lists and tuples, why not pattern-match the individual bits in a TCP packet?
• Binary comprehensions: List comprehensions for binaries, because sometimes you need to transform a stream of bytes and regular loops are just so... imperative
• Real-world applications: From parsing network protocols to implementing codec algorithms, because theory is nice but practice pays the bills

By the end of this exploration, you'll be able to look at a binary data format specification and translate it almost directly into working LFE code. You'll understand why binary pattern matching is one of the secret weapons that makes Erlang and LFE particularly well-suited for building telecommunications systems, distributed databases, and anything else that involves pushing large quantities of structured binary data through pipes at high velocity.

More importantly, you'll have in your toolkit an approach to binary data manipulation that doesn't feel like medieval torture. And in the grand scheme of programming, that's worth quite a lot.

A Note on Bitstrings

Before we dive in, a clarification: when the number of bits in your data is evenly divisible by 8, we call it a binary. When it's not—when you're dealing with, say, 13 bits or 47 bits—we call it a bitstring. Most operations work identically on both, but we'll use the more specific term when the distinction matters. Think of it as the difference between a byte-aligned data structure and one that wandered off the alignment grid to explore more interesting territory.

The convention is simple: binaries are the well-behaved, byte-aligned data structures your file I/O operations expect. Bitstrings are their more adventurous cousins that hang out at odd bit boundaries, usually because some protocol designer thought it would be clever to pack flags into 3 bits instead of wasting a whole byte.

Both are equally valid. Both are equally supported. And both, as we shall see, are equally elegant to work with in LFE.

Right, then. Shall we begin?

What Binaries Are

The Nature of the Beast

A binary, in the LFE (and Erlang) universe, is fundamentally a reference to a chunk of raw, untyped memory. If you're coming from a background in higher-level languages where everything must be wrapped in three layers of abstraction and a bow tie before it's considered respectable, this might feel a bit like meeting someone who still knows how to start a fire without matches. Primitive, perhaps, but undeniably useful when you need to cook dinner.

Binaries were originally conceived for a rather specific purpose: loading code over networks in the Erlang runtime system. Imagine trying to send compiled programs between machines in the 1980s, when network bandwidth was measured in units that would make modern developers weep. You needed something efficient, something compact, something that wouldn't spend more time being transferred than actually running. Thus, the binary was born—a data structure optimized for moving large quantities of bits around without the overhead of, say, converting everything to strings and hoping for the best.

What the designers discovered, rather like scientists who set out to invent a better glue and accidentally created Post-It notes, was that binaries were extraordinarily useful for many things. Socket-based communication. Protocol parsing. Codec implementation. Any situation where you need to deal with data in its raw, unadorned, byte-level glory.

Memory Representation

In memory, a binary is essentially a sequence of bytes (which are, if we're being pedantic, sequences of 8 bits each). The system represents them efficiently—more efficiently, in fact, than lists. This might come as a surprise if you've been taught that lists are the fundamental data structure and everything else is just lists in disguise. But consider: a list in Erlang (and thus LFE) is a linked list, where each element requires not only its own storage but also a pointer to the next element. A binary is more like a C array: a contiguous block of memory with minimal overhead.

For small binaries (less than 64 bytes), the data is typically stored directly in the process heap. For larger binaries, they're stored in a shared memory area and processes keep references to them. This means that when you send a large binary in a message between processes, you're not copying the entire thing—you're just passing a reference. It's rather like instead of photocopying a lengthy document and mailing it, you simply tell someone where the filing cabinet is.

This reference-based approach has pleasant implications for garbage collection as well. Large binaries can be shared among many processes without being duplicated, and they're only freed when the last reference disappears. It's communism for data structures, but the kind that actually works.

Visual Representation

When the REPL prints a binary, it uses the notation #B(...) (or sometimes <<...>> if you're looking at Erlang documentation). The contents are displayed as integers, each representing a byte:

lfe> #B(1 2 3)
#B(1 2 3)

If the binary happens to contain bytes that form printable ASCII characters, the shell might helpfully display it as a string:

lfe> #B(72 101 108 108 111)
#"Hello"

Note the #"..." notation for binary strings. This is the shell being clever, recognizing that the bytes 72 101 108 108 111 correspond to the ASCII codes for "Hello" and presenting them in a more readable format. The bytes haven't changed—only their cosmetic presentation has been upgraded for human consumption.

This is similar to how the number 42 and the string "42" are different beasts, even though they might occasionally be mistaken for each other at parties. One is a numeric value; the other is a sequence of character codes that, when interpreted by a human visual system, evoke the concept of forty-two. The distinction matters more than you might think.

The Binary/Bitstring Distinction

As mentioned in our introduction, binaries are the special case of bitstrings where the number of bits is evenly divisible by 8. A bitstring with 16 bits? That's a binary. A bitstring with 15 bits? That's still a bitstring, but it's not quite respectable enough to join the binary club. It's the data structure equivalent of showing up to a formal dinner in jeans and a t-shirt—perfectly valid, but people will notice.

The important point is that most operations work identically on both. Pattern matching doesn't care if you're byte-aligned or wandering off into the wilderness of arbitrary bit boundaries. The system handles both with equal grace, which is more than can be said for most programming languages' relationship with non-byte-aligned data.

You can verify what you're dealing with using the predicates (binary? x) and (bitstring? x):

lfe> (binary? #B(1 2 3))
true
lfe> (bitstring? #B(1 2 3))
true
lfe> (let ((b (binary (1 (size 9)))))  ; 9 bits
      (tuple (binary? b) (bitstring? b)))
#(false true)

Note that all binaries are bitstrings, but not all bitstrings are binaries. It's a classic is-a relationship, the kind that object-oriented programmers spend their entire careers thinking about but which here just... works, without requiring seventeen layers of inheritance and a PhD in type theory.

Why Not Just Use Lists?

A reasonable question. After all, you could represent binary data as a list of integers from 0 to 255. Many languages force you to do exactly that. But consider:

1. Space efficiency: A list needs space for both the data and the links between elements. A binary just needs space for the data. For large data structures, this difference can be substantial enough to matter, even in the age of abundant RAM.

2. Access patterns: Lists are optimized for sequential access from the front. Binaries can be indexed directly. If you need to extract the 47th byte, you don't have to walk through the first 46 elements holding hands with yourself.

3. Message passing: When you send a binary in a message, you're sending a reference. When you send a list, you're sending... well, the entire list, links and all. For large data structures, one of these is significantly less expensive than the other.

4. Pattern matching performance: Pattern matching on binary structures is highly optimized in the BEAM VM. It's the kind of optimization that the Erlang team spent years perfecting because they needed it for telecommunications. You get the benefits of their decades of effort for free.

Lists are wonderful data structures. They have their place, and that place is wherever you need flexible, dynamic, recursively-defined collections of things. But for raw bytes and bits? Binaries are simply better suited to the task, in the same way that a screwdriver is better suited to turning screws than is a particularly determined hammer.

An Important Property: Immutability

Like all data structures in LFE, binaries are immutable. Once created, they cannot be modified. If you need to "change" a binary, you create a new one. This might seem inefficient until you realize that the BEAM VM is rather clever about this—it can often share unchanged portions of the data structure and only allocate new space for the parts that actually differ.

This immutability has profound implications for concurrent programming, which is rather the point of Erlang and LFE. You never need to worry about another process modifying a binary out from under you, because it simply cannot happen. The binary you have is the binary you'll always have, unchanging and eternal, like a particularly stubborn philosophy professor or the value of e.

In practice, this means you can pass binaries between processes with abandon, secure in the knowledge that they won't suddenly transform into something else when you're not looking. In a world of concurrent processes and message passing, this guarantee is worth its weight in gold. Or at least in silicon.

Basic Binary Syntax

In the beginning, there was #B(...). Or possibly <<...>> if you're consulting Erlang documentation and wondering why everyone keeps using those cheerful angle brackets. Both notations refer to the same underlying concept, though LFE, having a certain aesthetic sensibility about parentheses, prefers the former. It's the difference between wearing a properly tailored suit and showing up in whatever was clean that morning—functionally equivalent, but one makes a better impression at job interviews.

The simplest possible binary is the empty one:

lfe> #B()
#B()

This binary contains exactly nothing, which might seem like an odd thing to create deliberately, but philosophers have been creating nothing deliberately for millennia and they're considered rather clever. In practice, the empty binary occasionally proves useful as a starting point for accumulation patterns, or when you need to pass a binary but have no actual data. It's the binary equivalent of a polite nod of acknowledgment.

Constructing Simple Binaries

To create a binary containing actual data, you simply list the values between the parentheses:

> #B(1 2 3)
#B(1 2 3)

Each integer you provide becomes a byte in the resulting binary. The values must be in the range 0 to 255, because that's what fits in 8 bits, and attempting to stuff larger numbers into bytes is a bit like trying to fit an elephant into a phone booth—technically possible if you're willing to ignore conventional constraints like physics, but generally frowned upon by those who have to clean up afterward.

> #B(42)
#B(42)
> #B(255)
#B(255)
> #B(256)
; This will result in an error, because 256 requires 9 bits

The system will politely inform you when you've exceeded the byte boundary, typically by throwing an exception. It's being helpful, really. Better to know at compile time than to discover later that your carefully constructed binary has been truncated in mysterious and exciting ways.

Binary Strings

A particularly convenient special case involves text. Since strings in many contexts are just sequences of character codes, you can create a binary directly from string literals:

lfe> #"Hello"
#"Hello"

This is exactly equivalent to:

lfe> #B(72 101 108 108 111)
#"Hello"

The ASCII codes for H, e, l, l, and o are 72, 101, 108, 108, and 111, respectively. You can verify this with a character code lookup, or by asking any passing computer scientist, though the latter approach may result in a lengthy discussion about character encodings and why Unicode was necessary and why UTF-8 is simultaneously brilliant and a source of ongoing existential dread.

Binary strings are particularly efficient for storing text that you don't plan to manipulate character-by-character. If you're reading a file and want to keep its contents in memory without converting everything to a list (which, remember, has pointer overhead for every element), binary strings are your friend. A friend who doesn't talk much and just does their job efficiently, like a particularly competent butler.

The syntax #"..." is exactly like the list string syntax "...", except the result is a binary instead of a list. Everything about escape sequences works identically:

lfe> #"Line 1\nLine 2"
#"Line 1\nLine 2"
lfe> #"Tab\there"
#"Tab\there"

You can even use the hexadecimal escape sequences:

lfe> #"\x48\x65\x6c\x6c\x6f"
#"Hello"

Though one might reasonably ask why you'd want to write "Hello" in hexadecimal, unless you're being deliberately obscure for job security purposes. Security through obscurity might not work for cryptography, but it works reasonably well for ensuring you're the only person who can maintain your code.

Mixing Integers and Strings

The #B(...) syntax allows you to mix integers and strings freely:

lfe> #B(1 2 "Hello" 3 4)
#B(1 2 72 101 108 108 111 3 4)

Note how the string "Hello" was expanded into its constituent byte values. The binary doesn't remember that those bytes once belonged to a string—it just stores the raw values. The distinction between "a byte containing 72" and "the letter H" exists only in our meat-based interpretation systems, not in the silicon.

This mixing capability is occasionally useful when you need to construct a binary containing both protocol-specific numeric values and actual text data. Which is to say, it's useful frequently, because protocols are designed by people who can never quite decide whether they want to use magic numbers or human-readable identifiers.

Nested Binaries

You can nest binaries within binary construction:

lfe> (let ((b1 #B(1 2))
        (b2 #B(3 4)))
    #B(b1 b2))
#B(1 2 3 4)

This concatenates the binaries, flattening them into a single contiguous sequence. It's the binary equivalent of appending lists, except more efficient because you're not walking linked structures and rebuilding chains of cons cells. You're just stacking memory blocks end-to-end like a particularly orderly game of digital Tetris.

The nested binary must, of course, be bound to a variable. Attempting to write #B(#B(1 2)) will confuse the parser, much like attempting to say "this sentence is false" will confuse certain types of philosophers and most logical paradox detection systems.

Variables in Binary Construction

You can use variables in binary construction, as we just demonstrated:

lfe> (let ((x 42))
    #B(x))
#B(42)

The variable is evaluated and its value is inserted into the binary. This works as long as the value is something sensible—an integer in the correct range, another binary, or a string. Attempting to insert tuples, lists, or atoms directly will result in the system politely declining your request, typically by throwing a bad argument exception.

If you want to insert complex terms into binaries, you'll need to serialize them first, which we'll discuss in a later chapter. For now, just remember: binaries like numbers and other binaries. They're not particularly sociable with the more structured data types, preferring to maintain their raw, untyped aesthetic.

Efficiency

When you construct a binary, the system allocates the appropriate amount of memory and copies the data into it. This is a relatively fast operation, but it's still worth being aware of. If you're building up a large binary piece by piece, you might want to accumulate the pieces in a list and construct the final binary all at once:

(let ((pieces (list #B(1 2) #B(3 4) #B(5 6))))
  (list-to-binary pieces))

This is more efficient than repeatedly appending to a binary, because immutable data structures mean that each append operation creates a new binary. Better to collect everything and do one allocation than to do many allocations as you build up the structure incrementally.

Of course, premature optimization is the root of all evil (or at least the root of spending three weeks optimizing code that runs once at startup and accounts for 0.0001% of your execution time). But when dealing with large binaries in performance-critical code, this distinction can matter. The BEAM VM is fast, but physics still applies, and copying gigabytes of data byte-by-byte is still gigabytes of data that need to be copied.

The List-to-Binary Bridge

Speaking of list-to-binary, this BIF (Built-In Function) is your primary tool for converting list-based data into binary form:

lfe> (list-to-binary '(72 101 108 108 111))
#"Hello"

It accepts what Erlang calls an "iolist"—a structure that can be a flat list of integers (0-255) and binaries, or a nested structure of the same. The nesting can be arbitrarily deep, because sometimes data structures grow organically and you don't want to flatten everything just to convert it to a binary:

lfe> (list-to-binary (list #B(1 2) (list 3 4) #B(5 6)))
#B(1 2 3 4 5 6)

Note how the nested list (list 3 4) was automatically flattened. The system understands that you want all the bytes, regardless of how they were structured in your list. It's remarkably forgiving, which is pleasant when you're dealing with data that was accumulated through various means and hasn't been tidied up into a perfectly flat structure.

Summary

Basic binary construction in LFE is refreshingly straightforward: put integers and strings between #B( and ), and out comes a binary. Variables work. Nesting works. It's the kind of intuitive design that makes you wonder why other languages make it so complicated. The answer, as is often the case, involves historical accidents, committee decisions, and the fact that once something is in a language specification, removing it requires an act of international treaty negotiation and possibly divine intervention.

For now, we have what we need: a simple, expressive syntax for creating binaries. In the next sections, we'll explore what to do with them once they exist, which is where things get interesting enough that people wrote their doctoral theses on it. Not because it's complicated—it isn't—but because it's powerful in ways that aren't immediately obvious until you try to accomplish the same things in languages that don't have bit syntax and realize you've been sent to extract data from a binary format armed only with a toothpick and determination.

Binary BIFs

The Erlang runtime system, in its infinite wisdom (or at least its well-considered pragmatism developed over three decades of telecommunications infrastructure), provides a collection of Built-In Functions—BIFs, to those in the know—for manipulating binaries. These are not functions written in Erlang or LFE themselves, but rather direct calls into the runtime's implementation in C, which means they're fast in the way that things are fast when they don't have to worry about abstractions or politeness and can just get on with the business of moving bytes around memory.

Think of BIFs as the difference between asking someone nicely to complete a task and simply reaching into their brain and adjusting the relevant neurons directly. More efficient, certainly, though probably less socially acceptable in most cultures.

list-to-binary: The Great Flattener

We've already met list-to-binary, but it deserves a more formal introduction:

(list-to-binary list-of-bytes-and-binaries)

This function takes what Erlang calls an "iolist"—a deeply recursive definition that essentially means "a list containing integers (0-255), binaries, or other iolists." The function then flattens this entire structure into a single contiguous binary, removing all the list overhead and leaving you with just the raw bytes.

lfe> (list-to-binary '(1 2 3))
#B(1 2 3)
lfe> (list-to-binary (list #B(1 2) 3 (list 4 5) #B(6)))
#B(1 2 3 4 5 6)

Note how nesting is handled transparently. The function doesn't care how your data is structured—it cares only about the bytes, and those it will find no matter how deeply you've hidden them in nested lists. It's like having a particularly determined detective who will find what they're looking for even if you've buried it under seventeen layers of bureaucratic paper trails and misdirection.

binary-to-list: The Decomposer

The inverse operation is equally straightforward:

(binary-to-list binary)

This extracts each byte from the binary and produces a list of integers:

lfe> (binary-to-list #B(1 2 3))
(1 2 3)
lfe> (binary-to-list #"Hello")
(72 101 108 108 111)

Why would you want to convert a space-efficient binary into a less-efficient list? Perhaps you need to manipulate individual elements and the list structure makes that more convenient. Perhaps you're interfacing with code that expects lists. Perhaps you're being deliberately perverse for artistic purposes. The function doesn't judge; it merely transforms.

There's also a variant that converts a slice of the binary:

(binary-to-list binary start end)

Where start and end are byte positions (1-indexed, because someone, somewhere, decided that counting should start at 1 despite computers preferring 0, and we're all just living with the consequences). This allows you to extract a portion of the binary without converting the entire thing to a list first.

split-binary: The Precision Slicer

When you need to divide a binary into two parts at a specific position:

(split-binary binary position)

This returns a tuple containing two binaries—the part before the position and the part from the position onward:

lfe> (split-binary #B(1 2 3 4 5 6 7 8 9 10) 3)
#(#B(1 2 3) #B(4 5 6 7 8 9 10))

Note that the position is 0-indexed (unlike the list version), because different functions in the standard library have different opinions about whether counting should start at 0 or 1, and consistency is apparently for lesser languages. This is the kind of thing that makes perfect sense when you understand the historical context and which committee member was arguing for what during the standardization process, and makes no sense whatsoever if you expect software design to be guided by principles like "intuitive interface design" or "not making users remember special cases."

The first binary contains position bytes, and the second contains everything else. If you try to split at a position larger than the binary's size, you'll get an error of the "what did you expect to happen?" variety. If you split at position 0, you get an empty binary and the original binary. If you split at the size of the binary, you get the original binary and an empty binary. Edge cases are handled with all the enthusiasm of someone filing tax returns, which is to say, correctly but joylessly.

term-to-binary: The Universal Serializer

This is where things get interesting enough that people write papers about it:

(term-to-binary any-erlang-term)

This function takes any Erlang/LFE term—atoms, integers, lists, tuples, maps, binaries, other terms nested arbitrarily deep—and converts it into a binary representation. This binary uses the "Erlang external term format," which is precisely specified and designed to be efficient both in space and encoding/decoding time.

lfe> (term-to-binary '#(test 12 true (1 2 3)))
#B(131 104 4 100 0 4 116 101 115 116 97 12 100 0 4 116 114
   117 101 107 0 3 1 2 3)

Those numbers might look like line noise, but they're actually a carefully structured representation of the tuple #(test 12 true (1 2 3)). The format includes type tags, length specifications, and the actual data, all packed efficiently into as few bytes as possible.

This is extraordinarily useful for:

• Storing complex data structures in files
• Sending data structures over network sockets
• Implementing persistence layers
• Debugging (though the binary output is not particularly human-readable without tools)
• Implementing distributed Erlang (which uses this format for inter-node communication)

The format is version-stable, meaning binaries created with older versions of Erlang can generally be read by newer versions. This is the kind of forward compatibility that database designers dream about and which distributed systems rely upon to avoid version upgrade nightmares where everything must be upgraded simultaneously or face the consequences (those consequences typically involving angry phone calls at 3 AM and rapidly updating one's résumé).

binary-to-term: The Reanimator

byte-size: The Measurer

bit-size: The Precise Measurer

Summary

Bit Syntax Fundamentals

The General Form

Segments: The Atomic Unit

Size Specifications

Default Sizes

Type Annotations

Combining Size and Type

Variables in Segments

Philosophical Implications

Type Specifiers

The Type Itself

Sign

Endianness

The Unit Specifier

Combining Specifiers

Special Cases and Edge Conditions

Summary

Size and Unit Specifications

Default Sizes

Explicit Size Specifications

The Unit Specification

The Constraint of Divisibility by Eight

Practical Example: Fixed-Width Records

The Pattern Matching Constraint

A Note on UTF Strings

Summary

Endianness

The Parable of Big-Endian and Little-Endian

The Endianness Specifiers in LFE

When Endianness Matters

The Native Option: Runtime Flexibility

Pattern Matching with Endianness

Cross-Platform Binary Protocols

A Historical Note

Practical Examples

Summary

Pattern Matching Binaries

The Fundamental Symmetry

Basic Pattern Matching Examples

The Rest of the Binary

Don't Care Variables

Pattern Matching in Function Clauses

Bit-Level Pattern Matching

The Dual Nature of Size Variables

Complex Real-World Example: Decoding a TLV Structure

Pattern Matching with Guards

Common Pitfalls and How to Avoid Them

Summary

Binary Comprehensions

The Basic Syntax

Binary to List: The Simplest Case

List to Binary: The Reverse Journey

Transforming Values

Binary to Binary: The Power Move

Bit-Level Manipulation

Practical Example: RGB Color Manipulation

Multiple Generators: The Cartesian Product

Bit String Manipulation

Complex Example: Run-Length Encoding

Advanced Pattern: Sliding Window

Performance Considerations

Summary

Bitstrings

What Bitstrings Are

Creating Bitstrings

Size Queries

Pattern Matching Bitstrings

Appending Bitstrings

Practical Example: Morse Code

Variable-Length Encoding Example

The Alignment Problem

Extracting Bits from Bytes

Why Bitstrings Matter

Performance Characteristics

Summary

Bitwise Operators

Bitwise AND

Bitwise OR

Bitwise XOR

Bitwise NOT

Bit Shift Left

Bit Shift Right

Practical Use: Extraction and Division

Combining Operators: Real-World Example

Performance Characteristics

When to Use Bitwise Operators

A Note on Negative Numbers

Summary

Serialization

The Basics

Why Serialize

A More Sophisticated Example: The Deep Space Signal

The Secret Sauce: What Gets Preserved

Practical Considerations: Size and Compression

Storage Patterns: Files and Mnesia

Versioning and Evolution: The Heat Death Problem

The Distributed Angle: Cross-Node Communication

Security Considerations: Trust No Binary

Summary

Real-World Applications

Example 1: Dissecting IPv4 Datagrams

Example 2: Building a Custom Binary Protocol

Example 3: Log File Format with Binary Efficiency

Performance Considerations: The Need for Speed

Debugging Binary Code: When Patterns Don't Match

Summary

Conclusion: The Binary Achievement

"For a moment, nothing happened. Then, after a second or so, nothing continued to happen."

— Douglas Adams

With binary data in LFE, however, quite a lot happens, and it happens with a precision and elegance that would make a Swiss watchmaker weep with professional jealousy.

What We've Learned

We began this journey in a state of innocence—or possibly ignorance—regarding the true nature of binary data. We end it having witnessed the full majesty of LFE's approach to bits and bytes, which can only be described as "what would happen if pattern matching and binary data had a child that was subsequently raised by telecommunications engineers with exacting standards and a distaste for unnecessary complexity."

Let us review what we've discovered, shall we?

The Fundamental Insights

1. Binaries Are First-Class Citizens
2. The Bit Syntax Is Pattern Matching for Bits
3. Construction and Deconstruction Are Symmetric
4. Bitstrings Liberate Us From Byte Boundaries
5. Binary Comprehensions Extend List Comprehensions
6. Bitwise Operators Complement the Bit Syntax
7. Serialization Provides Persistence

The Conceptual Shift

Perhaps the most important lesson is the conceptual shift that LFE's binary handling represents. In most languages, binary data manipulation is:

1. Imperative: You write code that explicitly manipulates bytes
2. Error-prone: Off-by-one errors, endianness mistakes, sign confusion
3. Obscure: The code doesn't resemble the data format specification
4. Performance-focused: Often written in low-level languages for speed

In LFE, binary data manipulation is:

1. Declarative: You describe the structure, the system handles the details
2. Correct: Pattern matching fails cleanly, no buffer overruns
3. Readable: Code documents the data format
4. Still Fast: The BEAM VM optimizes these operations extensively

This is the difference between:

// C version - manual bit manipulation
unsigned int value = (buffer[0] << 24) | (buffer[1] << 16) |
                     (buffer[2] << 8) | buffer[3];

And:

;; LFE version - declarative structure description
(binary ((value (size 32) big)) buffer)

Both extract a 32-bit big-endian integer, but only one makes you double-check the bit shift amounts and worry about whether you got the byte order right.

The Performance Reality

A reasonable concern: doesn't all this abstraction come with performance costs?

The answer is nuanced. The BEAM VM has spent decades optimizing binary operations because Erlang was designed for telecommunications systems handling millions of messages per second. The bit syntax compiles to specialized VM instructions that are highly optimized.

discipline to not micro-optimize code that runs perfectly adequately.

The Philosophical Takeaway

Binary data in LFE represents something deeper than just "a convenient way to handle bytes." It represents a philosophy of language design where:

1. Domain problems drive language features: Erlang needed to parse protocols, so it got bit syntax
2. Abstractions should clarify, not obscure: The bit syntax makes structure explicit
3. Similar tasks should have similar syntax: Binary comprehensions mirror list comprehensions
4. Safety and expressiveness aren't opposed: Pattern matching provides both

This is design by people who had real problems to solve and weren't satisfied with solutions that merely worked—they wanted solutions that worked well and could be understood six months later at 3 AM when production was down and the logs were unhelpful.

Where to Go From Here

Having absorbed these concepts, you're equipped to:

1. Parse binary protocols: Network protocols, file formats, data streams
2. Implement codecs: Compression, encryption, encoding schemes
3. Interface with hardware: Bit-level register manipulation, device protocols
4. Build distributed systems: Using serialization for persistence and communication
5. Optimize data structures: Bit packing, custom binary formats

A Final Note

In the grand taxonomy of programming language features, binary handling is rarely exciting. It's not as philosophically interesting as type systems, not as practically dramatic as garbage collection, and not as immediately gratifying as first-class functions. It's the plumbing of the programming world—essential, but not glamorous.

Yet LFE's approach to binary data is worth studying precisely because it takes something mundane and makes it elegant. It takes a domain where most languages say "here's malloc(), good luck" and provides sophisticated pattern matching, comprehensions, and a syntax that reads like the specifications you're implementing.

This is what good language design looks like: taking hard problems and making them approachable without sacrificing power or performance. It's the difference between a language that lets you solve problems and a language that helps you solve problems. The first provides tools. The second provides tools and shows you how to use them effectively.

Tuples

Property Lists

Maps

Arrays

Dicts

Graphs

Queues

Records

Pattern Matching

Generic Sequence Functions

Part III - Data as Code

Where Part II explored how LFE represents information through data structures, Part III reveals how those same structures spring to life as executable programs. In languages such as Lisp, data structures can be directly entered and evaluated as code, and LFE exemplifies this principle through its evaluation model and execution semantics.

The transition from data to code occurs through evaluation—the process by which static data structures become dynamic computations. Part III begins with expressions and functions, showing how lists like (factorial 5) transform from mere data into function calls that produce results. We explore how LFE's evaluator breathes life into these structures, turning symbols into function references and nested lists into complex program flows.

The concept of treating code as data and the manipulation and evaluation thereof becomes particularly powerful when we examine processes and message passing. In LFE, not only can you construct programs as data, but you can also send code between processes, enabling sophisticated distributed computing patterns. A process might receive a data structure that represents a function, evaluate it locally, and send the results elsewhere in the system.

The later chapters of Part III demonstrate how data-as-code principles scale to larger programming constructs. Modules and packages organize collections of functions and data, while still maintaining the fundamental property that program structure can be analyzed and manipulated as data. This makes metaprogramming easier than in a language without this property, as you can write LFE programs that generate, modify, or analyze other LFE programs using the same data manipulation techniques you use for ordinary programming tasks.

By Part III's conclusion, you will understand not just how to write LFE programs, but how LFE programs can write themselves—the ultimate expression of the data-as-code philosophy that makes Lisp family languages uniquely powerful for both practical programming and theoretical computer science.RetryClaude can make mistakes. Please double-check cited sources.

Expressions

Functions

Closures

Evaluation

Flow of Control

Processes

Messages and Their Passing

Objects and Flavors

I/O

Accessing Files

Modules

Packages

Scripting with LFE

Creating LFE Projects

Using rebar3

Project Layout Conventions

Part IV - Advanced Topics

Having firmly established in our minds the fundamental data structures, evaluation semantics, and basic programming constructs of LFE, Part IV ventures into the advanced territories where LFE truly demonstrates its power as both a Lisp and a dialect of Erlang. These chapters explore the sophisticated techniques and system-level capabilities that distinguish expert LFE programmers from casual users, covering everything from metaprogramming to distributed computing.

The journey begins with the practical realities of software development: errors and debugging. In LFE, error handling is not an afterthought but a fundamental design principle inherited from Erlang's "let it crash" philosophy. Lisp pioneered many ideas in computer science, including automatic storage management and error handling, and LFE extends these concepts with Erlang's supervision trees and process isolation. Understanding how to debug LFE programs means understanding both the introspective capabilities of Lisp and the fault-tolerance mechanisms of OTP.

Testing in LFE spans multiple paradigms, from traditional unit testing to property-based testing with Propr. The latter represents a particularly powerful approach where you specify the properties your functions should satisfy, and the testing framework generates thousands of test cases automatically. This mathematical approach to testing aligns naturally with LFE's functional programming roots while leveraging Erlang's QuickCheck heritage.

The chapter on macros reveals LFE's true metaprogramming capabilities. This property allows code to be treated as data and vice versa, enabling powerful metaprogramming techniques, and LFE macros exemplify this principle. Unlike preprocessor macros in other languages, LFE macros operate on the parsed abstract syntax tree, allowing you to extend the language itself. You can create new control structures, embed domain-specific languages, and transform code in ways that would be impossible in non-homoiconic languages.

Distributed LFE programming showcases one of the platform's greatest strengths. The actor model that underlies Erlang extends seamlessly across network boundaries, allowing LFE processes to communicate transparently whether they're running on the same core or on opposite sides of the globe. This capability, combined with OTP's battle-tested distribution protocols, enables building systems that scale horizontally while maintaining the same programming model.

Ports and port drivers represent LFE's gateway to the broader computing ecosystem. While the BEAM virtual machine provides excellent isolation and fault tolerance, real systems often need to interface with external programs, databases, or hardware. Ports allow you to communicate safely with external processes, while port drivers provide high-performance integration for systems programming tasks.

The final chapters on servers and clients bring together all these advanced concepts in the context of building production systems. LFE servers can handle thousands of concurrent connections while maintaining state consistency and fault tolerance. Client architectures demonstrate how to build resilient distributed applications that gracefully handle network partitions and service failures.

Part IV represents the transition from learning LFE to mastering it. These advanced topics require not just technical knowledge but a deep understanding of the design philosophies that make LFE effective: the principle that complexity should be managed through composition rather than monolithic design, that failure should be expected and planned for rather than avoided, and that the most powerful abstractions are often the simplest ones consistently applied. By the end of Part IV, you'll have the tools and knowledge to build LFE systems that are not just functional, but elegant, robust, and scalable.

Errors and Debugging

Writing Unit Tests

The Common Test Framework

The Propr Test Framework

The Compiler

Macros

Distributed LFE

Ports and Port Drivers

Servers

Clients

Part V - OTP

Open Telecom Platform (OTP) represents one of the most sophisticated and battle-tested frameworks for building distributed, fault-tolerant systems ever created. Originally developed by Ericsson for telecommunications equipment that must run continuously for years without failure, OTP has evolved into a general-purpose platform for building any system that demands high availability, scalability, and reliability. Part V explores how LFE leverages this remarkable foundation to create applications that embody the famous "nine nines" reliability (99.9999999% uptime) that has made Erlang legendary in industries where downtime is measured in millions of dollars per minute.

OTP is built around a fundamental insight: most distributed systems follow predictable patterns, and these patterns can be abstracted into reusable components called behaviours. Rather than reimplementing the complex logic of process supervision, state management, or event handling from scratch, OTP provides tested implementations that handle the difficult edge cases while allowing you to focus on your application's unique business logic. The behaviours chapter introduces these powerful abstractions - gen_server for stateful services, gen_statem for complex state machines, supervisor for fault tolerance, and others that form the building blocks of robust systems.

Applications in OTP are more than just programs - they are complete, self-contained units that can be started, stopped, and supervised as atomic entities. An OTP application encapsulates not just code, but also configuration, dependencies, and metadata about how the application should behave in various circumstances. This packaging model enables sophisticated deployment strategies and makes it possible to upgrade running systems without stopping them - a capability that has allowed some Erlang systems to achieve continuous operation measured in decades.

The progression from applications to releases represents the scaling of OTP concepts from individual components to complete systems. A release bundles multiple applications together with the Erlang runtime itself, creating a self-contained deployment artifact that can run on any compatible platform. OTP's release handling tools support hot code swapping, allowing you to upgrade production systems while they continue serving requests - a capability that seems almost magical until you understand the careful engineering that makes it possible.

Tables and databases in the OTP context extend far beyond simple data storage. ETS (Erlang Term Storage) and DETS (Disk ETS) provide high-performance, concurrent access to in-memory and persistent data structures, while Mnesia offers a distributed database designed specifically for telecom applications. These storage systems are deeply integrated with OTP's supervision and distribution mechanisms, ensuring that data remains consistent and available even in the face of node failures or network partitions.

The example OTP project serves as the capstone of Part V, demonstrating how all these concepts integrate into a complete, production-ready system. You'll see how behaviours coordinate to handle complex workflows, how applications compose to create larger systems, and how OTP's supervision trees ensure that failures in one component don't cascade throughout the system. Lisp pioneered many ideas in computer science, including automatic storage management and recursion, and OTP extends these concepts with process supervision and automatic restart capabilities that make systems truly self-healing.

What makes OTP particularly powerful in the context of LFE is how it preserves the interactive, exploratory nature of Lisp development while providing the industrial-strength reliability guarantees needed for production systems. You can experiment with OTP behaviours at the REPL, hot-swap code in running applications, and inspect system state using the same tools you use for everyday development. This combination of power and accessibility has made OTP the secret weapon behind some of the world's most reliable distributed systems.

By the end of Part V, you'll understand not just how to use OTP, but why it represents such a significant advancement in systems programming. You'll have the knowledge to build applications that gracefully handle the complexities of distributed computing - network failures, partial updates, cascading errors, and load balancing - using patterns that have been refined through decades of real-world deployment in some of the most demanding environments on Earth.

OTP Behaviours

Applications

Releases

Tables and Databases

Example OTP Project

Part VI - Tooling

The most elegant programming language in the world is only as powerful as the tools that support it. Part VI explores the ecosystem that transforms LFE from an interesting academic exercise into a practical platform for building real software. At the heart of this ecosystem lies rebar3, the de facto standard build tool for Erlang and LFE projects, extended with a comprehensive plugin that brings first-class LFE support to every aspect of the development lifecycle.

Rebar3 represents more than just a build system - it embodies the collective wisdom of the Erlang community about how software projects should be structured, built, tested, and deployed. Originally created to manage the complex dependencies and compilation requirements of Erlang applications, rebar3 has evolved into a sophisticated project management platform that handles everything from generating initial project scaffolding to creating production releases. The LFE plugin seamlessly integrates into this ecosystem, ensuring that LFE developers can leverage the same battle-tested workflows that have made Erlang development so productive.

The Quick Start section demonstrates how modern tooling can dramatically reduce the friction of getting started with a new language. What once required understanding complex compilation pipelines and dependency management can now be accomplished with a few simple commands. You'll go from an empty directory to a running LFE application in minutes, with a project structure that follows community conventions and includes everything needed for testing, documentation, and deployment.

The plugin reference reveals the depth of integration between LFE and the broader Erlang ecosystem. The rebar3 LFE plugin doesn't just compile LFE code - it understands LFE's unique characteristics and provides specialized support for features like macros, packages, and interactive development. Whether you're running unit tests with eunit, exploring code at the REPL, or building complex multi-application releases, the plugin ensures that LFE feels like a first-class citizen in the Erlang world.

Project creation commands showcase the flexibility of modern LFE development. Different project types - libraries, standalone applications, escripts, OTP applications, and full releases - each come with appropriate scaffolding and configuration. This isn't just convenience; it's the embodiment of best practices accumulated over years of real-world development. When you generate a new LFE project, you're not starting from scratch - you're beginning with a foundation that incorporates decades of lessons learned about how to structure maintainable, scalable software.

The testing integration deserves particular attention, as it demonstrates how tooling can elevate testing from an afterthought to a central part of the development process. The plugin supports multiple testing frameworks - eunit for traditional unit testing, ltest for LFE-specific testing patterns, and Common Test for integration and system testing. This multi-layered approach reflects the reality that different types of testing require different tools, and good tooling should make it easy to use the right tool for each job. Running and deployment commands bridge the gap between development and production. The ability to run code directly from the build system, execute escripts, or start full releases provides a seamless transition from experimentation to deployment. Hot code loading and release management - capabilities that seem almost magical in other languages - become routine operations supported by simple commands.

Perhaps most importantly, Part VI demonstrates how good tooling preserves the interactive, exploratory nature of Lisp development while providing the structure needed for serious software engineering. The REPL integration ensures that you can always drop into an interactive session to explore ideas, test hypotheses, or debug problems. This combination of structure and flexibility exemplifies the LFE philosophy: powerful abstractions that don't constrain creativity.

By the end of Part VI, you'll have mastered not just the mechanics of LFE development, but the workflows that make LFE development productive and enjoyable. You'll understand how to leverage community conventions while maintaining the flexibility to adapt tools to your specific needs. Most importantly, you'll have the confidence that comes from working with mature, well-designed tools that scale from quick experiments to large, complex systems.

rebar3

Quick Start

Introduction

This is a version of the LFE quick start that has been re-imagined with the LFE rebar3 plugin in mind.

We will cover the following:

• How to get started quickly with LFE using just rebar3 and your local installation of Erlang
• Creating a new LFE project
• Looking at LFE code in the REPL and in modules
• Provide a little insight on how this works
• Leave you with resources for jumping into LFE in more detail

About rebar3

Rebar3 is an Erlang tool that makes it easy to create, develop, and release Erlang libraries, applications, and systems in a repeatable manner.

Rebar3 will:

• respect and enforce standard Erlang/OTP conventions for project structure so they are easily reusable by the community;
• manage source dependencies and Erlang packages while ensuring repeatable builds;
• handle build artefacts, paths, and libraries such that standard development tools can be used without a headache;
• adapt to projects of all sizes on almost any platform;
• treat documentation as a feature, and errors or lack of documentation as a bug.

Rebar3 is also a self-contained Erlang script. It is easy to distribute or embed directly in a project. Tasks or behaviours can be modified or expanded with a plugin system flexible enough that even other languages on the Erlang VM will use it as a build tool.

The rebar3 site provides some nice instructions on installing rebar3.

Going Plaid

Remember: this is a quick-start; it's going to be a blur of colour! You're not going to get very many details here, but you will get to jump into the LFE REPL and see a little code in action.¹

The rest of this quick-start assumes that you've followed the links in the previous section and have installed both Erlang as well as rebar3, but to take things further, you'll need to do one more thing: set up the LFE plugin.

Each project you create with the LFE rebar3 plugin will generate a rebar.config file that automatically includes the plugin dependency, but that's only inside an LFE project. You need to bootstrap the LFE plugin by setting it up in your global rebar.config.

The rebar3 docs tell you this file is located at ~/.config/rebar3/rebar.config. To set this up, you can safely execute the following in a terminal, even if the file already exists:

mkdir -p ~/.config/rebar3/
touch ~/.config/rebar3/rebar.config

Then, in your preferred editor, open that file and add the entry for LFE rebar3 plugin. If that file is empty when you open it, then you can paste this whole thing in there:

{plugins, [
  {rebar3_lfe,
    {git, "https://github.com/lfe-rebar3/rebar3_lfe.git", {branch, "main"}}}
]}.

If you want to pin your project to a specific release of the plugin, you can view the list of released versions here:

• https://github.com/lfe-rebar3/rebar3_lfe/tags

And then use tag (with the version) instead of branch:

{plugins, [
  {rebar3_lfe,
    {git, "https://github.com/lfe-rebar3/rebar3_lfe.git", {tag, "x.y.z"}}}
]}.

If your global rebar3 config file already has one or more plugins in it, then simply add a comma after the last one and paste the {rebar3_lfe ...} line from above (with no trailing comma!).

Next Stop

Ready for some LFE? Next you'll learn how to create a new LFE project with just one command ...

Creating a New Project

A project? Already?! It sounds daunting, but it's easier than you might think. Open up a terminal window and do this in a directory of your choosing:

rebar3 new lfe-lib my-test-lib

It might take a minute or two to finish; here's what's happening:

• rebar3 downloads the LFE plugin
• Finds its dependencies, downloads those too
• Compiles all of them (plugins and dependencies)
• rebar3 then executes the new command, searches for (and finds!) the lfe-lib template
• Creates the project files for the given template

As that last step executes, you will see the following output:

===> Writing my-test-lib/README.md
===> Writing my-test-lib/LICENSE
===> Writing my-test-lib/rebar.config
===> Writing my-test-lib/.gitignore
===> Writing my-test-lib/src/my-test-lib.lfe
===> Writing my-test-lib/src/my-test-lib.app.src

It's as simple as that! Your new project is ready to go :-)

Next Stop

You can taste it, can't you? That LFE flavour coming your way? Yup, you're right. You're going to be looking at LFE code next ...

Hitting the Code

It may not seem like it, but we're off to a pretty fast start. If you had to do everything we've done, manually, you'd have given up by now. Seriously.

(Okay, maybe not.)

Time to put the brakes on, though, 'cause you're gonna want to see this next part in slow motion.

REPL Me Up!

Make sure you've cded into your new LFE project directory, and then do this:

$ rebar3 lfe repl

On windows first enter Erlang's repl then run lfe_shell:start().

D:\Lfe\my-test-lib
λ rebar3 lfe repl
===> Verifying dependencies...
===> Compiling my-test-lib
Eshell V10.3  (abort with ^G)
1> lfe_shell:start().

This should give you something that looks like the following:

Erlang/OTP 23 [erts-11.0.2] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [hipe] [dtrace]

   ..-~.~_~---..
  (      \\     )    |   A Lisp-2+ on the Erlang VM
  |`-.._/_\\_.-':    |   Type (help) for usage info.
  |         g |_ \   |
  |        n    | |  |   Docs: http://docs.lfe.io/
  |       a    / /   |   Source: http://github.com/rvirding/lfe
   \     l    |_/    |
    \   r     /      |   LFE v1.3-dev (abort with ^G)
     `-E___.-'

lfe>

Try setting a variable:

> (set my-list (lists:seq 1 6))
(1 2 3 4 5 6)

Here are some operations using more functions from the built-in Erlang lists module:

> (* 2 (lists:sum my-list))
42
> (* 2 (lists:foldl (lambda (n acc) (+ n acc)) 0 my-list))
42

Let's turn that into a function:

> (defun my-sum (start stop)
    (let ((my-list (lists:seq start stop)))
      (* 2 (lists:foldl
             (lambda (n acc)
               (+ n acc))
             0 my-list))))
my-sum

And try it out!

> (my-sum 1 6)
42

Enough with the fancy REPL-play ... What about some real code? What does a project look like?

Sample Code

Well, you've already seen some! But here is the full, if minimal, module generated by the LFE rebar3 plugin:

(defmodule my-test-lib
  (export (my-fun 0)))

;;; -----------
;;; library API
;;; -----------

(defun my-fun ()
  'hello-world)

You'll note that the function we define has been exported via the export form in the module definition. The number after the function is the arity of that function (Erlang views functions of the same name but different arity as different functions, and LFE does the same).

In the REPL you will have access to this module and its one function. Try it out:

lfe> (my-test-lib:my-fun)
hello-world

Let's add to this module our new my-sum function from the REPL jam session in the previous section. In another terminal window (or text editor pane) open up the src/my-test-lib.lfe file and paste the my-sum function at the bottom. Afterwards, add (my-sum 2) to the export section of defmodule at the top.

When you're done, the entire file should look like this:

(defmodule my-test-lib
  (export (my-fun 0)
          (my-sum 2)))

;;; -----------
;;; library API
;;; -----------

(defun my-fun ()
  'hello-world)

(defun my-sum (start stop)
  (let ((my-list (lists:seq start stop)))
    (* 2 (lists:foldl
            (lambda (n acc)
              (+ n acc))
            0 my-list))))

Then come back to the REPL sessions and compile the module with its new addition:

> (c "src/my-test-lib.lfe")
#(module my-test-lib)

And call the module functions:

> (my-test-lib:my-sum 1 6)
42
> (my-test-lib:my-sum 1 60)
3660
>

Here's something a little more involved you may enjoy, from the examples in the LFE source code:

(defun print-result ()
  (receive
    ((tuple pid msg)
      (io:format "Received message: '~s'~n" (list msg))
      (io:format "Sending message to process ~p ...~n" (list pid))
      (! pid (tuple msg))
      (print-result))))

(defun send-message (calling-pid msg)
  (let ((spawned-pid (spawn 'my-test-lib 'print-result ())))
    (! spawned-pid (tuple calling-pid msg))))

That bit of code demonstrates one of Erlang's core features in lovely Lisp syntax: message passing. When loaded into the REPL, that code can demonstrate bidirectional message passing between the LFE shell and a spawned process.

Want to give it a try? Add those two new functions to your module, and don't forget to update the export section, too! (note that one function has an arity of 0 and the other and arity of 2).

When you're done, your project module should look like this:

(defmodule my-test-lib
  (export (my-fun 0)
          (my-sum 2)
          (print-result 0)
          (send-message 2)))

;;; -----------
;;; library API
;;; -----------

(defun my-fun ()
  'hello-world)

(defun my-sum (start stop)
  (let ((my-list (lists:seq start stop)))
    (* 2 (lists:foldl
            (lambda (n acc)
              (+ n acc))
            0 my-list))))

(defun print-result ()
  (receive
    ((tuple pid msg)
      (io:format "Received message: '~s'~n" (list msg))
      (io:format "Sending message to process ~p ...~n" (list pid))
      (! pid (tuple msg))
      (print-result))))

(defun send-message (calling-pid msg)
  (let ((spawned-pid (spawn 'my-test-lib 'print-result ())))
    (! spawned-pid (tuple calling-pid msg))))

Message-Passing

We glossed over this in the previous section, but in LFE (and Erlang) you can compile on-the-fly in a REPL session. This is super-convenient when prototyping functionality for a new project where you want to use the REPL, but you also want the benefits of using a file, so you don't lose your work.

We made some changes to the sample code in the last section; let's compile it and take it for a spin:

> (c "src/my-test-lib.lfe")
#(module my-test-lib)

Next, let's send two messages to another Erlang process (in this case, we'll send it to our REPL process, (self):

> (my-test-lib:send-message (self) "And what does it say now?")
#(<0.26.0> "And what does it say now?")
Received message: 'And what does it say now?'
Sending message to process <0.26.0> ...
> (my-test-lib:send-message (self) "Mostly harmless.")
#(<0.26.0> "Mostly harmless.")
Received message: 'Mostly harmless.'
Sending message to process <0.26.0> ...

In the above calls, for each message sent we got a reply acknowledging the message (because the example was coded like that). But what about the receiver itself? What did it, our REPL process, see? We can flush the message queue in the REPL to find out.

What, what? Does each ...

Yup, every process in LFE (and Erlang, of course) has an inbox. You can see how many messages a given process has by looking at the process' info:

lfe> (erlang:process_info (self) 'message_queue_len)
#(message_queue_len 2)

And there you can see that our REPL process has two messages queued up in its inbox. Let's take a look!

> (c:flush)
Shell got {"And what does it say now?"}
Shell got {"Mostly harmless."}
ok

If you found this last bit interesting and want to step through a tutorial on Erlang's light-weight threads in more detail, you may enjoy this tutorial.

Next Stop

We did promise a bit more information, so we're going to do that next and then wrap up the quick start and point you in some directions for your next LFE adventures ...

Behind the Scenes

As we warned in the beginning, there's a lot going on behind the scenes: in rebar3 as well as LFE and Erlang (we haven't even touched OTP here ...!). This guide just gives you a quick taste of the LFE flavour :-) Before parting ways, though, there are some more bits we should share.

Some of those things are hinted at when just checking the current versions you are running using the LFE plugin's versions command (from a terminal where you have cded into your project directory):

rebar3 lfe versions

(#(apps (#(my-test-lib git)))
 #(languages
   (#(lfe "1.3-dev")
    #(erlang "23")
    #(emulator "11.0.2")
    #(driver_version "3.3")))
 #(tooling (#(rebar "3.10.0") #(rebar3_lfe "0.2.0"))))

To give a sense of what you'll encounter in the future: very often Erlang, LFE, and other BEAM language apps include more than just themselves when they are shipped. For instance, if you're in the REPL and you type (regs) you will see a list of applications that have been registered by name, currently running in support of the REPL. Usually, each app will have its own version. There is an LFE blog series on such things, showing you how to create and mess around with different types of LFE apps.

The LFE rebar3 plugin will also help you create OTP apps in LFE and perform other key tasks you may wish to integrate into your development workflow. You can learn more about those in the plugin's Command Reference

Next Stop

Where to go from here ...

Where Next?

We've mentioned the following resources so far:

• The LFE Tutorial
• The LFE rebar3 Plugin Command Reference
• The LFE Blog

But there are also these:

• The old-skool LFE User Guide
• Example code in the LFE repository
• The lfex org - Community-contributed libraries
• The lfe org - Core code, tools, and on-line resources (docs, blog, etc.)

True mastery of LFE is not matter of syntax, though: it requires a deep knowledge of Erlang/OTP and how to best apply that knowledge. The Erlang site has links to great Erlang books you can read.

Plugin Reference

Introduction

The rebar3_lfe project is a rebar3 plugin for the LFE language. It provides many of the conveniences one has come to expect of a programming language's build tool:

• Project Creation
• A REPL
• Compilation
• Maintenane Tasks (e.g., file cleanup)
• Metadata

Features

• Create new LFE projects:
  • rebar3 new lfe-lib
  • rebar3 new lfe-main
  • rebar3 new lfe-escript
  • rebar3 new lfe-app
  • rebar3 new lfe-release
• Start up an LFE REPL:
  • rebar3 lfe repl
• Compile LFE source code:
  • rebar3 lfe compile
• Run eunit tests
  • rebar3 eunit
• Run an LFE project's main/1 function as an lfescript (run rebar3 new lfe-main to see an example):
  • rebar3 lfe run
  • rebar3 lfe run -- 1 2 5
  • rebar3 lfe run -main some/path/main.lfe
• Escriptize an LFE escript project:
  • rebar3 ecsriptize
• Run an escriptized LFE escript project:
  • rebar3 lfe run-ecsript
• Generate an LFE/OTP release
  • rebar3 release
• Run an LFE/OTP release project's release script (COMMAND can be start , stop , status, ping, etc.):
  • rebar3 lfe run-release COMMAND
• Cleanup
  • rebar3 lfe clean
  • rebar3 lfe clean-build
  • rebar3 lfe clean-cache
  • rebar3 lfe clean-all
• Metadata
  • rebar3 lfe versions

More coming soon!

Background

This plugin originally started life as a shell script (lfetool -- there's even a T-shirt for it!), then it toyed with integrating with rebar (the original). Around that time, though, rebar3 was under initial development, and LFE took a chance on it as an early adopter. This lead to a whole series of LFE plugins, but after a few years momentum was lost.

Those early rebar3 efforts have been combined into a single plugin in this project, with many updates and using all the latest approaches developed in rebar3's now mature ecosystem.

Setup

Dependencies

In order to use the LFE rebar3 plugin, you need to have the following installed on your system:

• Erlang (only tested with versions 19 and above)
• rebar3 (tested with 3.10 and 3.12)

You don't need to download LFE; the plugin will do that for you.

Using the Plugin

After installing rebar3, the only thing you need to do in order to take full advantage of the LFE rebar3 plugin is add it to the plugins in your global rebar.config file.

{plugins, [
  {rebar3_lfe,
    {git, "https://github.com/lfe-rebar3/rebar3_lfe.git", {tag, "0.2.0"}}}
]}.

{plugins, [
  {rebar3_lfe,
    {git, "https://github.com/lfe-rebar3/rebar3_lfe.git", {branch, "release/0.3.x"}}}
]}.

Command Reference

The following sections detail actual usage of the suite of rebar3 lfe commands.

Compiling

The single most imporant convenience provided by the LFE rebar3 plugin is arguably the compiler. This allows any LFE project to be downloaded, compile, and used by any BEAM language that is also using rebar3 to manage its dependencies, etc.

To compile an LFE project:

rebar3 lfe compile

If you are publishing your LFE code, or using it in another project, you'll want to update your rebar.config file so that it is compile when a user (or script) executes the regular rebar3 compile command.

To ensure your LFE code will compile in other projects, add the following to your project's rebar.config:

{provider_hooks, [
  {pre, [{compile, {lfe, compile}}]}
  ]}.

Package Support

The LFE rebar3 plugin provides support for pseudo-packages. There is no such thing as a pckage in Erlang, but using this plugin, you can emulate some of the behaviours of packages.

This is accomplished by traversing top-level source directories for any subdirectories: if the plugin finds any .lfe or .erl files in subdirectories under configured source directories, it will create a dotted module name composed of the relative path to that file, and write that name to the ebin directory after successful compilation.

Here are some examples of how combinations of subdirectories and files will be transformed in their final form as .beam files:

./src/my.package1.lfe                      -> ebin/my.package1.beam
./src/my/package2.lfe                      -> ebin/my.package2.beam
./src/my/other/package.lfe                 -> ebin/my.other.package.beam
./src/my/really/deeply/nested/package1.lfe -> ebin/my.really.deeply.nested.package1.beam
./src/my/really/deeply/nested.package2.lfe -> ebin/my.really.deeply.nested.package2.beam
./src/my/really.deeply.nested.package3.lfe -> ebin/my.really.deeply.nested.package3.beam
./src/my.really.deeply.nested.package4.lfe -> ebin/my.really.deeply.nested.package4.beam

Running the REPL

An LFE project and all of its dependencies may be interacted with via a REPL that is started with the LFE rebar3 plguin, as rebar3 sets all of the correct library locations for use by shells and REPLs.

To start up a REPL:

rebar3 lfe repl

At which point you will be greeted with the familiar:

Erlang/OTP 25 [erts-13.2.2.2] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]

   ..-~.~_~---..
  (      \\     )    |   A Lisp-2+ on the Erlang VM
  |`-.._/_\\_.-':    |   Type (help) for usage info.
  |         g |_ \   |
  |        n    | |  |   Docs: http://docs.lfe.io/
  |       a    / /   |   Source: http://github.com/lfe/lfe
   \     l    |_/    |
    \   r     /      |   LFE v2.1.2 (abort with ^C; enter JCL with ^G)
     `-E___.-'

lfe>

The LFE banner is highly configurable in in rebar3_lfe and accepts the following configuration in your project's rebar.config file:

{lfe, [
   {repl, [
     {nobanner, false},
     {version, "9999.2"},
     {quit_message, "You can never LEEEEEAVE!"},
     {banner_template, "Weeee!~nLFE v~s ~s~n"}
   ]}
]}.

A project configuration with those values would have a banner like the following:

Erlang/OTP 25 [erts-13.2.2.2] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]

Weeee!
LFE v9999.2 You can never LEEEEEAVE!
lfe>

To disable the banner, use a configuration like this:

{lfe, [
   {repl, [
     {nobanner, true}
   ]}
]}.

Which would give:

Erlang/OTP 25 [erts-13.2.2.2] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]

lfe>

Testing

[In progress]

eunit

If you have written eunit tests in Erlang or LFE, they will be compiled by either the rebar3 compile or rebar3 lfe compile command and thus be available and ready to run using rebar3 as-is.

Run compiled eunit tests:

rebar3 eunit

ltest

[In progress]

Common Test

[In progress]

Creating Projects

The rebar3_lfe plugin is capable of creating several common project layouts. This and following sections provide details on those that are currently supported.

Creating LFE Libraries

Library projects are those with no running applications or scripts; they simply provide some core bit of functionality intended for use by applications, scripts, or other libraries.

To create a library project with the default name:

rebar3 new lfe-lib

This will generate the following output:

===> Writing my-lfe-lib/README.md
===> Writing my-lfe-lib/LICENSE
===> Writing my-lfe-lib/rebar.config
===> Writing my-lfe-lib/.gitignore
===> Writing my-lfe-lib/src/my-lfe-lib.lfe
===> Writing my-lfe-lib/src/my-lfe-lib.app.src

You can also explicitely name your project:

rebar3 new lfe-lib forty-two

Which will produce the following:

===> Writing forty-two/README.md
===> Writing forty-two/LICENSE
===> Writing forty-two/rebar.config
===> Writing forty-two/.gitignore
===> Writing forty-two/src/forty-two.lfe
===> Writing forty-two/src/forty-two.app.src

As mentioned abouve, the REPL offers a nice way to quickly interact your new project.

Start the REPL:

cd forty-two
rebar3 lfe repl

Call the generated/sample LFE function:

lfe> (mything:my-fun)
hello-world

Creating main Scripts

LFE supports the creation of scripts, and these can be integrated with libraries, allowing for the best of both worlds: a simple collection of useful functions (library) and a means of running them from the command line.

Create a library that also has a script with a main function:

rebar3 new lfe-main special-proj

Which generates:

===> Writing special-proj/README.md
===> Writing special-proj/LICENSE
===> Writing special-proj/rebar.config
===> Writing special-proj/.gitignore
===> Writing special-proj/src/special-proj.lfe
===> Writing special-proj/scripts/main.lfe
===> Writing special-proj/src/special-proj.app.src

The generated project's main script + function may then be run with:

cd mymain
rebar3 lfe run -- 42

Which will produce the following output:

Running script '/usr/local/bin/rebar3' with args [<<"42">>] ...
'hello-worl

Creating escripts

The LFE rebar3 plugin also supports generating escript-based projects in LFE. This is similar in nature to the main-based project, but is more standard in the BEAM family of languages.

To create an escript-based project:

rebar3 new lfe-escript myapp

===> Writing myapp/README.md
===> Writing myapp/LICENSE
===> Writing myapp/rebar.config
===> Writing myapp/.gitignore
===> Writing myapp/src/myapp.lfe
===> Writing myapp/src/myapp.app.src

Compile the LFE and then bundle all the code up by "escriptizing" it:

cd myapp
rebar3 lfe compile
rebar3 escriptize

Run the newly-created escript:

rebar3 lfe run-escript 1 2 5 no '3!'

Which will display the following:

Got args: ("1" "2" "5" "no" "3!")
Answer: 42

Creating OTP Applications

In the wider Erlang community, it is very common to see applications that run one or more gen_servers (or other server behaviours) managed by a supervision tree (with the appropriate restart strategy defined). The LFE rebar3 plugin provides the ability to generate OTP gen_server applications with the server managed in a supervision tree.

To create an LFE/OTP application:

rebar3 new lfe-app otp-lfe

===> Writing otp-lfe/README.md
===> Writing otp-lfe/LICENSE
===> Writing otp-lfe/rebar.config
===> Writing otp-lfe/.gitignore
===> Writing otp-lfe/src/otp-lfe.lfe
===> Writing otp-lfe/src/otp-lfe-app.lfe
===> Writing otp-lfe/src/otp-lfe-sup.lfe
===> Writing otp-lfe/src/otp-lfe.app.src

We can use the plugin's REPL command to demonstrate usage.

Start the REPL:

cd otp-lfe
rebar3 lfe repl

Start the app:

lfe> (application:ensure_all_started 'otp-lfe)
#(ok (otp-lfe))

Make sure the supervisor is running:

lfe> (erlang:whereis 'otp-lfe-sup)
#Pid<0.205.0>

Make an API call to the gen_server:

lfe> (otp-lfe:echo "testing the supervised gen_server ...")
"testing the supervised gen_server ..."

Creating OTP Releases

If you're going to be running an LFE application in production, you will very likely want to do so using the "release" functionality provided by OTP.

Create a release-based project with:

rebar3 new lfe-release prod-lfe

===> Writing prod-lfe/README.md
===> Writing prod-lfe/LICENSE
===> Writing prod-lfe/rebar.config
===> Writing prod-lfe/.gitignore
===> Writing prod-lfe/apps/prod-lfe/src/prod-lfe.lfe
===> Writing prod-lfe/apps/prod-lfe/src/prod-lfe-app.lfe
===> Writing prod-lfe/apps/prod-lfe/src/prod-lfe-sup.lfe
===> Writing prod-lfe/apps/prod-lfe/src/prod-lfe.app.src
===> Writing prod-lfe/config/sys.config
===> Writing prod-lfe/config/vm.args

Change directoy into your new app:

cd prod-lfe

Build the release:

rebar3 release

Start up the application:

rebar3 lfe run-release start

Check the status of the application:

rebar3 lfe run-release ping

pong

https://github.com/lfe-rebar3/rebar3_lfe/issues/33

rebar3 release
rebar3 lfe run-release start
rebar3 lfe run-release ping

In addition to using the LFE rebar3 commands to start the application, you can start up a release console and then switch to the LFE REPL.

Start the console:

./_build/default/rel/prod-lfe/bin/prod-lfe console

Eshell V11.0  (abort with ^G)
(prod-app@spacemac)1> lfe_shell:start().

(prod-app@spacemac)lfe> (erlang:whereis 'prod-lfe-sup)
#Pid<0.351.0>
(prod-app@spacemac)lfe> (prod-lfe:echo "testing from production!")
"testing from production!"

Running Code

The LFE rebar3 plugin provides developers the ability to run LFE code directly, taking advantage of rebar3's setting of library paths for dependencies, etc.

run

To run an LFE project that was generated with rebar3 new lfe-main:

rebar3 lfe run

run-escript

To run an LFE project that was generated with rebar3 new lfe-escript:

rebar3 lfe run-escript

run-release

To run an LFE project that was generated with rebar3 new lfe-release and has been compiled with rebar3 release:

rebar3 lfe run-release COMMAND

Where COMMAND is one of the non-interactive commands supported by the release script:

• start
• stop
• restart
• reboot
• pid
• ping

Others non-interactive commands not listed may also work, but they have not been tested.

https://github.com/lfe-rebar3/rebar3_lfe/issues/33

Cleanup Commands

There are a handful of tasks the author of the plugin has found useful when debugging LFE applications, rebar3 plugins, etc. These include various "cleanup" tasks that are not currently supported by rebar3 (or whose support is problematic).

clean

Remove the apps' ebin/* files:

rebar3 lfe clean

clean-build

Remove the _build directory:

rebar3 lfe clean-build

clean-cache

Remove the the cached directories for the app and the rebar3_lfe plugin, both global and local:

rebar3 lfe clean-cache

clean-all

Perform all clean tasks as well as remove the files erl_crash.dump, rebar3.crashdump, and rebar.lock:

rebar3 lfe clean-all

rebar3 Internals - A Developer's Reference

Introduction

The rebar3 compilation system is a sophisticated, multi-stage build system designed to compile Erlang/OTP applications and their dependencies efficiently. The system follows a provider-based plugin architecture, uses directed acyclic graphs (DAGs) for dependency tracking, and supports incremental and parallel compilation.

This document provides a bird's-eye view of the entire compilation process, outlining the major stages and their relationships. For detailed technical information about each stage, see the individual stage documents referenced throughout.

High-Level Architecture

The rebar3 compilation chain is built on several architectural principles:

1. Provider-Based Execution: All functionality is implemented as providers that declare dependencies and execution order
2. State Management: A central state object flows through all stages, accumulating configuration and results
3. Graph-Based Dependencies: File and application dependencies are tracked using directed acyclic graphs
4. Incremental Compilation: Only changed files and their dependents are recompiled
5. Parallel Compilation: Independent files can be compiled concurrently
6. Extensibility: Custom compilers and hooks allow modification of the build process

Major Compilation Stages

The compilation process can be divided into the following major stages:

1. Initialization and Configuration Loading
2. Dependency Resolution and Locking
3. Dependency Acquisition
4. Application Discovery
5. Compilation Order Determination
6. Dependency Compilation
7. Project Application Compilation
8. Single Application Compilation
9. Individual File Compilation
10. Build Verification and Completion

Key Dependencies

• Dependency resolution must precede acquisition
• Applications must be discovered before determining compilation order
• Dependencies must compile before project applications
• Compilation order is critical for correctness
• Verification ensures build integrity

Diagrams

Flow Diagram

graph TD
    A[Start: rebar3 compile] --> B[Initialization and Configuration Loading]
    B --> C[Dependency Resolution and Locking]
    C --> D[Dependency Acquisition]
    D --> E[Application Discovery]
    E --> F[Compilation Order Determination]
    F --> G{deps_only flag?}

    G -->|Yes| H[Dependency Compilation Only]
    G -->|No| I[Dependency Compilation]

    H --> H1[For Each Dependency]
    H1 --> H2[Single Application Compilation]
    H2 --> H3{More deps?}
    H3 -->|Yes| H1
    H3 -->|No| Z[End]

    I --> I1[For Each Dependency]
    I1 --> I2[Single Application Compilation]
    I2 --> I3{More deps?}
    I3 -->|Yes| I1
    I3 -->|No| J[Project Application Compilation]

    J --> J1[For Each Project App]
    J1 --> J2[Single Application Compilation]
    J2 --> J3{More apps?}
    J3 -->|Yes| J1
    J3 -->|No| K[Root Extra Dirs Compilation]

    K --> L[Build Verification and Completion]
    L --> Z[End]

    style A fill:#e1f5ff
    style Z fill:#e1f5ff
    style H2 fill:#fff4e1
    style I2 fill:#fff4e1
    style J2 fill:#fff4e1

Single Application Compilation Flow

graph TD
    SA[Single Application Compilation] --> SA1[Application Preparation]
    SA1 --> SA2[Pre-Compilation Hooks]
    SA2 --> SA3[Compiler Preparation Hooks]
    SA3 --> SA4[Source Compilation Loop]

    SA4 --> SC1[Leex Compiler: .xrl → .erl]
    SC1 --> SC2[Yecc Compiler: .yrl → .erl]
    SC2 --> SC3[MIB Compiler: .mib → .bin + .hrl]
    SC3 --> SC4[Erlang Compiler: .erl → .beam]
    SC4 --> SC5[Custom Compilers]

    SC5 --> SA5[Post-Compiler Hooks]
    SA5 --> SA6[App File Preparation Hooks]
    SA6 --> SA7[Application File Generation]
    SA7 --> SA8[App File Finalization Hooks]
    SA8 --> SA9[Compilation Finalization]
    SA9 --> SAE[Application Compiled]

    style SA fill:#fff4e1
    style SAE fill:#e1ffe1

Individual Compiler Execution Flow

graph TD
    C[Compiler Execution] --> C1[Load or Initialize DAG]
    C1 --> C2[Get Compiler Context]
    C2 --> C3[Scan Source Directories]
    C3 --> C4[Prune Deleted Files from DAG]
    C4 --> C5[Analyze Source Dependencies]

    C5 --> C6[For Each Source File]
    C6 --> C7[Parse Dependencies]
    C7 --> C8[Add to DAG]
    C8 --> C9{More sources?}
    C9 -->|Yes| C6
    C9 -->|No| C10[Propagate Timestamps]

    C10 --> C11[Determine Needed Files]
    C11 --> C12{Files to compile?}
    C12 -->|No| C20[Save DAG and Exit]
    C12 -->|Yes| C13[Separate First Files]

    C13 --> C14[Compile Parse Transforms]
    C14 --> C15[Split Sequential/Parallel]
    C15 --> C16[Compile Sequential Files]
    C16 --> C17[Compile Parallel Files]

    C17 --> C18[Store Artifact Metadata]
    C18 --> C19[Update DAG Timestamps]
    C19 --> C20
    C20 --> CE[Compiler Done]

    style C fill:#e1f5ff
    style CE fill:#e1ffe1

Stage Dependencies

The following diagram shows dependencies between major stages:

graph LR
    Init[Initialization] --> DepResolve[Dependency Resolution]
    DepResolve --> DepAcq[Dependency Acquisition]
    DepAcq --> AppDisc[Application Discovery]
    AppDisc --> CompOrder[Compilation Order]
    CompOrder --> DepComp[Dependency Compilation]
    CompOrder --> ProjComp[Project Compilation]
    DepComp --> ProjComp
    ProjComp --> Verify[Build Verification]

    style Init fill:#e1f5ff
    style Verify fill:#e1ffe1

Key Concepts

Provider

A provider is a unit of functionality in rebar3. Each provider:

• Has a unique name and namespace
• Declares dependencies on other providers
• Implements initialization and execution functions
• Can be extended or overridden by plugins

The compile provider depends on the lock provider, ensuring dependencies are resolved before compilation begins.

State

The rebar state is a record that flows through all stages, containing:

• Configuration options from rebar.config
• Lists of applications (project apps, dependencies, plugins)
• Code paths for different contexts
• Active profiles
• Registered providers and compilers
• Current execution context

Application Info

Each application (project or dependency) has an associated app info record containing:

• Application name and version
• Source and output directories
• Path to .app.src and .app files
• Application-specific configuration options
• Dependency list
• Profile information

Directed Acyclic Graph (DAG)

Each compiler maintains a DAG to track file dependencies:

• Vertices: Source files, header files, and artifacts with timestamps
• Edges: Dependency relationships
• Purpose: Incremental compilation, determining compilation order
• Persistence: Saved to disk between builds
• Invalidation: Recalculated if compiler version or critical options change

Compiler

A compiler is a module implementing the rebar_compiler behavior:

• Defines what source files it handles (extensions, directories)
• Analyzes dependencies between files
• Determines compilation order
• Executes compilation for individual files
• Tracks generated artifacts

Hooks

Hooks allow execution of custom code at specific points:

• Provider hooks: Run other rebar3 providers
• Shell hooks: Execute shell commands
• Pre/post hooks: Run before or after specific stages
• Platform-specific: Conditional execution based on OS

Profiles

Profiles allow configuration variations:

• Default profile: default
• Common profiles: test, prod
• Custom profiles: User-defined
• Profile stacking: Configuration merges from default + active profiles
• Separate build outputs: _build/PROFILE/

Incremental Compilation

Optimization that avoids unnecessary recompilation:

• Tracks file modification timestamps
• Propagates timestamps through dependency chains
• Recompiles only changed files and their dependents
• Detects compiler option changes
• Stores metadata with artifacts

Parallel Compilation

Performance optimization for independent files:

• Files with no interdependencies compiled concurrently
• Worker pool manages parallel tasks
• Parse transforms and dependent files compiled sequentially
• Configurable parallelism level

Initialization & Configuration

Purpose

The initialization stage prepares the rebar3 runtime environment by loading configuration files, setting up logging, initializing the state structure, loading plugins, and registering providers. This stage establishes the foundation for all subsequent compilation and build operations.

When It Executes

This is the first stage in the rebar3 execution chain. It runs immediately when:

• rebar3 compile (or any command) is invoked from the command line
• rebar3:run/2 is called from the Erlang API

Prerequisites

• Erlang/OTP runtime is available
• Current working directory is accessible
• Required Erlang applications (crypto, ssl, inets) can be started

Outputs

• Initialized rebar_state:t() record containing:
  • Loaded configuration from rebar.config
  • Merged global configuration (if ~/.config/rebar3/rebar.config exists)
  • Applied profiles
  • Registered providers
  • Installed plugins
  • Code paths for the build
  • Lock file data

Execution Flow

graph TD
    A[Start: rebar3 main/1] --> B[start_and_load_apps]
    B --> B1[Load rebar application]
    B1 --> B2[Start crypto, asn1, public_key]
    B2 --> B3{Offline mode?}
    B3 -->|No| B4[Start ssl, inets, httpc]
    B3 -->|Yes| C
    B4 --> C[init_config/0]

    C --> C1[Set HTTPC options]
    C1 --> C2[Initialize logging system]
    C2 --> C3[Consult rebar.config]
    C3 --> C4[Consult rebar.lock]
    C4 --> C5[Merge locks into config]
    C5 --> C6[Create initial state]
    C6 --> C7{Global config exists?}
    C7 -->|Yes| C8[Load global config]
    C7 -->|No| C9
    C8 --> C9[Set escript path]
    C9 --> C10[Initialize vsn cache]
    C10 --> D[run_aux]

    D --> D1[Set Unicode encoding]
    D1 --> D2{REBAR_PROFILE env set?}
    D2 -->|Yes| D3[Apply profile from env]
    D2 -->|No| D4
    D3 --> D4[Check minimum OTP version]
    D4 --> D5{HEX_CDN env set?}
    D5 -->|Yes| D6[Set CDN URL]
    D5 -->|No| D7
    D6 --> D7[Set compilers from app env]
    D7 --> D8[Create resources]
    D8 --> D9[Bootstrap test profile]
    D9 --> D10{REBAR_BASE_DIR env set?}
    D10 -->|Yes| D11[Override base_dir]
    D10 -->|No| D12
    D11 --> D12{REBAR_CACHE_DIR env set?}
    D12 -->|Yes| D13[Set global rebar dir]
    D12 -->|No| D14
    D13 --> D14[Create logic providers]
    D14 --> D15{REBAR_SKIP_PROJECT_PLUGINS?}
    D15 -->|No| D16[Install project plugins]
    D15 -->|Yes| D17
    D16 --> D17[Install top-level plugins]
    D17 --> D18[Set default opts]
    D18 --> D19[Parse command args]
    D19 --> D20{--offline flag?}
    D20 -->|Yes| D21[Set offline mode]
    D20 -->|No| D22
    D21 --> D22[Set code paths]
    D22 --> D23[init_command]
    D23 --> E[State Ready]

    style A fill:#e1f5ff
    style E fill:#e1ffe1

Detailed Steps

1. Application Loading (start_and_load_apps/1)

   • Load the rebar application
   • Start required applications: crypto, asn1, public_key
   • If not offline: start ssl, inets, and create an httpc profile

2. Base Configuration (init_config/0)

   • Set HTTPC options for package downloads
   • Initialize logging with appropriate verbosity level
   • Read rebar.config from project root
   • Read rebar.lock if it exists
   • Merge lock data into configuration
   • Create initial state with merged configuration

3. Global Configuration (in init_config/0)

   • Check for ~/.config/rebar3/rebar.config
   • If exists, load and merge with project configuration
   • Install global plugins (from global config)

4. Environment Setup (run_aux/2)

   • Set shell encoding to Unicode
   • Apply REBAR_PROFILE environment variable if set
   • Validate OTP version requirements
   • Configure Hex CDN URL if specified

5. Compilers and Resources

   • Load compilers from application environment
   • Create and register resource modules (for deps)
   • Bootstrap test profile with TEST macro and test directories

6. Directory Configuration

   • Set base_dir (default: _build, override with REBAR_BASE_DIR)
   • Set global_rebar_dir (default: ~/.cache/rebar3, override with REBAR_CACHE_DIR)

7. Provider and Plugin System

   • Register built-in providers
   • Install project plugins (unless REBAR_SKIP_PROJECT_PLUGINS is set)
   • Install top-level plugins
   • Merge all provider lists

8. Command Preparation

   • Parse command-line arguments
   • Set offline mode if requested
   • Initialize code paths
   • Call rebar_core:init_command/2 to dispatch to the actual command

Functions & API Calls

rebar3:main/1

Purpose: Entry point from escript

Signature:

-spec main([string()]) -> no_return().

Arguments:

• Args ([string()]): Command-line arguments

Returns: Does not return; calls erlang:halt/1

Flow:

• Calls rebar3:run/1
• Handles success/error
• Exits with appropriate code

rebar3:run/1

Purpose: Main execution entry point from command line

Signature:

-spec run([string()]) -> {ok, rebar_state:t()} | {error, term()}.

Arguments:

• RawArgs ([string()]): Command-line arguments

Returns:

• {ok, State}: Successful execution with final state
• {error, Reason}: Error occurred

Flow:

1. Start and load applications
2. Call init_config/0 to create base state
3. Set caller to command_line
4. Call set_options/2 to process global flags
5. Call run_aux/2 for actual execution

rebar3:init_config/0

Purpose: Set up base configuration and initial state

Signature:

-spec init_config() -> rebar_state:t().

Returns: Initialized state record

Flow:

1. Set HTTPC options via rebar_utils:set_httpc_options/0
2. Initialize logging via rebar_log:init/2
3. Read rebar.config via rebar_config:consult_root/0
4. Read rebar.lock via rebar_config:consult_lock_file/1
5. Merge locks into config via rebar_config:merge_locks/2
6. Create state via rebar_state:new/1
7. Load global config if exists
8. Set escript path
9. Initialize vsn cache

Example Usage:

BaseState = rebar3:init_config()

rebar_config:consult_root/0

Purpose: Read the main rebar.config file

Signature:

-spec consult_root() -> [term()].

Returns: List of configuration terms

Flow:

• Looks for rebar.config in current directory
• Parses as Erlang terms
• Returns empty list if file doesn't exist

Called From: Initialization & Configuration

rebar_config:consult_lock_file/1

Purpose: Read and parse the lock file

Signature:

-spec consult_lock_file(file:filename()) -> [term()].

Arguments:

• File (file:filename()): Path to rebar.lock

Returns: List of lock entries in internal format

Flow:

1. Read lock file
2. Detect version (beta, "1.2.0", etc.)
3. Parse locks based on version
4. Extract package hashes
5. Expand locks with hash information
6. Return internal lock format

Lock File Versions:

• Beta format: [Locks].
• Versioned format: {"1.2.0", Locks}. [Attrs].

rebar_state:new/0,1,2,3

Purpose: Create a new rebar state record

Signature:

-spec new() -> t().
-spec new(list()) -> t().
-spec new(t() | atom(), list()) -> t().
-spec new(t(), list(), file:filename_all()) -> t().

Arguments (for new/1):

• Config (list()): Configuration terms from rebar.config

Returns: New state record

Flow (for new/1):

1. Convert config list to dict via base_opts/1
2. Create base state with opts
3. Set current working directory
4. Set default opts

State Record:

-record(state_t, {
    dir                               :: file:name(),
    opts                = dict:new()  :: rebar_dict(),
    code_paths          = dict:new()  :: rebar_dict(),
    default             = dict:new()  :: rebar_dict(),
    escript_path                      :: undefined | file:filename_all(),
    lock                = [],
    current_profiles    = [default]   :: [atom()],
    namespace           = default     :: atom(),
    command_args        = [],
    command_parsed_args = {[], []},
    current_app                       :: undefined | rebar_app_info:t(),
    project_apps        = []          :: [rebar_app_info:t()],
    deps_to_build       = []          :: [rebar_app_info:t()],
    all_plugin_deps     = []          :: [rebar_app_info:t()],
    all_deps            = []          :: [rebar_app_info:t()],
    compilers           = []          :: [module()],
    project_builders    = []          :: [{project_type(), module()}],
    resources           = [],
    providers           = [],
    allow_provider_overrides = false  :: boolean()
}).

rebar_state:apply_profiles/2

Purpose: Apply configuration profiles to state

Signature:

-spec apply_profiles(t(), [atom()]) -> t().

Arguments:

• State (t()): Current state
• Profiles ([atom()]): List of profiles to apply (e.g., [test], [prod])

Returns: State with merged profile configuration

Flow:

1. Get profiles config from state
2. For each profile (except default):
   • Get profile-specific options
   • Merge with current opts using rebar_opts:merge_opts/3
3. Update current_profiles field
4. Return modified state

Profile Merging:

• Base configuration is from default profile
• Profile-specific configs override base
• Multiple profiles stack in order given

rebar_plugins:project_plugins_install/1

Purpose: Install plugins specified in rebar.config

Signature:

-spec project_plugins_install(rebar_state:t()) -> rebar_state:t().

Arguments:

• State (rebar_state:t()): Current state

Returns: State with plugins installed and providers registered

Flow:

1. Get project_plugins from configuration
2. For each plugin:
   • Fetch plugin (from Hex or Git)
   • Compile plugin
   • Load plugin application
   • Discover and register providers from plugin
3. Update state with new providers

rebar_state:create_logic_providers/2

Purpose: Register built-in providers with the state

Signature:

-spec create_logic_providers([module()], t()) -> t().

Arguments:

• Providers ([module()]): List of provider modules
• State (t()): Current state

Returns: State with providers registered

Flow:

1. For each provider module:
   • Call Module:init(State) to get provider record
   • Add to providers list
2. Return updated state

Built-in Providers (from application env):

• rebar_prv_app_discovery
• rebar_prv_compile
• rebar_prv_clean
• rebar_prv_install_deps
• rebar_prv_lock
• And many more...

rebar_core:init_command/2

Purpose: Initialize and dispatch to the requested command

Signature:

-spec init_command(rebar_state:t(), atom()) -> {ok, rebar_state:t()} | {error, term()}.

Arguments:

• State (rebar_state:t()): Current state
• Command (atom()): Command to execute (e.g., compile, test)

Returns:

• {ok, NewState}: Command executed successfully
• {error, Reason}: Command failed

Flow:

1. Handle special commands (do, as)
2. Call process_namespace/2 to resolve namespace
3. Call process_command/2 to execute provider chain

State Modification

Fields Set During Initialization

Configuration Keys in opts

Common configuration keys stored in the opts dictionary:

• deps: List of dependencies
• plugins: List of plugins
• project_plugins: List of project-scoped plugins
• profiles: Profile-specific configuration
• erl_opts: Erlang compiler options
• src_dirs: Source directories
• extra_src_dirs: Additional source directories
• minimum_otp_vsn: Minimum OTP version required
• base_dir: Build output directory
• artifacts: List of expected artifacts

Configuration

Configuration Files

rebar.config

Location: Project root directory

Format: Erlang terms

Common Options:

{erl_opts, [debug_info, warnings_as_errors]}.
{deps, [
    {jsx, "3.1.0"},
    {cowboy, {git, "https://github.com/ninenines/cowboy.git", {tag, "2.9.0"}}}
]}.
{plugins, [rebar3_hex]}.
{profiles, [
    {test, [
        {deps, [meck, proper]},
        {erl_opts, [{d, 'TEST'}]}
    ]}
]}.

rebar.lock

Location: Project root directory

Format: Versioned Erlang terms

Purpose: Lock dependency versions for reproducible builds

Example:

{"1.2.0",
[{<<"jsx">>, {pkg,<<"jsx">>,<<"3.1.0">>}, 0}]}.
[
{pkg_hash,[
 {<<"jsx">>, <<"...">>}]}
].

~/.config/rebar3/rebar.config

Location: User's global configuration directory

Purpose: Global settings and plugins

Common Uses:

• Global plugins (hex, dialyzer, etc.)
• Hex repository credentials
• Global compiler options

Environment Variables

Command-Line Options

Global options (available for all commands):

• -h, --help: Show help
• -v, --version: Show version information
• --offline: Run in offline mode (no network access)

Configuration Precedence

Configuration is merged in the following order (later overrides earlier):

1. Built-in defaults
2. ~/.config/rebar3/rebar.config (global config)
3. Project rebar.config
4. Profile-specific configuration
5. Environment variables
6. Command-line options

File System Operations

Files Read

Files Written

None in this stage. Files are written in later stages.

Directories Created

None in this stage. Directories are created during compilation.

Directories Accessed

• Current working directory (project root)
• ~/.config/rebar3/ (for global config)
• ~/.cache/rebar3/ (may be accessed for cached data)

Error Conditions

Missing Required Applications

Condition: Required Erlang application fails to start (e.g., crypto, ssl)

Detection: application:start/1 returns {error, Reason}

Error Message:

Rebar dependency <app> could not be loaded for reason <reason>

Recovery: None; rebar3 exits with code 1

Unsupported OTP Version

Condition: OTP version is below minimum_otp_vsn

Detection: rebar_utils:check_min_otp_version/2

Error Message:

ERROR: OTP release <version> does not match required regex <regex>

Recovery: None; must upgrade OTP

Blacklisted OTP Version

Condition: OTP version matches blacklisted_otp_vsns

Detection: rebar_utils:check_blacklisted_otp_versions/1

Error Message:

ERROR: OTP release <version> matches blacklisted version <regex>

Recovery: None; must change OTP version

Invalid Configuration Format

Condition: rebar.config contains invalid Erlang terms

Detection: Parse error in rebar_config:consult_file/1

Error Message:

ERROR: Error reading config file <file>: <parse error>

Recovery: None; must fix rebar.config

Invalid Profile Configuration

Condition: Profile configuration is not a list

Detection: rebar_state:apply_profiles/2

Error:

{profile_not_list, Profile, Other}

Error Message:

Profile config must be a list but for profile '<profile>' config given as: <other>

Recovery: None; must fix profile configuration

Newer Lock File Version

Condition: rebar.lock has a newer version than supported

Detection: Version check in rebar_config:consult_lock_file/1

Warning Message:

Rebar3 detected a lock file from a newer version. It will be loaded in compatibility mode,
but important information may be missing or lost. It is recommended to upgrade Rebar3.

Recovery: Loads in compatibility mode; may lose some information

Command Not Found

Condition: Requested command doesn't match any provider

Detection: rebar_core:process_namespace/2

Error Message:

Command <command> not found

Recovery: None; check command spelling or available commands

Edge Cases

No rebar.config File

Behavior: rebar3 uses built-in defaults

Impact:

• No dependencies
• Default compiler options
• Standard directory layout expected

Empty rebar.config

Behavior: Treated same as missing file

Impact: Uses all defaults

rebar.config.script

Behavior: If rebar.config.script exists, it's evaluated as Erlang code

Purpose: Dynamic configuration generation

Example:

CONFIG1 = case os:getenv("ENV") of
    "prod" -> [{erl_opts, [no_debug_info]}];
    _ -> []
end,
CONFIG1 ++ CONFIG.

Global Config Without Local Config

Behavior: Global config provides base configuration

Use Case: Setting up a new project with global plugins available

Multiple Profiles Applied

Behavior: Profiles merge in order, later values override earlier

Example:

rebar3 as test,custom compile

Applies profiles: default → test → custom

REBAR_PROFILE Environment Variable with as Command

Behavior: Environment variable overrides are replaced by as profiles

Recommendation: Use as command instead of environment variable

Cross References

Stages That Depend On This Stage

• All stages: Every stage requires initialized state
• Dependency Resolution & Locking: Uses lock data from initialization
• Dependency Acquisition: Uses configuration for app discovery
• Compilation Order Determination: Uses compiler configuration

Related Documentation

• State Management: Detailed state structure
• Configuration Reference: All configuration options

Example Scenarios

Scenario 1: Simple Project Initialization

Setup:

my_app/
├── rebar.config
└── src/
    └── my_app.erl

rebar.config:

{erl_opts, [debug_info]}.

Execution: rebar3 compile

Flow:

1. Read rebar.config → [{erl_opts, [debug_info]}]
2. No rebar.lock → empty lock
3. Create state with opts
4. Apply default profile
5. No plugins to install
6. Register built-in providers
7. Dispatch to compile command

Result: State initialized with debug_info enabled

Scenario 2: Project with Test Profile

rebar.config:

{erl_opts, [debug_info]}.
{profiles, [
    {test, [
        {deps, [meck]},
        {erl_opts, [nowarn_export_all]}
    ]}
]}.

Execution: rebar3 as test compile

Flow:

1. Load base config
2. Apply default profile
3. Apply test profile:
   • Add meck to deps
   • Add nowarn_export_all to erl_opts
4. Result: {erl_opts, [nowarn_export_all, debug_info]}

Scenario 3: Global Plugin Installation

~/.config/rebar3/rebar.config:

{plugins, [rebar3_hex]}.

Execution: rebar3 compile (in any project)

Flow:

1. Load project config
2. Detect global config exists
3. Load global config
4. Install rebar3_hex plugin globally
5. Register hex providers (publish, etc.)
6. Proceed with project compilation

Result: Hex commands available in all projects

Scenario 4: Environment Variable Override

rebar.config:

{erl_opts, [debug_info]}.
{profiles, [
    {prod, [{erl_opts, [no_debug_info]}]}
]}.

Execution: REBAR_PROFILE=prod rebar3 compile

Flow:

1. Load base config
2. Detect REBAR_PROFILE=prod
3. Apply prod profile
4. Result: {erl_opts, [no_debug_info]}

Scenario 5: Offline Mode

Execution: rebar3 compile --offline

Flow:

1. Parse --offline flag
2. Set REBAR_OFFLINE=1 environment variable
3. Skip starting ssl and inets
4. Set offline flag in state
5. Later stages skip network operations

Impact: Dependencies must already be cached

Dependency Resolution & Locking

Purpose

The dependency resolution and locking stage determines which dependencies are required for the project, resolves version constraints for package dependencies, builds a complete dependency tree, detects circular dependencies, and ensures reproducible builds through lock files.

When It Executes

This stage executes after Initialization & Configuration and is typically triggered by:

• The install_deps provider (dependency of compile)
• The lock provider (explicitly locking dependencies)
• Any command that requires dependencies to be resolved

The compile provider depends on the lock provider, which depends on install_deps.

Prerequisites

• State initialized with configuration loaded
• rebar.config parsed with dependencies list
• rebar.lock read (if exists) with locked versions
• Application discovery completed (for project apps)

Outputs

• Complete list of all dependencies (direct and transitive)
• Dependency tree with levels (depth from project root)
• Resolved versions for all package dependencies
• Updated rebar.lock file with locked versions
• Topologically sorted dependency list (compilation order)
• Updated state with:
  • all_deps: All resolved dependencies
  • deps_to_build: Dependencies that need compilation
  • lock: Current lock data

Execution Flow

graph TD
    A[Start: install_deps provider] --> B[Get current profiles]
    B --> C[Get project apps]
    C --> D[Check upgrade flag]
    D --> E[deps_per_profile]

    E --> E1[For each profile]
    E1 --> E2[Get parsed deps for profile]
    E2 --> E3{More profiles?}
    E3 -->|Yes| E1
    E3 -->|No| F[Get locks from state]

    F --> G[handle_profile_level]
    G --> G1{Deps remaining?}
    G1 -->|No| H[find_cycles]
    G1 -->|Yes| G2[For each dep at current level]

    G2 --> G3{Dep is top-level app?}
    G3 -->|Yes| G4[Skip this dep]
    G3 -->|No| G5{Dep already seen?}

    G4 --> G6{More deps at level?}
    G5 -->|Yes| G7[Check version conflict]
    G5 -->|No| G8[maybe_lock]

    G7 --> G9{Same source?}
    G9 -->|Yes| G6
    G9 -->|No| G10[Warn about skip]
    G10 --> G6

    G8 --> G11{In default profile?}
    G11 -->|Yes| G12[Add to lock list]
    G11 -->|No| G13[Skip locking]

    G12 --> G14[maybe_fetch]
    G13 --> G14

    G14 --> G15[handle_dep]
    G15 --> G16[Parse dep's deps]
    G16 --> G17[Add to next level]
    G17 --> G6

    G6 -->|Yes| G2
    G6 -->|No| G18[Process next level]
    G18 --> G1

    H --> H1[Build digraph of all apps]
    H1 --> H2[Add vertices for each app]
    H2 --> H3[Add edges for dependencies]
    H3 --> H4[digraph_utils:topsort]

    H4 --> H5{Acyclic?}
    H5 -->|Yes| H6[Return sorted list]
    H5 -->|No| H7[Find strongly connected components]
    H7 --> H8[Return cycles error]

    H6 --> I[cull_compile]
    I --> I1[Remove project apps from sorted list]
    I1 --> I2[Drop deps that don't need compile]
    I2 --> J[Update state: deps_to_build]

    J --> K[lock provider]
    K --> K1{Running in default profile?}
    K1 -->|No| L[Skip locking]
    K1 -->|Yes| K2[build_locks]

    K2 --> K3[Get all locked deps from state]
    K3 --> K4[For each dep not checkout]
    K4 --> K5[Create lock tuple]
    K5 --> K6[Sort locks by name]
    K6 --> K7[maybe_write_lock_file]

    K7 --> K8{Locks changed?}
    K8 -->|Yes| K9[Write rebar.lock]
    K8 -->|No| K10[Check format changed]
    K10 --> K11{Format different?}
    K11 -->|Yes| K9
    K11 -->|No| L

    K9 --> L[End: Dependencies resolved and locked]

    style A fill:#e1f5ff
    style L fill:#e1ffe1

Detailed Steps

1. Profile-Based Dependency Collection (deps_per_profile/3)

   • Get active profiles (e.g., [default], [default, test])
   • For each profile, get parsed dependencies
   • Load lock file data for consistency

2. Level-Order Dependency Traversal (handle_profile_level/7)

   • Process dependencies level by level (breadth-first)
   • Level 0: Direct dependencies from rebar.config
   • Level 1: Dependencies of level 0 dependencies
   • Continue until no more dependencies found

3. Dependency Processing (update_dep/9)

   • For each dependency:
     • Check if already seen (to avoid duplicates)
     • Check if it's a top-level app (skip if so)
     • Add to lock list if in default profile
     • Fetch the dependency (or verify it exists)
     • Parse the dependency's own dependencies
     • Add transitive deps to next level

4. Lock Management (maybe_lock/5)

   • Only lock dependencies in default profile
   • Skip checkout dependencies
   • Track dependency level (depth in tree)
   • Replace existing locks if new one is shallower

5. Cycle Detection (find_cycles/1)

   • Build directed graph of all applications
   • Vertices: application names
   • Edges: application → dependency relationships
   • Use digraph_utils:topsort/1 for topological sort
   • If sort fails, find strongly connected components (cycles)

6. Compilation Order (cull_compile/2)

   • Start with topologically sorted dependency list
   • Remove project applications (they compile separately)
   • Drop dependencies that don't need compilation (already built)
   • Return final list for deps_to_build

7. Lock File Writing (lock provider)

   • Only runs in default profile
   • Build lock entries from state
   • Each lock: {Name, Source, Level}
   • Sort locks alphabetically
   • Write to rebar.lock if changed
   • Preserve lock file format version

Functions & API Calls

rebar_prv_install_deps:do/1

Purpose: Main entry point for dependency resolution

Signature:

-spec do(rebar_state:t()) -> {ok, rebar_state:t()} | {error, string()}.

Arguments:

• State (rebar_state:t()): Current state with configuration

Returns:

• {ok, State}: Dependencies resolved and state updated
• {error, Reason}: Resolution failed

Flow:

1. Get current profiles
2. Get project apps
3. Check if upgrade mode is enabled
4. Resolve deps per profile via deps_per_profile/3
5. Update state with all deps
6. Build and update code paths
7. Detect cycles via find_cycles/1
8. Determine compile order via cull_compile/2
9. Update state with deps_to_build

Called From: Provider system (as dependency of lock and compile)

deps_per_profile/3

Purpose: Collect and resolve dependencies for all active profiles

Signature:

-spec deps_per_profile([atom()], boolean(), rebar_state:t()) ->
    {[rebar_app_info:t()], rebar_state:t()}.

Arguments:

• Profiles ([atom()]): Active profiles (e.g., [default, test])
• Upgrade (boolean()): Whether to upgrade dependencies
• State (rebar_state:t()): Current state

Returns:

• {Apps, State}: Resolved dependencies and updated state

Flow:

1. Get locks from state
2. For each profile, get parsed dependencies at level 0
3. Create RootSeen set with project app names
4. Call handle_profile_level/7 for traversal

handle_profile_level/7

Purpose: Level-order traversal of dependency tree across all profiles

Signature:

-spec handle_profile_level(
    [{Profile, Deps, Level}],
    Apps,
    RootSeen,
    Seen,
    Upgrade,
    Locks,
    State
) -> {Apps, State} when
    Profile :: atom(),
    Deps :: [rebar_app_info:t()],
    Level :: integer(),
    Apps :: [rebar_app_info:t()],
    RootSeen :: sets:set(),
    Seen :: sets:set(),
    Upgrade :: boolean(),
    Locks :: [term()],
    State :: rebar_state:t().

Arguments:

• Profile/Deps/Level tuples: Dependencies per profile at each level
• Apps: Accumulated resolved dependencies
• RootSeen: Set of top-level app names (never process as deps)
• Seen: Set of already-processed dependency names
• Upgrade: Whether upgrading
• Locks: Lock file data
• State: Current state

Returns: {Apps, State} with all resolved dependencies

Algorithm:

For each {Profile, Deps, Level}:
  For each Dep in Deps:
    If Dep is in RootSeen:
      Skip (it's a top-level app)
    Else if Dep is in Seen:
      Check for version conflicts, warn if needed
    Else:
      Lock the dependency
      Fetch the dependency
      Parse the dep's own dependencies
      Add new deps to next level

  If new deps were found:
    Append {Profile, NewDeps, Level+1} to queue

  Process next level

update_dep/9

Purpose: Process a single dependency

Signature:

-spec update_dep(
    AppInfo,
    Profile,
    Level,
    Deps,
    Apps,
    State,
    Upgrade,
    Seen,
    Locks
) -> {NewDeps, NewApps, NewState, NewSeen} when
    AppInfo :: rebar_app_info:t(),
    Profile :: atom(),
    Level :: integer(),
    Deps :: [rebar_app_info:t()],
    Apps :: [rebar_app_info:t()],
    State :: rebar_state:t(),
    Upgrade :: boolean(),
    Seen :: sets:set(),
    Locks :: [term()].

Arguments:

• AppInfo: Dependency to process
• Profile: Current profile
• Level: Current level in dependency tree
• Deps: Accumulated dependencies for next level
• Apps: All resolved apps so far
• State: Current state
• Upgrade: Upgrade flag
• Seen: Set of seen dependency names
• Locks: Lock data

Returns: Updated accumulator tuple

Flow:

1. Get dependency name
2. Check if already seen
3. If seen: check for conflicts, possibly warn
4. If not seen:
   • Lock the dependency
   • Fetch/verify the dependency
   • Handle the dependency (parse its deps)
   • Add to accumulated apps
   • Add transitive deps to next level

maybe_lock/5

Purpose: Add dependency to lock list if appropriate

Signature:

-spec maybe_lock(Profile, AppInfo, Seen, State, Level) -> {NewSeen, NewState} when
    Profile :: atom(),
    AppInfo :: rebar_app_info:t(),
    Seen :: sets:set(),
    State :: rebar_state:t(),
    Level :: integer().

Arguments:

• Profile: Current profile
• AppInfo: Dependency to potentially lock
• Seen: Set of seen dependencies
• State: Current state
• Level: Depth in dependency tree

Returns: {NewSeen, NewState} with updated lock

Logic:

• Skip if checkout dependency
• Skip if not in default profile
• If already in lock at deeper level, replace with shallower
• Otherwise add to lock list
• Always add to seen set

find_cycles/1

Purpose: Detect circular dependencies

Signature:

-spec find_cycles([rebar_app_info:t()]) ->
    {no_cycle, Sorted} | {cycles, Cycles} | {error, Error} when
    Sorted :: [rebar_app_info:t()],
    Cycles :: [[binary()]],
    Error :: term().

Arguments:

• Apps ([rebar_app_info:t()]): All applications (project + deps)

Returns:

• {no_cycle, Sorted}: No cycles; sorted topologically
• {cycles, Cycles}: Circular dependencies detected
• {error, Error}: Other error

Flow:

1. Call rebar_digraph:compile_order/1
2. Which creates digraph and calls digraph_utils:topsort/1
3. If sort succeeds: return sorted list
4. If sort fails: find strongly connected components
5. Filter components with length > 1 (these are cycles)
6. Return cycles

rebar_digraph:compile_order/1

Purpose: Build dependency graph and return topological sort

Signature:

-spec compile_order([rebar_app_info:t()]) ->
    {ok, [rebar_app_info:t()]} | {error, no_sort | {cycles, [[binary()]]}}.

Arguments:

• Apps ([rebar_app_info:t()]): Applications to sort

Returns:

• {ok, Sorted}: Topologically sorted applications
• {error, {cycles, Cycles}}: Circular dependencies found
• {error, no_sort}: Topological sort failed for other reason

Flow:

1. Create new digraph
2. For each app:
   • Add vertex with app name
   • Get all dependencies (from applications list and deps config)
   • Add edges from app to each dependency
3. Call digraph_utils:topsort/1
4. If successful: reverse the list (dependencies first)
5. If failed: determine if cyclic, extract cycles
6. Delete digraph
7. Return result

Example Graph:

my_app → [cowboy, jsx]
cowboy → [cowlib, ranch]
cowlib → []
ranch → []
jsx → []

Sorted: [cowlib, ranch, jsx, cowboy, my_app]

all_apps_deps/1

Purpose: Get all dependencies for an application

Signature:

-spec all_apps_deps(rebar_app_info:t()) -> [binary()].

Arguments:

• App (rebar_app_info:t()): Application to analyze

Returns: List of dependency names (binaries)

Flow:

1. Get applications list from .app file (runtime deps)
2. Get deps list from rebar.config (build deps)
3. Convert all to binaries
4. Sort and merge (union)

Why Both Sources:

• applications: Runtime dependencies declared in .app
• deps: Build-time dependencies from rebar.config
• Union ensures all dependencies are considered for build order

cull_compile/2

Purpose: Filter dependency list to those needing compilation

Signature:

-spec cull_compile([rebar_app_info:t()], [rebar_app_info:t()]) -> [rebar_app_info:t()].

Arguments:

• TopSortedDeps ([rebar_app_info:t()]): All apps in compile order
• ProjectApps ([rebar_app_info:t()]): Project's own applications

Returns: Dependencies that need compilation

Flow:

1. Remove project apps from sorted list (they're compiled separately)
2. Drop dependencies from start of list until finding one that needs compile
3. Return remaining list

Logic for "needs compile":

• Checkout dependencies always need compile
• Package dependencies from Hex usually don't (pre-compiled)
• Source dependencies (Git, etc.) need compile

rebar_prv_lock:do/1

Purpose: Write dependency locks to rebar.lock

Signature:

-spec do(rebar_state:t()) -> {ok, rebar_state:t()} | {error, string()}.

Arguments:

• State (rebar_state:t()): Current state with resolved deps

Returns: {ok, State} with locks saved

Flow:

1. Check if running in default profile (only lock in default)
2. Get old locks from state
3. Build new locks via build_locks/1
4. Sort locks alphabetically
5. Write to rebar.lock if changed
6. Update state with new locks
7. Report useless locks (removed dependencies)
8. Report checkout dependencies (can't be locked)

build_locks/1

Purpose: Convert state's lock data to lock file format

Signature:

-spec build_locks(rebar_state:t()) -> [lock_entry()].

Arguments:

• State (rebar_state:t()): Current state

Returns: List of lock entries

Lock Entry Format:

{Name :: binary(),
 Source :: lock_source(),
 Level :: integer()}

Example:

{<<"cowboy">>,
 {git, "https://github.com/ninenines/cowboy.git",
      {ref, "abc123def456..."}},
 0}

Flow:

1. Get all locked deps from state
2. Filter out checkout dependencies
3. For each dep:
   • Get name
   • Get lock source via rebar_fetch:lock_source/2
   • Get dependency level
   • Create tuple

State Modification

Fields Modified During This Stage

Dependency Information Stored

Each dependency (rebar_app_info:t()) contains:

• name: Dependency name
• source: Where it comes from (Hex, Git, etc.)
• dep_level: Depth in dependency tree (0 = direct)
• dir: Source directory
• out_dir: Build output directory
• is_checkout: Whether it's a checkout dependency
• is_lock: Whether from lock file
• deps: Transitive dependencies

Configuration

Dependency Specification Formats

Package Dependency (Hex)

Simple:

{deps, [
    jsx                          % Latest version
]}.

With Version:

{deps, [
    {jsx, "3.1.0"},             % Exact version
    {jsx, "~> 3.1"},            % Semantic versioning
    {jsx, ">= 2.0.0"}           % Version constraint
]}.

Source Dependency (Git)

{deps, [
    {cowboy, {git, "https://github.com/ninenines/cowboy.git", {tag, "2.9.0"}}},
    {ranch, {git, "git://github.com/ninenines/ranch.git", {branch, "master"}}},
    {gun, {git, "https://github.com/ninenines/gun.git", {ref, "abc123"}}}
]}.

Source Dependency (Mercurial)

{deps, [
    {my_dep, {hg, "https://bitbucket.org/user/my_dep", {tag, "1.0.0"}}}
]}.

Checkout Dependency

Not in rebar.config. Created by placing dependency in _checkouts/ directory.

Behavior:

• Always compiled
• Never locked
• Overrides any version specified in config

Profile-Specific Dependencies

{profiles, [
    {test, [
        {deps, [
            meck,
            proper
        ]}
    ]},
    {prod, [
        {deps, [
            recon
        ]}
    ]}
]}.

Merging:

• Profile deps added to default profile deps
• No removal; only addition
• Active profiles merge in order

Lock File Configuration

File: rebar.lock

Format Version: "1.2.0" (current)

Structure:

{"1.2.0",
[{<<"cowboy">>, {git, "https://github.com/ninenines/cowboy.git",
                      {ref, "abc123"}}, 0},
 {<<"jsx">>, {pkg, <<"jsx">>, <<"3.1.0">>}, 0},
 {<<"ranch">>, {git, "https://github.com/ninenines/ranch.git",
                     {ref, "def456"}}, 1}]}.
[
{pkg_hash, [
  {<<"jsx">>, <<"ABCD1234...">>},
  {<<"cowlib">>, <<"EFGH5678...">>}
]}
].

Components:

1. Version string
2. List of lock entries (name, source, level)
3. List of package hashes (for integrity verification)

File System Operations

Files Read

Files Written

Directories Accessed

• _checkouts/: Checkout dependencies
• _build/PROFILE/lib/: Dependency locations
• Various remote locations for fetching (if needed)

Error Conditions

Circular Dependency Detected

Condition: Dependency graph contains cycles

Detection: digraph_utils:topsort/1 returns false and digraph_utils:is_acyclic/1 returns false

Error Format:

{cycles, [[<<"app1">>, <<"app2">>, <<"app3">>], ...]}

Error Message:

Dependency cycle(s) detected:
applications: app1 app2 app3 depend on each other

Example:

app1 depends on app2
app2 depends on app3
app3 depends on app1
→ Cycle: [app1, app2, app3]

Recovery: None; must break the cycle in dependencies

Package Not Found in Registry

Condition: Hex package doesn't exist

Error Format:

{missing_package, Package, Version}
{missing_package, Package}

Error Message:

Package not found in registry: jsx-3.1.0
Package not found in registry: nonexistent_package

Recovery: Fix package name or check Hex registry

Bad Version Constraint

Condition: Invalid version specification

Error Format:

{bad_constraint, Name, Constraint}

Error Message:

Unable to parse version for package jsx: ~~>3.1.0

Recovery: Fix version constraint syntax

Package Not Buildable

Condition: Hex package exists but isn't rebar3-compatible

Error Format:

{not_rebar_package, Package, Version}

Error Message:

Package not buildable with rebar3: old_package-1.0.0

Recovery: Use different version or different package

Failed to Load Registry

Condition: Cannot download Hex registry

Error Format:

{load_registry_fail, Dep}

Error Message:

Error loading registry to resolve version of jsx. Try fixing by running 'rebar3 update'

Recovery: Run rebar3 update or check network connection

Dependency Application Not Found

Condition: Dependency fetched but application not in expected location

Error Format:

{dep_app_not_found, AppDir, AppName}

Error Message:

Dependency failure: Application my_dep not found at the top level of directory /path/to/dep

Possible Causes:

• Dependency doesn't have .app.src file
• Application name doesn't match directory name
• Dependency is not an OTP application

Recovery: Fix dependency structure or use different version

Edge Cases

Conflicting Dependency Versions

Scenario: Two dependencies require different versions of the same package

Behavior:

• First encountered version wins
• Later conflicts are warned about but skipped
• Lock file preserves the chosen version

Warning:

Skipping jsx (git source) as an app of the same name has already been fetched

Solution:

• Check if all dependencies can work with one version
• May need to upgrade/downgrade parent dependencies

Dependency at Multiple Levels

Scenario: Same dependency appears at different depths

Example:

my_app → dep_a → dep_common (level 2)
my_app → dep_b → dep_common (level 2)
my_app → dep_common (level 1)

Behavior:

• First occurrence wins (usually shallowest level)
• Lock file records the shallowest level
• Deeper occurrences skipped

Upgrade Mode

Triggered by: rebar3 upgrade

Behavior:

• Ignore lock file constraints
• Fetch latest versions matching constraints
• Update lock file with new versions

Effect on Warnings:

• Suppresses "already seen" warnings
• Shows upgrade activity

Checkout Dependencies

Location: _checkouts/dep_name/

Behavior:

• Always used, regardless of lock file
• Always compiled (never treated as binary)
• Never added to lock file
• Overrides any version specification

Use Cases:

• Local development on a dependency
• Testing unreleased dependency changes
• Temporary patches

Warning:

App my_dep is a checkout dependency and cannot be locked.

Empty Lock File

Behavior: All dependencies resolved from scratch

When This Happens:

• First time running rebar3
• After deleting rebar.lock
• In a new clone of the repository (if lock file not committed)

Profile Dependencies Don't Lock

Behavior: Only default profile dependencies are locked

Reason: Different profiles used in different contexts

Example:

{profiles, [
    {test, [{deps, [meck]}]}
]}.

Running: rebar3 as test compile

Result: meck resolved but NOT added to rebar.lock

Transitive Dependency Version Pinning

Scenario: Lock file pins version of transitive dependency

Example:

% rebar.config
{deps, [{cowboy, "2.9.0"}]}.

% rebar.lock includes:
{<<"ranch">>, {git, "...", {ref, "..."}}, 1}

Behavior:

• Even though ranch is not direct dependency, it's locked
• Ensures exact same version tree on every build
• Provides true reproducibility

Cross References

Dependencies

• Initialization & Configuration: Configuration and lock file loading
• Application Discovery: Discovering project apps (needed for RootSeen)

Dependents

• Dependency Acquisition: Fetching resolved dependencies
• Compilation Order Determination: Uses resolved deps
• Source File Compilation: Compiles resolved dependencies

Related

• State Management: State structure details
• Configuration Reference: Dependency configuration options

Example Scenarios

Scenario 1: Simple Project with Hex Dependencies

rebar.config:

{deps, [
    jsx,
    {cowboy, "2.9.0"}
]}.

Execution: rebar3 compile (first time, no lock file)

Flow:

1. Parse deps: jsx (latest), cowboy "2.9.0"

2. Query Hex registry for jsx latest → "3.1.0"

3. Resolve cowboy → "2.9.0"

4. Get cowboy's dependencies → [cowlib, ranch]

5. Resolve cowlib and ranch versions from cowboy's requirements

6. Build dependency graph:

   my_app → [jsx, cowboy]
   cowboy → [cowlib, ranch]
   jsx → []
   cowlib → []
   ranch → []
   

7. Topological sort → [cowlib, ranch, jsx, cowboy, my_app]

8. Write lock file with all versions

9. Return deps to build: [jsx, cowlib, ranch, cowboy]

rebar.lock created:

{"1.2.0",
[{<<"cowboy">>, {pkg, <<"cowboy">>, <<"2.9.0">>}, 0},
 {<<"cowlib">>, {pkg, <<"cowlib">>, <<"2.11.0">>}, 1},
 {<<"jsx">>, {pkg, <<"jsx">>, <<"3.1.0">>}, 0},
 {<<"ranch">>, {pkg, <<"ranch">>, <<"1.8.0">>}, 1}]}.
[{pkg_hash, [...]}].

Scenario 2: Circular Dependency Detection

Setup:

app_a/rebar.config: {deps, [app_b]}.
app_b/rebar.config: {deps, [app_c]}.
app_c/rebar.config: {deps, [app_a]}.

Execution: rebar3 compile

Flow:

1. Resolve app_a → depends on app_b
2. Resolve app_b → depends on app_c
3. Resolve app_c → depends on app_a
4. Build graph: app_a → app_b → app_c → app_a
5. Attempt topological sort → fails
6. Find strongly connected components → [[app_a, app_b, app_c]]
7. Return error

Error:

Dependency cycle(s) detected:
applications: app_a app_b app_c depend on each other

Scenario 3: Profile-Specific Dependencies

rebar.config:

{deps, [jsx]}.
{profiles, [
    {test, [{deps, [meck, proper]}]}
]}.

Execution: rebar3 as test compile

Flow:

1. Active profiles: [default, test]
2. Level 0, default profile: [jsx]
3. Level 0, test profile: [meck, proper]
4. Resolve all at level 0
5. Continue with transitive deps
6. Build graph with all apps
7. Topological sort
8. Write lock file with ONLY jsx (default profile only)

rebar.lock:

{"1.2.0",
[{<<"jsx">>, {pkg, <<"jsx">>, <<"3.1.0">>}, 0}]}.
[...].

Note: meck and proper NOT in lock file

Scenario 4: Using Checkout Dependencies

Setup:

my_app/
├── _checkouts/
│   └── my_dep/
│       ├── src/
│       └── rebar.config
├── rebar.config: {deps, [{my_dep, "1.0.0"}]}
└── src/

Execution: rebar3 compile

Flow:

1. Parse deps: my_dep version "1.0.0"
2. Discover checkout in _checkouts/my_dep/
3. Mark as checkout dependency
4. Skip fetching (use local version)
5. Resolve transitive deps from checkout's rebar.config
6. In locking stage: skip adding to lock file

Output:

App my_dep is a checkout dependency and cannot be locked.

rebar.lock: Does NOT include my_dep

Scenario 5: Dependency Upgrade

Initial rebar.lock:

{"1.2.0",
[{<<"jsx">>, {pkg, <<"jsx">>, <<"3.0.0">>}, 0}]}.

Execution: rebar3 upgrade jsx

Flow:

1. Set upgrade flag to true
2. Ignore locked version for jsx
3. Query Hex for latest version → "3.1.0"
4. Fetch new version
5. Resolve dependencies normally
6. Update lock file with new version

New rebar.lock:

{"1.2.0",
[{<<"jsx">>, {pkg, <<"jsx">>, <<"3.1.0">>}, 0}]}.

Scenario 6: Git Dependency with Transitive Deps

rebar.config:

{deps, [
    {my_git_dep, {git, "https://github.com/user/my_git_dep.git", {tag, "v1.0.0"}}}
]}.

my_git_dep/rebar.config:

{deps, [jsx]}.

Flow:

1. Resolve my_git_dep from Git

2. Fetch and clone the repository

3. Parse my_git_dep/rebar.config

4. Find transitive dependency: jsx

5. Resolve jsx from Hex

6. Build graph:

   my_app → [my_git_dep]
   my_git_dep → [jsx]
   jsx → []
   

7. Topological sort → [jsx, my_git_dep, my_app]

rebar.lock:

{"1.2.0",
[{<<"jsx">>, {pkg, <<"jsx">>, <<"3.1.0">>}, 1},
 {<<"my_git_dep">>, {git, "https://github.com/user/my_git_dep.git",
                          {ref, "abc123..."}}, 0}]}.

Note: jsx is level 1 (transitive), my_git_dep is level 0 (direct)

Dependency Acquisition

Purpose

The dependency acquisition stage fetches dependencies from their respective sources (Hex package registry, Git repositories, Mercurial repositories, local checkouts), verifies their integrity, extracts them to the appropriate locations, and prepares them for compilation. This stage ensures all required dependencies are available locally before compilation begins.

When It Executes

This stage executes as part of the dependency resolution process in Dependency Resolution & Locking, specifically within the update_unseen_dep/9 function via the maybe_fetch/5 call. It runs during:

• install_deps provider execution
• First compilation when dependencies aren't cached
• When dependencies need updating
• After modifying rebar.config dependencies

Prerequisites

• Dependencies resolved with specific versions/refs
• Source information available (Hex, Git URL, etc.)
• Network access (unless offline mode or dependencies cached)
• Git/Mercurial tools installed (for VCS dependencies)

Outputs

• Dependencies downloaded to _build/PROFILE/lib/DEPNAME/
• Dependency configurations parsed and validated
• Application info updated with actual dependency structure
• Dependencies ready for compilation

Execution Flow

graph TD
    A[Start: maybe_fetch] --> B{Offline mode?}
    B -->|Yes| C{Dep already exists?}
    B -->|No| D[download_source]

    C -->|Yes| E[Use cached version]
    C -->|No| F[Error: offline, no cache]

    D --> D1[Resource type detection]
    D1 --> D2{Source type?}

    D2 -->|pkg| G[Hex Package Download]
    D2 -->|git| H[Git Clone/Fetch]
    D2 -->|hg| I[Mercurial Clone]

    G --> G1[Check local cache]
    G1 --> G2{Cached with valid ETag?}
    G2 -->|Yes| G3{ETag matches server?}
    G2 -->|No| G4[Download from Hex]

    G3 -->|Yes| G5[Use cached tarball]
    G3 -->|No| G4

    G4 --> G6[r3_hex_repo:get_tarball]
    G6 --> G7[Verify checksum]
    G7 --> G8{Checksum valid?}
    G8 -->|Yes| G9[Store tarball in cache]
    G8 -->|No| G10[Error: bad checksum]

    G9 --> G11[Extract tarball to TmpDir]
    G5 --> G11

    H --> H1{Git version check}
    H1 --> H2{Branch, Tag, or Ref?}
    H2 -->|Branch| H3[git clone -b branch --single-branch]
    H2 -->|Tag| H4[git clone -b tag --single-branch]
    H2 -->|Ref| H5[git clone + checkout ref]

    H3 --> H6[Clone complete]
    H4 --> H6
    H5 --> H6

    I --> I1[hg clone]
    I1 --> I2{Tag or revision?}
    I2 -->|Tag| I3[hg update tag]
    I2 -->|Revision| I4[hg update -r rev]

    I3 --> I5[Clone complete]
    I4 --> I5

    G11 --> J[Move TmpDir to FetchDir]
    H6 --> J
    I5 --> J
    E --> J

    J --> K[Read dep's rebar.config]
    K --> L[Update app info with config]
    L --> M[find_app: Discover application]

    M --> M1{Application found?}
    M1 -->|Yes| N[Mark as available]
    M1 -->|No| O[Error: dep_app_not_found]

    N --> P[Return updated AppInfo]

    style A fill:#e1f5ff
    style P fill:#e1ffe1
    style F fill:#ffe1e1
    style G10 fill:#ffe1e1
    style O fill:#ffe1e1

Detailed Steps

1. Fetch Decision (maybe_fetch/5)

   • Check if dependency already exists locally
   • Check if update is needed
   • Skip if checkout dependency (already local)
   • Decide whether to fetch or use cached version

2. Source Type Detection

   • Determine resource type from source tuple:
     • {pkg, Name, Version, _, _} → Hex package
     • {git, URL, RefSpec} → Git repository
     • {hg, URL, RevSpec} → Mercurial repository

3. Offline Mode Handling

   • If offline mode and dependency not cached: error
   • If offline mode and dependency cached: use cache
   • Otherwise proceed with download

4. Package Download (Hex) (rebar_pkg_resource:download/4)

   • Check local cache directory for existing package
   • Read ETag file to get cached ETag
   • Request package from Hex with If-None-Match header
   • If 304 response: use cached tarball
   • If 200 response: download new tarball
   • Verify checksum against registry
   • Store tarball and ETag in cache
   • Extract tarball to temporary directory

5. Git Clone (rebar_git_resource:download/4)

   • Detect Git version for optimal clone command
   • For branches: git clone -b branch --single-branch
   • For tags: git clone -b tag --single-branch
   • For refs: git clone + git checkout ref
   • Clone to temporary directory
   • Handle authentication if required

6. Mercurial Clone (rebar_hg_resource:download/4)

   • Use hg clone to clone repository
   • Update to specific tag or revision
   • Clone to temporary directory

7. Extraction and Placement

   • Create temporary directory with ec_file:insecure_mkdtemp/0
   • Download/extract to temporary directory
   • Remove old cached version if exists
   • Move temporary directory to final location
   • Final location: _build/PROFILE/lib/DEPNAME/

8. Post-Download Verification

   • Read dependency's rebar.config
   • Update app info with dependency's configuration
   • Discover application structure with rebar_app_discover:find_app/4
   • Verify application is valid
   • Mark application as available

Functions & API Calls

maybe_fetch/5

Purpose: Determine if dependency needs fetching and fetch if necessary

Signature:

-spec maybe_fetch(AppInfo, Profile, Upgrade, Seen, State) -> {Cached | false, AppInfo} when
    AppInfo :: rebar_app_info:t(),
    Profile :: atom(),
    Upgrade :: boolean(),
    Seen :: sets:set(),
    State :: rebar_state:t(),
    Cached :: boolean().

Arguments:

• AppInfo: Dependency application info
• Profile: Current profile
• Upgrade: Whether in upgrade mode
• Seen: Set of already-seen dependencies
• State: Current rebar state

Returns: {Cached, UpdatedAppInfo}

Flow:

1. Check if checkout dependency → skip fetch
2. Check if already exists locally
3. If exists and not upgrade mode → check if needs update
4. If needs fetch/update → call rebar_fetch:download_source/2
5. Return updated app info

rebar_fetch:download_source/2

Purpose: Main entry point for downloading any dependency

Signature:

-spec download_source(rebar_app_info:t(), rebar_state:t()) ->
    rebar_app_info:t() | {error, any()}.

Arguments:

• AppInfo (rebar_app_info:t()): Dependency to download
• State (rebar_state:t()): Current state

Returns:

• Updated AppInfo with dependency downloaded
• {error, Reason} if download failed

Flow:

1. Check offline mode → error if offline and not cached
2. Call download_source_online/2
3. After download: read dependency's rebar.config
4. Update app info opts with dependency's config
5. Discover application with rebar_app_discover:find_app/4
6. Mark as available
7. Return updated app info

Example Usage:

AppInfo1 = rebar_fetch:download_source(AppInfo, State)

download_source_online/2

Purpose: Perform actual download operation

Signature:

-spec download_source_online(rebar_app_info:t(), rebar_state:t()) -> ok | {error, term()}.

Arguments:

• AppInfo: Dependency to download
• State: Current state

Returns: ok or {error, Reason}

Flow:

1. Get app directory from app info
2. Create temporary directory with ec_file:insecure_mkdtemp/0
3. Call rebar_resource_v2:download/3 with TmpDir
4. Resource module downloads to TmpDir
5. Ensure app directory exists
6. Remove old version from code path
7. Remove old fetch directory
8. Move TmpDir to final location (FetchDir)

rebar_resource_v2:download/3

Purpose: Dispatch to appropriate resource module for download

Signature:

-spec download(TmpDir, AppInfo, State) -> ok | {error, term()} when
    TmpDir :: file:filename(),
    AppInfo :: rebar_app_info:t(),
    State :: rebar_state:t().

Arguments:

• TmpDir: Temporary directory for download
• AppInfo: Dependency info with source
• State: Current state

Returns: ok or {error, Reason}

Flow:

• Determine resource type from source
• Call appropriate resource module's download/4:
  • rebar_pkg_resource:download/4 for Hex packages
  • rebar_git_resource:download/4 for Git repos
  • rebar_hg_resource:download/4 for Mercurial repos

rebar_pkg_resource:download/4

Purpose: Download Hex package

Signature:

-spec download(TmpDir, AppInfo, State, ResourceState) -> ok | {error, term()} when
    TmpDir :: file:name(),
    AppInfo :: rebar_app_info:t(),
    State :: rebar_state:t(),
    ResourceState :: rebar_resource_v2:resource_state().

Arguments:

• TmpDir: Temporary directory
• AppInfo: Package info
• State: Current state
• ResourceState: Resource-specific state (repos config)

Returns: ok or {error, Reason}

Flow:

1. Get package cache directory
2. Build cache path: ~/.cache/rebar3/hex/default/packages/name-version.tar
3. Build ETag file path: ~/.cache/rebar3/hex/default/packages/name-version.etag
4. Call cached_download/7 with cache info
5. Extract tarball contents to TmpDir

cached_download/7

Purpose: Download package with caching and ETag support

Signature:

-spec cached_download(TmpDir, CachePath, Pkg, State, ETag, ETagPath, UpdateETag) ->
    ok | {error, term()} when
    TmpDir :: file:name(),
    CachePath :: file:name(),
    Pkg :: package(),
    State :: rebar_state:t(),
    ETag :: binary() | undefined,
    ETagPath :: file:name(),
    UpdateETag :: boolean().

Flow:

1. Check if cached tarball exists
2. If exists: read ETag from ETag file
3. Request package from Hex with ETag (If-None-Match header)
4. If 304 Not Modified: use cached tarball
5. If 200 OK: download new tarball
6. Verify checksum: calc_checksum(Tarball) vs registry checksum
7. If valid: store tarball and ETag in cache
8. Extract tarball to TmpDir

r3_hex_repo:get_tarball/3

Purpose: HTTP request to Hex for package tarball

Signature:

-spec get_tarball(Config, Name, Version) ->
    {ok, {StatusCode, Headers, Body}} | {error, Reason} when
    Config :: map(),
    Name :: binary(),
    Version :: binary(),
    StatusCode :: integer(),
    Headers :: map(),
    Body :: binary(),
    Reason :: term().

Arguments:

• Config: Hex repo configuration (includes ETag)
• Name: Package name
• Version: Package version

Returns:

• {ok, {200, Headers, Tarball}}: New tarball downloaded
• {ok, {304, Headers, _}}: Cached version still valid
• {ok, {Code, _, _}}: Other HTTP status
• {error, Reason}: Network/HTTP error

Headers:

• <<"etag">>: ETag value for caching

rebar_git_resource:download/4

Purpose: Clone Git repository

Signature:

-spec download(TmpDir, AppInfo, State, ResourceState) -> ok | {error, term()} when
    TmpDir :: file:name(),
    AppInfo :: rebar_app_info:t(),
    State :: rebar_state:t(),
    ResourceState :: rebar_resource_v2:resource_state().

Arguments:

• TmpDir: Temporary directory for clone
• AppInfo: Git source info
• State: Current state
• ResourceState: Resource state

Returns: ok or {error, Reason}

Flow:

1. Ensure TmpDir exists
2. Detect Git version with git_vsn/0
3. Parse source ref spec (branch/tag/ref)
4. Call appropriate git_clone/5 variant
5. Clone to TmpDir/basename

git_clone/5

Purpose: Execute git clone with appropriate options

Signature:

-spec git_clone(Type, GitVersion, Url, Dir, RefSpec) -> ok | {error, term()} when
    Type :: branch | tag | ref | rev,
    GitVersion :: {Major, Minor, Patch} | undefined,
    Url :: string(),
    Dir :: file:name(),
    RefSpec :: string().

Arguments:

• Type: Type of ref (branch, tag, ref, rev)
• GitVersion: Detected Git version
• Url: Git repository URL
• Dir: Target directory
• RefSpec: Specific branch/tag/ref value

Commands Based on Git Version:

Branch (Git >= 2.3.0):

git clone [options] URL DIR -b BRANCH --single-branch

Branch (Git < 2.3.0):

git clone [options] URL DIR
cd DIR && git checkout -b BRANCH origin/BRANCH

Tag (Git >= 2.3.0):

git clone [options] URL DIR -b TAG --single-branch

Tag (Git < 2.3.0):

git clone [options] URL DIR
cd DIR && git checkout TAG

Ref:

git clone [options] URL DIR
cd DIR && git checkout REF

Environment:

• GIT_TERMINAL_PROMPT=0: Disable interactive prompts

rebar_git_resource:lock/2

Purpose: Get current commit ref for locking

Signature:

-spec lock(AppInfo, ResourceState) -> {git, Url, {ref, Ref}} when
    AppInfo :: rebar_app_info:t(),
    ResourceState :: rebar_resource_v2:resource_state(),
    Url :: string(),
    Ref :: string().

Arguments:

• AppInfo: Dependency app info
• ResourceState: Resource state

Returns: Lock source tuple with commit ref

Command:

git -C DIR rev-parse --verify HEAD

Result: Full commit SHA (40 characters)

rebar_git_resource:needs_update/2

Purpose: Check if Git dependency needs updating

Signature:

-spec needs_update(AppInfo, ResourceState) -> boolean() when
    AppInfo :: rebar_app_info:t(),
    ResourceState :: rebar_resource_v2:resource_state().

Arguments:

• AppInfo: Dependency info
• ResourceState: Resource state

Returns: true if update needed, false otherwise

Logic by Type:

Tag:

git describe --tags --exact-match

Compare with specified tag and URL

Branch:

git fetch origin BRANCH
git log HEAD..origin/BRANCH --oneline

Update needed if new commits exist

Ref:

git rev-parse --short=7 -q HEAD

Compare with specified ref (truncated to same length)

rebar_hg_resource:download/4

Purpose: Clone Mercurial repository

Signature:

-spec download(TmpDir, AppInfo, State, ResourceState) -> ok | {error, term()} when
    TmpDir :: file:name(),
    AppInfo :: rebar_app_info:t(),
    State :: rebar_state:t(),
    ResourceState :: rebar_resource_v2:resource_state().

Commands:

hg clone URL DIR
cd DIR && hg update -r REVISION

Or for tags:

hg clone URL DIR
cd DIR && hg update TAG

State Modification

No Direct State Modifications

This stage doesn't modify the global state directly. Instead, it:

• Downloads files to filesystem
• Updates rebar_app_info:t() records
• These updated records are accumulated in the resolution stage

AppInfo Modifications

For each fetched dependency, the rebar_app_info:t() record is updated:

Configuration

Resource Configuration

Hex Repository Configuration

In rebar.config:

{hex, [
    {repos, [
        #{name => <<"hexpm">>,
          url => <<"https://repo.hex.pm">>,
          auth_key => {<<"SOME_KEY">>}
        }
    ]}
]}.

Default Hex CDN: https://repo.hex.pm

Environment Variables:

• HEX_CDN: Override Hex CDN URL
• HEX_MIRROR: Alternative to HEX_CDN (for Mix compatibility)

Git Configuration

Authentication:

• SSH: Uses SSH keys from ~/.ssh/
• HTTPS: May prompt for credentials (disabled with GIT_TERMINAL_PROMPT=0)
• Token: Can embed in URL: https://token@github.com/user/repo.git

Git Version Requirements:

• Minimum: 1.8.5 (for locking with -C flag)
• Recommended: 2.3.0+ (for --single-branch optimization)

Cache Directories

Package Cache: ~/.cache/rebar3/hex/REPONAME/packages/

Contents:

• PACKAGE-VERSION.tar: Package tarball
• PACKAGE-VERSION.etag: ETag for cache validation

Environment Variable: REBAR_CACHE_DIR overrides cache location

File System Operations

Files Read

Files Written

Directories Created

Directories Removed

Error Conditions

Offline Mode Without Cache

Condition: Dependency not cached and offline mode enabled

Error Format:

{?MODULE, offline}

Error Message:

Cannot fetch dependency in offline mode

Recovery:

• Disable offline mode
• Or pre-cache dependencies

Network Failure

Condition: Cannot connect to remote server

Causes:

• No internet connection
• Server down
• Firewall blocking

Error Format:

{fetch_fail, Source}

Error Message:

Failed to fetch and copy dep: {git, "https://github.com/user/repo.git", {tag, "1.0.0"}}

Recovery: Check network and retry

Bad Registry Checksum

Condition: Downloaded package checksum doesn't match registry

Error Format:

{bad_registry_checksum, Name, Version, Expected, Found}

Error Message:

The checksum for package at jsx-3.1.0 (ABC123) does not match the checksum
expected from the registry (DEF456). Run `rebar3 do unlock jsx, update` and then try again.

Causes:

• Corrupted download
• Tampered package
• Cache corruption

Recovery:

rebar3 do unlock jsx, update

Git Clone Failure

Condition: Git clone command fails

Common Causes:

• Invalid URL
• Authentication failure
• Repository doesn't exist
• Network timeout

Error Message:

fatal: repository 'https://github.com/user/repo.git' not found

Recovery:

• Verify URL
• Check authentication
• Verify repository exists

Git Version Too Old

Condition: Git < 1.8.5 when locking

Error Message:

Can't lock git dependency: git version must be 1.8.5 or higher.

Recovery: Upgrade Git

Dependency Application Not Found

Condition: Downloaded dependency doesn't contain valid OTP application

Error Format:

{dep_app_not_found, Name}

Error Message:

Dependency failure: source for my_dep does not contain a recognizable project and can not be built

Causes:

• No .app.src file
• Invalid application structure
• Wrong directory layout

Recovery:

• Check dependency is valid OTP app
• Verify repository URL is correct

Hex Package Not Found

Condition: Hex returns 404 for package

Error: Handled in resolution stage, not acquisition

Edge Cases

Cached Package with Modified ETag

Scenario: Cached package exists but ETag file doesn't match

Behavior:

• Re-download package
• Verify checksum
• Update cache and ETag

Interrupted Download

Scenario: Download interrupted mid-process

Behavior:

• Temporary directory removed on error
• Next run will re-attempt download
• Cached packages may be incomplete

Protection: Always use temporary directory first, then atomic move

Git Shallow Clone Limitations

Behavior: rebar3 uses --single-branch for optimization

Limitation: Cannot switch to other branches without re-cloning

Impact: Minimal; dependencies shouldn't change branches

SSH vs HTTPS Git URLs

SSH: git@github.com:user/repo.git HTTPS: https://github.com/user/repo.git

Differences:

• SSH requires SSH keys configured
• HTTPS may prompt for credentials
• Both work equivalently for fetching

Lock File: Both normalize to same format

Git Tag vs Branch vs Ref

Tag: Immutable reference to specific commit Branch: Mutable, points to latest commit on branch Ref: Direct commit SHA

Recommendation: Use tags for releases, refs for locking

Example:

{deps, [
    {cowboy, {git, "https://github.com/ninenines/cowboy.git", {tag, "2.9.0"}}}
]}.

Lock File Result:

{<<"cowboy">>, {git, "https://github.com/ninenines/cowboy.git",
                     {ref, "abc123..."}}, 0}

Note: Tag converted to ref in lock file

Local Git URLs

Format: file:///path/to/repo.git or /path/to/repo

Warning:

Local git resources (file:///...) are unsupported and may have odd behaviour.
Use remote git resources, or a plugin for local dependencies.

Recommendation: Use _checkouts/ for local development

Package Download with Slow Connection

Behavior: May timeout after default httpc timeout

Solution: Configure longer timeout in ~/.config/rebar3/rebar.config:

{hex, [
    {http_timeout, 120000}  % 120 seconds
]}.

Hex CDN Changes

Multiple CDNs Supported:

• Official: https://repo.hex.pm
• Mirrors: Various regions

Configuration: Use HEX_CDN environment variable

Example:

HEX_CDN=https://hexpm.example.com rebar3 compile

Transitive Dependencies of Git Deps

Scenario: Git dependency has its own dependencies

Behavior:

• After cloning Git dep, read its rebar.config
• Resolve its dependencies recursively
• Add to dependency tree

Example:

my_app → git_dep (Git)
git_dep → hex_dep (Hex)

Both fetched, Git first, then Hex

Cross References

Dependencies

• Initialization & Configuration: Offline mode, cache directories
• Dependency Resolution & Locking: Calls maybe_fetch

Dependents

• Application Discovery: Discovers fetched applications
• Source File Compilation: Compiles fetched dependencies

Related

• State Management: AppInfo structure
• Error Handling: Error handling details

Example Scenarios

Scenario 1: First Hex Package Download

Dependency: {jsx, "3.1.0"}

Execution: rebar3 compile (first time)

Flow:

1. Resolution determines need for jsx 3.1.0
2. Call maybe_fetch → needs download
3. Call rebar_pkg_resource:download/4
4. Check cache: ~/.cache/rebar3/hex/hexpm/packages/jsx-3.1.0.tar doesn't exist
5. Request from Hex: GET /tarballs/jsx-3.1.0
6. Download tarball (HTTP 200)
7. Verify checksum: sha256(tarball) vs registry checksum
8. Store in cache with ETag
9. Extract to _build/default/lib/jsx/
10. Read jsx/rebar.config
11. Discover jsx application
12. Mark as available

Files Created:

• ~/.cache/rebar3/hex/hexpm/packages/jsx-3.1.0.tar
• ~/.cache/rebar3/hex/hexpm/packages/jsx-3.1.0.etag
• _build/default/lib/jsx/ (full source tree)

Scenario 2: Cached Hex Package (ETag Valid)

Dependency: {jsx, "3.1.0"}

Execution: rebar3 compile (after clean, cache still valid)

Flow:

1. Check cache: tarball exists
2. Read ETag file: "abc123"
3. Request from Hex with If-None-Match: "abc123"
4. Hex returns HTTP 304 Not Modified
5. Use cached tarball
6. Extract to _build/default/lib/jsx/
7. Discover application

Network: Minimal (only HTTP HEAD request equivalent)

Scenario 3: Git Dependency with Tag

Dependency:

{cowboy, {git, "https://github.com/ninenines/cowboy.git", {tag, "2.9.0"}}}

Execution: rebar3 compile (first time)

Flow:

1. Call rebar_git_resource:download/4

2. Detect Git version: 2.30.0

3. Execute:

   git clone --depth 1 --no-single-branch \
     https://github.com/ninenines/cowboy.git \
     /tmp/rebar-abc123/cowboy \
     -b 2.9.0 --single-branch
   

4. Clone completes to temporary directory

5. Move to _build/default/lib/cowboy/

6. Read cowboy/rebar.config:

   {deps, [cowlib, ranch]}.
   

7. Discover cowboy application

8. Add cowlib and ranch to dependency queue

Subsequent Runs:

• Check if update needed: git describe --tags --exact-match
• Compare with "2.9.0"
• If match: skip re-download

Scenario 4: Git Dependency with Branch

Dependency:

{my_dep, {git, "https://github.com/user/my_dep.git", {branch, "main"}}}

First Run:

1. Clone with -b main --single-branch
2. Extract to _build/default/lib/my_dep/

Subsequent Runs:

1. Check for updates:

   cd _build/default/lib/my_dep
   git fetch origin main
   git log HEAD..origin/main --oneline
   

2. If output not empty: new commits available

3. Pull updates

4. Re-compile

Warning: Branches are mutable; can cause non-reproducible builds

Scenario 5: Offline Mode with Cache

Setup:

• jsx 3.1.0 previously downloaded and cached
• Offline mode enabled

Execution: REBAR_OFFLINE=1 rebar3 compile

Flow:

1. Attempt fetch for jsx
2. Detect offline mode
3. Check cache: tarball exists
4. Use cached tarball without network request
5. Extract and proceed

Success: Works without network

Scenario 6: Offline Mode Without Cache

Setup:

• Fresh dependency not in cache
• Offline mode enabled

Execution: REBAR_OFFLINE=1 rebar3 compile

Flow:

1. Attempt fetch for new_dep
2. Detect offline mode
3. Check cache: not found
4. Error: cannot fetch in offline mode

Error:

Cannot fetch dependency in offline mode

Solution: Disable offline mode or pre-cache

Scenario 7: Corrupted Package Cache

Setup: Cached package has wrong checksum (corruption or tampering)

Execution: rebar3 compile

Flow:

1. Check cache: jsx-3.1.0.tar exists
2. Use cached tarball
3. Calculate checksum
4. Compare with registry
5. Mismatch detected
6. Clear ETag file
7. Error reported

Error:

The checksum for package at jsx-3.1.0 (WRONG_HASH) does not match
the checksum expected from the registry (CORRECT_HASH).
Run `rebar3 do unlock jsx, update` and then try again.

Recovery:

rm ~/.cache/rebar3/hex/hexpm/packages/jsx-3.1.0.*
rebar3 compile

Scenario 8: Authentication Required for Private Git Repo

Dependency:

{private_dep, {git, "git@github.com:company/private_dep.git", {tag, "1.0.0"}}}

Execution: rebar3 compile

Prerequisites: SSH key added to GitHub account

Flow:

1. Git uses SSH authentication
2. Reads ~/.ssh/id_rsa or configured key
3. Authenticates with GitHub
4. Clone proceeds normally

If Authentication Fails:

Permission denied (publickey).
fatal: Could not read from remote repository.

Solution:

• Add SSH key to GitHub
• Or use HTTPS with token: https://TOKEN@github.com/company/private_dep.git