Posted on August 27th, 2018.
I’ve gotten a bunch of emails asking for advice on how to learn Common Lisp in
the present day. I decided to write down all the advice I’ve been giving
through email and social media posts in the hopes that someone might find it
useful.
One disclaimer up front: this is a road to Common Lisp, not the road to
Common Lisp. It’s what I followed (without some of the dead ends) and has
a lot of my personal opinions baked in, but it is by no means the only way to
learn the language.
This post has been translated into
Japanese.
I can’t vouch for the accuracy of any translations.
Context
I think it’s important to have a sense of where Common Lisp came from and what
kind of a language it is before you start learning it. There are some things
that will seem very strange if you’re coming straight from modern languages,
but will make more sense if you’ve got a bit of background context.
History
Common Lisp has a long, deep history. I’m not going to try to cover it all here
— if you’re interested you should check out some of the following (in roughly
increasing order of detail):
- Wikipedia’s History of Lisp and History of Common Lisp.
- The Where it Began section in Practical Common Lisp.
- The History: Where did Lisp come from? section of the comp.lang.lisp FAQ.
- Common Lisp: the Untold Story by Kent Pitman.
- The Evolution of Lisp by Guy Steele and Richard Gabriel.
I realize you probably won’t want to read all of the links above immediately, so
here’s a whirlwind tour of sixty years of Lisp.
Lisp began in the late 1950’s. It was invented by John McCarthy at MIT.
Over the next twenty or so years various versions and dialects of Lisp grew and
flourished. Some of the more notable dialects were Maclisp, BBN Lisp/Interlisp,
Franz Lisp, Spice Lisp, and Lisp Machine Lisp. There were others too. The
point is that there were a lot of different implementations, all growing,
changing, and trying out different things.
(Scheme also originated in this time frame, but took a very different route and
diverged from the path we’re looking at. I won’t cover Scheme in this post.)
In the early 1980s people decided that having a whole slew of
mutually-incompatible dialects of Lisp might be not be ideal. An effort was
made to take these different languages that had grown organically and produce
one common language that would satisfy the needs of everyone (or at least
a reasonable subset of “everyone”). In 1984 the first edition of Guy Steele’s
Common Lisp: the Language was published.
If you do some math you’ll see that at the time the book was published Lisp had
around twenty-five years of real-world use, experimentation, experience, and
history to draw upon. Even so, the book alone didn’t quite satisfy everyone and
in 1986 a committee (X3J13) was formed to produce an ANSI specification for
Common Lisp.
While the committee worked on the standardization process, in 1990 the second
edition of Common Lisp: the Language was published. This was more
comprehensive and contained some of the things the committee was working on
(see the comp.lang.lisp FAQ linked above for more on this). At this point the
Lisp family of languages had over thirty years of experience and history to
draw upon. For comparison: Python (a “modern” language many people think of as
also being “kind of old”) was released for the first time the
following year.
In 1992 the X3J13 committee published the first draft of the new Common Lisp
ANSI standard for public review (see Pitman’s paper). The draft was approved in
1994 and the approved specification was finally published in 1995. At this
point Lisp was over thirty-five years old. The first version of Ruby was
released in December of that year.
That’s the end of the history lesson. There has not been another revision of
the ANSI specification of Common Lisp. The version published in 1995 is the one
that is still used today — if you see something calling itself “an
implementation of Common Lisp” today, that is the specification it’s referring
to.
Consequences
I wanted to give you a quick overview of the history of Common Lisp because I
want you to know what you’re getting yourself into. I want you to realize that
Common Lisp is a stable, large, practical, extensible, ugly language.
Understanding these characteristics will make a lot of things make more sense
as you learn the language, and I want to talk a little bit more about each of
them before I start offering recommendations.
Escaping the Hamster Wheel of Backwards Incompatibility
If you’re coming from other languages, you’re probably used to things breaking
when you “upgrade” your language implementation and/or libraries. If you want
to run Ruby code you wrote ten years ago on the latest version of Ruby, it’s
probably going to take some effort to update it. My current day job is in Scala,
and if a library’s last activity is more than 2 or 3 years old on Github I just
assume it won’t work without a significant amount of screwing around on my part.
The Hamster Wheel of Backwards Incompatibility we deal with every day is a fact
of life in most modern languages, though some are certainly better than others.
If you learn Common Lisp, this is usually not the case. In the next section of
this post I’ll be recommending a book written in 1990. You can run its code,
unchanged, in a Common Lisp implementation released last month. After years of
jogging on the Hamster Wheel of Backwards Incompatibility I cannot tell you how
much of a relief it is to be able to write code and reasonably expect it to
still work in twenty years.
Of course, this is only the case for the language itself — if you depend on any
libraries there’s always the chance they might break when you update them. But
I’ve found the stability of the core language is contagious, and overall the
Common Lisp community seems fairly good about maintaining backwards
compatibility.
I’ll be honest though: there are exceptions. As you learn the language and
start using libraries you’ll start noticing some library authors who don’t
bother to document and preserve stable APIs for their libraries, and if
staying off the Hamster Wheel is important to you you’ll learn to avoid relying
on code written by those people as much as possible.
Practicality Begets Purity
Another thing to understand about Common Lisp is that it’s a large, practical
language. The second edition of Common Lisp: the Language (usually abbreviated
as “CLtL2” by Common Lisp programmers) is 971 pages long, not including the
preface, references, or index. You can get a surprising amount done by writing
pure Common Lisp without much extra support.
When programming applications in Common Lisp people will often depend on
a small(ish) number of stable libraries, and library writers often try to
minimize dependencies by utilizing as much of the core language as possible.
I try to stick to fewer than ten or so dependencies for my applications and no
more than two or three for my libraries (preferably zero, if possible), but I’m
probably a bit more conservative than most folks. I really don’t like the
Hamster Wheel.
It’s also worth noting that since Common Lisp has been around and stable for so
long, it has libraries older and more stable than many programming languages.
For example: Bordeaux Threads (the de-facto threading library for Common Lisp)
was first proposed in 2004 and released soon after (2006 at the latest but
possibly earlier, it’s hard to tell because so many links are dead now), which
makes it about fourteen years old. So yes, threading is handled by a library,
but I’m not worried about it breaking my code in the next decade or two.
My advice is this: as you learn Common Lisp and look for libraries, try to
suppress the voice in the back of your head that says “This project was last
updated six years ago? That’s probably abandoned and broken.” The stability of
Common Lisp means that sometimes libraries can just be done, not abandoned,
so don’t dismiss them out of hand.
Extensibility
Part of Common Lisp’s practicality comes from its extensibility. No one has
been clamoring for a new version of the specification that adds features
because Common Lisp’s extensibility allows users to add new features to the
language as plain old libraries, without having to alter the core language.
Macros are what might come to mind when you hear “Lisp extensibility”, and of
course that’s part of it. Macros allow users to write libraries that would
need to be core language features in other languages.
Common Lisp doesn’t include string interpolation. You want it? No problem, you
don’t have to wait for Scala
2.10 or
Python 3.6, just use
a library.
Want to try some nondeterministic programming without any boilerplate? Grab
a library.
Pattern matching syntax can make for some really beautiful, readable code.
Common Lisp doesn’t include it, but of course there’s a library.
Enjoying algebraic data types in Haskell or Scala? Here’s your
library.
All of these libraries rely on macros to make using them feel seamless. Of
course you could do all of that without macros, but you’ve have to add a layer
of boilerplate to manage evaluation. This:
(match foo
'(list x y z) (lambda (x y z) (+ x y z))
'(vector x y) (lambda (x y) (- x y)))
just doesn’t flow off the fingers like:
(match foo
((list x y z) (+ x y z))
((vector x y) (- x y)))
No one’s up in arms trying to get a new revision of the Common Lisp standard to
add pattern matching because you can write it as a library and get 90% or more
of what you’ve get if it were built in. The language gives you enough power to
extend it in a way that feels like the extension was there from the beginning.
Having things that are core features in other languages be provided by libraries
might seem at odds with the previous section about minimizing dependencies, and
to some extent that’s true. But I think there’s a happy medium where you can
write stable libraries in the core language and then depend on a small number of
those libraries in your applications to add exactly the features you need for
any particular problem.
Power
Macros are one of the things that make Lisp so extensible, because they let you
transform arbitrary code into other arbitrary code. This is true for macros in
languages like C too, but Common Lisp macros are different because they’re part
of the language.
In C you have a layer of macros on top, written in a preprocessor macro
language. The macro layer and the language layer are separate from each other,
with the macro layer providing one one extra level of abstractive power (which,
don’t get me wrong, is certainly useful).
In Common Lisp, you write macros in Common Lisp itself. You can then use
those macros to write functions, and use those functions to write more macros.
Instead of two stratified layers it’s a feedback loop of abstractive power.
But macros aren’t the only thing about Common Lisp that make it so practical and
extensible. Something people often don’t realize is that while Common Lisp is
an extremely high-level language thanks to macros, it also has plenty of
low-level facilities as part of the language. It’s never going to be as
low-level as something like C, Rust, or Forth, but you might be surprised at
some of the things that the ANSI spec includes.
Want to see the assembly code a particular function compiles down to?
DISASSEMBLE it!
Want to stack-allocate something to avoid some garbage collection? X3J13
thought of that.
Need arrays of unboxed floats to ship to a graphics card? The standard allows
for that.
Think GOTO should be considered helpful, not harmful? Well, okay, we’re all
adults here. Good luck, try not to shoot your foot off.
Need to do unsigned 8-bit arithmetic in your Game Boy emulator, but would prefer
it to compile down to just a machine instruction or two? It’s
possible.
Not all Common Lisp implementations actually perform all these optimizations,
but the designers of Common Lisp had the foresight to include the language
features needed to support them. You can write vanilla Common Lisp as defined
by the standard and trust that it will run everywhere, and implementations that
do support these kinds of things will take advantage of the optimization
opportunities.
This combination of supporting extremely high-level programming with macros and
a reasonable amount of low-level optimization mean that even though the
specification is over twenty years old, it’s still a good solid base to build on
today. The thirty years of experience and history the designers were drawing
from allowed them to create a very practical language that has survived for
decades.
Ugliness
It’s also important to realize that while Common Lisp might be very practical,
the need to accommodate existing users and dialects means that there are plenty
of ugly parts. If you buy a paper copy of the second edition of Common Lisp:
the Language and look up “kludges” in the index you’ll find this:
Common Lisp is not a beautiful crystal of programming language design. It’s
a scruffy workshop with a big pegboard wall of tools, a thin layer of sawdust on
the floor, a filing cabinet in the office with a couple of drawers that open
perpendicular to the rest, and there’s a weird looking saw with RPLACD written
on the side sitting off in a corner where no one’s touched it for twenty years.
This historical baggage is a price paid to ensure Common Lisp had a future. It
made it practical for people using the old dialects to actually adopt Common
Lisp with a reasonable amount of effort. If the designers had tried to make it
perfect and beautiful this could have made it too different to port
implementations and code to and might have resulted in the language being
ignored, instead of being adopted and embraced.
A Road to Learning Common Lisp
If all of this hasn’t scared you away from the language, let’s talk about how
you can learn it in 2018.
If you search around on the internet for Common Lisp tutorials and guides,
you’re not going to find as much as you might expect. This is because a lot of
Common Lisp reference material was created before or during the infancy of the
internet. There are a lot of books about Common Lisp out there. Some are
better than others. I’ll recommend the ones I think are the best, but don’t
hesitate to browse around and find others.
Get a Lisp
To get started with Common Lisp you’ll need to install a Common Lisp
implementation. Common Lisp is an ANSI specification, so there are multiple
implementations of it, which gives you choices. There are a bunch of options,
but I’ll make it simple for you:
- If you’re using MacOS and want a single GUI app you can download from the App
Store, choose ClozureCL (often abbreviated “CCL”). - Otherwise, choose SBCL.
That’s Clozure with a Z. Clojure is something entirely different that just
happens to have a confusingly similar name.
You might also hear of something called CLISP, which sounds like it might be
what you want. It’s not. CLISP is just another implementation, but it hasn’t
had a release in eight years (even though development is still ongoing in its
source repos!) and it’s not as commonly used as CCL or SBCL, so it’ll be harder
to find help if you have questions about the installation, etc.
You might also hear about something called Roswell. Don’t use Roswell, you don’t
need it (yet (or at all)).
Just install SBCL or CCL for now, you can explore the other options once you’ve
got your bearings a bit better.
Pick an Editor
You might hear people tell you that you must learn Emacs before learning
Common Lisp. They’re wrong. You can get started learning the language just
fine in whatever text editor you’re comfortable in.
If you don’t have a preference, CCL itself comes bundled with a text editor on
MacOS. That one will work just fine to start.
Emacs, Vim, Sublime Text, Atom, whatever, for now it doesn’t matter. As long as
it can balance parentheses, highlight comments and strings, and autoindent Lisp
code that’s all you need to start. Worry about shaving the editor yak once
you’re more comfortable in the language.
Hello, Lisp
To check that you’ve got everything set up properly, make a hello.lisp file
with the following contents:
(defun hello ()
(write-line "What is your name?")
(let ((name (read-line)))
(format t "Hello, ~A.~%" name)))Don’t worry about what this means yet, it’s just a check that everything’s
working properly.
Open an SBCL or CCL REPL (Read/Eval/Print
Loop) and
load the file by entering (load "hello.lisp"), then call the function and make
sure it works. It should look something like this if you picked SBCL:
$ sbcl
* (load "hello.lisp")
T
* (hello)
What is your name?
Steve
Hello, Steve.
NIL
*
Or if you chose CCL but still want to use the command line, rather than the
MacOS app (the command line program might be annoyingly named ccl64 if you’re
on a 64-bit system):
$ ccl64
Clozure Common Lisp Version ...
? (load "hello.lisp")
#P"/home/sjl/Desktop/hello.lisp"
? (hello)
What is your name?
Steve
Hello, Steve.
NIL
?
If your arrow keys and backspace don’t work in the REPL, use rlwrap
to fix that. rlwrap sbcl will give you a non-miserable REPL. rlwrap is
a handy tool to have in your toolbox anyway.
A Gentle Introduction
The best book I’ve found for getting started in Common Lisp is Common Lisp:
A Gentle Introduction to Symbolic Computation. This book really
does strive to be gentle. Even if you’ve programmed before I’d still recommend
starting here because it eases you into the language.
The 1990 edition is available free from the site, and there’s a 2013 reprint
which fixes some minor errors in the 1990 version. If you can afford it I’d
recommend buying the 2013 edition, but the 1990 version will also do fine.
Go through the book and do all the exercises. This will take a while, and
is mainly meant to get you started overcoming some of the main obstacles to
being comfortable in Common Lisp, such as:
- How am I ever going to remember all these weird function names?
- Why do people use strings so rarely?
- When do I need the god damn quotation mark?
If you find the book is moving too slow, just skim forward a bit. Skimming is
a very useful skill to practice as a programmer. I think it’s better for
authors to err on the side of explaining too much when writing books and
documentation — expert readers should be comfortable skimming if you explain too
much, but new users will be stuck wallowing in confusion if you’re too terse.
Creating hours of newbie misery and confusion to save a few flicks of an
expert’s scroll wheel is a poor tradeoff to make.
You should also join the #clschool channel on the Freenode IRC network so you
can ask questions if you get stuck. For the most part people there are friendly
and helpful, though I’ll warn you in advance that there’s at least one person
who can sometimes be abrasive. There’s also a #clnoobs channel, but that was
mostly abandoned durin
