I've been obsessed with Lisp for a long time. For most of that time, I've had dreams and aspirations of building a new Lisp dialect that compiled down to machine code as a standalone static executable. I reasoned that if only I could ship static binaries, no one would care what language I wrote the software in, and I could finally use Lisp!
Last week, that all changed, when it finally dawned on me that nobody cares what language I write my software in, because most of the time it's getting stuffed into:
- A BOSH Release
- A Docker Image
- A Cloud Foundry Application
Suddenly, I was without reasons blocking me from using Lisp! I packaged up SBCL into a BOSH Release, and then I went off to build a Common Lisp buildpack for Cloud Foundry! It all worked! Yay!
Now, I needed to get down to the business of actually writing software (the scary part)! So now I had to go find solutions for all the things I was used to getting in other languages, like JSON libraries, web frameworks, etc.
This is the story of building a small web API server in Common Lisp. We're going to play with some cool stuff, and it won't be as painful as you might think.
Here goes.
The Quicklisp-ening
Libraries are a programmers best friend. Networks for finding and installing libraries on-demand are even more dear.
For this little foray, we'll be using Quicklisp. Go read the docs and get it installed if you're going to play along at home. We're going to be leveraging three libraries, Hunchentoot, Drakma, and cl-json.
;; put this at the top of your Common Lisp source file
(load "~/quicklisp/setup.lisp")
(ql:quickload :hunchentoot)
(ql:quickload :drakma)
(ql:quickload :cl-json)
Shouting On The Internet
I don't have time to come up with a toy application, so we're just going to build a real one.
Shout! is a notifications gateway that I've wanted to build for a long time. Eventually, it will grow into a flexible means of applying business logic to break/fix style notifications, determing how, when, and where to dispatch to real messaging systems like email, SMS, Slack, etc.
For now, it's just going to send all the messages to Slack, 24/7. This is a blog post, not a Ken Burns documentary.
Shout! operates on a stream of events, and maintains a set of states that result from said events. If you've ever dealt with a failing Concourse pipeline, you've probably seen a string of failure messages (pipeline's broke. pipeline's broke. pipeline's broke.) followed by silence (by which you know the pipeline is now fixed). With Shout!, the first failure elicits an "it's broke" notification, and the next success sends the all-clear.
You Could Be A Data Model, Baby
Let's start with the types of objects we'll be dealing with.
An event is a single input from the outside world:
(defclass event ()
((message
:initarg :message
:accessor message)
(ok
:initarg :ok
:accessor event-ok?)))
A state tracks events occurring to a "topic":
(defclass state ()
((name
:initarg :name
:accessor state-name)
(status
:initarg :status
:initform "unknown"
:accessor status)
(last-event
:initarg :last-event
:accessor last-event)))
Then we can define an association list of states, indexed by topic:
(defvar *states* ())
We're going to need a way of applying a new event to a state, and thereby modify the status, to go from "working" to "broken", for example:
(defun still-ok? (e1 e2)
(and (event-ok? e1)
(event-ok? e2)))
(defun transition (e1 e2)
(cond ((still-ok? e1 e2) "working")
((event-ok? e2) "fixed")
(t "broken")))
still-ok? checks two events, which should have occurred in sequence, and returns whether or not the state they belong to is still ok. There's no point in notifying people that things are still good all the time.
The transition function takes two events (also in sequence) and returns the word that describes the transition. We'll use this in our Slack notification to say stuff like the ice cream parlor is still broken and the political divide is now fixed.
(defun update-state (state event)
(let ((prev (last-event state)))
(setf (status state) (transition prev event)
(last-event state) event)
(when (not (still-ok? prev event))
(notify-about state))
state))
(defun create-state (key topic event)
(let ((state (make-instance 'state
:name topic
:last-event event
:status (if (event-ok? event)
"working" "broken"))))
(setf *states* (acons key state *states*))
state))
(defun ingest (topic event)
(let* ((key (intern topic))
(state (cdr (assoc key *states*))))
(if state
(update-state state event)
(create-state key topic event))))
Finally, ingest takes a topic (string), and an incoming event, and does the needful with states; if we already have a state for the given topic, we determine if there was a transition, handle the bookkeeping, and even send out a notification. Otherwise, a new state gets created and put into the association list (via acons).
We'll skip notify-about for right now.
My Old Friend, JSON
Before we can get to the webby stuff, we need JSON representations for our event and state objects. One thing I've found is that whenever I need a library in Common Lisp, it's out there. A good place to start is Quicklisp. Sure enough, they have a cl-json library!
let's build some helper functions to transform our objects into structures suitable for JSON-ification:
(defun event-json (event)
(when event
`((message . ,(message event))
(ok . ,(event-ok? event)))))
(defun state-json (state)
(when state
`((name . ,(state-name state))
(status . ,(status state))
(last . ,(event-json (last-event state))))))
We can pass the output of these two functions directly to the JSON library to get back JSONified strings.
Serving Up Hot, Fresh HTTP
Now we need to go get ourselves a web server. Luckily, Common Lisp has a pretty nice one, the eminently google-able Hunchentoot. It works like most other web dispatch frameworks — set up some handlers and let the library do the heavy lifting.
We only really need one endpoint, POST /events. We're going to build two. GET /states will return the full states database, in JSON form.
(defun api (port)
(defun handle-get-states ()
(setf (content-type* *reply*) "application/json")
(format nil (encode-json-to-string *states*)))
(push (create-prefix-dispatcher "/states" 'handle-get-states)
*dispatch-table*)
(start (make-instance 'easy-acceptor :port port)))
handle-get-states is our worker function; it sets the Content-Type HTTP header to indicate that we're sending down JSON, and then JSONifies the states database.
Next up, POST /events takes a JSON object with message and ok keys, tracks the event, and reacts. Mostly this is just a call to ingest, with some serialization functions.
(defun json-body ()
(decode-json-from-string
(raw-post-data :force-text t)))
(defun attr (object field)
(cdr (assoc field object)))
(defun api (port)
(defun handle-get-states ()
(setf (content-type* *reply*) "application/json")
(format nil (encode-json-to-string *states*)))
(push (create-prefix-dispatcher "/states" 'handle-get-states)
*dispatch-table*)
(defun handle-post-events ()
(setf (content-type* *reply*) "application/json")
(let ((b (json-body)))
(format nil "~A~%"
(encode-json-to-string
(state-json
(ingest
(attr b :topic)
(make-instance 'event :ok (attr b :ok)
:message (attr b :message))))))))
(push (create-prefix-dispatcher "/events" 'handle-post-events)
*dispatch-table*)
(start (make-instance 'easy-acceptor :port port)))
I introduced some helper utilities. json-body decodes the raw HTTP request body (for our POST endpoint). attr is just shorthand for retrieving the value of a key from an association list. This is what Lisp is all about to me: writing small, composable utility functions that make the rest of the codebase clearer.
Now we have everything we need in our api function. We could stop here, but I couldn't shake a bad feeling while I was writing that last version. It's a lot of repetition, and it's a bit awkward. For each endpoint, we define a function, then we hook it into the dispatcher.
I'd rather do this:
(handle "/path/to/register"
;; body of the handler
;; do stuff, and return an object
*states*)
You may not be able to do this in other languages easily, but this is Lisp!
Let's write a macro.
(defmacro handle (url &body body)
(let ((fn (gensym)))
`(progn
(defun ,fn ()
(setf (content-type* *reply*) "application/json")
(format nil "~A~%" (encode-json-to-string (progn ,@body))))
(push (create-prefix-dispatcher ,url ',fn) *dispatch-table*))))
(defun api (port)
(handle "/states" *states*)
(handle "/events"
(let ((b (json-body)))
(state-json
(ingest
(attr b :topic)
(make-instance 'event :ok (attr b :ok)
:message (attr b :message))))))
(start (make-instance 'easy-acceptor :port port)))
With this macro, we've simultaneously cut down on the number of lines of code, and increased the clarity of the api function. And since its a macro, there's no runtimepenalty! Win!
Slacking Off
The last piece of this puzzle is integrating with Slack. It's high time we implemented notify-about. For that, we'll need an HTTP client library, and that's what Drakma is for.
If you recall, I plan to eventually do more than just Slack, so we're going to implement a standalone function for sending messages via Slack.
(defun slack (ok summary details)
(http-request
*slack-webhook*
:method :post
:content (encode-json-to-string
`((text . ,summary)
(username . "shout!bot")
(icon_url . "https://bit.ly/2AC9vAV")
(attachments
((text . ,details)
(color . ,(if ok "good" "danger"))))))))
Then notify-about just becomes a thin wrapper:
(defun notify-about (state)
(let ((event (last-event state)))
(slack (event-ok? event)
(format nil "~A is ~A"
(state-name state)
(status state))
(message event))))
Final Thoughts
Here's the final code, which you can take for a spin by running this in the SBCL REPL:
(load "shout.lisp")
(api 8080)
This (complete, working) implementation clocks in at 122 lines of code. That's pretty impressive, and speaks to both the breadth of the language, the extent of the standard library, and the power of macros.
In a future post, I'll go into more detail about how I package up Common Lisp code like Shout! into BOSH, and how I built the Cloud Foundry Buildpack for Common Lisp. Stay tuned!
James (@iamjameshunt) works on the Internet, spends his weekends developing new and interesting bits of software and his nights trying to make sense of research papers.
Currently exploring Kubernetes, as both a floor wax and a dessert topping.