Testing Clojure Web Applications with Selenium

This article is brought with ❤ to you by Semaphore.

Selenium is a commonly used set of tools for automating browsers. It allows you to drive a browser interaction with a web page by writing code. It's most often used to write browser-based tests for web applications. Tests can be executed in a development environment or even on a real application as part of a smoke test suite.

Selenium provides support for most popular languages and browsers. This tutorial explains how to use Selenium for testing Clojure web applications. Our setup will be based on clj-webdriver and Firefox, and we will use Compojure to write a very simple web application.

Setting Up the Project

Prerequisites

For developing a Clojure application using this tutorial you will need:

  • Java JDK version 6 or later.
  • Leiningen 2.
  • Firefox 39 or lower.

At the time of this writing, if you have a newer version of Firefox, the example might not work for you. If it doesn't, you will need to downgrade Firefox. You can see what's the last version of Firefox that Selenium officially supports on the Selenium changelog page. If you plan to use Selenium-based tests regularly, you might want to hold Firefox updates until Selenium starts supporting them.

Create a Hello World Compojure Application

Compojure is a routing library for Ring, and a popular choice for writing web applications in Clojure. Leiningen provides a Compojure template that allows us to get started with Compojure quickly.

Create a Compojure-based Clojure project:

lein new compojure clj-webdriver-tutorial

The second parameter compojure is the name of the template that's going to be used for creating the application. The last parameter, clj-webdriver-tutorial, is the name of your project.

Navigate to the project directory:

cd clj-webdriver-tutorial

Start the server:

lein ring server-headless

After the server starts, visit http://localhost:3000 in a browser and you should see the Hello World greeting from the application:

Hello World

Compojure Application Structure

The structure of your application should look like this:

├── project.clj
├── README.md
├── resources
│   └── public
├── src
│   └── clj_webdriver_tutorial
│       └── handler.clj
├── target
│   ├── ...
└── test
    └── clj_webdriver_tutorial
        └── handler_test.clj

The file that we're interested in is src/clj_webdriver_tutorial/handler.clj. If you open it, it should contain the following code:

(ns clj-webdriver-tutorial.handler
  (:require [compojure.core :refer :all]
            [compojure.route :as route]
            [ring.middleware.defaults :refer [wrap-defaults site-defaults]]))

(defroutes app-routes
  (GET "/" [] "Hello World")
  (route/not-found "Not Found"))

(def app
  (wrap-defaults app-routes site-defaults))

It defines the access point to the application (/ - the root path), and we can see that this is where that "Hello World" is coming from.

We can also notice that Leiningen created the handler_test.clj file that's using clojure.test to test the handler. Since we're concentrating on clj-webdriver instead, let's remove the test:

rm test/clj_webdriver_tutorial/handler_test.clj

Install clj-webdriver

Install clj-webdriver by adding the project development dependencies to project.clj:

(defproject clj-webdriver-tutorial "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :min-lein-version "2.0.0"
  :dependencies [[org.clojure/clojure "1.6.0"]
                 [compojure "1.3.1"]
                 [ring/ring-defaults "0.1.2"]]
  :plugins [[lein-ring "0.8.13"]]
  :ring {:handler clj-webdriver-tutorial.handler/app}
  :profiles
  {:dev {:dependencies [[clj-webdriver "0.7.1"]
                        [org.seleniumhq.selenium/selenium-server "2.47.0"]
                        [javax.servlet/servlet-api "2.5"]
                        [ring-mock "0.1.5"]
                        [ring/ring-jetty-adapter "1.4.0"]]}})

There are several new things in project.clj:

  • We added clj-webdriver "0.7.1".
  • Next, we explicitly added the selenium-server that supports at least Firefox
    1. If you have a newer version of Firefox, you can try upgrading selenium-server to the latest available Selenium version.
  • We also added ring-jetty-adapter to run the application before executing tests.

First clj-webdriver Test

Create the features directory where you will put clj-webdriver tests:

mkdir test/clj_webdriver_tutorial/features

Open test/clj_webdriver_tutorial/features/config.clj and add some common configurations that we will use in tests:

(ns clj-webdriver-tutorial.features.config)

(def test-port 5744)
(def test-host "localhost")
(def test-base-url (str "http://" test-host ":" test-port "/"))

The default configuration states that tests will be executed against the application running on http://localhost:5744. 5744 is the default port for Selenium.

Our first test will check if the home page really displays the "Hello World" message. Since we're testing by opening a real browser, the test needs some setup and teardown. Here are the steps that need to be executed:

  1. Start the server for the application.
  2. Open the root path in the browser.
  3. Check if the "Hello World" message is present on the page.
  4. Close the browser.
  5. Shut down the server.

Let's write a skeleton of that code in test/clj_webdriver_tutorial/features/homepage.clj:

(ns clj-webdriver-tutorial.features.homepage
  (:require [clojure.test :refer :all]
            [ring.adapter.jetty :refer [run-jetty]]
            [clj-webdriver.taxi :refer :all]
            [clj-webdriver-tutorial.features.config :refer :all]
            [clj-webdriver-tutorial.handler :refer [app-routes]]))

(deftest homepage-greeting
  (let [server (start-server)]
    (start-browser)
    (to test-base-url)
    (is (= (text "body") "Hello World"))
    (stop-browser)
    (stop-server server)))

The most important parts are the to and text functions that are used for navigating to a page and extracting text from a node, respectively. They are part of the clj-webdriver Taxi API.

Before running the test, we need to implement the start-server, start-browser, stop-browser and stop-server functions.

The start-server function is the most complex one, as it starts the jetty server on the test port and waits for the server to be started:

(defn start-server []
  (loop [server (run-jetty app-routes {:port test-port, :join? false})]
    (if (.isStarted server)
      server
      (recur server))))

The other functions are much simpler:

(defn stop-server [server]
  (.stop server))

(defn start-browser []
  (set-driver! {:browser :firefox}))

(defn stop-browser []
  (quit))

As they are actually wrappers against respective functions in the clj-webdriver, they can be used directly in a real application test.

Putting in all together and our first code in test/clj_webdriver_tutorial/features/homepage.clj looks like this:

(ns clj-webdriver-tutorial.features.homepage
  (:require [clojure.test :refer :all]
            [ring.adapter.jetty :refer [run-jetty]]
            [clj-webdriver.taxi :refer :all]
            [clj-webdriver-tutorial.features.config :refer :all]
            [clj-webdriver-tutorial.handler :refer [app-routes]]))

(defn start-server []
  (loop [server (run-jetty app-routes {:port test-port, :join? false})]
    (if (.isStarted server)
      server
      (recur server))))

(defn stop-server [server]
  (.stop server))

(defn start-browser []
  (set-driver! {:browser :firefox}))

(defn stop-browser []
  (quit))

(deftest homepage-greeting
  (let [server (start-server)]
    (start-browser)
    (to test-base-url)
    (is (= (text "body") "Hello World"))
    (stop-browser)
    (stop-server server)))

Run the tests suite:

lein test

And you will see that the test passed:

Ran 1 tests containing 1 assertions.
0 failures, 0 errors.

You now have a basic setup for testing Clojure web applications with Selenium.

Cleaning Up

This setup works well, but we have to remember to start the server and the browser before each test, and to shut them down after the tests are done. To make things easier, we can implement test fixtures that will do this automatically before and after every test.

The fixture for handling the server can be implemented as follows:

(defn with-server [t]
  (let [server (start-server)]
    (t)
    (stop-server server)))

The t parameter stands for test case function. It starts the server before the test case, executes the test function, and stops the server.

The fixture for handling the browser is similar:

(defn with-browser [t]
  (start-browser)
  (t)
  (stop-browser))

Using fixtures, we can write a much cleaner test:

(ns clj-webdriver-tutorial.features.homepage
  (:require [clojure.test :refer :all]
            [ring.adapter.jetty :refer [run-jetty]]
            [clj-webdriver.taxi :refer :all]
            [clj-webdriver-tutorial.features.config :refer :all]
            [clj-webdriver-tutorial.handler :refer [app-routes]]))

;; Fixtures

(defn start-server []
  (loop [server (run-jetty app-routes {:port test-port, :join? false})]
    (if (.isStarted server)
      server
      (recur server))))

(defn stop-server [server]
  (.stop server))

(defn with-server [t]
  (let [server (start-server)]
    (t)
    (stop-server server)))

(defn start-browser []
  (set-driver! {:browser :firefox}))

(defn stop-browser []
  (quit))

(defn with-browser [t]
  (start-browser)
  (t)
  (stop-browser))

(use-fixtures :once with-server with-browser)

;; Tests

(deftest homepage-greeting
  (to test-base-url)
  (is (= (text "body") "Hello World")))

Note that we passed the :once parameter to the use-fixtures function. This means that the browser will be started once before all tests, and stopped after all tests are finished. The same goes for the server. This should significantly speed up the tests in the file.

In a real application, you can move fixture functions to a separate namespace that is shared by all tests.

Conclusion

Selenium is a valuable tool for testing web applications, and it is indispensable if an application is heavily using JavaScript. Setting up Selenium with Clojure requires several steps covered in the tutorial. Using fixtures, a test setup can be reused, which results in cleaner and faster tests.

You can find more information about common functions for interacting with web pages and inspecting page content in the clj-webdriver Taxi API documentation.

This article is brought with ❤ to you by Semaphore.

Permalink

Creating serverless applications with ClojureScript and Firebase

Earlier this year, I traveled to India and gave a presentation at IN/Clojure. I talked about building serverless ClojureScript applications that use Firebase to persist and sync data between clients.

I was pretty pleased with how the talk went. The people who talked to me after seemed to enjoy the presentation and were inspired to try out some of the techniques and tools I mentioned.

Here is the talk. I hope you enjoy it. It was fun to give.

Permalink

The Expression Problem in Go

I've written about the expression problem here and later also here. Between them, these posts presented various solutions and non-solutions in C++, Haskell and Clojure.

Today I want to add another language into the mix - Go. It turns out that Go interfaces allow us to solve the expression problem, or at least a limited variant thereof.

The Go way

Go's main vehicle of abstraction and polymorphism is interfaces. Go encourages the use of small interfaces, such that types could be easily implementing multiple interfaces if needed.

It turns out this tool and philosophy is exactly suitable for tackling the expression problem in Go.

The expression problem in Go

For a recap of the expression problem, please refer to the past posts linked at the top. Without further ado, here's a Go solution.

type Expr interface {
}

// Types
type Constant struct {
  value float64
}

type BinaryPlus struct {
  left  Expr
  right Expr
}

Note that the Expr interface is empty - any type implements it. We need something like this so that we can specify the types for the left and right members of BinaryPlus; it really means left and right can be of any type. We'll get back to this discussion shortly.

Now an interface for expressions we can evaluate:

type Eval interface {
  Eval() float64
}

And implementations for our existing types:

func (c *Constant) Eval() float64 {
  return c.value
}

func (bp *BinaryPlus) Eval() float64 {
  return bp.left.(Eval).Eval() + bp.right.(Eval).Eval()
}

The instance for Constant is straightforward, but what's going on with BinaryPlus? Recall that left and right can have arbitrary types at compile time. But at run-time, we need these to be Eval, hence the type casts. This code will fail with a cast error at run-time if the objects occupying b's left or right slots don't, in fact, imlement Eval.

Note that we could force Expr to have at least an Eval method - all expressions have to be evaluable, after all. This would make the Expr interface a bit less empty, and also would remove the need for casts in implementations of Eval. We'd still need casts for other interfaces though, as we'll see shortly.

OK, now we have the basics. Let's add another operation, without modifying existing code:

type Stringify interface {
  ToString() string
}

func (c *Constant) ToString() string {
  return strconv.FormatFloat(c.value, 'f', -1, 64)
}

func (bp *BinaryPlus) ToString() string {
  ls := bp.left.(Stringify)
  rs := bp.right.(Stringify)
  return fmt.Sprintf("(%s + %s)", ls.ToString(), rs.ToString())
}

No surprises here. How about adding a new type:

type BinaryMul struct {
  left  Expr
  right Expr
}

func (bm *BinaryMul) Eval() float64 {
  return bm.left.(Eval).Eval() * bm.right.(Eval).Eval()
}

func (bm *BinaryMul) ToString() string {
  ls := bm.left.(Stringify)
  rs := bm.right.(Stringify)
  return fmt.Sprintf("(%s * %s)", ls.ToString(), rs.ToString())
}

Again, very similar code to what we've written for other types. Finally, let's write some client code that uses these types and operations:

func CreateNewExpr() Expr {
  c11 := Constant{value: 1.1}
  c22 := Constant{value: 2.2}
  c33 := Constant{value: 3.3}
  bp := BinaryPlus{left: &BinaryPlus{left: &c11, right: &c22}, right: &c33}
  return &bp
}

func main() {
  ne := CreateNewExpr()
  fmt.Printf("ne Eval = %g\n", ne.(Eval).Eval())
  fmt.Printf("ne ToString = %s\n", ne.(Stringify).ToString())
}

This snippet demonstrates how we can create new expressions at runtime (the return type of CreateNewExpr is simply Expr) and observe their implementation of the interfaces we've defined. When run, it prints:

ne Eval = 6.6
ne ToString = ((1.1 + 2.2) + 3.3)

Discussion

So, what works well for Go here, and what doesn't?

On one hand, this seems too easy. In my initial post, I've shown an attempt to implement the same thing in C++, and encountered a fundamental problem very early - C++ has to know, at compile-time, which interfaces a class implements. Go doesn't, and herein lies the difference. Go is more like Clojure than like C++ in this respect - methods are defined outside of types, which provides two strong advantages:

  1. Given a new type, we can make it implement a bunch of interfaces without modifying these interfaces.
  2. Given a new interface, we can make existing types implement it without modifying their code, just by adding new code. One could say that Go satisfies the open/closed principle very well.

On the other hand, it's important to understand that the casts involved in this solution lose the static type safety guarantees of the language. Again, here Go is more like Clojure than like C++. The compiler can not verify at compile-time that we assign the right sub-expressions to a BinaryPlus, for example. We can write non-sensical code like this:

bp1 := BinaryPlus{left: "john", right: Constant{value: 2.2}}
fmt.Printf("bp1 Eval = %g\n", bp1.Eval())

And it will compile just fine! It will fail at run-time, though:

panic: interface conversion: string is not main.Eval: missing method Eval

goroutine 1 [running]:
main.(*BinaryPlus).Eval(0xc420047f30, 0xc420047df0)
  goexprsolution.go:47 +0x47
[...]

In fact, one could reasonably claim this is not a valid solution to the expression problem as originally defined by Philip Wadler:

The goal is to define a datatype by cases, where one can add new cases to the datatype and new functions over the datatype, without recompiling existing code, and while retaining static type safety (e.g., no casts).

Note the explicit ban on casts.

Permalink

Overview of the nREPL

In this post I'm going to explain in a rough high level about how the nREPL you use every day actually works. This isn't intended to be 100% accurate, but close enough that you can build a mental model for understanding it when it goes wrong.

Client/Server

A common confusion is that lein repl is not the same as nREPL. When you run lein repl, two things happen:

  1. A server is started by running tools.nrepl/start-server.
  2. REPL-y, a terminal nREPL client is launched

The server aspect is the focus of this post. REPL-y is a terminal UI nREPL client, it allows you to evaluate code from the terminal. Another client is CIDER.el, which communicates via nREPL to enhance emacs with features. It also has an nREPL server extension, CIDER-nREPL, which is written in Clojure.

Transport Layer

The nREPL has the notion of a transport. This is an abstract concept of communicating with the nREPL, for sending & receiving messages. The default transport (ergo most popular) is bencode over tcp. Bencode is a simple encoding format designed by Bittorrent for encoding torrent files.

It can be described concisely:

;; A number, i prefix, e marks the end
i10e ;; 10

;; A string, <length>: prefix. Length is bytes (for unicode)
4:spam ;; "spam"
6:poop💩 ;; "poop💩" the unicode symbol is 2 bytes

;; A list, l prefix, e marks the end, elements nest
li1ei2ei3ee ;; [1 2 3]
l4:spami2ei3ee ;; ["spam" 2 3]

;; An associative array (hash map)
d3:foo3:bari1e4:spame ;; {"foo" "bar", 1 "spam"}

This provides a very simple format for reading data structures over the network. Other transports exist such as drawbridge, a HTTP transport; and nrepl-hornetq which provides a HornetQ transport for tools.nrepl server.

Each client has to support the various transports natively within it. Some clients are open to extension, but the primary transport remains bencode over tcp. I don't doubt this is related to the high cost of rewriting multiple transports for each client.

Messages

A message is simply a map; there are two kinds of message, a request and a response. There are common keys in a request message:

  • id: Is a unique id for the message, that should be included in the request message. Responses relating to this request should also include this same id.
  • op: Specifies what to do. In a way, it's specifying which handler to run; eval is a common operation, as is stdin.
  • session: This specifies an id which relates to a set of thread bindings. I'll cover this in more detail later.

An unencoded request message to an nREPL server could look like this:

{:op "eval" :code "(println (+ 1 2))" :id "some-unique-id"}

The nREPL server then handles this message however it pleases. I'll cover specifically how tools.nrepl handles this internally later in the post.

Responses have a similar shape. A single request can elicit many responses. When the request has finished being processed, it should include "done" in the list of :status values. It's important that the response :id is the same request as the :id, this allows a client to track particular responses, this is due to the nrepl server being asyncronous in design. You can have multiple requests in flight at a time. For this reason, it's important that the :id is globally unique, I generally use a UUIDv4.

For the example request above, I would expect 2 response maps like such:

{:id "some-unique-id" :out "3"}
{:id "some-unique-id" :value "nil" :status ["done"]}

Sessions

Sessions persist dynamic vars (collected by get-thread-bindings) against a unique lookup. This is allows you to have a different value for *e from different REPL clients (e.g. two separate REPL-y instances). An existing session can be cloned to create a new one, which then can be modified. This allows for copying of existing preferences into new environments.

Sessions become even more useful when different nREPL extensions start taking advantage of them. debug-repl uses sessions to store information about the current breakpoint, allowing debugging of two things separately. piggieback uses sessions to allow host a clojurescript repl alongside an existing clojure one.

An easy mistake is to confuse a session with an id. The difference between a session and id, is that an id is for tracking a single message, and sessions are for tracking remote state. They're fundamental to allowing simultaneous activities in the same nREPL.

Internals of tools.nrepl Message Handling

This section deals specifically with how tools.nrepl, the most popular implementation of nREPL, is implemented. Messages are handled via middleware. This will feel familiar if you have used Ring.

It's a unix system

tools.nREPL middleware works slightly differently to how ring middleware works, because you can handle the message and pass it on to the next handler. This is because responses are asyncronous.

A :transport key is added to messages as they are handled. The value of this key is an implementation of a protocol representing the transport layer. By replacing the transport with a new one, middleware can control the responses of middleware after it. Usually the replacement delegates back to the original transport, after updating the message or doing something with it.

;; Print out messages as they happen
(fn [next-fn transport]
  (next-fn (reify TransportProtocol
             (send [_ msg]
               (prn msg)
               (send transport msg)))))

;; Add pretty printed versions of values
(fn [next-fn transport]
  (next-fn (reify TransportProtocol
             (send [_ msg]
               (send transport
                     (assoc msg :pprint-value (pprint (:value msg))))))))

The default functionality of the nREPL is implemented entirely using middleware. Even the fundamental parts like sessions and eval. Default functionality like eval is provided in order to be useful out of the box.

Middleware Descriptors

A message handler can set a descriptor upon itself. This is metadata which tools.nrepl uses to order middleware automatically and generate documentation.

:requires This specifies middleware which must be present before this middleware in the stack. Specifying a string indicates that anything which provides this op must be present, you can also also use vars here.

:expects This specifies middleware which must be present after this middleware in the stack. Again, use vars or strings here.

:handles This is a documentation key. It includes text strings for human explanations of your returns, arguments, etc.

Using the :requires and :expects key, tools.nrepl is able to automatically order the middleware and handlers automatically. This is why you do not have to manually order middleware like so in .lein/profile.clj:

:nrepl-middleware (cider (pr-values (eval)))

Practical Example: Piggieback

Piggieback is an nREPL middleware by Chas Emerick. It underlies the ability to execute Clojurescript over the nREPL. It's used in figwheel and boot-cljs-*

To launch a Piggieback session you clone the current nREPL session, and update a dynamic var in Piggieback. The dynamic var stores a repl env, which is a clojurescript protocol for evaluation.

Piggieback uses the nREPL middleware dependency system to specify that eval should be after it in the chain. This allows it to "hijack" an incoming eval. It will use it's dynamic vars to determine whether or not Clojurescript is activated.

If it's dynamic var is set, it will use the clojurescript api to send code to the repl-env. Usually, it's a browser env, so it will be sent to your browser for evaluation, the result will be captured by the javascript & sent back. This communication is done via websockets, out of band.

This setup allows you to evaluate Clojure and Clojurescript in the same nREPL, by changing the session you're using to control where the evaluation happens.

CIDER-nrepl's place in all this

CIDER-nrepl provides a lot of functionality which works around the limitations imposed by bencode, providing additional semantic data not available when using eval. All eval responses get pr-str'd in order to prevent issues with encoding.

For example, it turns the classpath into a bencode list, as opposed to to the string "(\"src/\" \"resources/\")" that you'd get from using eval to do the same thing.

It also bundles dependencies in a unique way, via Mr. Anderson, so that they don't conflict with your project. This means for example, that it brings tools.namespace in order to provide a refresh, without conflicting with the version already in your project.

The open design of CIDER-nrepl has allowed unexpected clients such as vim-fireplace to have a number of additional features with only integration effort.

Permalink

Access Control in Datomic Cloud

In this article, we will look at Datomic access control, covering

  • authentication and authorization
  • network-level access
  • how the bastion works
  • some example deployments

Authentication and Authorization

Datomic integrates with AWS Identity and Access Management (IAM) for authentication and authorization, via a technique that we'll call S3 proxying. Here's how it works:

Every Datomic permission has a hierarchical name. For example, read-only access to database Jan is named access/dbs/db/Jan/read.

Permission names have a 1-1 correspondence with keys in the Datomic system S3 bucket.

The Datomic client signs requests using AWS's Signature Version 4. But instead of using your IAM credentials directly, the Datomic client uses your IAM credentials to retrieve a signing key from S3.

Thus, IAM read permissions of S3 paths act as proxies for Datomic permissions. As a result, you can use all of the ordinary IAM tools (roles, groups, users, policies, etc.) to authorize use of Datomic.

After decades of experience with racing to log into new servers to change the admin password, we think that this "secure by default" is pretty cool. But that is not the end of the story, as clients also must have network-level access to Datomic.

Network-Level Access

Datomic Cloud is designed to be accessed by applications running inside a VPC, and (unlike a service!) is never exposed to the Internet. You must make an explicit choice to access Datomic. You could:

  • run an EC2 instance in the Datomic VPC and in the Datomic applications security group
  • peer another VPC with the Datomic VPC 
  • configure a VPN Connection

Each of these approaches has its place for application access, and I will say more about them in a future article. For easy access to Datomic from a developer's laptop we offer the bastion.

How the Bastion Works

The bastion is a dedicated machine with one job only: to enable developer access to Datomic. When you turn the bastion on, you get a barebones AWS Linux instance that does exactly one thing: forwards SSH traffic to your Datomic system.

To connect through the bastion:
  1. run the Datomic socks proxy script on your local machine
  2. add a proxy port argument when creating a system client
  3. the Datomic client sees the proxy port argument and connects to the socks proxy 
  4. the socks proxy forwards encrypted SSH traffic to the bastion 
  5. the bastion forwards Datomic client protocol traffic to Datomic


Access to the bastion is secured using the same IAM + S3 proxying technique used earlier for auth. The bastion has an auto-generated, ephemeral private key that is stored in S3 and secured by IAM.

The bastion is dynamic, and you can turn it on or off an any time. And the client support means that the bastion is entirely transparent to your code, which differs only in the argument used to create the client.

A Concrete Example

As a concrete example, here is how the Datomic team configures access for some of our Datomic systems:

Our 'ci' system is dedicated to continuous integration, supporting several dozen Jenkins projects. The ci system contains no sensitive data, only canned and generated examples. The ci system runs the Solo Topology with the bastion enabled to allow access by automated tests.

Our 'devdata' system contains non-sensitive data used by the development team (think departmental and sample apps). The devdata system runs the Solo Topology with the bastion enabled to allow access by developers.

Our 'applications' system supports applications and contains real-world data. The applications system is reachable only by deployed application code, and needs to be highly available. So the applications system uses the Production Topology with the bastion disabled, and also uses fine-grained IAM permissions to limit applications to the databases they need.

Conclusion

Datomic is secure by default, integrating directly with AWS IAM and VPC capabilities. The bastion makes it easy for developers to get connected, so you can be up and transacting on a new system in minutes.

To learn more, check out

Or just dive in and get started.

Permalink

PurelyFunctional.tv Newsletter 265: The one after the first Clojure SYNC

Issue 265 – February 19, 2018 · Archives · Subscribe

Hi Clojurers,

The world’s first Clojure SYNC happened Thursday and Friday last week. And it was a total success! And because of that, I don’t have much to link to this week. I usually link to things I read/listen to during the week, but I have not been reading much this week. There was Mardi Gras then the conference.

About the conference, I’m still processing all that happened. Briefly, it validated a few things that I believed in but was still worried about. First, Q&A can actually be worth doing. Other conferences I’ve been to seem to have given up on it. They don’t reserve time for it and they don’t include it in the recordings. But with a little curation and some curiosity, you can ask good questions.

Second was that the setting can be a big part of the conference. The French Quarter is so dense with stuff to do that your conference venue doesn’t have to serve every purpose. You can ditch the big box hotels if you’re in the middle of everything.

Third was that the sponsor, the volunteers, the speakers, and the attendees were super supportive. I’m very lucky to have such great people in all of those roles. Thanks to everyone involved!

Check out the twitter stream of all the great tweets. There’s still a contest running (until tomorrow) for the most retweeted picture and the most retweeted tweet. You can vote, too, by retweeting!

I’ve learned a lot and have big hopes for next year. If you’re interested in hearing more about it, get on the Clojure SYNC mailing list. I’ll be writing about my lessons and sharing it there. My biggest regret is not being able to be more social with all 100+ attendees.

Please enjoy the issue.

Rock on!
Eric Normand <eric@purelyfunctional.tv>

PS Want to get this in your email? Subscribe!


The REPL

I was about to link to three or four things this week. Then I realized they were all from The REPL and I decided just to link to The REPL. It’s a great Clojure newsletter. Subscribe if you’re not already.

The post PurelyFunctional.tv Newsletter 265: The one after the first Clojure SYNC appeared first on PurelyFunctional.tv.

Permalink

Data-Driven Ring with Reitit

The first part introduced the new Reitit routing library. This is the second part and walks through the reitit-ring module and some of the problems it's trying to resolve.

Ring

Ring is the de facto abstraction for web apps in Clojure. It defines the following concepts:

  • request and response as maps
  • handlers and middleware as functions
  • adapters to bridge the concepts to HTTP

A simple Ring application:

(require '[ring.adapter.jetty :as jetty]) ; [ring "1.6.3"]

(defn handler [request]
  {:status 200
   :body "Hello World"})

(jetty/run-jetty app {:port 8080})

Ring is basically just functions and data. Because of this, pure Ring apps are easy to reason about. There are lot of adapters, routing libraries and middleware available and some great template-projects like Luminus and Duct. The async ring provides common abstractions for both Clojure and ClojureScript (via Macchiato).

In short, Ring is awesome.

Middleware

Middleware is a higher-level function that add additional functionality to handler function. Here's an example (from Ring Wiki):

(defn wrap-content-type [handler content-type]
  (fn [request]
    (let [response (handler request)]
      (assoc-in response [:headers "Content-Type"] content-type))))

The middleware concept is simple, but is it too simple? Composing the middleware chain is not always trivial as the ordering can matter. Only way to make sure that it's composed correctly is to invoke the chain, which is just an opaque function. We should have better tools for managing the middleware chain.

Common pattern with Ring is to mount a tower of middleware before any routing is applied. It's easy to do, but performance-wise this is not good, as not all routes need "everything". We should route first and apply only the middleware that is relevant for the given route.

Welcome Reitit-ring

reitit-ring is a standalone routing library for Ring that builds on top of reitit-core. It inherits all the features from the core and adds the following:

  • :request-method based routing
  • support for ring middleware
  • transformation of the middleware chain
  • dynamic extensions
  • data-driven middleware
  • middleware compilation
  • partial route specs
  • pluggable http-coercion

To get started, let's add a dependency:

[metosin/reitit-ring "0.1.0"]

Here's a simple app, with some middleware, handlers and a default-handler, handling the routing misses (HTTP Statuses 404, 405 and 406).

(require '[reitit.ring :as ring])

(defn handler [_]
  {:status 200, :body "ok"})

(defn wrap [handler id]
  (fn [request]
    (update (handler request) :via (fnil conj '()) id)))

(def app
  (ring/ring-handler
    (ring/router
      ["/api" {:middleware [#(wrap % :api)]}
       ["/ping" handler]
       ["/admin" {:middleware [[wrap :admin]]}
        ["/db" {:middleware [[wrap :db]]
                :delete {:middleware [[wrap :delete]]
                         :handler handler}}]]]
      {:data {:middleware [[wrap :top]]}}) ;; all routes
    (ring/create-default-handler)))

Handlers and middleware can be defined as route data either to top top-level (all methods) or under request-methods.

(app {:request-method :delete
      :uri "/api/admin/db"})
; {:status 200
;  :body "ok"
;  :via (:top :api :admin :db :delete)}

Having a fast data-driven router is cool, but let's not stop there.

Transforming the chain

Middleware chain is stored as data and can be queried from the router. We can transform the chain via router option :reitit.middleware/transform.

Adding debug-middleware between other middleware:

(require '[reitit.middleware :as middleware])

(def app
  (ring/ring-handler
    (ring/router
      ["/api" {:middleware [[wrap :api]]}
       ["/ping" handler]]
      {::middleware/transform #(interleave % (repeat [wrap :debug]))
       :data {:middleware [[wrap :top]]}})))

(app {:request-method :get,
      :uri "/api/ping"})
; {:status 200
;  :body "ok"
;  :via (:top :debug :api :debug)}

Dynamic extensions

One of the concerns with ring has been that middleware is too isolated as it doesn't know where the request is heading. We can fix that.

ring-handler matches the route before any middleware is applied. After a successful match, a routing Match is injected into the request. Middleware can read the Match at runtime and access all the route data, including other Middleware on the chain. This enables us to write systems where the (route) data and functions to interpret it (middleware) are cleanly separated.

Here's a middleware that reads :roles from a route data and if set, compares to :roles from a request.

(require '[clojure.set :as set])

(defn wrap-enforce-roles [handler]
  (fn [{:keys [roles] :as request}]
    (let [required (some-> request (ring/get-match) :data :roles)]
      (if (and (seq required) (not (set/subset? required roles)))
        {:status 403, :body "forbidden"}
        (handler request)))))

(def app
  (ring/ring-handler
    (ring/router
      ["/api"
       ["/ping" handler]
       ["/admin" {:roles #{:admin}}
        ["/ping" handler]]]
      {:data {:middleware [wrap-enforce-roles]}})))

Invoking the endpoint:

(app {:request-method :get, :uri "/api/ping"})
; {:status 200, :body "ok"}

(app {:request-method :get, :uri "/api/admin/ping"})
; {:status 403, :body "forbidden"}

(app {:request-method :get, :uri "/api/admin/ping", :roles #{:admin}})
; {:status 200, :body "ok"}

Dynamic extensions are an easy way to extend the system, but are ad-hoc in nature.

Data-driven middleware

In Pedestal, Interceptors are data, enabling better docs and advanced chain manipulation. There is no reason why Middleware couldn't be data too. Let's have data-driven middleware.

Instead of just functions, Reitit allows middleware also to be defined maps or instances of Middleware Records. They should have at least :wrap key defined, which is used in the actual request processing - with zero overhead.

(def wrap-middleware
  {:name ::wrap
   :description "Middleware that conjs the :id into :via in the response"
   :wrap wrap})

Internally, all forms of middleware are expanded into Middleware Records via IntoMiddleware protocol.

Extensions can use the Middleware data however they want. Route data validation uses :spec key and we could build an automatic middleware chain dependency resolution with this.

Compiling middleware

Each route knows the the exact Middleware chain defined for it. We can use this to optimize the chain for the route at router creation time. For this, there is a :compile key in Middleware. It expects a function of route-data router-opts => ?IntoMiddleware. The compilation is recursive, the results are merged to the parent and by returning nil the Middleware is removed from the chain.

Authorization middleware re-written as Middleware:

(require '[clojure.spec.alpha :as s])

(s/def ::role #{:admin :manager})
(s/def ::roles (s/coll-of ::role :into #{}))

(def enforce-roles-middleware
  {:name ::enforce-roles
   :spec (s/keys :req-un [::roles])
   :compile (fn [{required :roles} _]
              (if (seq required)
                {:description (str "Requires roles " required)
                 :wrap (fn [{:keys [roles] :as request}]
                         (if (not (set/subset? required roles))
                           {:status 403, :body "forbidden"}
                           (handler request)))}))})
  • mounts only if route has :roles defined
  • human readable description with required roles
  • defines a partial spec for the route data validation
  • faster as it does less work at request-processing time

Validation

Route validation works just like with the core router, but we should use a custom validator reitit.ring.spec/validate-spec! to support both the :request-method endpoints and the :spec from Middleware mounted for that route.

Here's an example using Expound:

(require '[reitit.ring.spec :as rrs])
(require '[reitit.spec :as rs])
(require '[expound.alpha :as e]) ; [expound "0.5.0"]

(def app
  (ring/ring-handler
    (ring/router
      ["/api"
       ;; no middleware -> not validated or enforced
       ["/ping" {:roles #{:pinger}
                 :handler handler}]
       ;; middleware -> enforced and validated           
       ["/roles" {:middleware [enforce-roles-middleware]}
        ["/admin" {:get {:roles #{:adminz}
                         :handler handler}}]]]
      {:validate rrs/validate-spec!
       ::rs/explain e/expound-str})))
; CompilerException clojure.lang.ExceptionInfo: Invalid route data:
;
; -- On route -----------------------
;
; "/api/admin" :get
;
; -- Spec failed --------------------
;
; {:middleware ..., :roles #{:adminz}, :handler ...}
;                            ^^^^^^^
;
; should be one of: :admin, :manager

Coercion

Coercion is the process of transforming and validating input and output parameters between external formats and clojure data. Reitit-ring ships with full-blown pluggable http-coercion (thanks to work in compojure-api), supporting clojure.spec, data-specs and Plumatic Schema as separate modules.

Unlike with the core router, coercion is applied automatically by coercion Middleware from reitit.ring.coercion. These include:

  • coerce-request-middleware - request parameter coercion
  • coerce-response-middleware - response body coercion
  • coerce-exceptions-middleware - coercion exceptions as http responses

Syntax for :parameters and :responses is adopted from ring-swagger. All have :spec defined, so you can validate the syntax too.

Example application with data-specs coercion:

(require '[reitit.ring.coercion :as rrc])
(require '[reitit.coercion.spec]) ; [metosin/reitit-spec "0.1.0"]
(require '[reitit.ring :as ring])

(def app
  (ring/ring-handler
    (ring/router
      ["/api"
       ["/ping" (fn [_]
                  {:status 200
                   :body "pong"})]
       ["/plus/:z" {:post {:coercion reitit.coercion.spec/coercion
                           :parameters {:query {:x int?}
                                        :body {:y int?}
                                        :path {:z int?}}
                           :responses {200 {:body {:total pos-int?}}}
                           :handler (fn [{:keys [parameters]}]
                                      ;; parameters are coerced
                                      (let [x (-> parameters :query :x)
                                            y (-> parameters :body :x)
                                            z (-> parameters :path :z)
                                            total (+ x y z)]
                                        {:status 200
                                         :body {:total total}}))}}]]
      {:data {:middleware [rrc/coerce-exceptions-middleware
                           rrc/coerce-request-middleware
                           rrc/coerce-response-middleware]}})))

Valid request:

(app {:request-method :post
      :uri "/api/plus/3"
      :query-params {"x" "1"}
      :body-params {:y 2}})
; {:status 200, :body {:total 6}}

Invalid request:

(app {:request-method :post
      :uri "/api/plus/3"
      :query-params {"x" "abba"}
      :body-params {:y 2}})
; {:status 400,
;  :body {:spec "(spec-tools.core/spec {:spec (clojure.spec.alpha/keys :req-un [:$spec8596/x]), :type :map, :keys #{:x}, :keys/req #{:x}})",
;         :problems [{:path [:x], :pred "clojure.core/int?", :val "abba", :via [:$spec8596/x], :in [:x]}],
;         :type :reitit.coercion/request-coercion,
;         :coercion :spec,
;         :value {:x "abba"},
;         :in [:request :query-params]}}

Performance

As Reitit-ring provides a rich set of features, so it must be slow right? No, the abstractions actually make it faster and thanks to snappy routing algorithms in reitit-core, it's really, really fast.

Below are the average route resolution times of a mid-size example REST route tree. As all perf tests, the tests may contain errors. Please let us know.

Coercion is the slowest part of the default stack, the this is because the actual coercion is done in 3rd party libraries. Reitit precompiles all the coercers and the overhead is as small as possible.

Road ahead

Next steps is to support Swagger and OpenAPI as a separate modules. Requires still some work to get working with ClojureScript.

As everything is data, it would be great to visualize the routing system including Middleware chains. Also, runtime chain visualization and debugging tools would be nice. If you interested in implementing this kind of things, please let us know.

Despite being modular, Reitit-ring is just a routing library, so where can one find more data-driven Middleware like CORS, OAuth, Content Negotiation etc? Should we re-package and host all the common Middleware as optimized data-driven Middleware? Should there be a community repo or an Github organisation for this? Could some parts of Reitit be part of the next Ring Spec? Ideas and comments welcome.

As reitit-ring is fully compatible with Ring, it would also be great to see it as optional routing engine for the established template projects like Luminus and Duct.

Final Words

Reitit-ring is an new data-driven web-router for Clojure(Script). It's fast, easy to extend and enables has some new concepts like first-class route data and data-driven middleware. Tools like route data validation and route conflict resolution help keep the routing trees sound.

Big thanks to existing Clojure routing and web libraries for ideas and showing the way. These including Bidi, Compojure, Compojure-api, Kekkonen, Pedestal, Ring-swagger and Yada. Also to HttpRouter for some perf stuff.

For discussions, there is #reitit channel in Clojurians slack.

Pointers to get started:

Happy ring-routing.

Permalink

The REPL

The REPL

AI fire alarms, Parinfer, Clojure book deals
View this email in your browser

The REPL

-main

Libraries & Books.

  • Humble Bundle has a deal on a bunch of O'Reilly FP books, including Living Clojure, Clojure Cookbook, and Clojure Programming.

People are worried about Types. ?

Foundations.

  • Project Detroit seeks to bring the V8 JavaScript engine to OpenJDK
  • The Java SE roadmap has been updated with support for Java 8 until January 2019.

Tools.

  • Cursive now supports Parinfer v3
  • Pyro is a new library that looks like it will vastly improve the experience of Clojure stacktraces, by showing the actual code that failed in the stacktrace.
  • The suite of core Clojure tooling continues to expand with test-runner.

Recent Developments.

Learning.

Misc.

  • "In 1901, two years before helping build the first heavier-than-air flyer, Wilbur Wright told his brother that powered flight was fifty years away. In 1939, three years before he personally oversaw the first critical chain reaction in a pile of uranium bricks, Enrico Fermi voiced 90% confidence that it was impossible to use uranium to sustain a fission chain reaction." - There’s No Fire Alarm for Artificial General Intelligence
Copyright © 2018 Daniel Compton, All rights reserved.


Want to change how you receive these emails?
You can update your preferences or unsubscribe from this list

Email Marketing Powered by MailChimp

Permalink

Using Clojure macros for nicer error handling

In July 2017, I found myself editing some Clojure code that looked approximately like this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
(defn validate-required-fields [params]
  (when-not (contains? params :source)
    "Missing source field"))

(defn validate-invariants [params]
  (when (>= (:lower params) (:higher params))
    "lower field must be smaller than higher"))

;; route handler taken out of other routes
(GET "/event-redirect/:event_type" request []
  (let [params (:params request)]
    (if-let [field-error (validate-required-fields params)]
      {:status 400 :body field-error}
      (if-let [invariant-error (validate-invariants params)]
        {:status 400 :body invariant-error}
        (publish-and-redirect params)))))

This route handler validates its inputs, and if they fail validation, then it returns an error response. I found this pretty ugly. This small chunk of code has numerous if branches and quite a bit of nesting. All of this makes it hard to read and hurts understanding.

While adding a new feature to it, I remembered some code I wrote with Case back in late 2015. Back then we were working on Lumanu and wrote a Clojure macro that we called halt-on-error->>. This macro worked similarly to ->>, except it allowed any step in the processing pipeline to halt execution and trigger an error handler. We were working on a web crawler at the time, and this macro significantly improved the readability of our data processing pipeline. There was a lot of error handling code throughout the web crawler, and this macro helped keep it readable.

I realized that using a similar macro would make this code easier to follow. I recreated halt-on-error->> to allow any form to cause it to return early. The above code could then be written like below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
(defn validate-required-fields [params]
  (if (contains? params :source)
    params
    (exec/halt {:status 400 :body "Missing source field"})))

(defn validate-invariants [params]
  (if (< (:lower params) (:higher params))
    params
    (exec/halt {:status 400 :body "lower field must be smaller than higher"})))

(GET "/event-redirect/:event_type" request []
  (exec/halt-on-error->> request
                         :params
                         validate-required-fields
                         validate-invariants
                         publish-and-redirect))

Once you understand halt-on-error->>, this chunk of code is much easier to read.

Let’s implement halt-on-error->>.

Implementing halt-on-error->>

Here are some tests for that specify how halt-on-error->> should work.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
(ns halt.execution-test
  (:require  [halt.execution :as exec]
             [clojure.test :refer :all]))

(def produce-error (constantly (exec/halt {:x "foobar"})))

(defn success-fn
  "Weird function that appends suffix to s"
  [suffix s]
  (str s suffix))

(deftest single-step
  (is (= "first" (exec/halt-on-error->> (success-fn "first" "")))))

(deftest two-steps-with-no-error
  (is (= "firstsecond" (exec/halt-on-error->> (success-fn "first" "")
                                              (success-fn "second")))))

(deftest error-as-first-step
  (is (= {:x "foobar"} (exec/halt-on-error->> (produce-error))))
  (is (= {:x "foobar"} (exec/halt-on-error->> (produce-error)
                                              (success-fn "first")))))

(deftest error-after-first-step
  (is (= {:x "foobar"} (exec/halt-on-error->> (success-fn "first" "")
                                              (produce-error)
                                              (success-fn "second")))))

(deftest works-with-anonymous-functions
  (is (= 1 (exec/halt-on-error->> (success-fn "first" "")
                                  ((fn [x] (exec/halt 1)))))))

Below is an implementation of halt-on-error->>.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
(ns halt.execution)

(defrecord Stopper [x])

(defn halt [data]
  (Stopper. data))

(defmacro halt-on-error->> [form & forms]
  (let [g (gensym)
        pstep (fn [step] `(if (instance? Stopper ~g) ~g (->> ~g ~step)))]
    `(let [~g ~form
           ~@(interleave (repeat g) (map pstep forms))]
       (if (instance? Stopper ~g)
         (.x ~g)
         ~g))))

So what is this macro doing? First, it uses gensym to get a symbol with a unique name and stores this in g. It then defines a helper function called pstep for use in the code generation part of the macro.

This macro generates a let block that repeatedly executes a form and assigns the return value back to g. g is then checked to confirm execution should continue before it is threaded into the next form. If g is ever an instance of a Stopper, execution halts and the value wrapped in the Stopper is returned.

Looking at an expanded version of a macro can be easier to understand than a written explanation. Below is a macroexpanded version of one of the tests.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
;; What is being expanded
(macroexpand-1 '(exec/halt-on-error->> (success-fn "first" "")
                                       (produce-error)
                                       (success-fn "second")))

;; The expansion
(let [G__15365 (success-fn "first" "")
      G__15365 (if (instance? halt.execution.Stopper G__15365)
                 G__15365
                 (->> G__15365 (produce-error)))
      G__15365 (if (instance? halt.execution.Stopper G__15365)
                 G__15365
                 (->> G__15365 (success-fn "second")))]
  (if (instance? halt.execution.Stopper G__15365)
    (.x G__15365)
    G__15365))

Looking at that expansion, you can see how we are using a let block to repeatedly assign to the same symbol and we check that return value before executing the next stop.

This isn’t a new pattern. There are libraries that implement similar ideas. At IN/Clojure 2018, Varun Sharma gave a talk about how this cleaned up their code. You can even get bogged down and throw around words like monad when talking about it.

I’d encourage you to look at your code and see if you have areas where error handling code is detracting from the readability. This might be an area where this, or something similar to it, would help.

Permalink

Reitit, Data-Driven Routing with Clojure(Script)

As many things we do, it started as a small experiment but ended up as a standalone library. We are happy to introduce Reitit, a new fast data-driven router for Clojure(Script)!

Another routing library?

There are already many great routing libraries for Clojure(Script), and we at Metosin have used most of them: for frontend - Secretary, Silk and Bidi and for backend Compojure, Bidi and some Pedestal. Still, none of the existing solutions felt perfect. There should be a routing library that:

  • works with both Clojure & ClojureScript
  • has simple data-driven syntax (for humans)
  • supports first-class route data
  • supports bi-directional routing
  • has pluggable parameter coercion
  • supports middleware &/ interceptors
  • handles route conflicts
  • is modular and extendable
  • is fast

Reitit does all of those and more.

Simplest thing that works

To use just the routing core, we need a dependency to:

[metosin/reitit-core "0.1.0"]

It provides the basic routing abstractions and implementation for Clojure(Script). It works best with clojure.spec but the only mandatory dependency is meta-merge.

(require '[reitit.core :as r])

(def router
  (r/router
    [["/api/ping" ::ping]
     ["/api/orders/:id" ::order]]))

(r/match-by-path router "/api/ping")
; #Match{:template "/api/ping"
;        :data {:name ::ping}
;        :result nil
;        :path-params {}
;        :path "/api/ping"}

(r/match-by-name router ::order {:id 2})
; #Match{:template "/api/orders/:id",
;        :data {:name ::order},
;        :result nil,
;        :path-params {:id 2},
;        :path "/api/orders/2"}

The route syntax and basic usage is described in detail in the docs.

Different routers

The core abstraction in reitit is the Router Protocol:

(defprotocol Router
  (router-name [this])
  (routes [this])
  (options [this])
  (route-names [this])
  (match-by-path [this path])
  (match-by-name [this name] [this name path-params]))

Reitit ships with multiple implementations for a Router and by default, reitit.core/router chooses the best one after analysing the route tree. The implementations include:

  • :linear-router - naive, but works with all route trees
  • :segment-router - prefix-tree-based, fast for wildcards routes
  • :lookup-router - hash-lookup, fast, only for non-wildcard routes
  • :single-static-path-router - super fast, for one static route
  • :mixed-router - for route trees with both static & wildcard routes
(r/router-name
  (r/router
    [["/ping" ::ping]
     ["/api/:users" ::users]]))
; :mixed-router

(r/router-name
  (r/router
    [["/ping" ::ping]
     ["/api/:users" ::users]]
    {:router r/linear-router}))
; :linear-router

The original routing code was ported from Pedestal and thanks to that, it's really fast. Below are the average route resolution times of a mid-size example REST route tree. As all perf tests, the tests may contain errors. Please let us know.

Route data

The key feature of Reitit is first-class route data. Any map-like data can be attached to routes, either directly to leaves or to branches. When router is created, paths are flattened and route data is expanded and meta-merged into leaves. Data can optionally be validated and compiled.

The following two routers are effectively the same:

(def nested-router
  (r/router
    ["/api" {:interceptors [api-interceptor]}
     ["/ping" {:handler ping-handler}]
     ["/admin" {:roles #{:admin}}
      ["/users" {:handler user-handler}]
      ["/db" {:interceptors [db-interceptor]
              :roles ^:replace #{:db-admin}
              :handler db-handler}]]]))
(def flat-router
  (r/router
    [["/api/ping" {:interceptors [api-interceptor]
                   :handler ping-handler}]
     ["/api/admin/users" {:interceptors [api-interceptor]
                          :roles #{:admin}
                          :handler user-handler}]
     ["/api/admin/db" {:interceptors [api-interceptor db-interceptor]
                       :roles #{:db-admin}
                       :handler db-handler}]]))

Full route data can be queried via r/routes and is available per route in a successful match:

(r/match-by-path nested-router "/api/admin/users")
; #Match{:template "/api/admin/users",
;        :data {:interceptors [api-interceptor],
;               :roles #{:admin},
;               :handler user-handler},
;        :result user-handler
;        :path-params {},
;        :path "/api/admin/users"}

Interpretation of the data is left to the client application.

Validation

As route data can be everything, it's easy to forget or misspell keys. We can use clojure.spec to validate both the route syntax and the route data. Route data specs can be defined via router options. Routing components like Middleware and Interceptors can also contribute to specs, with a scope of only the routes these components are mounted to.

An example router requiring :roles for all routes, using Expound:

(require '[clojure.spec.alpha :as s])
(require '[expound.alpha :as e]) ; [expound "0.5.0"]
(require '[reitit.spec :as rs])

(s/def ::role #{:admin :manager})
(s/def ::roles (s/coll-of ::role :into #{}))

(defn router [routes]
  (r/router
    routes
    {:spec (s/merge (s/keys :req-un [::roles]) ::rs/default-data)
     ::rs/explain e/expound-str
     :validate rs/validate-spec!}))

(router
  ["/api" {:handler identity}])
; CompilerException clojure.lang.ExceptionInfo: Invalid route data:
;
; -- On route -----------------------
;
; "/api"
;
; -- Spec failed --------------------
;
; {:handler identity}
;
; should contain key: `:roles`
;
; |    key |                                   spec |
; |-------------+-----------------------------------|
; | :roles | (coll-of #{:admin :manager} :into #{}) |

(router
  ["/api" {:handler identity
           :roles #{:adminz}}])
; CompilerException clojure.lang.ExceptionInfo: Invalid route data:
;
; -- On route -----------------------
;
; "/api"
;
; -- Spec failed --------------------
;
; {:handler ..., :roles #{:adminz}}
;                         ^^^^^^^
;
; should be one of: `:admin`,`:manager`

By design, Spec can't report errors on extra keys, but we are working on integrating Figwheel-grade error reporting to spec-tools and with that, to Reitit.

Coercion

Coercion is the process of transforming and validating input and output parameters between external formats and clojure data. Reitit ships with full-blown pluggable http-coercion (thanks to work in compojure-api), supporting clojure.spec, data-specs and Plumatic Schema as separate coercion modules.

Reitit defines routing and coercion as two separate processes: if coercion is enabled for a route, a successful routing Match will contain enough data for the client application to apply the coercion. There is a full coercion guide, but here's a simple example on using data-specs to coerce the path-parameters.

(require '[reitit.coercion :as coercion])
(require '[reitit.coercion.spec]) ; [metosin/reitit-spec "0.1.0"]

(def router
  (r/router
    ["/:company/users/:user-id"
     {:name ::user-view
      :coercion reitit.coercion.spec/coercion
      :parameters {:path {:company string?
                          :user-id int?}}}]
    {:compile coercion/compile-request-coercers}))

(defn match-by-path-and-coerce! [path]
  (if-let [match (r/match-by-path router path)]
    (assoc match :parameters (coercion/coerce! match))))

(match-by-path-and-coerce! "/metosin/users/123")
; #Match{:template "/:company/users/:user-id",
;        :data {:name :user/user-view,
;               :coercion <<:spec>>
;               :parameters {:path {:company string?,
;                                   :user-id int?}}},
;        :result {:path #object[reitit.coercion$request_coercer$]},
;        :path-params {:company "metosin", :user-id "123"},
;        :parameters {:path {:company "metosin", :user-id 123}}
;        :path "/metosin/users/123"}

(match-by-path-and-coerce! "/metosin/users/ikitommi")
; => ExceptionInfo Request coercion failed...

Route conflicts

Last but not least, Reitit does full route conflict resolution when router is created. This is important because in real life route trees are usually merged from multiple sources and we should fail fast if there are routes masking each other.

(r/router
  [["/ping"]
   ["/:user-id/orders"]
   ["/bulk/:bulk-id"]
   ["/public/*path"]
   ["/:version/status"]])
; CompilerException clojure.lang.ExceptionInfo: Router contains conflicting routes:
;
;    /:user-id/orders
; -> /public/*path
; -> /bulk/:bulk-id
;
;    /bulk/:bulk-id
; -> /:version/status
;
;    /public/*path
; -> /:version/status
;

As mostly everything, conflict resolutions can be configured via router options.

That's all?

No. This was just a quick walkthrough of the core features. In the second part I'll walk through the optional reitit-ring module, including data-driven middleware, middleware compilation, partial route specs and full http-coercion.

We are big fans of the Pedestal and are incubating a separate full interceptor module as an alternative to the reitit-ring. Goal is to distill common abstractions for both the frontend and the backend. Still work to do here.

Swagger and OpenAPI will be supported as a separate modules. Requires still some work to get working with ClojureScript.

For the browser, we have been using Reitit successfully with Keechma-style stateful routing controllers together with Re-frame. These will too be published as small helper module(s).

Final words

Even though Reitit is a new library, the ideas have evolved over many years and the code is originally based on more proven libraries so the core api should be quite stable. We are really exited about it.

Big thanks to existing Clojure(Script) routing and web libraries for ideas and showing the way. These including Bide, Bidi, Compojure, Compojure-api, Keechma, Kekkonen, Pedestal, Ring-Swagger and Yada. Also to HttpRouter for some perf stuff.

This is just a first release and there is still a lot of things to do, so comments, ideas and contributions are welcome! Roadmap is mostly written in issues, spanning multiple repos.

For discussions, there is #reitit channel in Clojurians slack.

Pointers to get started:

Happy routing.

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.