Typing Toccata

Typing Toccata

The mission of Toccata is "To be the most productive programming language possible." (Everyone needs goals. :) ) Another way of saying this is that Toccata should help typical programmers write typical software as quickly and correctly as possible. Obviously, (to me, at least) the Functional Programming paradigm is the first step. But then what? For many, static type checking would be the next requirement. Yet languages like Scala and Haskell seem to be pretty strong in that area already. And while they're popular, there are large numbers of programmers who prefer dynamically typed languages for the flexibility and ease they offer in getting software written. And they might have trouble switching between programming at the "type level" and programming at the "value level". I certainly do. Toccata is intended for those people. This post starts to describe Toccata's "killer feature" enabling this.

Gradually

Let's start with a function from the core library.

116: (defn add-numbers [x y]
117:   (assert (instance? Integer x))
118:   (assert (instance? Integer y))
119:   (inline C Integer "..."))

(I don't usually include line numbers in my code snippets, but they'll be needed later.)

Let's start at the bottom. The inline C expression has an optional type annotation field which in this case is said to be Integer. This means that the C code (which I've ellided) will always produce an Integer value. Therefore, add-numbers will always return an Integer.

But what if we tried to pass a string to add-numbers? That's what the two (seemingly) runtime assertions deal with. They assert that both x and y have to be Integers, which is checked at runtime. Or is it? This establishes a contract for add-numbers that it is only valid to call it with two integers and that it will always return an integer. It's like there's a wall or a fence between the C code which implements this functionality which is safe, and the rest of the program, which is not guaranteed to be safe.

And here's the kicker; the compiler can tuck that contract for add-numbers away for use later.

Let's look at another core library function.

709: (defn +
710:   ([] 0)
711:   ([x y]
712:    (add-numbers x y))
713:   ([& xs]
714:    (reduce xs 0 add-numbers)))

Just focus on the 2-arity for the moment. It calls add-numbers with the two values that it is called with. I refer to this call to add-numbers as a 'call site' and since there's only one target for this call site, the compiler can check the contract it remembered when it compiled add-numbers and move the fence guaranteeing the safety of adding two number up the call tree one level to here. It can also know that the 2-arity of + will always return an integer. So the runtime assertions that were ostensibly compiled into add-numbers can instead be moved to this (and every other) call site which targets add-numbers. And this in turn supplies a contract for the 2-arity of +. That it only accepts integers and always returns an integer.

As I understand it, this expanding the boundaries of code that is known to be safe is referred to as 'Gradual Typing'. Feel free to correct me in the comments.

But wait ...

So let's take it one more step. Check out this function

915: (defn inc [x]
916:   ;; Add 1 to an integer
917:   (+ 1 x))

inc calls the 2-arity of + with a literal 1 and whatever value x is. Since the compiler can know exactly what function is targeted at this call site, it can move the 'safe code' fence out to here. And since 1 is a literal integer, it can determine at compile time that it doesn't need to be checked. And it also knows precisely what the contract for inc is. It accepts one integer and returns an integer.

In practice

Now what happens when we run this program?

1: (main [_]
2:      (println (inc "bogus string")))

That's a trick question. This code doesn't even compile. Instead, we get this error.

*** Failed type assertion at tricky.toc: 2.
Expected 'Integer', got 'String'.
Assertion from
 $TOCCATA_DIR/core.toc: 917
 $TOCCATA_DIR/core.toc: 712
 $TOCCATA_DIR/core.toc: 118

By moving the safety fence up the call tree, the compiler can tell us exactly where the bug is at compile time rather than make us sift through a stack trace at runtime. In essence, the compiler gives a "reverse stack trace" that tells us where the bug is and the path to where the failure would have occurred. Also, one of the runtime assertions can be eliminated. And all the runtime checks between the call in the main function and add-numbers can be eliminated as well. All this happens with no additional effort from the programmer who wrote 'tricky.toc'. It happens automatically.

But it gets so much better than that...

Next time.

Permalink

Clojure in London: Gower Street

As we’ve grown as a business we’re now using Clojure for almost everything…​

Distributing Movies

Gower Street Analytics.

Gower Street Analytics is an analytics company working in the film industry. Gower Street provides a platform for major distributors of films to manage release dates, helping them to decide exactly when a film should be released.

They use complex algorithms built in Clojure to simulate the performance of box offices around the world, trying out various release dates for movies. These simulations take into account population demographics, release dates and the expected performance of other movies.

I chatted with Gower Street co-founder Dimitrios Mitsinikos and his team to discuss their decision to use Clojure to power their business.

Jon: What is the goal of Gower Street?

Dimitrios: We want to build the analytics platform that we think is missing in the industry. There are many companies doing film analytics, but most are only doing 'green light' comparisons; looking at how well a film will do based on which actors are in it, for example.

We instead look at what films are being released around the same time as others, alongside the likelihood of a film hitting its estimate based on a variety of data points. We also monitor the market closely, improving the accuracy of our predictions as we get closer to a film’s release date.

All the large studios are now using our software, and we are in the process adding more countries to the platform.

Jon: How did Gower Street get started?

Dimitrios: We have a lot of experience in the movie industry - for the last 15 years I’ve worked in analytics at Universal Studios. We started two years ago, spending a lot of time on research and development, processing data feeds to track how movies are performing across the world, and figuring out exactly what platform the industry might need.

Jon: So if a studio wants to put out a film, then they can go to you to find out when would be the optimum time?

Dimitrios: Yes, that’s right. A studio can go online using our system and see what the film estimates are for the next 12-18 months. They can play with scenarios, for example: "Let’s swap Star Wars with Suicide Squad and see how much more or less each film might make."

We create algorithms that accurately model the box office, so customers can see the impact of choosing different release dates and monitor what is happening in a very competitive environment. For example our models show that one film could have made an extra $2.5m by releasing on the date that our system suggested.

It’s fascinating that even in a mature market like the UK, you can still significantly increase revenue by using analytics in this way.

gower st

Technology choices

Jon: Why did you choose Clojure to power the business?

Dimitrios: The core of our business is the simulation of the global box office. It’s all about processing complex and constantly changing data as fast as possible. When we first started our simulations could take five hours. It’s now about a minute. We have found functional programming a great fit for our team, especially those with a scientific background.

Team: Clojure might seem an unusual choice for performance-intensive simulation work; but it’s working out well. It’s a great fit for the problem, performant enough for our needs and it’s extremely expressive and readable, allowing us to iterate quickly on complex simulation algorithms.

Jon: How do you get it so fast?

Dimitrios: Through lots of optimisation and the ability to easily deploy using Amazon AWS.

Team: We’ve had to think outside the box, just throwing more machines at the problem might help, but then we’d have more problems with managing a distributed processing infrastructure e.g. a Hadoop cluster. To say more would be to enter a larger discussion of simulation and stochastic models that we probably don’t have time for here!

Jon: How did you go about adopting Clojure?

Dimitrios: Initially we started with Racket. Racket was nice and functional, but it was lacking support in certain areas. The developers came up with the idea to move to Clojure.

Team: Because the webapp for a previous iteration of our product was built in Racket, we were no strangers to parentheses. We tried Clojure for certain parts of the system at first, and it worked out very well. It has a larger ecosystem than Racket, so we’ve ended up adopting it across our stack. The usual reason of 'it works well with our existing Java systems' wasn’t a factor for us.

Dimitrios: As we’ve grown as a business we’re now using Clojure for almost everything - plain Clojure for back-end services and ClojureScript on the front-end.

Team: While we’ve gotten a lot from Clojure, we’re not obsessive, and recognise that it isn’t the best tool for every job. We’ve been experimenting with Serverless recently and this is one area where a lighter-weight solution may work better.

Jon: What is your team size?

Dimitrios: Our development team is currently five people and will be six shortly. We’re a fully remote team, scattered across the UK.

Team: You might think that remote would be an issue, but we’ve developed a way of collaboratively working together using appear.in video conferencing, Slack and tmux - we pair-program remotely much of the time and although we’re close to XP we’d prefer to refer to our team as agile with a small 'a'.

We get together for two days a month on-site in London as a team to work, eat and play - it helps us see each other as people rather than just co-workers. We have a flat-hierarchy and use a consensus-based decision-making model, so we’re not exactly your normal corporate sweatshop in this regard; we very much agree with Dan Pink’s ideas about Autonomy, Mastery and Purpose.

Hiring

Jon: Tell me about hiring?

Dimitrios: The team has grown organically, and based on word of mouth we’ve hired people. From my experience, it’s best for the development team to grow slowly, there’s a risk with teams of growing big too quickly.

Team: We’re actively creating an environment which promotes employee well-being and we take intentional steps to reduce inherent bias and to promote diversity. We focus on people and robust discussions rather than processes or tools.

We don’t require people to know Clojure to join us; just to have valuable experience to contribute and a willingness to learn and be open to new things. We’ve found that Clojure is a big draw for us - it attracts the kind of people who are more likely to see things as a holistic system rather than a collection of classes.

Permalink

Timeout Effect in Re-frame

Sometimes you need to delay an Event a certain amount of time. Let's say you want an alert to appear for twenty seconds before disappearing. We can store the alert message in the Database, but how do we make it disappear after the right time? We could set a timeout, using setTimeout(), but Event handlers should be pure to make them more testable. We need to turn setting a timeout into an Effect.

Let's make the shell of an Effect:

(rf/reg-fx
  :timeout
  (fn [{}]
    ))

We can pass in a time and an Event to dispatch. We then call setTimeout().

(rf/reg-fx
  :timeout
  (fn [{:keys [event time]}]
    (js/setTimeout
      (fn []
        (rf/dispatch event))
      time)))

That's great! Another feature of the JavaScript timeouts is the ability to cancel them with clearTimeout(). setTimeout() will return an id, which we later pass to clearTimeout().

We cancel timeouts if we store the id somewhere. Let's make a new atom to store these ids.

(defonce timeouts (reagent/atom {}))

We can use our own id when we create the Event so we can refer to it later:

(rf/reg-fx
  :timeout
  (fn [{:keys [id event time]}]

If we already have a timeout for this id, let's cancel it:

(rf/reg-fx
  :timeout
  (fn [{:keys [id event time]}]
    (when-some [existing (get-in @timeouts id)]
      (js/clearTimeout existing))

Then, we dissoc it from the timeouts atom.

(rf/reg-fx
  :timeout
  (fn [{:keys [id event time]}]
    (when-some [existing (get-in @timeouts id)]
      (js/clearTimeout existing)
      (swap! timeouts dissoc id))
    (when (some? event)
      (swap! timeouts assoc id
        (js/setTimeout
          (fn []
            (rf/dispatch event))
          time)))))

There! Now we can use it like the following. We can set the alert message and trigger its removal after 20 seconds.

(rf/reg-event-fx
  :set-alert
  (fn [cfx [_ msg]]
    {:db (assoc (:db cfx) :alert-message msg)
     :timeout {:id :alert-message
               :event [:remove-alert]
               :time 20000}}))

(rf/reg-event-db
  :remove-alert
  (fn [db]
    (dissoc db :alert-message)))

;; ...

(rf/dispatch [:alert-message "Please have a nice day!!"])

Try it out!

Permalink

Business benefits of Clojure SYNC

Does your manager know how many benefits their programmers will get from Clojure SYNC? And do they know how direct those benefits will be?

In this video, I go over the talks that are directly relevant to using Clojure at work–from hiring to improving the maintainability of our code. I also discuss the networking opportunities.

The post Business benefits of Clojure SYNC appeared first on Clojure SYNC.

Permalink

Optimistic Update in Re-frame

We’ve got a little bit of a problem on the web, especially with frontend apps. Our users are enjoying the fast, interactive experience of the app, which is running in the browser. Then some change needs to be propagated on the server. The server has to be the source of truth, since the browser tab is ephemeral. It could be reloaded at any time! Let’s say we click the “Like” button on a picture on social media. Obviously, we need to tell the server. The server knows all the likes and until it gets to the server, no other user can see the like. However, we want the user to know that the like happened or is at least in the process of happening.

In the ideal case, we send a message to the server, get a successful response, and update the UI; and all of that happens so fast the user never wonders whether it happens.

When the user clicks the like button, we dispatch the :like event, which sends a POST to the server. Then on success, we store some state for that post id in a hash map. If there's an error, we put an error message in the database.

(rf/reg-event-fx :like
  (fn [cofx [_ post-id]]
    {:http-xhrio {:url (str "http://server.com/" post-id "/like")
                  :method :post
                  :on-success [:confirm-like post-id]
                  :on-failure [:show-error "Could not like."]}}))

(rf/reg-event-db :confirm-like
  (fn [db [_ post-id]]
    (assoc-in db [:likes post-id] true)))

(rf/reg-event-db :show-error
  (fn [db [_ msg]]
    (assoc db :last-error msg)))

However, this ideal is rare. The message to the server and back is too slow. The user wants instant feedback. What can we do?

Pessimistic update

Well, one option is to save something to the database saying that we're trying to confirm the like. That way, we can show a loading spinner. In addition to sending the POST, we can store that we're trying. Let's create a new structure to house all of the state we'll need to keep track of. Let's use a map. It will have a status and a value.

{:status :stable
 :value  true}

 {:status    :loading
 :value      true
 :next-value false}

There are two statuses: :stable and :loading. Stable means we don't anticipate a response from the server. The value is what it is and is not changing. Loading means we are awaiting a response from the server. This is the case where we'd show a loading spinner.

We can make three operations on this new map. The first one, begin-load, we call when we start the Ajax request. It stores the value we're hoping it will change to under :next-value.

There's also an operation for succeeding, succeed-load, which sets the status to stable and updates the value.

And for failure responses, fail-load sets it back to stable, but leaves the value alone. I've also defined some Events that call these on a certain path in the Database.

(defn begin-load [state next-value]
  (cond
    (nil? state) ;; consider nil a stable value
    {:status :loading
     :value nil
     :next-value next-value}

    (= :stable (:status state))
    {:status :loading
     :value (:value state)
     :next-value next-value}

    (= :loading (:status state))
    (assoc state :next-value next-value)))

(defn succeed-load [state]
  (cond
    (nil? state)
    state

    (= :stable (:status state))
    state

    (= :loading (:status state))
    {:status :stable
     :value (:next-value state)}))

(defn fail-load [state]
  (cond
    (nil? state)
    state

    (= :stable (:status state))
    state

    (= :loading (:status state))
    {:status :stable
     :value (:value state)}))

(rf/reg-event-db :succeed-load
  (fn [db [_ path]]
    (update-in db path succeed-load)))

(rf/reg-event-db :fail-load
  (fn [db [_ path msg]]
    (-> db
      (update-in path fail-load)
      (assoc :last-error msg))))

Alright! Now how do we use this little bit of machinery? We use begin-load to start it off, telling it we're trying to make it true. We can use the generic :succeed-load and :fail-load events, giving them the path that they'll need when they are dispatched.

(rf/reg-event-fx :like
  (fn [cofx [_ post-id]]
    ;; let's start the ajax request
    {:http-xhrio {:url (str "http://server.com/" post-id "/like")
                  :method :post
                  ;; we can call our new Events
                  :on-success [:succeed-load [:likes post-id]]
                  :on-failure [:fail-load    [:likes post-id] "Failed to like post."]}
     ;; and let's begin the load with next value true
     :db (update-in (:db cofx) [:likes post-id] begin-load true)}))

In our Component, if the status is stable, we show the value. Otherwise, we show a spinner. That's one option. We show exactly the truth: we're loading. The actual state of things is in limbo.

Optimistic update

But there's another way, which is to tell a 90% truth. 90% of the time, the Ajax request will succeed with no problem. So 90% of the time, you can just pretend like it's going to work, showing what it would look like if it were a success immediately. You then you have to deal with those 10% of cases where it didn't work--you actually lied to the user. But the lucky thing is that your users will get feedback immediately. That feedback is correct the vast majority of the time. And sometimes, that's good enough.

In this app, liking something is not an essential feature, so we can update it optimistically. With just a few small changes, we can do an optimistic load. The first thing is to immediately set the value to the next value, and store the old value.

{:status :stable
 :value  true}

 {:status    :loading
 :value      false
 :old-value true}

We change our begin-load function so it doesn't save the current value and the next value anymore. Now, it will save the current value, which it optimistically sets to next-value, and the old value, which we'll need in case of failure. Success means we throw away the old value. And failure means we set the current value to the old value.

(defn begin-load [state next-value]
  (cond
    (nil? state)
    {:status :loading
     :value next-value
     :old-value nil}

    (= :stable (:status state))
    {:status :loading
     :value next-value
     :old-value (:value state)}

    (= :loading (:status state))
    (assoc state :value next-value)))

(defn succeed-load [state]
  (cond
    (nil? state)
    state

    (= :stable (:status state))
    state

    (= :loading (:status state))
    {:status :stable
     :value (:value state)}))

(defn fail-load [state]
  (cond
    (nil? state)
    state

    (= :stable (:status state))
    state

    (= :loading (:status state))
    {:status :stable
     :value (:old-value state)}))

I didn't repeat the Events because they're the same. We just had to change these three operations. Well, we also have to change the view. It should never show the loading spinner now. It should just show the current value and there should still be a message when there's a failure.

Permalink

Gettin’ schwifty with Clojure’s core.async

A few months ago, somebody in the office pointed us to an interesting  job offer from CartoDB which looked as follows: What follows is a technical test for this job offer at CARTO: https://boards.greenhouse.io/cartodb/jobs/705852#.WSvORxOGPUI Build the following and make it run as fast as you possibly can using Python 3 (vanilla). The faster it runs, the ...

Permalink

Step inside :cljsbuild

In an ongoing effort to document more of the Clojure ecosystem, I’m going to share with you a day’s headache around the Clojurescript build process. Don’t get me wrong! I adore Clojurescript. However it’s Leiningen configuration is opaque and lends to confusion. What’s worse, the mystery is papered over with calls to lein new and we hope for the best!

No more! I shall rescue you. Here’s a bucket. We’ll save the Titanic yet!

For interested parties, I wanted to build a Clojure/Clojurescript portable library. It was important that the test suite execute in both environments as well as being easy for the programmer to execute.

profile.clj

Here’s the interesting bits of the configuration all in one place.

:plugins [[lein-cljsbuild "1.1.7"]]
:source-paths ["src"]
:test-paths ["test"]
:aliases
{"cljs-test" ["cljsbuild" "test" "unit-tests"]
"test-all" ["do" "clean," "test," "cljsbuild" "once"]
"cljs-auto-test" ["cljsbuild" "auto" "tests"]}
:cljsbuild
{:test-commands {"unit-tests" ["node" "target/unit-tests.js"]}
:builds
{:tests
{:source-paths ["src" "test"]
:notify-command ["node" "target/unit-tests.js"]
:compiler {:output-to "target/unit-tests.js"
:optimizations :none
:target :nodejs
:main your.awesome.code.core-test}}
:production
{:source-paths ["src"]
:compiler {:output-to "target/production.js"
:optimizations :advanced}}}}

:plugins

The cljsbuild project is the central control. It empowers Leiningen with commands like lein cljsbuild test. It handles the compiler through the configuration given here.

:aliases

These serve a duel purpose in my projects. On the one hand they make convenient shortcuts for frequent commands. As importantly, it serves as breadcrumbs for others to understand expectations of how the build tools are configured.

In the first instance, we expose cljs-test as a command to power the test and test-command unit-tests. More on :test-command below, but note that it references cljsbuild and node.

test-all is a set of commands that clean the repository, run the Clojure tests, then compile and run the Clojurescript tests. This is accomplished with the leiningen do command and the :notify-command below.

Finally, cljs-auto-test continuously compiles and runs the tests on every source code change.

:cljsbuild

This is the meat-and-potatoes of most Clojurescript projects, and probably the least understood. I won’t pretend it’s easy (I did spend a day pulling my hair out) but it’s understandable in pieces.

:test-commands

One of the most important lines here, and rarely mentioned. While Clojurescript will happily compile all your code and tests, that code must be run against a real-live Javascript engine. The vector is the shell command to invoke the Javascript engine with your code. The path portion used will match the :output-to key below.

All of this is called by lein cljsbuild test <name>. In this case, the name is unit-tests. This compiles our Clojurescript into Javascript, then runs NodeJS with our newly minted Javascript code.

:builds

I have a pet peeve. This key supports both a vector as input and a map input. This leads to chaos and confusion when you look at other people’s code.

Here I use the map form, and I would encourage others as well to stick to this convention. Although I would love to hear thoughtful benefits of the vector form.

Pet peeve aside, the :builds values contain configurations for the Clojurescript compiler and build configurations. This is how we control where and how our Javascript is generated.

:source-paths

Here we point the compiler to all the folders we want to build from. In the :tests config, we want both the source code and test code. In :production the source code is enough.

:notify-command

It’s a callback when the compiler finishes. It calls a shell command of your choice. Here I call NodeJS to run our generated Javascript+Testcode on every change to the source code so I can see my unit-test results.

:output-to

Once your Javascript is built, where should the compiler put it? This is important, as other tools like NodeJS need to know this location to run our test suite.

When building Single Page Applications and other advanced Clojurescript applications, it is critical you know where to place your generated JS files. Often other tools pick up on output from the Clojurescript compiler.

:optimizations

How aggressive is the code optimization and minification? For :tests I have to avoid using :whitespace as NodeJS complains loudly. In :production I use :advanced to get the smallest deployable.

:output-dir

Often builds and compilations will create extra files. This allows you to specify where those go. This must be unique across the different :builds configurations, otherwise incremental builds will break and worse!

:target :nodejs

Normally, you target a browser. That means you have documents you write to. However, NodeJS lacks these browser features, so this key forces the compiler to generate the raw Javascript for NodeJS consumption.

:main

Usually this is the main entrypoint of your application. In the case of the unit tests, this points to the namespace where my test runner starts the unit tests.

You see, because you send your JS to another Javascript Engine to run tests, it must explicitly call the (run-tests) function somewhere. In this case, it is pointed at the test suite’s main namespace, where the function is called directly in the root.

Extra details

In your test suite namespace, you need two functions: enable-console-print! and run-tests.

(ns your.awesome.code.core-test
(:require [cljs.test :refer [run-tests]]
[your.awesome.code.test-set-a]
[your.awesome.code.test-set-b])
; Turn on console printing. Node can't print to *out* without.(enable-console-print!)
; This must be a root level call for Node to pick it up.
(run-tests 'your.awesome.code.test-set-a
'your.awesome.code.test-set-b)

Compile Onward!

I spent quite some time pouring over the documentation and source files to figure out how all these fit together. Hopefully I saved you a bit of time.

More information

If you’re seeing *print-fn* errors.

How to set the clojurescript *print-fn* fn in nodejs

An exhaustive list of :compiler options available.

ClojureScript - Compiler Options

Sample cljsbuild configuration, annotated.

emezeske/lein-cljsbuild

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.