Learning to Build Clojure Webapps

A while ago I gave a talk at an internal event at Zenika Singapore. We were free to choose a topic so I chose something I thought I didn’t know enough about – what it feels like to build a web app in Clojure. This post is a transcript of the talk. I’ll go into ...

Permalink

Presentation: Functional / Microservices in Real-time Financials

Vitor Olivier presents how Nubank has built the system of record based on functional programming principles, the tools they used (Clojure, Datomic, Kafka), the challenges they faced when taking it to scale, and the benefits of their approach, including data science modeling, real-time customer visibility, guaranteed conservation of money, and customer account histories.

By Vitor Olivier

Permalink

Playing with Event-driven Web-based 3D-Visualization via A-Frame and bowerick

This year in February, I heard for the first time of A-Frame (at RWDSL@CGO). In a nutshell, A-Frame aims on easing web-based 3D visualisation.

Based on my previous work involving Event-driven Architecture (EDA) and Message-oriented Middleware (MoM), I was curious if A-Frame could be used for visualizing near real-time data from EDA/MoM in a web browser. In this post, I briefly write about some very early experiments and how you can run them on your own.

In this post, my focus is on a simple example, which, I hope, you can easily reproduce on your own computer with a few simple steps. If you are interested in more technical details please refer to:

If everything works out, you should be able to see a simple 3D animation in your web browser. Thereby, the animated coordinates are calculated outside of your browser and sent to the browser via the MoM.

Below the results are shown in two screenshots. In the first screenshot, a simple sphere is used. In the second screenshot, I used a tetrahedron to which I added a “trail” to better visualize the animated movement. Note that I deliberately enabled shadows for the sphere example and disabled shadows for the tetrahedron-trail example.

For easing the reproducibility of the examples, I made the corresponding web-pages available via the github pages of the bowerick project:

Alternatively, you can also find the web pages in the examples directory of the bowerick repository.

In addition to the web pages, you need a MoM broker and an event/message producer for generating messages with the animated coordinates. To ease the use of this example, I included a corresponding event/message generator in bowerick.

In the simplest way of using it, the event/message generator will be created along with a broker instance. Thus, you only need to download an appropriate bowerick version, start the example, and connect to the broker using the example web pages mentioned above. Below, commands for optionally downloading bowerick and running the example are shown:

 
# Optionally, use, e.g., wget or curl to download bowerick:
# wget https://github.com/ruedigergad/bowerick/raw/master/dist/bowerick-2.2.2-standalone.jar
# curl -o https://github.com/ruedigergad/bowerick/raw/master/dist/bowerick-2.2.2-standalone.jar
# Start the example including an appropriate broker:
java -jar bowerick-2.2.2-standalone.jar -A

The example web pages use default settings for the broker URL and topic name that matches this simple setup of running the example on localhost. Thus, to see the animated visualization, you simply need to press the “Connect” button of the example web pages.

For another example and as an outlook on a post I plan on more sophisticated event/message generation, you can try to run the bowerick event/message generation as shown below. As shown in the image underneath, this example plots a heart shape using a more complex mathematical function than the simple circle above (This is actually an Easter egg dedicated to family.).

 
java -jar bowerick-2.2.2-standalone.jar -G heart4family -D /topic/aframe -u "[\"ws://127.0.0.1:1864\"]"

I hope you consider bowerick and this post useful. If you have constructive criticism and comments just let me know. I am very much looking forward to helpful feedback.


Permalink

in which a path is charted through the coming apocalypse

I've long counted myself among the grumpy old-timers who grudgingly accept the shift towards web-based-everything and just try to make the most of it, wistfully remembering the days when I could just do everything from within Emacs. One of my core survival strategies in this web-first world has been to trick my browser into at least having the decency to pretend to be Emacs. I accomplished this in Firefox1 with the Keysnail extension. Keysnail has remarkable flexibility in how it overrides Firefox's default key bindings to match those of Emacs, and everything has been more or less great.

Unfortunately, a soon-to-be-released update to Firefox will remove the extension mechanism used by Keysnail.

laptop at Green Lake

I have felt very conflicted about this, because the old state of affairs is admittedly untenable. Firefox currently uses Gecko, a decades-old rendering engine written in C++, and like much software written in C++ it has a pretty distressing security track record. Version 57 of Firefox replaces parts of Gecko with functionality from Servo, a browser engine implemented in the Rust programming language. Most of the bugs in Gecko which have led to embarrassing security flaws are simply impossible in Servo. The fact that so much safety-critical code is still being written in C++ and similar languages is a sad state of affairs, and we should celebrate changes that mean end users will no longer bear the penalty for programmers' reluctance to move beyond the technology of the 1980s.

But on the other hand, losing the ability to shape your computing environment to your whims is awful. I lost track of how many times (when using Chromium or other keysnail-less browsers) I've wanted to throw my laptop out the window when I held down ctrl-n to scroll down and it opened seventeen new windows instead. I can't remember ever wanting to open a new browser window in the past decade; why should I be stuck with a key bound to that command and no way to disable it?

Of course, the new Firefox will still have an extension mechanism, but it's a pale shadow of the old one. Citing the flimsy2 excuse of security, key bindings like C-n are hard-coded into the browser and forbidden from being overridden.

Tumwater Falls

Things were looking bleak for me, and I contemplated whether I would switch to curl or just give up software development altogether for a career in goat-herding. I ended up finding a solution from a most unlikely place.


I had heard of EXWM a while ago, and it struck me as a quixotic curiosity. The X Window System uses a network socket for its control protocol, allowing a lot of flexibility including native forwarding of interfaces for remote programs. The developer of EXWM had taken an XML description of the specification for the network protocol and written a compiler to turn it into a library of Emacs Lisp functions which he then used to implement a window manager in pure Emacs Lisp. While I admired the chutzpah this must have taken, I assumed it was a novelty that could never be practical.

Eventually the Firefox conundrum prompted me to give it a second look due to a feature called Simulation Keys. The exwm-input-set-simulation-keys function allows you to define a translation mapping so that a certain key combination will be intercepted by EXWM when a non-Emacs program has focus, and a different set of key input events will be sent instead. It seemed too good to be true; I could let go of Keysnail and instead get the same features applied to every program I use3.

I'm happy to report that EXWM does actually function startlingly well as a window manager. The simulation keys feature is amazing and puts my Firefox-related fears at ease, and having all configuration written in a single language simplifies my setup dramatically. Every X window you launch is given an Emacs buffer, and all your normal splits and window resizing commands work great with it. With the tiling window managers I used in the past, it was so unusual for me to use something other than the "one fullscreen window per display" setup that I would often forget the key bindings for splitting and rearranging windows. EXWM even integrates "system tray" programs into the Emacs echo area, so your wifi connect tool shows up unobtrusively in the bottom right corner.

There are a handful of gotchas. Emacs Lisp lacks general-purpose concurrency features, but it does allow for concurrency when dealing with subprocesses and network communication. Most well-written Emacs Lisp will never block the main event loop, which is good because when using EXWM that means the entire window manager is stuck until the blocking operation completes. I only came across two exceptions to this rule. One of them is smtpmail-send-it, which can be replaced by the smtpmail-async library. The other is the racket-run command, which I was able to patch in about an hour to remove the blocking call4.

Other folks might run into more problems if they use other third-party libraries which don't take care to use the network functions properly. But for my use5, it's been very smooth, and I'm thrilled to have it.


[1] I used Conkeror for several years, but eventually things got to the point where browsing without Noscript became untenable, and I could never get the two to work well together.

[2] The rationale of "it's for security" would stand up to a little more scrutiny if it weren't for the fact that extensions can rebind C-t, a key which is used hundreds if not thousands of times more often than C-n.

[3] Granted gnumeric is the only program I use outside the browser and Emacs, but it's still greatly appreciated. I also use the Saka Key extension, which implements Keysnail's ability to trigger links from the keyboard even if they don't have text attached to them.

[4] I feel that the increasing "Emacs needs concurrency!" calls tend to overstate the problem. Yes, of course it would be nicer for the programmer to code using coroutines (coming in Emacs 26!) instead of callbacks, but in the end this is a convenience for the author, not for the end user.

[5] My customizations largely revolve around replacing my xbindkeys config with elisp, mapping workspace numbers to physical displays, and some eshell commands to give one eshell buffer per workspace. EXWM has XMonad-style workspaces where you can change the workspace for each display independently rather than forcing you to change them all at once like many more conventional WMs, and I'm very glad it does.

Permalink

PurelyFunctional.tv Newsletter 248: Weapons, GraphQL, and the return of an old acquaintance

Issue 248 – October 23, 2017

Hi Clojurers,

I’m deep in content-production mode on my next course. That means I did not watch or read as much this week as I sometimes do.

Please enjoy the shorter issue!

Rock on!
Eric Normand <eric@purelyfunctional.tv>

PS Want to get this in your email? Subscribe!


Weapons of Math Destruction Book

As computers give us more help, and computers make more decisions for us in our lives, we should be aware of the dangers inherent in how they work. Algorithms are considered neutral, yet it is this neutrality that makes them amplify existing problems.

Some example dangers:

  • Training a model with historical data may train the model on pre-existing biases.
  • Once an algorithm is used to make decisions, it could become a positive-feedback loop, creating the evidence for its own efficacy.
  • Current algorithms go against many of the ideals we hold dear. For instance, we want to judge a person by their actions instead of what a group they belong to has done in the past. Algorithms do bucket people into groups (often called clusters) based on statistical correlations, then treat everyone in that group the same.

I was initially afraid to read this book because I already dislike technology enough. I didn’t want another bleak outlook on the future. But I faced it and read it anyway. The book has made me rethink my relationship to algorithms. If anything, I want to make sure a person is somewhere in any loop involving a statistical system.


The REPL

I’m very happy to see the REPL is back with its excellent newsletter. I missed it while Daniel was away for the winter.


The Ultimate list of Functional Programming Conferences

I’ve made a big list of all of the functional programming conferences I could find. Is your favorite conference on the list? Have you found any errors? Let me know!


Clojure SYNC

I am continuing work on Clojure SYNC, the conference I’m running in New Orleans. Please buy a ticket. If you’re thinking about sponsoring the conference, that program is coming soon!

Do check out the speaker line-up. I personally invited each of them to ensure that we had nothing but consistently engaging speakers.

And if you have any questions, just reply to this email. In addition, I’d love to hear what’s stopping you from buying a ticket 🙂


The Case for Clojure and GraphQL: Replacing Django

Rodrigo Landerdahl had quite a positive experience with Clojure and GraphQL. Short version: ecosystem is not as good a Django, but the Clojure codebase was smaller, had fewer bugs, and caused fewer arguments.


How can more layers be more efficient?

From the archives: 18 months ago, people were still wondering how ClojureScript + React could be faster than React, and how React could be faster than manipulating the DOM by hand. Each layer is adding indirection. So how can it get faster the more you add? Read the article for my take.

Permalink

The Toccata Way

Using Toccata

Every tool is designed with a certain use in mind and using the tool in the way it was designed is often much easier. So starting with this post, I want describe how I intended Toccata to be used; The Toccata Way.

However, programming languages are incredibly malleable, so what I describe as The Toccata Way could very well change as people use it and discover better ways of doing things. Nothing I write here is cast in stone and I would love to be shown better ways of creating software using Toccata.

The programming triple

All programming boils down to telling the computer to do something in very small steps and there are 3 fundemental constructions used to build a whole program out of these small steps.

  • Executing steps in sequence one after the other.
  • Choosing one of a number of sequences of steps to execute.
  • Executing a sequence of steps repeatedly with different inputs

Together, I consider these the "control-flow triple" and it appears everywhere I look at all levels of abstraction from assembly language to the highest level languages. Even declarative logic programming languages like miniKanren (or Prolog), have analogous constructions.

In Toccata, and most other languages, sequencing steps is so trivial that it's almost invisible

  (fn [x y]
    (expr1 x)
    (expr2 y (expr3))

expr1 is done first followed by expr3 and finally expr1 which produces the final result.

I'll address repeatedly executing a sequense of steps in the next post. So now, let's look at choosing.

A Time for Choosing

Let's revisit the hello-world program:

  (main [args]
    (println "Howdy, Folks"))

"Folks" is a little generic, so let's personalize it a little by letting a name be passed in on the command line.

The args parameter to the main function is a list of strings that are the command line arguments to the program. The first string in the list is the name of program that was executed which we'll mostly ignore. So we need to get the second item from the list. The second function from the core library does exactly that. Except it returns a Maybe value that we need to extract the actual name from. But there might not have been a name passed in on the command line. So we need to account for that.

  (main [args]
        (println 'args args)
        (println "Howdy," (either (second args)
                                  "Folks")))

I encourage you to run this for yourself.

Short-circuit

Let's say we want to personalize it further by having a table of names mapped to a corresponding greeting.

  (def greetings {"Jim" "Howdy"
                  "Frank" "Hello"
                  "Maria" "Bonjour"
                  "Tina" "Beautiful"
                  "Human" "Greetings"})

So we need to get the second arg as above, and then use that name to get the salutation. But several things could go wrong. The name may not be in our map, or there may not be a name given on the command line at all. We'll use the flat-map function, which is defined for Maybe values, to chain some calls to functions together sequentially. But if any of them return nothing, the whole expression short-circuits and returns nothing.

  (main [args]
    (println (either (or (flat-map (second args)
                                   (fn [name]
                                     (map (get greetings name)
                                          (fn [salutation]
                                            (str salutation ", " name)))))
                         (map (second args)
                              (fn [name]
                                (str "Howdy, " name))))
                     "Howdy, Folks")))

In this case, the map expressions both apply a function to a Maybe value. The anonymous fns only get called if the Maybe values are not nothing. Hopefully that construction is obvious given the explanation in the Containers post.

Let's clean this up a little. We can use a for expression to refactor the flat-map expression.

  (main [args]
    (println (either (or (for [name (second args)
                               salutation (get greetings name)]
                           (str salutation ", " name))
                         (map (second args)
                              (fn [name]
                                (str "Howdy, " name))))
                     "Howdy, Folks")))

Having two calls to second is a little ugly, so let's pull that out

  (main [args]
    (let [maybe-name (second args)]
      (println (either (or (for [name maybe-name
                                 salutation (get greetings name)]
                              (str salutation ", " name))
                           (map maybe-name
                               (fn [name]
                                 (str "Howdy, " name))))
                          "Howdy, Folks"))))

I'm sure there are other ways to write this little ditty that improve on this.

Permalink

Embedded Interop between Clojure, R, and Python with GraalVM

https://images-na.ssl-images-amazon.com/images/M/MV5BOTViY2Y0ZGItMTg2OC00YzEzLWJhYjYtZjg4OTMyOWE4YzM1XkEyXkFqcGdeQXVyNTQ1NzU4Njk@._V1_.jpg" title="" >

In my talk at Clojure Conj I mentioned how a project from Oracle Labs named GraalVM might have to potential for Clojure to interop with Python on the same VM. At the time of the talk, I had just learned about it so I didn’t have time to take a look at it. Over the last week, I’ve managed to take it for a test drive and I wanted to share what I found.

Are you ready?

In this example, we will be using an ordinary Leinengen project and using the REPL we will interop with both R and python.

But first will need a bit of setup.

We will download the Graal project so we can use its java instead of our own.

Once we have it downloaded we will configure our PATH to use Graal’s java instead of our own.

```

export PATH=/path/to/graalAndTruffle/bin:$PATH

```

Now, we can create a new lein project and run lein repl and begin the fun.

The Polyglot Context

In our new namespace, we just need to import the Polyglot Context to get started:

```clojure (ns graal-test.core (:import (org.graalvm.polyglot Context)))

;; note that is also supports Ruby, LLVM, and JS (def context (Context/create (into-array [“python” “R”]))) ```

Now, we are ready to actually try to run some R and Python code right in our REPL. Let’s start first with R.

Interoping with R

The main function we are going to use is the eval function in the context. Let’s start small with some basic math.

clojure (.eval context "R" " 3^2 + 2^2 ") ;=> #object[org.graalvm.polyglot.Value 0x7ff40e4d "13.0"]

Wow! It actually did something. It returned something called a Polyglot Value with what looks like the right answer in it.

Emboldened by our early success, let’s try something a little more complicated like calling a function.

clojure (def result1 (.eval context "R" " sum.of.squares <- function(x,y) { x^2 + y^2 } sum.of.squares(3,4) ")) ;=> #object[org.graalvm.polyglot.Value 0xc3edd92 "25.0"]

Again, it looks like it worked. Let’s try to get the result back into Clojure as a value we can work with. We could ask the result what sort of type it is with

clojure (.isNumber result1) ;=> true

but let’s just use clojure.edn to read the string and save some time.

``` (defn –>clojure [polyglot-value] (–> polyglot-value

  (.toString)
  (clojure.edn/read-string)))

(–>clojure result1) ;=> 25 ```

It would be nice to have a easier way to export symbols and import symbols to and from the guest and host language. In fact, Graal provides a way to do this but to do this in Clojure, we would need something else called Truffle.

Truffle is part of the Graal project and is a framework for implementing languages with the Graal compliler. There are quite a few languages implemented with the Truffle framework. R is one of them.

https://image.slidesharecdn.com/polyglotonthejvmwithgraalenglish-170521104613/95/polyglot-on-the-jvm-with-graal-english-14-638.jpg?cb=1495364615">

My understanding is that if Clojure was implemented as a truffle lang, then interop could be much more seamless like this example in Ruby

https://image.slidesharecdn.com/polyglotonthejvmwithgraalenglish-170521104613/95/polyglot-on-the-jvm-with-graal-english-37-638.jpg?cb=1495364615">

But let’s continue in our exploration. What about doing something more interesting, like importing a useful R library and using it. How about the numDeriv package that supports Accurate Numerical Derivatives?

First we import the package using cran.

clojure (.eval context "R" " install.packages(\"numDeriv\", repos = \"http://cran.case.edu/\") ")

If you are doing this at your REPL, you can will see lots of text going on in your lein repl process at this point. It’s going out and figuring out what deps you need and installing them in your /graalvm-0.28.2/jre/languages/R directory structure.

After it is done, we can actually use it!

```clojure (def result2 (.eval context “R” “ library(numDeriv) grad(sin, (0:10)2pi/10) ”)) result2 ;=> #object[org.graalvm.polyglot.Value 0x76765898 “c(1,

    ;0.809016994367249, 0.309016994372158, -0.309016994373567,
    ;-0.809016994368844, -0.999999999993381, -0.809016994370298,
    ;-0.309016994373312, 0.309016994372042, 0.809016994369185,
    ;0.999999999993381)"]

```

This has a bit more interesting result as an array. But the Context has ways of dealing with it.

```clojure (.hasArrayElements result2) ;=> true (.getArraySize result2) ;=> 11

(for [i (range 10)] (–> (.getArrayElement result2 i) (–>clojure))) ;=> (1.0 0.8090169943672489 0.3090169943721585 -0.3090169943735675 ;-0.8090169943688436 -0.9999999999933814 ; -0.8090169943702977 -0.3090169943733122 0.30901699437204233 ; 0.8090169943691851) ```

So, we’ve showed basic interop with R – which is pretty neat. What about Python?

Interoping with Python

Truffle is scheduled to fully support Python in 2018, but there is already an early alpha version in the Graal download that we can play with.

clojure (.eval context "python" " import time; time.clock() ") ;=> #object[org.graalvm.polyglot.Value 0x4a6b3b70 "1.508202803249E9"]

Neat!

It is still a long way for import numpy or import tensorflow but cPython compatibility is the goal. Although the c-extensions are the really tricky part.

So keep an eye on Graal and Truffle for the future and wish the Oracle Labs team the best on their mission to make the JVM Polyglot.

Footnotes

If you are interested in playing with the code. I have a github repo here graal-test. If you are interested in watching a video, I really liked this one. There are also some really nice examples of running in polyglot mode with R and Java and JS here https://github.com/graalvm/examples.

Permalink

The REPL Returns

The REPL Returns

Effective programs, Java 9, and Clojure 1.9
View this email in your browser

The REPL

Notes.

Welcome back to The REPL. I've been taking a break after our new son was born, but I'm ready to get back to it. There was so much good stuff in Clojure over the last few months, and unfortunately I won't have time to highlight it all. Thanks for reading, and enjoy!

-main

  • Clojure/Conj 2017 has just happened, and the stand-out talk for me was (of course) Rich's talk Effective Programs - 10 Years of Clojure. I found it helped me understand the philosophy of Clojure even better.
  • Eric Normand is running a new conference Clojure SYNC in New Orleans in February 2018. There's some good speakers lined up, and it looks like a neat event.
  • Help ship Clojure 1.9!
  • Matthew Gilliard has had a great series of articles on improving Clojure startup time with some new Java 9 features.

Libraries & Books.

  • clojerl: Clojure for the Erlang VM

People are worried about Types. ?

  • But expound makes the error messages better for developers. I've used this fairly heavily and find it makes a big improvement to understanding spec errors.
  • Phrase is another tool for error messages, but focusing on presenting the errors to the user. I haven't used it, but it looks neat too.

Foundations.

Tools.

  • Leiningen 2.8.0 was recently released, and also recently turned 8.
  • Colin Fleming was on Talking Kotlin, talking about how he uses Kotlin with Cursive
  • jmh-clojure looks like a great way to use the very accurate JMH benchmarking tool

Recent Developments.

  • CLJ-2194: Spec metadata support
  • CLJ-1074: Added new literals for Infinity and NaN

Learning.

Misc.

Copyright © 2017 Daniel Compton, All rights reserved.


Want to change how you receive these emails?
You can update your preferences or unsubscribe from this list

Email Marketing Powered by MailChimp

Permalink

Becoming Foolish

This post was originally published on my blog.

The book The Pragmatic Programmer: From Journeyman to Master by Andy Hunt and Dave Thomas suggests that as developers, we should "learn at least one new language every year." (pg. 14)

When I recently asked a roomful of developers, if there's anyone who had learned a new language this year, only very few hands went up. A year ago today, that would have been me in the audience, keeping my hands down.

I had been a happy C# developer for over a decade, starting professionally sometime before .NET 2.0 was released. During my career I had the privilege of working in a very specialized field - building tools for other developers, mainly in form of extensions to Visual Studio itself. My job often required from me to understand the code better than the compiler, in fact, I often had to take a peek under the hood to understand what the compiler, MSIL, JIT, and sometimes native code were doing, all long before tools like Roslyn even existed. I knew implementation details of the runtime that were not documented anywhere. I knew most of the corner cases and language gotchas. But overall, my language, C#, was a very powerful hammer, capable of handling any kind of nail.

Which caused me to pretty much ignore any other languages out there, which weren't relevant to my day job. Whenever people from the F# camp were boasting their lack of nulls, pattern matching, or not needing semicolons or braces, I would typically shrug it away. I have spent years of my career learning to use productivity tools, such as ReSharper and IntelliJ IDEA, to the point where every bit of boilerplate was one keystroke away. I simply didn't care! Having a few nice features was never a good enough reason for me to switch a programming language -- why would I even bother, when my language that I'm an expert in, could kinda-sorta do that too? Why would a different syntax make a difference in how I write software? After all, doesn't it all compile down to the same thing?

Because when you really get down to it, (almost) all programming languages are, essentially, this:

public static void main(String[] args)
{
    ...
}

In most programming languages that we've ever experienced, the world as we know it lives and dies in the confines of the opening and closing brace of the main function, which is the entry point to our application. Everything that happens in our program, the universe our program interacts with, happens between those two braces. By its very definition, our program returns an exit code, specifying whether it succeeded or not, and everything else happens as a side-effect.

The implication, that all the work is done between the two braces of the main method, is what drives everything we know about this model - every design pattern, every practice, every discipline, every tool, library, or framework - were created to let us manage this model of our world that exists between those two braces. We have decades of knowledge in how to do this properly in almost any language.

For me, this was the core, the very foundation on which all the software is built! It was the one thing that was the ultimate truth, regardless of the programming language.

And it was like this, until Haskell ruined it for me!

I will not go into details - I inadvertently did that in my previous post. But during watching a talk on seemingly unrelated topic, the speaker managed to sneak some Haskell in there! Until that moment, I haven't really seen any Haskell code before - I've always dismissed it as irrelevant - and there it was, staring me in the face.

main :: IO () -- reads as 'main, having the return type of IO ()'

What I saw being explained is that Haskell doesn't actually do any work inside of its main function. Instead, the program "describes" in a declarative way what is going to have happen when your program executes, and this "description" will be returned into the Haskell runtime, and exactly what was described will happen - with no possible side-effects or surprises. It flips the entire idea of execution on its head - the work, the actual execution of effects is not being done inside of main - it's done on the outside!

Seeing this for the first time, trying to wrap my head around it, had created a bug inside my head. It had conflicted with what I knew - that all languages worked the same, and the difference was merely syntax! But here, I was seeing something that just worked differently, and did not fit my world view! I struggled with trying to understand it, and it took me a couple of more re-watches, until one day, it finally clicked!

In one single moment, I suddenly understood the idea of Haskell, of why it works the way it does. Everything just clicked into place and I finally "understood monads"! To me, it felt like Neo finally coming to terms with the fact that there is no spoon, and only then he's able to bend it.

I am thankful for Andrea Magnorsky's excellent keynote talk, Inviting Everyone to the Party, which helped me realize that not only what I had experienced had a name, it was, in fact, a textbook definition of the phenomenon: what I had experienced is called a paradigm shift.

The term "paradigm shift", coined by Thomas Kuhn in his 1962 book The Structure of Scientific Revolutions, describes a process where if an established model no longer fits the world view, it is rejected and simultaneously replaced by another. Kuhn says:

The decision to reject one paradigm is always simultaneously the decision to accept another, and the judgment leading to that decision involves the comparison of both paradigms with nature and with each other.
Thomas Kuhn, The Structure of Scientific Revolutions: 50th Anniversary Edition (pg. 78)

I had two conflicting ideas, and when finally that other idea started making sense, my mind accepted it, and immediately switched to that new idea - the switch was instantaneous! I was able to see things from a different perspective, understanding a whole new world model, which is precisely the definition of Kuhn's "paradigm shift".

But something else happened -- I realized just how little I knew about software, there was an entire world out there that was unknown to me! What I knew until that moment, all my expertise in a narrow field was just a tip of the iceberg. And this realization removed a barrier I had that prevented me from trying to learn new languages and experimenting with new things. Where previously I would dismiss these things as irrelevant, I suddenly gained interest in things that I never had interest in before. In the past year alone I've used Haskell, Clojure, and Rust. I've looked at Erlang, Elm and PureScript. I am working my way through SICP and learning Category Theory. I've done all that because for the first time in a long time, I am genuinely curious about what these languages/paradigms can teach me! I really don't remember the last time I was this excited about learning. Genuinely proud of being able to say - "I don't know everything!" - is incredibly liberating!

This exact sentiment is also echoed in Bret Victor's fantastic, and highly entertaining keynote talk, The Future of Programming, where he pretended he was a software developer in 1973, and speculated about the next 40 years of programming. The last 5 minutes of that talk has some of the best advice I ever heard, and I wish I understood it sooner. He said,

...you have to say to yourself - "I don't know what I'm doing. We, as a field, don't know what we're doing. We don't know what programming is, we don't know what computing is, we don't even know what a computer is." And once you truly understand that, once you truly believe that -- then you're free, and you can think anything.
Bret Victor - The Future of Programming

Stay hungry. Stay foolish. Be free!

Permalink

Tooling For Toccata

Conjing

After being in Baltimore for the Clojure Conj last week, it's time to get back in the saddle and write about Toccata. It was great hanging out with Clojure community, seeing old friends and just generally geeking out. Thanks to everyone who came to the unsession on Toccata despite the late hour.

I got lots of positive feedback from folks when I mentioned what I was working on. The most common statement was "That would be great for embedded software.". Which is really gratifying, since that's what my career was before I started doing Clojure work. Toccata isn't specifically targeted to the embedded space and since it uses ref counting garbage collection, there are some places it's not suitable. But it seems there's quite a few Clojure programmers who want to do embedded stuff.

Tools

And now; tooling.

In programming, it's easy to assume that if you can just make the language "good enough", all problems go away and it's sunshine, rainbows, and ponies from then on. I don't believe that, so one of my goals with Toccata is to make it easy to write great tooling. I would like to see tools for writing, formatting, debugging, mapping, refactoring, and working with software written in Toccata. And yes, that includes static analysis, ala. types. One primary goal is to enable the creation of a tool to assist in reading Toccata code that you can point at a source file and it'll help you understand what's going on and why.

Given those goals, it's quite appropriate that the first merge to the toccata repo from someone else was by Michael Nygard (who has a new edition of his classic "Release It" coming out) that added support for CMake and also a Docker container with a self contained Toccata dev environment.

This is huge. I've always used lldb to debug my code and just recently was able to enable source level debugging in it. It's kind of a hack, but you can set break points in your Toccata source and the debugger will stop at those points. Then you can single step from that point and also examine variables. The Toccata symbol names are a little mangled and have a numeric identifier appended to the end, but you can probably figure it out.

Fire!

As I work on the Toccata compiler, I'm pretty focused on performance. I want the compiler to be fast so the edit/compile/run debug loop is takes as little time as possible. So tools for measuring the performance and identifying hotspots is prety important. One tool is the flamegraph. Unfortunately, the way the compiler works limits the utility of flame graphs. But thanks to the docker container, I got it working for Toccata so anyone should be able to generate them from their code.

One thing these graphs do show in stark detail is the effect compiler optimizations have on the Toccata compiler. Here's a flamegraph of Toccata compiling regression-tests/main1.toc from the repo. Obvious to see the massive call stack that builds up due to the structure of the recursive descent parser and the compiler.

However, this flamegraph is for a run where the compiler was compiled using the -O1 flag in Clang. That optimization level includes sibling call elimination and the difference in the flamegraph is dramatic. So much so, that I don't really trust it. At some point, I may dig into that further. OTOH, the difference in performance is dramatic.

FYI, if you intend to generate flamegraphs using the Docker container, you'll have to run it with the --privileged option so that perf will work properly.

Graphical Debugging

I would think it would be doable to enable gdbgui or something similar and get graphical debugging of Toccata code. It'll probably be awhile before I can do it, so let me know if get that working before I do.

Time Travel

But that's not the biggest thing.

Ever since I saw the rr project, I've wanted it for Toccata. But it doesn't work on OSX, which is what I mostly develop on. Now that there's a container, I thought it would be possible to get it working much like the flamegraphs. Sadly, this isn't the case and I have to drop that effort to make progress in other areas. I would love it if someone would take that up. Then we could have record/replay debugging at the Toccata source level. How cool would that be?

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.