Building a Purescript web server with Stetson and Pinto

All the posts so far..

Useful links

  • demo-ps The demo codebase we're talking about here
  • erl-pinto (the opinionated bindings to OTP we're using)
  • erl-stetson (the opinionated bindings to Cowboy we're using)

Having now gotten a basic OTP application up and running, it'd be nice to get some data out to the world.

Cowboy

Cowboy is the defacto web server in the Erlang world, each route loosely maps to a module which has callbacks defined in it to handle various stages of the decisioning process.


-module(my_handler).

-export([init/2,
         get_text/2,
         content_types_provided/2
        ]).

init(Req, _Opts) ->
  { cowboy_rest, Req, #state{} }.

content_types_provided(Req, State) ->
  {[{ <<"text/plain">>, get_text}], Req, State}.

get_json(Req, State) ->
  { <<"Hello World">>, Req, State }.

This is directly representable in Purescript, using erl-cowboy.

module MyHandler where

init :: forall a. InitHandler a a
init = mkEffectFn2 \req c -> pure (initResult c req)

content_types_provided :: forall s. ContentTypesProvidedHandler s
content_types_provided =  mkEffectFn2 \req s -> pure $
  restResult
    (contentTypesProvidedResult $ fromFoldable
      [ tuple2 (ContentType "text" "plain" AnyParams) (ProvideCallback $ atom "asText") ]
    )
    s req


asText :: forall s. EffectFn2 Req s (Tuple3 String Req s)
asText = mkEffectFn2 \req s -> pure $ tuple3 "Hello World" req s

However, this doesn't make the best use of Purescript itself - and writing out a few dozen handlers like this would soon get a bit tedious, which is why I went away and wrote Stetson. Purescript is a functional programming language and it makes sense that rather than provide a bunch of loosely typed callbacks referred to by name using strings, that we built an API that took functions to do all the heavy lifting.


helloWorld :: StetsonHandler Unit
helloWorld =
  Rest.handler (\req -> Rest.initResult req unit)
    # Rest.contentTypesProvided (\req state -> Rest.result (tuple2 "text/html" asText) req state)
    # Rest.yeeha
    where 
      asText req state = do
        Rest.result "Hello World" req state)

The idea of course being that we can configure Stetson/Cowboy at the top level by providing a pile of functions and abstract over the common bits like "This is an accept handler that turns state into JSON because State has the typeclass "WriteForeign", and start to get rid of a lot of duplication across our routes.


init :: BookWebStartArgs -> Effect State
init args = do
  Stetson.configure
    # Stetson.route "/api/books" books
    # Stetson.route "/api/books/:isbn" book
    # Stetson.static "/assets/[...]" (PrivDir "demo_ps" "www/assets")
    # Stetson.static "/[...]" (PrivFile "demo_ps" "www/index.html")
    # Stetson.port args.webPort
    # Stetson.bindTo 0 0 0 0
    # Stetson.startClear "http_listener"
  pure $ State {}

where books and book are handlers as described above. In our own applications, we have ended up with pretty much the entire web server and all routes in a single file - which is in stark contrast to our usual Erlang apps where we have a folder containing dozens of separate erlang modules.

books :: StetsonHandler (List Book)
books =
  Rest.handler (\req -> do
                        state <- BookLibrary.findAll
                        Rest.initResult req state)
    # Rest.allowedMethods (\req state -> Rest.result (Stetson.POST :  Stetson.HEAD : Stetson.GET : Stetson.OPTIONS : nil) req state)
    # Rest.contentTypesProvided (\req state -> Rest.result (jsonWriter : nil) req state)
    # Rest.contentTypesAccepted (\req state -> Rest.result ((tuple2 "application/json" acceptJson) : nil)
                                req state)
    # Rest.yeeha
    where 
          acceptJson req state = do
            body <- allBody req mempty
            result <- either (pure <<< Left <<< show) BookLibrary.create $ readJSON $ unsafeCoerce body
            case result of
                 Left err -> Rest.result false (setBody err req) state
                 Right c -> Rest.result true req state

jsonWriter :: forall a. WriteForeign a => Tuple2 String (Req -> a -> (Effect (RestResult String a)))
jsonWriter = tuple2 "application/json" (\req state -> Rest.result (writeJSON state) req state)

So that's a handler that has a state of type 'List Book', which it gets from our BookLibrary via a call (as in the previous blog entry), jsonWriter being a function as decscribed above - simply taking that model and spitting it out as JSON, leveraging our model which happens to implement that type class.

We'll look more into that in the next entry, where we talk about that model and how we're using it on both client and server.

Permalink

Grokking Simplicity Launch

My new book, Grokking Simplicity, all about functional programming, is now available in early access. The first three chapters are ready to read. Go to https://lispcast.com/gs, add the book to the cart, and use discount code MLNORMAND for 50% off.

https://www.youtube.com/watch?v=jYN-b404Qds

The post Grokking Simplicity Launch appeared first on LispCast.

Permalink

Monads in the real world

Monads are real, y’all. They are all around us. In this metaphor-free episode, I’ll share two real-world monads you interact with all the time. No burritos or space suits, I promise! Plus, we’ll see why monads are useful in Haskell.

Transcript

Eric Normand: Today, we’re looking at monads in the real world. By the end of this episode, we will have a pretty good idea of what they are based on real, concrete, actual, real world stuff. I hope we can see a little bit of how they’re defined mathematically.

I just want to also say that this is going to be a completely metaphor-free monad explanation. These are not metaphors. I am trying to look for monads in the real world.

We can talk about other categories and how we see them in the real world, for instance, monoids. Lots of monoids in the real world, so when you have a toy train set, you can detach them and then reattach them, and they maintain the order when you reattach them, so it’s a monoid. It’s an associative. It has an identity.

There’s a whole episode on that. If you want to learn about monoids, which are cooler than monads, by the way, go check out the monoid episode. This one’s all about monads though. Metaphor-free, the real monads that you have dealt with before.

Before I get to the real-world things, I want to briefly talk about what a monad is in the abstract. We’ll touch on these and explain this through real-world examples.

Monads are mathematical objects. They’re values that have a couple of properties. One is that they’re functors. I have a whole episode on functors. Basically, it means you can map over them.

Some people think of functors as containers, but don’t do that, that’s a metaphor. I’m not doing that in this one. A functor is something that you can map over, so a list, obviously, an array, you can map over an array.

You can map over a maybe, meaning you apply the function if it has a value. If it doesn’t have the value, if it’s nothing or none, then you don’t apply that function.

Now, the second thing, so all monads are functors. Let’s just get that out of our way. There’s a whole episode on functors, you should watch that.

In addition to being a functor, a monad has a join operation, and that’s the one we are going to go deeply into in a couple of real-world examples. So why are monads important? I’m going to try to explain this with an example you might be familiar with.

Let’s say I was giving a gift to my friend, and I told my friend, his name is Bill. “Bill, I’m giving you a box of chocolates, happy birthday.” He takes the box, you know, kind of a sizable box, and he thinks, “Wow thank you.”

He goes home, and at home, he opens the box, and inside are lots of little boxes, boxes that fit inside the bigger box. He opens those boxes up, and inside of those boxes are the chocolates. Now, is he going to call me and say, “Hey Eric, you said it was a box of chocolates, but it was actually a box of boxes of chocolates.”

No, he is not going to do that, he’s going to eat the chocolates and just be happy. A type checker would balk at you. It would say, “Wohoho, you said a box of chocolates, but there is an extra nesting of boxes in there, that you forgot to mention.”

What do you do when you have this type error? It is a type error. I was not very specific. I was not as pedantic as a type checker needs to be. I’m admitting that. I was acting [laughs] as a birthday gift, this was like very human level.

That’s what this join operation does. It allows you to go from box of boxes of chocolates, to box of chocolates. Removes one level of nesting in a structured way that makes sense for that monad. That’s all it is. It just removes that one level of nesting. In Scala and probably other languages I am not familiar with, functor is called mappable because it has the map method on it.

Monad, it also has the map. Remember, all monads are functors, but it has flat mappable. That flat is like you’re flattening one level of nesting. That’s called join. Let’s look at another example. Let’s have a to-do list.

You know it’s like household chores and stuff, but then I get to one thing that is actually a big enough task that it has five subtasks. I want to make sure to get all of it done.

As I’m going through, I just go through one, two, three, four and then when I get to five, I have to do 5.1, 5.2, 5.3 etc. I’m still doing them in order. It’s just they’re nested in one list more.

I can do a join operation to flatten it into a single list, which is what I’m doing in my mind. As I read it, I flatten it. I read them sequentially from top to bottom. I ignore the nesting.

This join operation lets us do the nesting in the structure of our data, but then undo it when we want it to be flat. That’s what join is all about. This to-do list example gives a clue about why something like Haskell uses monads for sequencing I/O operations.

Sometimes, an I/O operation actually has several steps in it. What it needs to do is flatten them all out into just like a long string of things to do. It doesn’t want to have to go nesting there and become like a stack, as a tree, because it would turn into a tree, it can flatten it all out and just run straight down.

That’s one part of the monad that Haskell uses. I hope that by showing this as a real-world example that you see that these aren’t that hard to understand. Now, when you take them out of the real world and into a very abstract expression, there’s not much to them because every monad is different really.

Every monad has a different structure and different way of joining different semantics for how they join. There’s not that much in common, except that it’s a functor.

It has a join operation. There’s a few monad laws because it has to be associative and it has to have an identity. Those usually are pretty straightforward. Now, in Haskell and in Scala with flat map, remember I talked about flat map, the equivalent in Haskell is called “bind.”

What bind does is it’s really joining the map from the functor joining. It’s composing the map with a join. It’s sometimes more convenient to use bind than to use the map and join. It’s more convenient to use flat map than the map and flatten.

That’s why we see bind together defined as one of the two operations of monad, but really, I think that the join is easier to work with from real-world stuff. The map already comes from it being a functor, so you don’t have to define two different operations. I want to go over Maybe, which is a type in Haskell. It has equivalents in other languages.

It’s called Option in Scala, but it’s really got the same semantics. Maybe as a type that has two possible cases two constructors. One is when you have a value and one is when you don’t.

It’s used for representing optional values. I either have the value or I don’t. This has the two cases. In a lot of languages, you might use a value or null to represent I don’t have it. In Haskell, there’s a type that represents this.

That helps you, if you don’t have a Maybe, you know you will not get null. There’s only certain cases that you have to specifically call out. I may or may not have an answer on this function.

In that case, the two cases are called just and nothing. Just is when you have the value. You might say, “I have just five. It’s just five.” There’s no other meaning behind it besides I have it and the nothing is when you don’t have the answer.

How does join work on a Maybe because Maybe is a monad? First, let’s do the functor. How does map work on a monad?

Let’s say I want to have five and I want to map increment over that just five. We have to cover the two cases, let’s do that just case first. I want to map increment over this value and it’s a just. It’s just five.

What it’s going to do, because it’s a just, it’s going to return five plus one in a new just. It’s going to make a new Maybe with a just constructor. That’s got the answer to reaching into the Maybe, grabbing the value, calling the function on it, get an answer, put it back in the type.

It’s a just with six. If it has a nothing, that’s easy, there’s nothing to call the function on so you just return a nothing again. Those are the two cases. You’ve covered it all. That’s what map does over Maybe.

Now, what about join? In join, we have to deal with the nesting. I’ll handle the easiest case first. This is the type, signature would be Maybe/Maybe Int. Nested inside the Maybe is a Maybe Int.

The easiest form of this type, the easiest value of this type, is if the outer maybe is a nothing. There’s actually several cases that we have to deal with now because we have the nesting. The outer Maybe is a nothing, so that’s easy.

I just return nothing. That’s what the join us. Now, the other cases if the outer one is a just and the inner one is a nothing, we have to get rid of the nesting. I don’t have a value. That’s also nothing.

We have two cases that result in nothing, but if I have a just, just five, then when I join that, it becomes just five. It gets rid of one of the layers of Maybe on it. Those are the three possible cases.

If the outer one is nothing, it returns nothing. If the inner one is nothing, it returns nothing. Otherwise, it just collapses it into a single just instead of two justs. That’s it. That’s monad.

They’re used a lot in Haskell because of this property of being able to unnest stuff in a way that you can expect and reason about. If I have a calculation that returns a Maybe and then I want to do something to the thing inside of it, but sometimes it’s nothing, I don’t have to think about that.

I don’t have to put an if statement that says, if it’s nothing then do this, but if it’s something, do this. I don’t have to write that anymore. It’s that conditional that those two cases are handled automatically by the join.

That is why they use it in Haskell because it makes it easy to write the nice case where everything is returning values and the case where something returns a nothing happens automatically for you. That’s why they do that.

This has been a deeper episode than I thought it would be. We’ve got into what a monad is mathematically. It’s a functor with a join operation. We went over two real-world examples, “A box of boxes of chocolates,” and also a to-do list where there’s like nested lists in there.

We also saw how Maybe works in Haskell, and why it’s used in Haskell, what it’s useful for. Awesome. I hope this helped. If it did, you might want to see other episodes.

You can go to lispcast.com/podcast, and see all the old episodes. Each one includes audio, video, and text. You can also find links to subscribe wherever you want to subscribe and social media so you can get in touch with me and complain that I’ve created yet another monad tutorial.

This has been my thought on functional programming. My name is Eric Normand. Thank you for being there. Rock on.

The post Monads in the real world appeared first on LispCast.

Permalink

Building on top of OTP with Purescript with Pinto

All the posts so far..

Useful links

  • demo-ps The demo codebase we're talking about here
  • erl-pinto (the opinionated bindings to OTP we're using)
  • erl-stetson (the opinionated bindings to Cowbou we're using)

We left the last post demonstrating that an end-to-end Purescript project was essentially a pile of Purescript written in the right place, in order that the usual Erlang application/rebar3/etc can just work with it.

That entry point again then

We looked at BookApp.purs, which compiled into an Erlang module called bookApp@ps. which ends up in src/compiled_ps and gets picked up with the usual Erlang structure. Let's look at that file and see what we see:

module BookApp where

import Prelude
import BookSup as BookSup

import Pinto.App as App

start = App.simpleStart BookSup.startLink

That simpleStart function is just a Pinto helper that describes an entry point that doesn't worry about inputs, and provided a Supervisor will start that Supervisor when the application is started.

This gets compiled into the following Erlang

% Generated by purs version 0.12.3
-module(bookApp@ps).
-export([start/0, start/2]).
-compile(nowarn_shadow_vars).
-compile(nowarn_unused_vars).
-compile(no_auto_import).
-file("src/BookApp.purs", 8).
start() -> (pinto_app@ps:simpleStart((bookSup@ps:startLink()))).
-file("src/BookApp.purs", 8).
start(_@0,_@1) -> ((pinto_app@ps:simpleStart((bookSup@ps:startLink())))(_@0, _@1)).

If get rid of the cruft, that's essentially just

-module(bookApp@ps).
-export([start/0, start/2]).
start() -> (pinto_app@ps:simpleStart((bookSup@ps:startLink()))).
start(_@0,_@1) -> ((pinto_app@ps:simpleStart((bookSup@ps:startLink())))(_@0, _@1)).

This is just a standard application module that you'd find in an Erlang application, exporting the usual start function that calls into a supervisor to start it. This is the only time I'll be loading up the compiled Purescript as it's the simplest example - but nearly all of the Pinto helpers are about making sure we can end up exposing a module that looks like the equivalent OTP erlang module.

The supervisor? More of the same

module BookSup where

-- [[ imports redacted ]]

startLink :: Effect Pinto.StartLinkResult
startLink = Sup.startLink "book_sup" init

init :: Effect SupervisorSpec
init = do
  connectionString <- BookConfig.connectionString
  webPort <- BookConfig.webPort
  pure $ buildSupervisor
                # supervisorStrategy OneForOne
                # supervisorChildren ( ( buildChild
                                       # childType Worker
                                       # childId "book_web"
                                       # childStart BookWeb.startLink  { webPort } )
                                       : 
                                       ( buildChild
                                       # childType Worker
                                       # childId "book_library"
                                       # childStart BookLibrary.startLink { connectionString } )
                                        : nil)

Sup.startLink is a Pinto helper that'll end up calling our usual supervisor:start_link under the hood, with our init function being invoked within the context of that created supervisor and returning a supervisor spec when invoked.

The supervisor spec structure is quite a complicated tangle of maps in Erlang, and while possible to directly represent these in Purescript with records, there are convenience builders/etc provided by Pinto to describe these specs in a more Purescripty and type safe manner - all of these values are pretty much relatable right back to the OTP documentation which is handy and intentional.

Now, let's break down one of these gen servers to see what we can see - we'll have to do this piece-meal as there is a lot to uncover. We'll be looking at BookLibrary.purs, invoked above in the second child of the supervision tree with BookLibrary.startLink and the code for which is shown below..


serverName :: ServerName State
serverName = ServerName "book_library"

type BookLibraryStartArgs = {
  connectionString :: ConnectionString
}

startLink :: BookLibraryStartArgs -> Effect StartLinkResult
startLink args =
  Gen.startLink serverName $ init args

So there is nothing too special about this, we're exporting a function called startLink that takes some configuration from the supervisor, and that calls into the Pinto function Gen.startLink with the serverName, and an init function to invoke within the context of the started GenServer along with those args. This isn't disimilar to how this looks in the Erlang world so far.

That serverName construct represents both the unique identifier for this started process, and also encapsulates the type of the 'state' that is held by the GenServer, and will be used in all interactions with the Gen module.

That init function?


init :: BookLibraryStartArgs -> Effect State
init args = do
  connection <- Redis.open args.connectionString
  pure $ { connection }

Takes place inside the context of the newly started GenServer, and is responsible for effectfully creating the initial state of the GenServer, which is just a Redis connection (We'll talk about that later). This is actually all we need to create a running GenServer as everything else is entirely optonal.

Obviously that's useless, so let's look at how we can externally call into this GenServer to do something useful


findAll :: Effect (List Book)
findAll = 
  Gen.doCall serverName \state@{ connection } -> do
    books <- Redis.findAll dbPrefix connection
    pure $ CallReply books state

We can export a plain ol' Purescript function called findAll that returns a plain ol' Effect producing a List of Book, and we can do the rest of the work by invoking a 'call' with the Gen module, by giving Gen.doCall our serverName construct, we are able to then provide a typed callback that will be invoked within the context of the gen-server as part of a handle_call, and therefore gain access to the state and return some books by calling into the Redis connection.

The original Erlang would of course look a little like this.


-export([start_link/1, 
         init/1,
         find_all/0]).

-record(args, {
    connection_string :: connection_string()
    }).

-record(state, {
    connection :: redis:connection()
  }).

find_all() ->
  gen_server:call({via, gproc, {n, l, ?MODULE}}, find_all).

start_link(Args) ->
  gen_server:start_link({via, gproc, {n, l, ?MODULE}}, ?MODULE, [Args], []).

init([#args { connection_string = ConnectionString }]) ->
  { ok, Connection } = redis:open(ConnectionString),
  {ok, #state { connection = Connection }}.

handle_call(find_all, _Sender, State = #state { connection = Connection }) ->
  { ok, Result } = redis:find_prefix(dbPrefix, Connection),
  { reply, Result, State }.

This is a bit unwieldy, lacks any notion of type safety across the calls being made, but is still pleasantly relatable to the Purescript variant.


serverName :: ServerName State
serverName = ServerName "book_library"

type BookLibraryStartArgs = {
  connectionString :: ConnectionString
}

type State = {
  connection :: RedisConnection
}

startLink :: BookLibraryStartArgs -> Effect StartLinkResult
startLink args =
  Gen.startLink serverName $ init args

init :: BookLibraryStartArgs -> Effect State
init args = do
  connection <- Redis.open args.connectionString
  pure $ { connection }

findAll :: Effect (List Book)
findAll = 
  Gen.doCall serverName \state@{ connection } -> do
    books <- Redis.findAll dbPrefix connection
    pure $ CallReply books state

Note: Wrapping up a connection behind a genserver is nearly always not the thing you want to do (effectively it introduces a read/write lock), but sample code gonna sample code.

Next up, we'll look at how we could use this GenServer from Stetson to provide a restful JSON API to our client.

Permalink

PurelyFunctional.tv Newsletter 340: Fewer side-effects is better than more

Issue 340 – August 19, 2019 · Archives · Subscribe

Clojure Tip 💡

Fewer side-effects is better than more

You probably know that if a function doesn’t have any side-effects, it is considered pure. When a function is a pure function, it can be used without regard to when it is called or how many times it is called. It makes parallel programming easier, it makes testing easier, it makes higher-order operations easier.

However, is it better to go from 2 side-effects down to 1? The benefit is not so clear. You’re not into pure function territory, but you’ve made things easier. What about from 10 down to 9? Certainly, if you can go from 10 to 0, you’re in the functional programming sweet spot. But do you get benefits for having fewer side-effects if you can’t get down to zero?

I believe that you do. It is better to have fewer side-effects, even if you can’t get down to zero. Zero is the magic number, but 1 is better than 2. The same benefits apply: a function is 1 side-effect is easier to make parallel, easier to test, and easier to use in higher-order operations, than a function with 2 side-effects. That means it’s worth it to refactor even if you can’t go all the way.

Awesome book 📖

Apollo: The race to the moon by Charles Murray and Catherine Bly Cox.

What an adventure the Apollo program was! This book was first published in 1989. It tells an amazing story of engineering skill and organizational genius that went into building, testing, and commanding the Apollo missions. It focuses much less on the astronauts and much more on the politics and engineering. I really loved to learn about how the major decisions were made and how mission control came to be.

One last point: I’m not sure about the pedigree of the authors—Murray also wrote The Bell Curve, which is a pseudoscientific racist screed. I didn’t know that before reading Apollo, but I wish I had. I didn’t notice anything amiss save for perhaps the description of the german rocket scientists and their relationships to the Nazi party. They received very little attention in the book, but if I were to read it again, I’d take everything about them with a grain of salt.

Book status 📖

The book’s title has changed and it now has a cover. It used to be called Taming Complex Software. Now it’s Grokking Simplicity.

It’s still not available yet. You can check out the cover, watch me read the first reviews that have come back (they’re positive!), and sign up to learn when it launches. Click here.

Currently recording 🎥

I am now recording a course called Property-Based Testing with test.check. Property-based testing (PBT) is a powerful tool for generating tests instead of writing them. You’ll love the way PBT makes you think about your system. And you can buy it now in Early Access. Of course, PF.tv members have had access since the beginning, along with all of the other courses.

New lessons this week include:

  1. Behind the scenes: shrinkage — in which we shine a microscope on the shrinkage process and watch it happen, step-by-step. Shrinkage is how test.check reports useful failing cases. Without shrinkage, the failing cases would be too big and random.
  2. Building complex generators: mutation testing — in which we generate values that are almost correct.

Last week I said that there were three more lessons for properties, and here I am showing you only two. I reviewed the notes for the third one and I had already covered everything. Soon we’re going to get into real testing examples. But before that, we have to talk about shrinkage.

I’ve made the first and fifth lessons free to watch. Go check them out.

Members already have access to the lessons. The Early Access Program is open. If you buy now, you will get the already published material and everything else that comes out at a serious discount. There are already 7 hours of video, and looking at my plan, this one might be 9-10 hours. But that’s just an estimate. It could be more or less. The uncertainty about it is why there’s such a discount for the Early Access Program.

Clojure Challenge 🤔

Last week’s challenge

The puzzle in Issue 339 was to write a symbolic differentiator.

You can check out the submissions here.

This week’s challenge

symbolic differentiation, pt 2

Now that we’ve got a basic symbolic differentiator (and the numeric one), we could simplify the expressions. SICP gives some basic simplifications.

  1. (+ 0 x) => x or additive identity
  2. (* 1 x) => x or multiplicative identity
  3. (+ 1 2) => 3 or constant simplification
  4. (* 3 2) => 6 or constant simplification

Here’s the relevant link into SICP, for reference.

The challenge this week is to integrate these easy simplifications into the differentiator. If you didn’t do last week’s challenge, grab someone else’s code and work from that.

As usual, please send me your implementations. I’ll share them all in next week’s issue. If you send me one, but you don’t want me to share it publicly, please let me know.

Rock on!
Eric Normand

The post PurelyFunctional.tv Newsletter 340: Fewer side-effects is better than more appeared first on PurelyFunctional.tv.

Permalink

Simple Autoencoder

Perfect mirror

If you look long enough into the autoencoder, it looks back at you.

The Autoencoder is a fun deep learning model to look into. Its goal is simple: given an input image, we would like to have the same output image.

It’s sort of an identity function for deep learning models, but it is composed of two parts: an encoder and decoder, with the encoder translating the images to a latent space representation and the encoder translating that back to a regular images that we can view.

We are going to make a simple autoencoder with Clojure MXNet for handwritten digits using the MNIST dataset.

The Dataset

We first load up the training data into an iterator that will allow us to cycle through all the images.

1
2
3
4
5
6
(def train-data (mx-io/mnist-iter {:image (str data-dir "train-images-idx3-ubyte")
                                   :label (str data-dir "train-labels-idx1-ubyte")
                                   :input-shape [784]
                                   :flat true
                                   :batch-size batch-size
                                   :shuffle true}))

Notice there the the input shape is 784. We are purposely flattening out our 28x28 image of a number to just be a one dimensional flat array. The reason is so that we can use a simpler model for the autoencoder.

We also load up the corresponding test data.

1
2
3
4
5
6
(def test-data (mx-io/mnist-iter {:image (str data-dir "t10k-images-idx3-ubyte")
                                  :label (str data-dir "t10k-labels-idx1-ubyte")
                                  :input-shape [784]
                                  :batch-size batch-size
                                  :flat true
                                  :shuffle true}))

When we are working with deep learning models we keep the training and the test data separate. When we train the model, we won’t use the test data. That way we can evaluate it later on the unseen test data.

The Model

Now we need to define the layers of the model. We know we are going to have an input and an output. The input will be the array that represents the image of the digit and the output will also be an array which is reconstruction of that image.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
(def input (sym/variable "input"))
(def output (sym/variable "input_"))

(defn get-symbol []
  (as-> input data
    ;; encode
    (sym/fully-connected "encode1" {:data data :num-hidden 100})
    (sym/activation "sigmoid1" {:data data :act-type "sigmoid"})

    ;; encode
    (sym/fully-connected "encode2" {:data data :num-hidden 50})
    (sym/activation "sigmoid2" {:data data :act-type "sigmoid"})

    ;; decode
    (sym/fully-connected "decode1" {:data data :num-hidden 50})
    (sym/activation "sigmoid3" {:data data :act-type "sigmoid"})

    ;; decode
    (sym/fully-connected "decode2" {:data data :num-hidden 100})
    (sym/activation "sigmoid4" {:data data :act-type "sigmoid"})

    ;;output
    (sym/fully-connected "result" {:data data :num-hidden 784})
    (sym/activation "sigmoid5" {:data data :act-type "sigmoid"})

    (sym/linear-regression-output {:data data :label output})))

From the model above we can see the input (image) being passed through simple layers of encoder to its latent representation, and then boosted back up from the decoder back into an output (image). It goes through the pleasingly symmetric transformation of:

784 (image) –> 100 –> 50 –> 50 –> 100 –> 784 (output)

We can now construct the full model with the module api from clojure-mxnet.

1
2
3
4
5
6
7
(def data-desc (first (mx-io/provide-data-desc train-data)))

(def model (-> (m/module (get-symbol) {:data-names ["input"] :label-names ["input_"]})
               (m/bind {:data-shapes [(assoc data-desc :name "input")]
                        :label-shapes [(assoc data-desc :name "input_")]})
               (m/init-params {:initializer  (initializer/uniform 1)})
               (m/init-optimizer {:optimizer (optimizer/adam {:learning-rage 0.001})})))

Notice that when we are binding the data-shapes and label-shapes we are using only the data from our handwritten digit dataset, (the images), and not the labels. This will ensure that as it trains it will seek to recreate the input image for the output image.

Before Training

Before we start our training, let’s get a baseline of what the original images look like and what the output of the untrained model is.

To look at the original images we can take the first training batch of 100 images and visualize them. Since we are initially using the flattened [784] image representation. We need to reshape it to the 28x28 image that we can recognize.

1
2
3
4
(def my-batch (mx-io/next train-data))
(def images (mx-io/batch-data my-batch))
(ndarray/shape (ndarray/reshape (first images) [100 1 28 28]))
(viz/im-sav {:title "originals" :output-path "results/" :x (ndarray/reshape (first images) [100 1 28 28])})

originals

We can also do the same visualization with the test batch of data images by putting them into the predict-batch and using our model.

1
2
3
4
5
;;; before training
 (def my-test-batch (mx-io/next test-data))
 (def test-images (mx-io/batch-data my-test-batch))
 (def preds (m/predict-batch model {:data test-images} ))
 (viz/im-sav {:title "before-training-preds" :output-path "results/" :x (ndarray/reshape (first preds) [100 1 28 28])})

before-training-preds

They are not anything close to recognizable as numbers.

Training

The next step is to train the model on the data. We set up a training function to step through all the batches of data.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
(def my-metric (eval-metric/mse))

(defn train [num-epochs]
  (doseq [epoch-num (range 0 num-epochs)]
    (println "starting epoch " epoch-num)
    (mx-io/do-batches
     train-data
     (fn [batch]
       (-> model
           (m/forward {:data (mx-io/batch-data batch) :label (mx-io/batch-data batch)})
           (m/update-metric my-metric (mx-io/batch-data batch))
           (m/backward)
           (m/update))))
    (println "result for epoch " epoch-num " is " (eval-metric/get-and-reset my-metric))))

For each batch of 100 images it is doing the following:

  • Run the forward pass of the model with both the data and label being the image
  • Update the accuracy of the model with the mse (mean squared error metric)
  • Do the backward computation
  • Update the model according to the optimizer and the forward/backward computation.

Let’s train it for 3 epochs.

1
2
3
4
5
6
starting epoch  0
result for epoch  0  is  [mse 0.06460866]
starting epoch  1
result for epoch  1  is  [mse 0.033874355]
starting epoch  2
result for epoch  2  is  [mse 0.027255038]

After training

We can check the test images again and see if they look better.

1
2
3
4
5
;;; after training
(def my-test-batch (mx-io/next test-data))
(def test-images (mx-io/batch-data my-test-batch))
(def preds (m/predict-batch model {:data test-images} ))
(viz/im-sav {:title "after-training-preds" :output-path "results/" :x (ndarray/reshape (first preds) [100 1 28 28])})

after-training-preds

Much improved! They definitely look like numbers.

Wrap up

We’ve made a simple autoencoder that can take images of digits and compress them down to a latent space representation the can later be decoded into the same image.

If you want to check out the full code for this example, you can find it here.

Stay tuned. We’ll take this example and build on it in future posts.

Permalink

Ep 042: What Does It Mean to Be 'Data-Oriented'?

Each week, we answer a different question about Clojure and functional programming.

If you have a question you’d like us to discuss, tweet @clojuredesign, send an email to feedback@clojuredesign.club, or join the #clojuredesign-podcast channel on the Clojurians Slack.

This week, the question is: “What does it mean to be ‘data-oriented’?” We merge together different aspects of Clojure’s data orientation, and specify which of those help make development more pleasant.

Selected quotes:

  • “Clojure has the corner on data.”
  • “Other languages have data too, it’s just locked in little cages.”
  • “Data is inert, it can’t harm you.”
  • “Because Clojure is expressed in its own data structures, and those structures are simple, that makes Clojure syntax simple.”
  • “Find a good way to represent the information that you want to work with, in a way that feels appropriate for the subject matter.”
  • “If you find a good way of representing your information, that representation tends to be pretty stable. All of the change is in the functions you use to work with it.”

Permalink

Getting Started With Clojure CLI Tools

Clojure Command Line Interface (CLI) tools provide a fast way for developers to get started with Clojure and simplify an already pretty simple experience. With tools.deps it also provides a more flexible approach to including libraries, including the use of code from a specific commit in a Git repository.

Practicalli Clojure 35 - Clojure CLI tools - an introduction is a video of a live broadcast of this content (inclucing typos)

Clojure CLI tools provide:

  • Running an interactive REPL (Read-Eval-Print Loop)
  • Running Clojure programs
  • Evaluating Clojure expressions
  • Managing dependencies via tools.deps

Clojure CLI tools allow you to use other libraries to, referred to as dependencies or ‘deps’. These may be libraries you are writing locally, projects in git (e.g. on GitHub) or libraries published to Maven Central or Clojars.

The Clojure CLI tools can cover the essential features of Clojure Build tools Leiningen and Boot, but are not designed as a complete replacement. Both these build tools are mature and may have features you would otherwise need to script in Clojure CLI tools.

This article is a follow on from new Clojure REPL Experience With Clojure CLI Tools and Rebel Readline

Getting started

Clojure is packaged as a complete library, a JVM JAR file, that is simply included in the project like any other library you would use. You could just use the Java command line, but then you would need to pass in quite a few arguments as your project added other libraries.

Clojure is a hosted language, so you need to have a Java runtime environment (Java JRE or SDK) and I recommend installing this from Adopt OpenJDK. Installation guides for Java are covered on the ClojureBridge London website

The Clojure.org getting started guide covers instructions for Linux and MacOXS operating systems. There is also an early access release of clj for windows

Basic usage

The installation provides the command called clojure and a wrapper called clj that provides a readline program called rlwrap that adds completion and history once the Clojure REPL is running.

Use clj when you want to run a repl (unless you are using rebel readline instead) and clojure for everything else.

Start a Clojure REPL using the clj command in a terminal window. This does not need to be in a directory containing a Clojure project for a simple REPL.

1
clj

A Clojure REPL will now run. Type in a Clojure expression and press Return to see the result

Clojure CLI Tools REPL

Exit the REPL by typing Ctrl+D (pressing the Ctrl and D keys at the same time).

Run a Clojure program in a the given file. This would be useful if you wanted to run a script or batch jobs.

1
clojure script.clj

Aliases can be added that define configurations for a specific build task:

1
clojure -A:my-task

You can use and legal Clojure keyword name for an alias and include multiple aliases with the clojure command. For example in this command we are combining three aliases:
clojure -A:my-task:my-build:my-prefs

What version of Clojure CLI tools are installed?

The deps.edn file allows you to specify a particular version of the Clojure language the REPL and project use. You can also evaluate *clojue-version* in a REPL to see which version of the Clojure language is being used.

clj -Sdescribe will show you the version of the Clojure CLI tools that is currently installed.

clojure cli tools - describe install version

clj -Sverbose will also show the version of Clojure CLI tools used before it runs a REPL.

deps.edn

deps.edn is a configuration file using extensible data notation (edn), the language that is used to define the structure of Clojure itself.

Configuration is defined using a map with top-level keys for :deps, :paths, and :aliases and any provider-specific keys for configuring dependency sources (e.g. GitHub, GitLab, Bitbucket).

~/.clojure/deps.edn for global configurations that you wish to apply to all the projects you work with

project-directory/deps.edn for project specific settings

The installation directory may also contain a deps.edn file. On my Ubuntu Linux system this location is /usr/local/lib/clojure/deps.edn and contains the following configuration.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
:paths ["src"]
:deps {
org.clojure/clojure {:mvn/version "1.10.1"}
}
:aliases {
:deps {:extra-deps {org.clojure/tools.deps.alpha {:mvn/version "0.6.496"}}}
:test {:extra-paths ["test"]}
}
:mvn/repos {
"central" {:url "https://repo1.maven.org/maven2/"}
"clojars" {:url "https://repo.clojars.org/"}
}
}

Note: the install deps.edn is now depreciated and will not be included in a future version of the Clojure CLI tools.

The deps.edn files in each of these locations (if they exist) are merged to form one combined dependency configuration. The merge is done in the order above install/config/local, last one wins. The operation is essentially merge-with merge, except for the :paths key, where only the last one found is used (they are not combined).

You can use the -Sverbose option to see all of the actual directory locations.

Much more detail is covered in the Clojure.org article - deps and cli

Using Libraries - deps.edn

deps.edn file in the top level of your project can be used to include libraries in your project. These may be libraries you are writing locally, projects in git (e.g. on GitHub) or libraries published to Maven Central or Clojars.

Include a library by providing its name and other aspects like version. This information can be found on Clojars if the library is published there.

Libraries as JAR files will be cached in the $HOME/.m2/repository directory.

Example clj-time

Declare clojure.java-time as a dependency in the deps.edn file, so Clojure CLI tools can downloaded the library and add it to the classpath.

1
2
3
{:deps
{org.clojure/clojure {:mvn/version "1.10.1"}
clojure.java-time {:mvn/version "0.3.2"}}}

Writing code

For larger projects you should definately find an editor you find productive and has great CLojure support. You can write code in the REPL and you can just run a specific file of code, if you dont want to set up a full project.

Create a directory what-time-is-it.

Create a deps.edn file in this directory with the following code:

1
2
3
{:paths ["src"]
:deps {org.clojure/clojure {:mvn/version "1.10.1"}
clojure.java-time {:mvn/version "0.3.2"}}}

Create a src directory and the source code file src/practicalli/what_time_is_it.clj which contains the following code:

1
2
3
4
5
6
(ns practicalli.what-time-is-it
(:require [java-time :as time]))
(defn -main []
(println "The time according to Clojure java-time is:"
(time/local-date-time)))

The code has a static entry point named -main that can be called from Clojure CLI tools. The -m option defines the main namespace and by default the -main function is called from that namespace. So the Clojure CLI tools provide program launcher for a specific namespace:

1
2
3
clojure -m practicalli.what-time-is-it
The time according to Clojure java-time is: #object[java.time.LocalDateTime 0x635e9727 2019-08-05T16:04:52.582]

Using libraries from other places

With deps.edn you are not limited to using just dependencies from JAR files, its much easier to pull code from anywhere.

TODO: Expand on this section in another article with some useful examples

rebl readline

Rebel readline enhances the REPL experience by providing multi-line editing with auto-indenting, language completions, syntax highlighting and function argument hints as you code.

clj-new

clj-new is a tool to generate new projects from its own small set of templates. You can also create your own clj-new templates. It is also possible to generate projects from Leiningen or Boot templates, however, this does not create a deps.edn file for Clojure CLI tools, it just creates the project as it would from either Leiningen or Boot.

Add clj-new as an alias in your ~/.clojure/deps.edn like this:

1
2
3
4
5
6
7
8
{
:aliases
{:new {:extra-deps {seancorfield/clj-new
{:mvn/version "0.7.6"}}
:main-opts ["-m" "clj-new.create"]}}
}

Create a Clojure CLI tools project using the clj-new app template

1
2
3
clj -A:new app myname/myapp
cd myapp
clj -m myname.myapp

The app template creates a couple of tests to go along with the sample code. We can use the cognitec test runner to run these tests using the :test alias

1
clj -A:test:runner

clj-new currently has the following built-in templates:

app – a deps.edn project with sample code, tests and the congnitect test runner, clj -A:test:runner. This project includes :gensys directive, so can be run as an application on the command line via clj -m
lib – the same as the app template, but without the :gensys directive as this is mean to be a library.
template – the basis for creating your own templates.

figwheel-main

Use the figwheel-main template to create a project for a simple Clojurescript project, optionally with one or reagent, rum or om libraries.

Defining aliases

An alias is a way to add optional configuration to your project which is included when you use the alias name when running clojure or clj.

We will cover examples of using aliases as we discover more about Clojure CLI tools. For now, take a look at Clojure CLI and deps.edn - video by Sean Corfield

Multiple versions of Clojure CLI tools

Installing CLI tools downloads a tar file that contains the installation files, the executables, man pages, a default deps.edn file, an example-deps.edn and a Jar file.

Clojure cli tools - install - tar contents

The jar file is installed in a directory called libexec is not removed when installing newer versions of the Clojure CLI tools, so you may find multiple versions inside the libexec directory.

Clojure CLI tools - install - multiple versions

Summary

Despite the seemingly stripped-down set of options available in deps.edn (just :paths, :deps, and :aliases), it turns out that the :aliases feature really provides all you need to bootstrap a wide variety of build tasks directly into the clojure command. The Clojure community is building lots of tools on top of Clojure CLI tools to provide richer features that can simply be added as an alias.

What I really like best about this approach is that I can now introduce new programmers to Clojure using command line conventions that they are likely already familiar with coming from many other popular languages like perl, python, ruby, or node.

References

Thank you.
@jr0cket

Permalink

We need to talk about JSON

Like many in the Clojure community, I’ve considered JSON as a rather horrible syntax (compared to the obvious superiority of EDN).

As a configuration format, JSON is used almost everywhere: npm’s package.json, AWS CloudFormation, Hashicorp Terraform, Open API, .NET, the list is endless.

For me, the worst issue is the exclusion of comments. Of all the problems we’re leaving to future generations, the complete lack of explanation in the configuration files of future legacy software systems must be among the most irksome. Understanding is good, clear explanations are good; discouraging the use of helpful and clarifying comments is inexcusable.

The situation would improve if developers stopped adopting JSON for configuration files - and there are JSON-like formats available that are much better suited to this task, such as Hjson, Jsonnet, YAML and others. The downside of these is that they require special format reading software which is rarely built-in to languages. At JUXT, we use Aero, which we consider to be the best of all worlds.

Change the perspective

Let’s start again and think about what JSON is:

{
"list": ["string" {"map": {"key": "value"}}]
}

It’s maps, lists, maps-of-lists, lists-of-maps, lists-of-lists, maps-of-maps.

In fact, from a Clojure perspective, things couldn’t be better: Clojure is, hands down, the most adept language at processing this near-ubiquitous format. Of course, we’re a little bit biased at JUXT!

Back when everything was XML, and everyone wrote Java (or C#), I remember working on a large project. We spent weeks (literally, weeks!) working on serialisation and de-serialisation of data from XML into Java and back. I/O is a critical part of any program. Getting data in and out of a program shouldn’t be that hard, and with Clojure and JSON it isn’t.

Data format Ideal processing language

CSV

SQL

SGML

DSSSL

XML

XSLT

JSON

CLJ/S (Clojure)

It’s almost as if, behind the scenes, there’s been this Grand Lisp Conspiracy to get everyone programming in Lisp, whether or not they realise it.

guy steele

By disguising itself with some ornamental punctuation, JSON has infected the whole world with Lisp.

JSON Schema

JSON Schema is an emerging standard which is used to define the structure of JSON documents. It’s mostly used to validate that JSON documents conform to expectations. However, since JSON allows anyone to define new keywords and add them to 'vocabularies', I believe JSON Schema should be thought of as a more general way of specifying meta-data.

When combined with meta-data, data takes on some magical super-powers.

react jsonschema form

One example is Mozilla’s React JSON Schema Form which demonstrates a React component that can dynamically produce forms via introspecting JSON data and its schema.

I believe there will be plenty of other uses of JSON Schema in the future.

json schema

One example I’m particularly interested in is the use of JSON Schema as a pull-syntax for our Crux database. JSON Schema allows you to declare the structure of the document you want to 'pull' out of Crux, Crux will perform the necessary graph queries against its documents. The result documents will successfully validate against the schema, as the query is the schema.

jinx

I wanted to explore the potential of JSON Schema to provide a way of accessing metadata about documents in order to make the data more useful and powerful.

Thanks to some help from some colleagues (in particular, Shivek Khurana and Sunita Kawane), these explorations have resulted in a new Clojure(Script) library called jinx, which provides schema-powered capabilities, which of course includes validation. We’re hoping this can be used for a variety of applications, not just to validate data documents but to access the annotations made via the JSON Schema document.

A quick example:

(juxt.jinx.validate/validate
  {"firstName" "John"
   "lastName" "Doe"
   "age" 21}
  (juxt.jinx.schema/schema
   {"$id" "https://example.com/person.schema.json"
    "$schema" "http://json-schema.org/draft-07/schema#"
    "title" "Person"
    "type" "object"
    "properties"
    {"firstName"
     {"type" "string"
      "description" "The person's first name."}
     "lastName"
     {"type" "string"
      "description" "The person's last name."}
     "age" {"description" "Age in years which must be equal to or greater than zero."
            "type" "integer"
            "minimum" 0}}}))

=>

{:instance {"firstName" "John", "lastName" "Doe", "age" 21},
 :annotations
 {"title" "Person",
  :properties
  {"firstName" {"description" "The person's first name."},
   "lastName" {"description" "The person's last name."},
   "age"
   {"description"
    "Age in years which must be equal to or greater than zero."}}},
 :type "object",
 :valid? true}

Our jinx library is still in ALPHA status but it would be great to hear all feedback and comments. It’s available on GitHub at https://github.com/juxt/jinx.

Permalink

7 Easy functional programming techniques in Go

There is a lot of hype around functional programming(FP) and a lot of cool kids are doing it but it is not a silver bullet. Like other programming paradigms/styles, functional programming also has its pros and cons and one may prefer one paradigm over the other. If you are a Go developer and wants to venture into functional programming, do not worry, you don't have to learn functional programming oriented languages like Haskell or Clojure(or even Scala or JavaScript though they are not pure functional programming languages) since Go has you covered and this post is for you.

If you are looking for functional programming in Java then check this out

I'm not gonna dive into all functional programming concepts in detail, instead, I'm gonna focus on things that you can do in Go which are in line with functional programming concepts. I'm also not gonna discuss the pros and cons of functional programming in general.

What is functional programming?

As per Wikipedia,

Functional programming is a programming paradigm—a style of building the structure and elements of computer programs—that treats computation as the evaluation of mathematical functions and avoids changing-state and mutable data.

Hence in functional programming, there are two very important rules

  • No Data mutations: It means a data object should not be changed after it is created.
  • No implicit state: Hidden/Implicit state should be avoided. In functional programming state is not eliminated, instead, its made visible and explicit

This means:

  • No side effects: A function or operation should not change any state outside of its functional scope. I.e, A function should only return a value to the invoker and should not affect any external state. This means programs are easier to understand.
  • Pure functions only: Functional code is idempotent. A function should return values only based on the arguments passed and should not affect(side-effect) or depend on global state. Such functions always produce the same result for the same arguments.

Apart from these there are functional programming concepts below that can be applied in Go, we will touch upon these further down.

Using functional programming doesn't mean its all or nothing, you can always use functional programming concepts to complement Object-oriented or imperative concepts in Go. The benefits of functional programming can be utilized whenever possible regardless of the paradigm or language you use. And that is exactly what we are going to see.

Functional programming in Go

Golang is a multi-paradigm language so let us see how we can apply some of the functional programming concepts above in Go.

First-class and higher-order functions

First-class functions(function as a first-class citizen) means you can assign functions to variables, pass a function as an argument to another function or return a function from another. Go supports this and hence makes concepts like closures, currying, and higher-order-functions easy to write.

A function can be considered as a higher-order-function only if it takes one or more functions as parameters or if it returns another function as a result.

In Go, this is quite easy to do

func main() {
    var list = []string{"Orange", "Apple", "Banana", "Grape"}
    // we are passing the array and a function as arguments to mapForEach method.
    var out = mapForEach(list, func(it string) int {
        return len(it)
    })
    fmt.Println(out) // [6, 5, 6, 5]

}

// The higher-order-function takes an array and a function as arguments
func mapForEach(arr []string, fn func(it string) int) []int {
    var newArray = []int{}
    for _, it := range arr {
        // We are executing the method passed
        newArray = append(newArray, fn(it))
    }
    return newArray
}

Closures and currying are also possible in Go

// this is a higher-order-function that returns a function
func add(x int) func(y int) int {
    // A function is returned here as closure
    // variable x is obtained from the outer scope of this method and memorized in the closure
    return func(y int) int {
        return x + y
    }
}

func main() {

    // we are currying the add method to create more variations
    var add10 = add(10)
    var add20 = add(20)
    var add30 = add(30)

    fmt.Println(add10(5)) // 15
    fmt.Println(add20(5)) // 25
    fmt.Println(add30(5)) // 35
}

There are also many built-in higher-order-functions in Go standard libraries. There are also some functional style libraries like this and this offering map-reduce like functional methods in Go.

Pure functions

As we saw already a pure function should return values only based on the arguments passed and should not affect or depend on global state. It is possible to do this in Go easily.

This is quite simple, take the below this is a pure function. It will always return the same output for the given input and its behavior is highly predictable. We can safely cache the method if needed.

func sum(a, b int) int {
    return a + b
}

If we add an extra line in this function, the behavior becomes unpredictable as it now has a side effect that affects an external state.

var holder = map[string]int{}

func sum(a, b int) int {
    c := a + b
    holder[fmt.Sprintf("%d+%d", a, b)] = c
    return c
}

So try to keep your functions pure and simple.

Recursion

Functional programming favors recursion over looping. Let us see an example for calculating the factorial of a number.

In traditional iterative approach:

func factorial(num int) int {
    result := 1
    for ; num > 0; num-- {
        result *= num
    }
    return result
}

func main() {
    fmt.Println(factorial(20)) // 2432902008176640000
}

The same can be done using recursion as below which is favored in functional programming.

func factorial(num int) int {
    if num == 0 {
        return 1
    }
    return num * factorial(num-1)
}
func main() {
    fmt.Println(factorial(20)) // 2432902008176640000
}

The downside of the recursive approach is that it will be slower compared to an iterative approach most of the times(The advantage we are aiming for is code simplicity and readability) and might result in stack overflow errors since every function call needs to be saved as a frame to the stack. To avoid this tail recursion is preferred, especially when the recursion is done too many times. In tail recursion, the recursive call is the last thing executed by the function and hence the functions stack frame need not be saved by the compiler. Most compilers can optimize the tail recursion code the same way iterative code is optimized hence avoiding the performance penalty. Go compiler, unfortunately, does not do this optimization.

Now using tail recursion the same function can be written as below, but Go doesn't optimize this, though there are workarounds, still it performed better in benchmarks.

func factorialTailRec(num int) int {
    return factorial(1, num)
}

func factorial(accumulator, val int) int {
    if val == 1 {
        return accumulator
    }
    return factorial(accumulator*val, val-1)
}

func main() {
    fmt.Println(factorialTailRec(20)) // 2432902008176640000
}

I ran some benchmarks with all 3 approaches and here is the result, as you can see looping is still the most performing followed by the tail recursion.

goos: linux
goarch: amd64
BenchmarkFactorialLoop-12           100000000           11.7 ns/op         0 B/op          0 allocs/op
BenchmarkFactorialRec-12            30000000            52.9 ns/op         0 B/op          0 allocs/op
BenchmarkFactorialTailRec-12        50000000            44.2 ns/op         0 B/op          0 allocs/op
PASS
ok      _/home/deepu/workspace/deepu105.github.io/temp  5.072s
Success: Benchmarks passed.

Consider using recursion when writing Go code for readability and immutability, but if performance is critical or if the number of iterations will be huge use standard loops.

Lazy evaluation

Lazy evaluation or non-strict evaluation is the process of delaying evaluation of an expression until it is needed. In general, Go does strict/eager evaluation but for operands like && and || it does a lazy evaluation. We can utilize higher-order-functions, closures, goroutines, and channels to emulate lazy evaluations.

Take this example where Go eagerly evaluates everything.

func main() {
    fmt.Println(addOrMultiply(true, add(4), multiply(4)))  // 8
    fmt.Println(addOrMultiply(false, add(4), multiply(4))) // 16
}

func add(x int) int {
    fmt.Println("executing add") // this is printed since the functions are evaluated first
    return x + x
}

func multiply(x int) int {
    fmt.Println("executing multiply") // this is printed since the functions are evaluated first
    return x * x
}

func addOrMultiply(add bool, onAdd, onMultiply int) int {
    if add {
        return onAdd
    }
    return onMultiply
}

This will produce the below output and we can see that both functions are executed always

executing add
executing multiply
8
executing add
executing multiply
16

We can use higher-order-functions to rewrite this into a lazily evaluated version

func add(x int) int {
    fmt.Println("executing add")
    return x + x
}

func multiply(x int) int {
    fmt.Println("executing multiply")
    return x * x
}

func main() {
    fmt.Println(addOrMultiply(true, add, multiply, 4))
    fmt.Println(addOrMultiply(false, add, multiply, 4))
}

// This is now a higher-order-function hence evaluation of the functions are delayed in if-else
func addOrMultiply(add bool, onAdd, onMultiply func(t int) int, t int) int {
    if add {
        return onAdd(t)
    }
    return onMultiply(t)
}

This outputs the below and we can see that only required functions were executed

executing add
8
executing multiply
16

There are also other ways of doing it using Sync & Futures like this and using channels and goroutines like this. Doing Lazy evaluations in Go might not be worth the code complexity most of the times, but if the functions in question are heavy in terms of processing then its is absolutely worth it to lazy evaluate them.

Type system

Go has a strong type system and also has pretty decent type inference. The only thing missing compared to other functional programming languages are something like case classes and pattern matching.

Referential transparency

From Wikipedia:

Functional programs do not have assignment statements, that is, the value of a variable in a functional program never changes once defined. This eliminates any chances of side effects because any variable can be replaced with its actual value at any point of execution. So, functional programs are referentially transparent.

Unfortunately, there are not many ways to strictly limit data mutation in Go, however by using pure functions and by explicitly avoiding data mutations and reassignment using other concepts we saw earlier this can be achieved. Go by default passes variables by value, except for slices and maps. So, avoid passing them by reference(using pointers) as much as possible.

For example, the below will mutate external state as we are passing a parameter by reference and hence doesn't ensure referential transparency

func main() {
    type Person struct {
        firstName string
        lastName  string
        fullName  string
        age       int
    }
    var getFullName = func(in *Person) string {
        in.fullName = in.firstName + in.lastName // data mutation
        return in.fullName
    }

    john := Person{
        "john", "doe", "", 30,
    }

    fmt.Println(getFullName(&john)) // johndoe
    fmt.Println(john) // {john doe johndoe 30}
}

If we pass parameters by the value we can ensure referential transparency even if there is an accidental mutation of passed data within the function

func main() {
    type Person struct {
        firstName string
        lastName  string
        fullName  string
        age       int
    }
    var getFullName = func(in Person) string {
        in.fullName = in.firstName + in.lastName
        return in.fullName
    }

    john := Person{
        "john", "doe", "", 30,
    }

    fmt.Println(getFullName(john))
    fmt.Println(john)
}

We cannot rely on this when passed parameters are maps or slices.

Data structures

When using functional programming techniques it is encouraged to use functional data types such as Stacks, Maps and Queues.
Hence maps are better than arrays or hash sets in functional programming as data stores.

Conclusion

This is just an introduction for those who are trying to apply some functional programming techniques in Go. There are a lot more that can be done in Go and with the addition of generics in the next major version, this should be even easier. As I said earlier functional programming is not a silver bullet but it offers a lot of useful techniques for more understandable, maintainable and testable code. It can co-exist perfectly well with imperative and object-oriented programming styles. In fact, we all should be using the best of everything.

I hope you find this useful. If you have any question or if you think I missed something please add a comment.

If you like this article, please leave a like or a comment.

You can follow me on Twitter and LinkedIn.

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.