What is Immutability?

Functional Programmers will talk about immutable values. What do they mean? How can you write software where none of the values change?

Transcript

Eric Normand: What do we mean when functional programmers talk about immutability? Do things really have to never change?

Hi, my name is Eric Normand. These are my thoughts on functional programming. Immutability is one of those things that comes up a lot when people are talking about functional programming.

I want to talk about some of the more practical aspects of it, not just the theoretical. The theoretical says that an immutable value cannot be changed. Now, in practice, there are several ways that you can enforce that.

Sometimes you don’t have immutability in your language so you have to do something else. It’s really just how you interpret that cannot. One thing is, the language could enforce it or the particular object, the data itself, might not have any methods on it that allow it to be mutated.

That is one way of enforcing this discipline, this rule, that things shouldn’t be mutated, they can’t be mutated. Another way is you could just have a, let’s call it, a developer policy that says, “We don’t mutate.”

Maybe the object is mutable. You could but you don’t. You just apply the rule, but you have no help from the language, your compiler, or anything. That’s the practice of it.

If you’re in JavaScript and you have some mutable thing like a JavaScript object or you have a JavaScript array, those are mutable. You could practice immutability by, in practice, never changing them even though you could. Given that you could change them, there are…

You need to make modified versions of these things. Even if you aren’t changing the thing, you want to be able to add a new element to your object or to your array. How do you do this?

There’s actually two and a half disciplines for doing this. The first one is called copy-on-write. This means whenever you need to make a modification to an object, you don’t know who else has a pointer to that object so you don’t want to change that because then you’re breaking your rule of immutability. You make a copy and then change the copy.

You’re changing it. You’re using mutation, but no one has a reference to it yet. No one has seen it but you. Then, once you give it out now, boom, the immutability rule is now enforced and no one else can change it. You’re not going to change it and you’re saying no one else can change it.

The rule has to be applied unilaterally. Once you give away a pointer to it, you cannot change it. If you get something that you didn’t create yourself, meaning you know exactly the code path that it went through to get to you, you can’t change it either because someone else might already have it.

Those are the two rules. If you get something, you can’t change it. If you need to make a change, you have to make a copy and then change that copy that you control. That’s called copy-on-write.

There’s another one called copy-on-read. This one is, if you get something from someone and you’re not sure of this library, this API that you’re using, believes the same things you do about immutability.

If it’s even implemented correctly, if they enforce the rule properly, so you don’t believe that this thing, this system, is immutable. What do you do? You make your own copy of the thing.

That means that the library can modify the one that it gave you all it wants. What’s important is that the one that you keep is under your control and you’re only going to pass it to things that you believe will be immutable.

Let’s talk about the rules of this. If you get something from an untrusted source and you don’t trust that it enforces immutability, you make a copy and you throw away the old one. You throw away the one they gave you. You just immediately make a copy and that’s the one you keep.

Then, if you’re going to give your ostensibly immutable thing to something else, you don’t trust that it’s going to maintain your discipline of immutability, you have to make a copy before and give them the copy.

You can say like, “You can do whatever you want with this. I’ve got the original copy. You can tear that up. You can write all over it. I’ve got the original.”

Those are the two main disciplines. If you’re living in a language or a runtime that allows for immutability of your main objects, you’re going to have to do those kinds of things, those two disciplines.

One is defensive and one is more offensive. You’re going to probably have to do both depending on the situation. If you’re dealing with untrusted code, you want to do copy-on-read.

If you’re dealing with your own objects that you maintain, you’re going to do copy-on-write. Because, within your own code, you trust that you’re being immutable, that you’re enforcing your rule of immutability. You don’t have to copy every time you read. You only have to copy when you need to modify it.

The half one, I said there were two and a half. The other one is called append-only. This is only a half because it doesn’t really apply in most cases. It’s just not applicable. You can’t use it.

There are certain data structures that let you apply an append-only discipline which means you don’t modify the thing, you only add to the thing in a way that doesn’t modify the original.

Just as an example, if you have a stack and let’s say you only push, you never pop. It’s append-only. It’s push-only in this case. If you have this stack, things can push without modifying anything underneath the top of the stack. You can share this thing around because the stack is only growing. None of the old stuff is being modified.

It’s immutable in that way. Now, if it’s an immutable implementation of a stack, meaning every time you push, you’re actually creating a copy, then that’s even better. I’m just talking about a regular immutable stack where you’re adding stuff.

All you can do is add. You can’t remove and you can’t modify anything that’s on there already. That has immutability to it. It’s only growing in a known way without modifying the old stuff.

In theory, if you had a pointer to this stack up to this height, you know that stuff is only getting added on top. I can still read this safely and be sure it’s never going to change.

Of course, this isn’t just another discipline. This is append-only. Does your language help you do this? Does it have a pop method? Does your stack have a pop method that someone might accidentally call that doesn’t realize that it shouldn’t do that in your system?

That’s how much the language affords doing immutability. Some languages like Clojure enforce these automatically. It does not enforce copy-on-read. It doesn’t need to because the objects are immutable. There are no methods on them that can change the state inside the objects. They are immutable.

It does do a copy-on-write. If you want to add a key value to a HashMap, it actually makes a new HashMap with all the existing key values plus the new one that you’re adding. It does that very efficiently because it’s going to share most of what already exists with the old copy.

You can have both copies in memory and even if there’s a thousand keys and values in that HashMap and you add a new one, in theory, you have two HashMaps with 1000 key values and a 1001 key values.

Most if it is shared between the two. Only a very small amount of memory has to be duplicated to have both. That’s called persistent data structures. What persists is the structure of the preexisting HashMap when you make a copy to a new HashMap.

That’s immutability. I hope that explains what it is and how it’s done. I didn’t really explain very much about why, but that could be a topic for another of these episodes.

Thank you. See you later.

The post What is Immutability? appeared first on LispCast.

Permalink

Static Methods, Companion Objects, and Testing

I’ve been working with Kotlin a ton recently, both at work and for fun. It’s a fantastic language on the JVM that combines almost all of my favorite Scala features and adds in several new features as well. If you haven’t given it a shot yet and you’re a Java reader, give it a try! You’ll love how expressive it can be and how simple and concise some of the code can be. If you’re a Scala reader, you’ll feel familiar with a lot of the syntax and ideas (Objects, replacing statics for example). One thing Kotlin does that still feels a bit weird is it promotes the use of companion objects. A companion object sits inside a class definition and houses all the static methods which that class will own. Let’s convert a piece of Java code to Kotlin so you can see what I mean.

class Foo {
  public Foo() {}
  public static String getBar() { return "Bar"; }

  public boolean amIFoo() { return true; }
}

 

We define a class with two methods, one of which is static. The first method, the static one, simple returns a string and the second method, the instance method just returns true. Since it’s an instance method, we’d have to new up a Foo to be able to call it. Let’s convert this to Kotlin.

class Foo() {
  companion object {
    fun getBar(): String { return "Bar" }
  }

  fun amIFoo(): Boolean { return true }
}

You’ll likely notice some differences:

  1. We don’t have a constructor in our Kotlin class. Well… we do have a primary constructor and it’s this defined by the () in this block class Foo().
  2. We have a companion object. We’ll talk more about that in a moment.
  3. We don’t need semi-colons.
  4. We define functions differently, specifically using the fun keyword. That’s because Kotlin is fun! 😉

It’s worth mentioning that we can actually simplify our Kotlin code to be even more idiomatic. Let’s do that first and then we’ll talk about the companion object.

class Foo() {
  companion object {
    fun getBar() = "Bar"
  }

  fun amIFoo() = true
}

The only thing I wanted to point out here is that simple functions (as in, one-liner function bodies) can be inlined instead of using {} expression blocks. Anyways, Companion Objects!

Companion Objects

Companion Objects are how you define static methods in Kotlin. A companion object is built from two Kotlin keywords: companion and object. To fully understand a companion object, we must first understand an object. The object keyword starts an object declaration. It’s important to note that this is a declaration and not an expression, so it cant be used on the value side of an assignment. That being said, object allows you to define a Singleton object. So then what is a companion? A companion is paired with a class and is used to indicate that the object we’re about to define can be called simply by using the class qualifier that this object is nested in. This means that in our above example we can call Foo.getBar() in Kotlin. Why did I say “in Kotlin?” That’s because in Java, you’ll actually reference the companion object’s functions like so: Foo.Companion.getBar(). There’s arguments and merits to this syntax (but why would you write Java when you could write Kotlin?) and I won’t go into those here but I will say that I’m a fan of it, because it explicitly calls out the Kotlin interop from your Java code.



Companion Objects – Why?

So why in the world do we have companion objects? Kotlin is an extremely object oriented language (with some “functional” love as well). The language tends to favor the use of classes and objects over random functions so much that it even allows you to extend existing objects and provide new methods onto them (at compile time; they compile out to statics). Coming from a heavy functional background, I really started to question why this was but the more I looked into it the more I saw similarities between functional languages (like Clojure) and the heavy use of singleton objects like in Scala and Kotlin. In most of the functional languages, you have namespaces. In Kotlin and Scala, you have objects. These are different but they both provide one similar thing — They’re both a target for mocks when testing. In Clojure, you can easily mock out a namespace. In Kotlin, you can easily mock out an object. This allows you to write static methods with all the benefits of them (they can be static at run-time) but to be able to mock them for testing purposes. With a library like Mockito (there’s even Kotlin bindings and extensions), testing (what would be) statics is a complete breeze.

Permalink

Something NLP-ish in Clojure

I started writing this article almost 2 years ago. While some things have changed (interests, time, priorities), I thought about this the other day and thought it was too neat to just let die. I came back into this article today and tried to fill out a few of the lesser detailed areas and decided to just hit publish to ensure the code, thoughts, and process was shared with you all. Sorry that it’s not of my usual quality.

Hey all! I haven’t written anything in awhile and wanted to jump back in with something fun and cool, but first an apology — I haven’t had much time to write anything due to me losing a bit of interest in Clojure. I’ve had a few issues with the language lately (another post, maybe?) and decided to explore Rust with the hopes of my complaints being alleviated. Sadly, the grass isn’t always greener and I’m deciding to stick with Clojure for the time being.

I left the above for prosperity but most of my efforts at this time are focused on Kotlin and Android, with some spare efforts going to Scala and Clojure.

A Bayesian Classifier for Song Lyrics

I’ve been spending a lot of my recent time exploring Natural Language Processing. So today, I’m sharing with you a fun little project that was inspired by something I found online written in Python. We’re analyzing the lyrics of a few popular bands and then asking the user to input a few lyrics of their own. Finally, we inform the user which Artist (based off the lyrics we have analyzed) is most likely to use the user’s input in their song.

Text files for Lyrics

I’m storing the lyrics in text files for simplicities sake. We’ll write code to trim and clean these files as we process them, but if you look at the source code, you’ll also notice that the lyrics aren’t separated by blank lines or anything that you’d get from a standard copy/paste job.

The songs and artists that I’m using for this exercise (but feel free to use your own!) are the following :

  • Heat of the Moment by Asia
  • War Pigs by Black Sabbath
  • Peace of Mind by Boston
  • I Believe in a Thing Called Love by The Darkness

Side bar: If you haven’t heard all of these songs, you should definitely look them up. They’re classics and are very powerful works of art.
Side bar#2: I’m not including the lyrics here as I do not own them or have the right to them, you’ll have to look up your own lyrics!

I Believe in a Thing Called Code

Alright, let’s get to it! I’m going to break the code up piece by piece and talk about each block.

(ns clj-bayes.core
  (:gen-class))

(def test-data {
   :asia          (slurp "resources/asia-heat-of-the-moment.txt")
   :black-sabbath (slurp "resources/black-sabbath-war-pigs.txt")
   :boston        (slurp "resources/boston-peace-of-mind.txt")
   :the-darkness  (slurp "resources/the-darkness-i-believe.txt")})

Nothing crazy here, we’re defining a map of test data and slurping that text from the filesystem. You may need to change the resource paths in your project.

(defn- clean-string [s]
  (-> s
      (clojure.string/lower-case)
      (clojure.string/replace "n" " ")
      (clojure.string/replace #"[^A-Za-z0-9 ]" "")))

Next, we’re defining a function that’ll help us clean the text input from these files. We’re taking in that string that we slurped, and piping it through the following transformations:

  • Lowercasing the string
  • Replacing line breaks with spaces
  • Removing special characters (because some people write lyrics like “Yeah!!!!!!!!!”)
(defn- clean-data [test-map]
  (let [d (for [[k v] test-map] [k (clean-string v)])]
    (into {} d)))

This function takes the entire map of test data that we defined earlier and calls the above function (clean-string) on all of the values. It takes this cleaned data, and shoves it into a new map for us and returns that.

The NLP Part

Alright, we’re getting into the NLP part now. Now’s a good time to tell you that we’re doing a Naive Bayes classification here, so it’s not 100% accurate but should be close. This form of classification is commonly used to classify emails as spam or not. As with most forms of text classification, the more data you shove into your test-set, the more accurate it will be.

The algorithm that we’ll be using for this example is written like so:
Probability of X being of Class TODO

; Given 4 bands with equal representation, we have a 1/4 chance.
(defn- get-probability-of-single-band []
  (/ 1 (count (keys test-data))))

Next, we define a function that calculates the probability of the song lyrics being from a specific band. This is really straightforward for us, as we only have four songs from four unique bands, so a one-in-four chance.

(defn- get-probability-of-word-for-band-song [w t]
  (let [d (for [[k v] t]
            [k (frequencies (clojure.string/split v #"s+"))])
        probw (map (fn [[k v]] {k (get v w)}) d)]
    probw))
(defn- get-total-word-count-for-band-songs [t]
  (let [d (for [[k v] t]
            [k (count (clojure.string/split v #"s+"))])]
    d))
(defn- get-probs [input times total]
  (let [times (into {} times)]
    (println times total input)
    (map #([input % total]) (vals times))))



Putting it all together

Finally, we’ll need to pipe together all of the functions that we’ve written. We’ll read from the console and then sanitize that a bit, and finally figure out what band is most likely to say the words you gave us, based off of the bayes implementation that we’ve written above.

(defn -main
  [& args]
  (println "Enter some text and I'll tell you which band is most likely to say it [Asia, Black Sabbath, Boston, The Darkness]")
  (let [data (clean-data test-data)
        n (read-line)
        inputs (clean-string n)
        inputs (clojure.string/split inputs #"s+")
        band-prob (get-probability-of-single-band)
        times-appeared (map #(get-probability-of-word-for-band-song % data) inputs)
        total-word-count (get-total-word-count-for-band-songs data)
        word-prob (map get-probs inputs times-appeared total-word-count)]
    (println word-prob)))

Final Code

Here’s all the code together, feel free to copy/paste!

(ns clj-bayes.core
  (:gen-class))

(def test-data {
   :asia          (slurp "resources/asia-heat-of-the-moment.txt")
   :black-sabbath (slurp "resources/black-sabbath-war-pigs.txt")
   :boston        (slurp "resources/boston-peace-of-mind.txt")
   :the-darkness  (slurp "resources/the-darkness-i-believe.txt")})

(defn- clean-string [s]
  (-> s
      (clojure.string/lower-case)
      (clojure.string/replace "n" " ")
      (clojure.string/replace #"[^A-Za-z0-9 ]" "")))

(defn- clean-data [test-map]
  (let [d (for [[k v] test-map] [k (clean-string v)])]
    (into {} d)))

; P(Band | word) = 

; Given 4 bands with equal representation, we have a 1/4 chance.
(defn- get-probability-of-single-band []
  (/ 1 (count (keys test-data))))

(defn- get-probability-of-word-for-band-song [w t]
  (let [d (for [[k v] t]
            [k (frequencies (clojure.string/split v #"s+"))])
        probw (map (fn [[k v]] {k (get v w)}) d)]
    probw))

(defn- get-total-word-count-for-band-songs [t]
  (let [d (for [[k v] t]
            [k (count (clojure.string/split v #"s+"))])]
    d))

(defn- get-probs [input times total]
  (let [times (into {} times)]
    (println times total input)
    (map #([input % total]) (vals times))))

(defn -main
  [& args]
  (println "Enter some text and I'll tell you which band is most likely to say it [Asia, Black Sabbath, Boston, The Darkness]")
  (let [data (clean-data test-data)
        n (read-line)
        inputs (clean-string n)
        inputs (clojure.string/split inputs #"s+")
        band-prob (get-probability-of-single-band)
        times-appeared (map #(get-probability-of-word-for-band-song % data) inputs)
        total-word-count (get-total-word-count-for-band-songs data)
        word-prob (map get-probs inputs times-appeared total-word-count)]
    (println word-prob)))

Once again, sorry that this isn’t of the best quality. I just did not want to leave this post unposted. Thanks for reading!

Permalink

PurelyFunctional.tv Newsletter 283: Spec, Error messages!, Games

Issue 283 – July 16, 2018 · Archives · Subscribe

Hi Clojurists,

It used to be, maybe five years ago, that I could keep up with everything in the Clojure community. There were few enough conference talks, blog posts, and libraries that I could feel informed about everything going on. We passed that point a long time ago and it’s feeling more and more difficult to keep up. Every time I talk to someone, there’s something I’ve missed.

In Apropos 12, we talked about the lack of blogs, podcasts, and events around Clojure. So this is a request: please blog, podcast, and host events. There’s plenty of room for more! I love reading The REPL, Daniel Compton’s email newsletter. It covers different stuff from this newsletter, and I enjoy the different perspective. There’s so much room, we’re not competing. We link to eachother’s stuff.

Thank you to everyone who does blog about stuff! I enjoy:

What are your favorite places to learn about Clojure? Like always, just hit reply 🙂

Rock on!
Eric Normand <eric@purelyfunctional.tv>

PS Want to get this in your email? Subscribe!


Clojure Career Workshop ALPHA

There are a handful of tickets left for the Clojure Career Workshop. I want to help you get a job in Clojure. This workshop will support you to improve your resume and find jobs. I’ll also connect you to people I know who can help you on your quest.

This is an ALPHA version, meaning 2 things: you get more attention and it’s cheaper. I haven’t worked out all of the kinks, so to make sure you get what you’re looking for, I’m going to have to give everyone special attention. And because it may be rougly hewn, I’m making it cheaper. But there will only be ten total people in the workshop.

Click the link above to buy tickets and for more information.


Ghostwheel GitHub

One of the cool things about Spec is that it’s a bunch of composable pieces. They’re also optional. Ghostwheel takes advantage of that, providing a powerful library of tools, built on spec. There’s a lot to explore and the documentation looks great.


You should think about some states Podcast

Kevin Lynaugh was on the Future of Coding podcast, talking about everything from state machines to TLA+.


Hyperfiddle

This looks like a cool web framework built on Datomic.


Formally Specifying UIs

Starting from Kevin Lynaugh’s Sketch.systems, Hillel Wayne shows how we can formally specify the links between UI screens so that we can verify interesting properties.


improvements to exception messages and printing Clojure JIRA

Stuart Halloway and Alex Miller are working on improving error messages in Clojure. There’s a much more detailed analysis on the wiki.

Stuart Halloway has been actively discussing error messages on Twitter for the past few weeks. It seems like that discussion is bearing fruit. This is not a reversal on Stuart’s part. In fact, this is a doubling-down on the same principles that make Clojure great but were incompletely applied to error messages.

The changes in the Jira ticket are all about separating (read: decomplecting) the error message (from getMessage()) from its printing. The error messages now are terser so that they can compose better and so that they are reusable. The printer is in charge of displaying relevant information to the user, including the class of the exception.

The default printer, as the Jira ticket now stands, does not print the entire data component of ex-info exceptions. The data component could be a huge nested data structure, and so dominate the printed output. Now the default printer just prints the keys at the top level, which usually fit on one line. They merely give you a first idea about what you will find in the data component when you explore it interactively. All in all, the default error printing seems to be improving, while increasing the freedom and power of the end programmer to customize them.

Though I think error messages can be improved, I know that making good error messages is really hard. If you just focus on error messages, other parts of the system can suffer. I’m glad Stuart has dived into this and found a path forward. And I hope he continues to explain the design choices that make Clojure special.

We talk about this Jira ticket on the latest episode of Apropos.


Nil Punning (or null pointers considered not so bad)

Clojure makes extensive use of nil punning, which means nil can have different meanings in different contexts. I don’t like nil in general (it’s the Billion Dollar Mistake). But the mistake was made in Java and JavaScript, and Clojure needs to live with it. Nil punning is a way for Clojure to make the best of a bad situation.


Making a tale tick

Bryce Covert explains why he chose Clojure to build Tick Tales, his point-and-click adventure game.


Clojure core.async Video course

core.async is a powerful library that makes many kinds of concurrent programming easy. This course teaches the basic concepts of the language and how to apply them to solve problems. core.async changed the game for me, especially in ClojureScript. While core.async has a lot of features, they’re all based on the fundamental concepts of channels and go blocks. This course will help you master them.

The post PurelyFunctional.tv Newsletter 283: Spec, Error messages!, Games appeared first on PurelyFunctional.tv.

Permalink

What does it mean for programs to be built using “whole values”?

John Hughes, FP researcher extraordinaire, says Whole Values is one of the principles of Functional Programming. But what does he mean? We explore this important concept.

Transcript

Eric Normand: One of John Hughes’s principles of functional programming is to use whole values. What does that mean?

Hi, my name is Eric Normand. These are my thoughts on functional programming.

Whole values is a subtle idea. It’s something that I think needs to be shored up and made more definite. He doesn’t ever explain it in definite terms anywhere. He just suggests we use whole values and gives some interesting examples of it.

I would like to try to define it, but I don’t know if I’m ready. I’m just going to explore the idea a little bit and see where it goes. The thing is I do believe that he’s right that this is an important thing in functional programming, it really helps.

I do believe in it. I use it myself, and it really feels right, but that doesn’t mean I could put what I’m doing into words. That’s what I’m trying to do right now.

Let’s imagine that we have some imperative algorithm, and this algorithm requires us to keep track of two different numbers. We make two variables, and we update each as they need to be updated.

For instance, if you wanted to calculate the average of sum of a list of numbers, you could initialize two variables to zero. One of them is called the sum and one of them is called the count. You iterate through the list. Every time you see a number, you add it to the sum, and store it back in sum.

You add one to the count and store it back in count. At the end you divide them out and you have figured out the answer. At each stage of the loop, each iteration through the loop you got these two numbers. You got the sum and the count, but they’re separate. There’s no indication in your program that they go together. They definitely do.

As you know, I’ve talked about calculating an average using the ratio, so that instead of storing the numbers separately, you put them into a tuple where you have a sum and a count in the tuple. That’s an example of keeping the values together and making them whole. Separately they’re not meaningful, but as a whole they are.

This lets you pass it around like an argument. You can have a much more coherent set of operations on them because you have both values all the time.

When I’ve been reading about Smalltalk — again I bring up Smalltalk for some reason — but I think a lot of this stuff was known by the Smalltalk group. The original group in Smalltalk, the whole purpose of objects is to bundle two values together that need to be in a relationship, so that you can transactionally modify them to maintain that relationship.

That’s all we’re talking about is bundling values together into a composite value that those two values are kept in a relationship. It might be three, whatever.

Let’s come up with another example. I was doing some coding yesterday. It was a recursive descent parser. One of the things I needed was two values coming out of a…let’s say I parsed a number out of this stream of characters. I needed two things, the number that I parsed and the rest of the stream like where did I get to when I finished parsing the numbers. I bundled those up as a tuple.

If you extrapolate that to the whole architecture of the entire parser, not just parsing integers but parsing everything, you start to see that this bundling is very useful to be able to say, “Here’s the value that I’ve got and here’s the rest of the stream at all times.”

Then you could be adding to that stream, I mean sorry, adding to that value. Let’s say, you’re parsing a list, you have to parse one item at a time. Add it to the list. You’re maintaining the entire state of the parser in one value, one single value.

Instead of ad hoc, passing these things around as arguments and sometimes you need one and sometimes you need the other, you just always bundle them together. It just makes for a much cleaner interface to everything. I would like to give one more example. This is the example that John Hughes gave.

He was talking about a system that someone came up with to draw graphics. This was quite an old example. The example was that it was a system where an image was represented as a function. The arguments to that function where the axes on which is supposed to draw that picture.

You skew it to fit in the axis. You could have it whether orthogonal, the two axes and wherever the point is where they meet. That’s the bottom left corner. Then, if you were to…sorry, you’re backwards to what I’m seeing. Is you were to skew them like this or move it, it will move the image to another spot and will squish it.

That’s interesting, but really what it’s saying is that that function was complete as a whole value. It could be drawn anywhere on the screen, contained everything it needed to draw. The thing that was variable, where the arguments like, where it’s going to be drawn or what the axes look like.

It was a complete unit. It did nothing. It could be completely opaque because its interface was very clear. It could be composed in very known ways. That’s the example that he gave. I think that’s another interesting example.

This idea of the whole value. Instead of having an image that was a ray of pixels, and then an algorithm, like a function or something else. I guess a function that would take the image and the two axes, make a new image that would skew it. Then take this skewed image. You could then draw that to the screen.

It was done in a much more elegant way that allowed for interesting patterns because you could pass in the axes to a function and it would change the axes, pass them to the image, made it recursively do it so you had this fractal effect.

It’s much more interesting to do cool stuff with. It’s much easier to do that functional recursive stuff when you’re dealing with it at that level.

I don’t know about how beautiful these images were by the way. I do find that that kind of thing helps where you have the whole value. I don’t know about any other examples, but I’m going to try to define it now.

The real essence of it is that you take two or more values, usually it’s a small number of things, and you put them into a composite value. That could be a tuple or you could define a new type.

You put them into this composite value and the composite value represents a semantic whole that you can then define new operations on top of. This is very much like…When I say it like this, it just sounds so much like object-oriented programming where you are grouping state together and putting an interface on it.

That’s not what I see most people doing with object-oriented programming. Usually what they’re doing is…They’re not thinking in terms of the relationships between the data and how to make operations to maintain those relationships.

They’re thinking more like, “What’s all the data we can put in here? Let’s group it together and call it a person or let’s group it together and call it an inventory manager.” I don’t know what they would call it.

Then one of those operations that we need to do on it. We need to read it. We need to write it. We need…They’re not doing this much more relationship based analysis.

Here’s the tip. If you’re doing object-oriented programming, you should look into doing it like this where you’re thinking about…Instead of doing this for-loop and maintaining two different variables, those two variables should be bundled together.

Does it make sense to bundle them together? Once you’ve bundled them together, it could be that the operations…This make it functionally again. The operations on them have interesting properties.

If they’re associative, that means you can do much more free recursion on them. If they are commutative, it means you can distribute your algorithm, not worry about what order things come back in. You’re going to want to bundle those values together anyway.

Maybe if it’s associative, you can find an identity value, you’ve got a monoid. There’s things that you can do with them once they’re bundled together. You can start to analyze the properties of those operations that maintain the relationship.

It’s actually another principle he talks about. He talks about combining forms, it’s what he calls them. Call them operations, but he’s talking about you take two of these whole values and you combine them in some way.

If it’s an average, you’re taking the two averages, you add them up, you have another average. Take two of these images that take the same skewed axes and you can overlay them or you can put them side-by-side or something like that.

It’s a combining form. I’m about to get on the greenway here, a lot of bikes. It’s like a highway for bikes. I’m trying to define this. Trying to define this, this idea of whole values.

Before I define it, I did want to talk a little bit about this talk I saw by Fred George. Fred George is a…I guess he’s a product manager now or project manager, something like that. He’s an old timer, very experienced programmer.

I started getting into him because he talked about what his management style, which is called Developer Anarchy, or something like that. The idea is just give the programmers some KPI, some goals to hit and let them hit it.

Let them go and figure out ways to increase those metrics and don’t give them user stories and stuff like that. Let them come up with them on their own. It required a whole architecture, like a microservices architecture so that they could work independently without breaking the entire system.

He also had another talk called The Secrets of Agile Programming or something like that. Secret Practices of Agile Programming. It was trying to say that when a lot of the agile practices were invented, a lot of them came from extreme programming.

It was assuming that you were programming, basically, like can’t back. Do all these practices, do pair programming. Do continuous deployment. Do test-driven development. All these things that we think of as agile style things modern software development practices.

They also assume that your programming style, your coding style, was in a certain way, so we’ve described the few of these things. One of them, one of the things they said, was your classes should have two or at most three fields.

The audience was like, “What? That’s not possible.” And he’s like, “Go, analyze your classes and the fields that they have. I’ll bet that you’ll see that some of those fields have a stronger relationship to each other than they do to the others.”

If you have a person class, you’ll see that, “Hey, this street name goes together with the zip code much more than it goes together with their salary. Maybe, those should start to be group together to represent that relationship.”

That the whole point of this is to start representing the semantic information in ways that your programming language supports. If you’re just throwing them together like a big bag, you’re not going to be able to move this fast to program as quickly. You just going to have this big bag objects that just do everything.

The secret is to break them out into smaller objects that have much more coherent between the fields. I think that’s very much related to this functional programming idea of whole values. Don’t have a bunch of values that have a relationship and that they’re split apart.

Like they’re split into even something as simple as the two arguments to a function. Put them together. You’ll notice that composite might have certain properties.

There’s also the idea in a whole value that you could have partial solutions to problems. Instead of saying, “Well, we either have the solution or we don’t,” you might have a partial solution.

It’s like this parser example that I gave where you have some value, and then the rest of the problem is being stored or is part of the answer. I parsed four characters for you, here’s the number that that represents, and here’s the rest of the stream.

This is all the characters that were part of that number. You put them together, and it makes sense together.

I’m going long now. I just want to say thanks for listening. It really helps me to get my ideas out, walking and talking, and sharing them with people. I love it that you…

The post What does it mean for programs to be built using “whole values”? appeared first on LispCast.

Permalink

Designing good DSL

DSLs are great tool to reduce complexity and define problems in a compact and succinct way. In case you need to design your own, these are a few common traps to avoid.

Make it verbose

Most well-known example of non-verbose DSL is regular expressions:

([A-Z][a-z]*[,.!?]\s+)*

Even though regular expressions are quite popular, it’s not a good design to copy.

First, most of its syntax beyond the very basics like "X+" or "[^X]" is impossible to remember. It’d be nice to know what "(?<!X)" does without having to look it up first.

Second: how do I google something like that?

And third: it’s really hard to read. There are no clear boundaries, so it’s hard to understand what is part of what. I always struggle even on simple cases like "https?" (it’s either "http" or "https" but it’s really hard to see that at first sight). And when things get complex—and they do get complex quick—you’ll spend more time parsing regex in your head than you would spend writing it anew. I mean, try reading this:

[a-z]{3,10}://([^/?#]*)([^?#]*)(?:\?([^#]*))?(?:#(.*))?

There’s nothing tricky going on, but it’s way too hard to read than it needs to be. That’s why they call Perl write-only language.

Another non-verbose DSL example is Java date and time format string:

"YYYY-MM-DD'T'HH:mm:ss.SSSZ"

More readable (the domain is way simpler) but also less widespread, so you still have to look it up every time. Not a problem while writing, but consider this: you just walking by that code and want to make simple modification (change short month to full month) or even fool-proof that it uses padded 24-hour for hours. Now, is "HH" 24-hour format or 12-hour? Is "MM" the format of month you really want here? Is it even a month, or minutes? It’s all still write-only and ungooglable.

Things get worse if we move to Clojure land. Clojure started as a language that assigned contextual meaning to a few language-defined syntax structures: vector might be a let-binding in let expression, function arguments list in fn and just a data structure. Again, more or less fine in a language because language is the same for all its users, finite and fixed. Learn it once and move along (still, mature developers keep discovering that e.g. case expression treats lists specially etc).

It got worse when library authors treated it as a design guideline. E.g. Datomic rules are specified like this:

[[(descendant? ?p ?c)
  [?p :parent ?c]]
 [(descendant? ?p ?c)
  [?p :parent ?x]
  (descendant? ?x ?c)]]

No words at all, just three-level deep mix of lists and vectors. If you don’t a priori know what’s going on it’s hard to even formulate a question about THAT. Also, as a user, I have to admit it’s really annoying placing all these brackets and parents right every time.

Core.async has the same problem. Here’s how you specify get operation in alt!:

(alt! [c t] ...)

And this is put (value to channel):

(alt! [[c v]] ...)

I mean, this doesn’t correspond to anything in core language or even in core.async itself! It’s just a structure-based syntax for the sake of structure-basedness. Strange that the very same alt! has named options—compare how clearer those are!

(alt! [c t]     ...
      [[c v]]   ...
      :default  nil
      :priority true) 

How to solve? Simple:

Always give alternative options/behaviors/branches long, descriptive names.

E.g. core.match does this right:

(match [x]
  ([1 2] :seq)    ...
  (:or 1 2 3)     ...
  (_ :guard odd?) ...
  :else           ...)

Notice those :seq :or and :guard. Those are the explicit words marking different behaviour, not some implicit shape-defined structure.

Clojure.spec does good job too. It still uses short non-descriptive + and ? (those are borrowed from regexes) but rest is perfectly readable and googlable words: alt, cat, keys, req, coll-of, map-of, every, tuple:

(spec/alt
 :arity-1 ::args+body
 :arity-n (spec/cat
           :bodies (spec/+ (spec/spec ::args+body))
           :attr   (spec/? map?)))

This is how datetime formatting could be done, aided by this principle (example from Tongue):

{hour12}:{minutes-padded}{dayperiod} {month-numeric}/{day}/ {year-2digit}

Verbose, yes, but why would you care about just a few more characters here, when readability is at stake? By making it verbose, even without any prior experience with Tongue, anyone can fool-proof that this code will output something like "1:05PM 7/13/18" and not "13:5PM 07/13/2018". Or even add 0-padding to hours, day and month and maybe even change 12-hour to 24-hour format if needed. Without even looking at the docs! Can you say the same about "H:mm a d/M/y"?

As for regexes, there’re VerbalExpressions:

var tester = VerEx()
    .startOfLine()
    .then('http')
    .maybe('s')
    .then('://')
    .maybe('www.')
    .anythingBut(' ')
    .endOfLine();

Don’t invent second syntax

Sometimes your DSL starts simple and then you figure there’s no way to handle complex cases. So you extend it with advanced ways to do stuff. E.g. Datomic Pull syntax let you simply list attributes you need, attribute = keyword:

[:artist/name :artist/gid ...]

But that way there’s no way to specify additional options: nesting, limits, etc. So even though attributes started as keywords in a list, sometimes they might be specified as a map:

{:artist/tracks [:track/name :track/gid ...]}

Or a list can be used instead of keyword:

(:artist/endYear :default "N/A")

Now even I am confused how to combine default and nesting in a single expression.

Sometimes you start with all-powerful solution and then figure it’s really too much writing for simple cases. So you add shortcuts. E.g. Datomic query let you shorthand this:

(d/q {:find  [?a]
      :where [[?a :artist/name "Beatles"]]})

to this (less brackets):

(d/q [:find  ?a
      :where [?a :artist/name "Beatles"]])

Both ways, now your language has two+ ways of saying the same thing.

Having more than one way to do something is not a virtue, it’s a curse. Your users now have to learn two syntaxes. They need twice as much examples. Answers found on internet might not work because they use different syntax, etc.

My suggestion:

One way to do it + an escape hatch.

You cover most of your users’ needs (that’s the primary value of your DSL anyway) and let users figure the rest in plain old code—best of both worlds.

Don’t be too liberal

Almost the same as the previous one, but on a smaller scale. Sometimes DSLs let you get away with small variations: e.g. in Datomic you can specify default and limit as either keyword, symbol or a string—all are fine (apparently, you can even change order):

(:artist/tracks :limit 10)
(:artist/tracks "limit" 10)
(limit :artist/tracks 10)

Hiccup lets you drop empty attributes map:

[:div {} "Hello, world"]
[:div "Hello, world"]

Core.match lets you drop vector in case you’re matching a single value:

(match [x]
  [:a] ...
  :a   ...) // the same

The thing with special cases, same as with multiple syntaxes, is that you have to learn them, remember about them and recognize them. That’s a problem: it eats up cognitive resources better spent elsewhere.

Also, people in your/other teams will have preferences or would simply not care about consistency. Hence:

The only way to force everybody to always do the same thing is to ban any variations.

Being DSL designer, it’s your responsibility alone.

Keep it small

Rule of thumb:

Your DSL should be entirely memorizable.

That makes using it quick (once you’ve learned it), and this is where the value comes from. If you don’t use it too often or there’s too much syntax, you’ll be required to look at documentation each time before using or reading it. That slows things down, making using DSL an effort and eventually leading to DSL being replaced with a simpler solution.

Don’t try to help too much

Many DSLs were designed to reduce amount of non-DSL code to the absolute zero. They try to help too much. Your stance here should be:

Do in DSL what you know how to do well, leave rest for the users to figure out.

E.g. routing libraries might extract parameters but shouldn’t try to coerce them to other types—it’s really simple to do in your own code. Your users will have full programming language to handle those tricky/rare/exceptional cases.

Sometimes not doing stuff in DSL and leaving specific cases to users leads to less overall complexity.

Conclusion

That’s more or less all I can think of right now. Designing DSLs is a fun and rewarding activity, but as with any design, it’s really hard to get right. I wish you luck on that tricky path, and let me know if it helps.

Permalink

Clojure Don’ts: Thread as

A brief addendum to my last post about Clojure’s threading macros. As I was saying … I said you could use as-> to work around the placement of an argument that doesn’t quite fit into a larger pattern of ->. My example was: (-> results :matches (as-> matches (filter winning-match? matches)) (nth 3) :scores (get…

Read the full article

Permalink

Re-implementing boodle database layer

When I wrote boodle, I didn’t think the database layer thoroughly. The model was simple enough to figure out: one namespace for each table. However, I stuck the queries at the top of these namespaces, using plain strings to compose them.

A working solution, yes, but far from being optimal. Let’s be honest: it’s just plain ugly and prone to error. White spaces, characters to escapes. Not a mess, but neither a smooth ride.

At 7bridges I recently had the chance to play with honeysql. At first I wrongly mistook it for syntactic sugar. A DSL on top of SQL? The horrors of ORM sprang to mind in a rush of anxiety, but I set aside my fears and gave it a chance anyway.

It took me ten minutes to fall in love with honeysql. And turning to boodle for a proper refactoring was the following thought.

A quick example of how prettier queries are now:

(defn select-aims-with-transactions
  []
  (-> (hh/select :a.id [:a.name :aim] :a.target :t.amount)
      (hh/from [:transactions :t])
      (hh/right-join [:aims :a] [:= :a.id :t.id_aim])
      (hh/where [:= :a.achieved false])
      hc/build
      db/query))

No strings, no white spaces, no escaping characters and quoting values. Heaven.

honeysql is also extensible, which makes it simple to add support for operators.

(:require [honeysql.format :as fmt])

;; … other code …

(defmethod fmt/fn-handler "ilike" [_ col qstr]
  (str (fmt/to-sql col) " ilike " (fmt/to-sql qstr)))

During the refactoring process, I noticed something else that needed a better solution: dates.

I was relying on PostgreSQL TO_DATE to format my dates in the queries. honeysql is database-agnostic, so it pushed me to look for a better option.

(:require [java-time :as jt])

;; … other code …

(extend-protocol jdbc/IResultSetReadColumn
  Date
  (result-set-read-column [v _ _]
    (-> v
        jt/local-date
        ud/format-date))

  Timestamp
  (result-set-read-column [v _ _]
    (-> v
        jt/local-date
        ud/format-date)))
        
(extend-protocol jdbc/ISQLValue
  java.time.LocalDateTime
  (sql-value [v] (jt/sql-timestamp v))
  java.time.LocalDate
  (sql-value [v] (jt/sql-timestamp v)))
        
;; … other code …

format-date is just a utility function:

(:require [java-time :as jt])

;; … other code …

(defn format-date
  "Return `date` in dd/MM/yyyy format."
  [date]
  (jt/format "dd/MM/yyyy" date))

If you have any experience with Java, you know dealing with dates and times had not been a piece of cake before Java 8. I briefly mentioned LocalDate when reporting on my experience with Java 8.

Now that I am working with Clojure, dates and times strike back. And thanks to Vadim Platonov, I can safely use Java 8 Date-Time API with clojure.java-time.

Permalink

Clojure from the ground up: functions

We left off last chapter with a question: what are verbs, anyway? When you evaluate (type :mary-poppins), what really happens?

user=> (type :mary-poppins) clojure.lang.Keyword

To understand how type works, we’ll need several new ideas. First, we’ll expand on the notion of symbols as references to other values. Then we’ll learn about functions: Clojure’s verbs. Finally, we’ll use the Var system to explore and change the definitions of those functions.

Let bindings

We know that symbols are names for things, and that when evaluated, Clojure replaces those symbols with their corresponding values. +, for instance, is a symbol which points to the verb #<core$_PLUS_ clojure.core$_PLUS_@12992c>.

user=> + #<core$_PLUS_ clojure.core$_PLUS_@12992c>

When you try to use a symbol which has no defined meaning, Clojure refuses:

user=> cats CompilerException java.lang.RuntimeException: Unable to resolve symbol: cats in this context, compiling:(NO_SOURCE_PATH:0:0)

But we can define a meaning for a symbol within a specific expression, using let.

user=> (let [cats 5] (str "I have " cats " cats.")) "I have 5 cats."

The let expression first takes a vector of bindings: alternating symbols and values that those symbols are bound to, within the remainder of the expression. “Let the symbol cats be 5, and construct a string composed of "I have ", cats, and " cats".

Let bindings apply only within the let expression itself. They also override any existing definitions for symbols at that point in the program. For instance, we can redefine addition to mean subtraction, for the duration of a let:

user=> (let [+ -] (+ 2 3)) -1

But that definition doesn’t apply outside the let:

user=> (+ 2 3) 5

We can also provide multiple bindings. Since Clojure doesn’t care about spacing, alignment, or newlines, I’ll write this on multiple lines for clarity.

user=> (let [person "joseph" num-cats 186] (str person " has " num-cats " cats!")) "joseph has 186 cats!"

When multiple bindings are given, they are evaluated in order. Later bindings can use previous bindings.

user=> (let [cats 3 legs (* 4 cats)] (str legs " legs all together")) "12 legs all together"

So fundamentally, let defines the meaning of symbols within an expression. When Clojure evaluates a let, it replaces all occurrences of those symbols in the rest of the let expression with their corresponding values, then evaluates the rest of the expression.

Functions

We saw in chapter one that Clojure evaluates lists by substituting some other value in their place:

user=> (inc 1) 2

inc takes any number, and is replaced by that number plus one. That sounds an awful lot like a let:

user=> (let [x 1] (+ x 1)) 2

If we bound x to 5 instead of 1, this expression would evaluate to 6. We can think about inc like a let expression, but without particular values provided for the symbols.

(let [x] (+ x 1))

We can’t actually evaluate this program, because there’s no value for x yet. It could be 1, or 4, or 1453. We say that x is unbound, because it has no binding to a particular value. This is the nature of the function: an expression with unbound symbols.

user=> (fn [x] (+ x 1)) #<user$eval293$fn__294 user$eval293$fn__294@663fc37>

Does the name of that function remind you of anything?

user=> inc #<core$inc clojure.core$inc@16bc0b3c>

Almost all verbs in Clojure are functions. Functions represent unrealized computation: expressions which are not yet evaluated, or incomplete. This particular function works just like inc: it’s an expression which has a single unbound symbol, x. When we invoke the function with a particular value, the expressions in the function are evaluated with x bound to that value.

user=> (inc 2) 3 user=> ((fn [x] (+ x 1)) 2) 3

We say that x is this function’s argument, or parameter. When Clojure evaluates (inc 2), we say that inc is called with 2, or that 2 is passed to inc. The result of that function invocation is the function’s return value. We say that (inc 2) returns 3.

Fundamentally, functions describe the relationship between arguments and return values: given 1, return 2. Given 2, return 3, and so on. Let bindings describe a similar relationship, but with a specific set of values for those arguments. let is evaluated immediately, whereas fn is evaluated later, when bindings are provided.

There’s a shorthand for writing functions, too: #(+ % 1) is equivalent to (fn [x] (+ x 1)). % takes the place of the first argument to the function. You’ll sometime see %1, %2, etc. used for the first argument, second argument, and so on.

user=> (let [burrito #(list "beans" % "cheese")] (burrito "carnitas")) ("beans" "carnitas" "cheese")

Since functions exist to defer evaluation, there’s no sense in creating and invoking them in the same expression as we’ve done here. What we want is to give names to our functions, so they can be recombined in different ways.

user=> (let [twice (fn [x] (* 2 x))] (+ (twice 1) (twice 3))) 8

Compare that expression to an equivalent, expanded form:

user=> (+ (* 2 1) (* 2 3))

The name twice is gone, and in its place is the same sort of computation–(* 2 something)–written twice. While we could represent our programs as a single massive expression, it’d be impossible to reason about. Instead, we use functions to compact redundant expressions, by isolating common patterns of computation. Symbols help us re-use those functions (and other values) in more than one place. By giving the symbols meaningful names, we make it easier to reason about the structure of the program as a whole; breaking it up into smaller, understandable parts.

This is core pursuit of software engineering: organizing expressions. Almost every programming language is in search of the right tools to break apart, name, and recombine expressions to solve large problems. In Clojure we’ll see one particular set of tools for composing programs, but the underlying ideas will transfer to many other languages.

Vars

We’ve used let to define a symbol within an expression, but what about the default meanings of +, conj, and type? Are they also let bindings? Is the whole universe one giant let?

Well, not exactly. That’s one way to think about default bindings, but it’s brittle. We’d need to wrap our whole program in a new let expression every time we wanted to change the meaning of a symbol. And moreover, once a let is defined, there’s no way to change it. If we want to redefine symbols for everyone–even code that we didn’t write–we need a new construct: a mutable variable.

user=> (def cats 5) #'user/cats user=> (type #'user/cats) clojure.lang.Var

def defines a type of value we haven’t seen before: a var. Vars, like symbols, are references to other values. When evaluated, a symbol pointing to a var is replaced by the var’s corresponding value:

user=> user/cats 5

def also binds the symbol cats (and its globally qualified equivalent user/cats) to that var.

user=> user/cats 5 user=> cats 5

When we said in chapter one that inc, list, and friends were symbols that pointed to functions, that wasn’t the whole story. The symbol inc points to the var #'inc, which in turn points to the function #<core$inc clojure.core$inc@16bc0b3c>. We can see the intermediate var with resolve:

user=> 'inc inc ; the symbol user=> (resolve 'inc) #'clojure.core/inc ; the var user=> (eval 'inc) #<core$inc clojure.core$inc@16bc0b3c> ; the value

Why two layers of indirection? Because unlike the symbol, we can change the meaning of a Var for everyone, globally, at any time.

user=> (def astronauts []) #'user/astronauts user=> (count astronauts) 0 user=> (def astronauts ["Sally Ride" "Guy Bluford"]) #'user/astronauts user=> (count astronauts) 2

Notice that astronauts had two distinct meanings, depending on when we evaluated it. After the first def, astronauts was an empty vector. After the second def, it had one entry.

If this seems dangerous, you’re a smart cookie. Redefining names in this way changes the meaning of expressions everywhere in a program, without warning. Expressions which relied on the value of a Var could suddenly take on new, possibly incorrect, meanings. It’s a powerful tool for experimenting at the REPL, and for updating a running program, but it can have unexpected consequences. Good Clojurists use def to set up a program initially, and only change those definitions with careful thought.

Totally redefining a Var isn’t the only option. There are safer, controlled ways to change the meaning of a Var within a particular part of a program, which we’ll explore later.

Defining functions

Armed with def, we’re ready to create our own named functions in Clojure.

user=> (def half (fn [number] (/ number 2))) #'user/half user=> (half 6) 3

Creating a function and binding it to a var is so common that it has its own form: defn, short for def fn.

user=> (defn half [number] (/ number 2)) #'user/half

Functions don’t have to take an argument. We’ve seen functions which take zero arguments, like (+).

user=> (defn half [] 1/2) #'user/half user=> (half) 1/2

But if we try to use our earlier form with one argument, Clojure complains that the arity–the number of arguments to the function–is incorrect.

user=> (half 10) ArityException Wrong number of args (1) passed to: user$half clojure.lang.AFn.throwArity (AFn.java:437)

To handle multiple arities, functions have an alternate form. Instead of an argument vector and a body, one provides a series of lists, each of which starts with an argument vector, followed by the body.

user=> (defn half ([] 1/2) ([x] (/ x 2))) user=> (half) 1/2 user=> (half 10) 5

Multiple arguments work just like you expect. Just specify an argument vector of two, or three, or however many arguments the function takes.

user=> (defn add [x y] (+ x y)) #'user/add user=> (add 1 2) 3

Some functions can take any number of arguments. For that, Clojure provides &, which slurps up all remaining arguments as a list:

user=> (defn vargs [x y & more-args] {:x x :y y :more more-args}) #'user/vargs user=> (vargs 1) ArityException Wrong number of args (1) passed to: user$vargs clojure.lang.AFn.throwArity (AFn.java:437) user=> (vargs 1 2) {:x 1, :y 2, :more nil} user=> (vargs 1 2 3 4 5) {:x 1, :y 2, :more (3 4 5)}

Note that x and y are mandatory, though there don’t have to be any remaining arguments.

To keep track of what arguments a function takes, why the function exists, and what it does, we usually include a docstring. Docstrings help fill in the missing context around functions, to explain their assumptions, context, and purpose to the world.

(defn launch "Launches a spacecraft into the given orbit by initiating a controlled on-axis burn. Does not automatically stage, but does vector thrust, if the craft supports it." [craft target-orbit] "OK, we don't know how to control spacecraft yet.")

Docstrings are used to automatically generate documentation for Clojure programs, but you can also access them from the REPL.

user=> (doc launch) ------------------------- user/launch ([craft target-orbit]) Launches a spacecraft into the given orbit by initiating a controlled on-axis burn. Does not automatically stage, but does vector thrust, if the craft supports it. nil

doc tells us the full name of the function, the arguments it accepts, and its docstring. This information comes from the #'launch var’s metadata, and is saved there by defn. We can inspect metadata directly with the meta function:

(meta #'launch) {:arglists ([craft target-orbit]), :ns #<Namespace user>, :name launch, :column 1, :doc "Launches a spacecraft into the given orbit.", :line 1, :file "NO_SOURCE_PATH"}

There’s some other juicy information in there, like the file the function was defined in and which line and column it started at, but that’s not particularly useful since we’re in the REPL, not a file. However, this does hint at a way to answer our motivating question: how does the type function work?

How does type work?

We know that type returns the type of an object:

user=> (type 2) java.lang.long

And that type, like all functions, is a kind of object with its own unique type:

user=> type #<core$type clojure.core$type@39bda9b9> user=> (type type) clojure.core$type

This tells us that type is a particular instance, at memory address 39bda9b9, of the type clojure.core$type. clojure.core is a namespace which defines the fundamentals of the Clojure language, and $type tells us that it’s named type in that namespace. None of this is particularly helpful, though. Maybe we can find out more about the clojure.core$type by asking what its supertypes are:

user=> (supers (type type)) #{clojure.lang.AFunction clojure.lang.IMeta java.util.concurrent.Callable clojure.lang.Fn clojure.lang.AFn java.util.Comparator java.lang.Object clojure.lang.RestFn clojure.lang.IObj java.lang.Runnable java.io.Serializable clojure.lang.IFn}

This is a set of all the types that include type. We say that type is an instance of clojure.lang.AFunction, or that it implements or extends java.util.concurrent.Callable, and so on. Since it’s a member of clojure.lang.IMeta it has metadata, and since it’s a member of clojure.lang.AFn, it’s a function. Just to double check, let’s confirm that type is indeed a function:

user=> (fn? type) true

What about its documentation?

user=> (doc type) ------------------------- clojure.core/type ([x]) Returns the :type metadata of x, or its Class if none nil

Ah, that’s helpful. type can take a single argument, which it calls x. If it has :type metadata, that’s what it returns. Otherwise, it returns the class of x. Let’s take a deeper look at type‘s metadata for more clues.

user=> (meta #'type) {:ns #<Namespace clojure.core>, :name type, :arglists ([x]), :column 1, :added "1.0", :static true, :doc "Returns the :type metadata of x, or its Class if none", :line 3109, :file "clojure/core.clj"}

Look at that! This function was first added to Clojure in version 1.0, and is defined in the file clojure/core.clj, on line 3109. We could go dig up the Clojure source code and read its definition there–or we could ask Clojure to do it for us:

user=> (source type) (defn type "Returns the :type metadata of x, or its Class if none" {:added "1.0" :static true} [x] (or (get (meta x) :type) (class x))) nil

Aha! Here, at last, is how type works. It’s a function which takes a single argument x, and returns either :type from its metadata, or (class x).

We can delve into any function in Clojure using these tools:

user=> (source +) (defn + "Returns the sum of nums. (+) returns 0. Does not auto-promote longs, will throw on overflow. See also: +'" {:inline (nary-inline 'add 'unchecked_add) :inline-arities >1? :added "1.2"} ([] 0) ([x] (cast Number x)) ([x y] (. clojure.lang.Numbers (add x y))) ([x y & more] (reduce1 + (+ x y) more))) nil

Almost every function in a programming language is made up of other, simpler functions. +, for instance, is defined in terms of cast, add, and reduce1. Sometimes functions are defined in terms of themselves. + uses itself twice in this definition; a technique called recursion.

At the bottom, though, are certain fundamental constructs below which you can go no further. Core axioms of the language. Lisp calls these "special forms”. def and let are special forms (well–almost: let is a thin wrapper around let*, which is a special form) in Clojure. These forms are defined by the core implementation of the language, and are not reducible to other Clojure expressions.

user=> (source def) Source not found

Some Lisps are written entirely in terms of a few special forms, but Clojure is much less pure. Many functions bottom out in Java functions and types, or, for CLJS, in terms of Javascript. Any time you see an expression like (. clojure.lang.Numbers (add x y)), there’s Java code underneath. Below Java lies the JVM, which might be written in C or C++, depending on which one you use. And underneath C and C++ lie more libraries, the operating system, assembler, microcode, registers, and ultimately, electrons flowing through silicon.

A well-designed language isolates you from details you don’t need to worry about, like which logic gates or registers to use, and lets you focus on the task at hand. Good languages also need to allow escape hatches for performance or access to dangerous functionality, as we saw with Vars. You can write entire programs entirely in terms of Clojure, but sometimes, for performance or to use tools from other languages, you’ll rely on Java. The Clojure code is easy to explore with doc and source, but Java can be more opaque–I usually rely on the java source files and online documentation.

Review

We’ve seen how let associates names with values in a particular expression, and how Vars allow for mutable bindings which apply universally. and whose definitions can change over time. We learned that Clojure verbs are functions, which express the general shape of an expression but with certain values unbound. Invoking a function binds those variables to specific values, allowing evaluation of the function to proceed.

Functions decompose programs into simpler pieces, expressed in terms of one another. Short, meaningful names help us understand what those functions (and other values) mean.

Finally, we learned how to introspect Clojure functions with doc and source, and saw the definition of some basic Clojure functions. The Clojure cheatsheet gives a comprehensive list of the core functions in the language, and is a great starting point when you have to solve a problem but don’t know what functions to use.

We’ll see a broad swath of those functions in Chapter 4: Sequences.

My thanks to Zach Tellman, Kelly Sommers, and Michael R Bernstein for reviewing drafts of this chapter.

Permalink

Datomic Cloud Monitoring and Ion Cast

The new datomic.ion.cast library lets your application produce monitoring data that are integrated with Datomic's support for AWS CloudWatch.

Datomic Cloud and AWS CloudWatch

AWS CloudWatch provides a powerful set of tools for monitoring a software system running on AWS:
  • Collect and track CloudWatch Metrics -- variables that measure the behavior of your system.
  • Configure CloudWatch Alarms to notify operations or take other automated steps when potential problems arise.
  • Monitor, store, and search CloudWatch Logs across all your AWS resources.
  • Create CloudWatch Dashboards that provide a single overview for monitoring your systems.
Datomic Cloud is fully integrated with all of these AWS monitoring tools. On the producing side, Datomic creates metrics and logs; and on the consuming side, Datomic organizes metrics in custom dashboards like this Production Dashboard:

Production Dashboard

datomic.ion.cast

With the introduction of Datomic Ions, your entire application can run on Datomic Cloud nodes. The datomic.ion.cast namespace lets Ion application code add your own monitoring data alongside the monitoring data already being produced by Datomic.  Cast supports four categories of monitoring data:
  1. An event is an ordinary occurence that is of interest to an operator, such as start and stop events for a process or activity.
  2. An alert is an extraordinary occurrence that requires operator intervention, such as the failure of some important process.
  3. Dev is information of interest only to developers, e.g. fine-grained logging to troubleshoot a problem during development. Dev data can be much higher volume than events or alerts.
  4. A metric is a numeric value in a named time series, such as the latency for an operation.
To get started using ion.cast, you can

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.