The REPL

The REPL

Ions, Dhall, new clj-new
View this email in your browser

The REPL

-main

Libraries & Books.

  • Kaocha is a new test runner from Lambda Island with a bunch of nice features including test randomization, fail fast, and profiling slow tests. There's also some nice docs on cljdoc.
  • dhall-clj lets you consume Dhall expressions from Clojure. Dhall is a very Haskelly looking non-turing complete language, suited for writing config, templates, e.t.c. 

People are worried about Types. ?

  • But Ambrose Bonnaire-Sergeant is proposing to do a thesis on automatically generating Typed Clojure annotations and specs.

Foundations.

  • Did you know there are two reasons you can get a NoClassDefFoundError on the JVM? And that one of them can happen when the class is present?
  • ClojuTRE was last week and Jan Stępień had a talk on building native binaries with Graal. The slides are up here, and the talk is already out.

Tools.

  • clj-new has a new version out: 21ca1b27f46dc324be084ba839beca555aeda387. Not sure how I feel about SHA's as the new version number.

Learning.

  • I've mentioned Heart of Clojure before here, a new Clojure conference. They're now looking for sponsors for their first event next year.
  • Lambda Island has some new screencasts on data science using kixi.stats from Mastodon C
Misc.
  • Make sure you're not drinking anything when you read Just JUXT #34. I won't spoil it for you here.

I'm Daniel Compton. I maintain public Maven repositories at Clojars, private ones at Deps, and help fund OSS Clojure projects (along with tons of generous members like JUXT and Metosin) at Clojurists Together. If you've enjoyed reading this, tell your friends to sign up at therepl.net, or post a link in your company chatroom. If you've seen (or published) a blog post, library, or anything else Clojure/JVM related please reply to this to let me know about it.
Copyright © 2018 Daniel Compton, All rights reserved.


Want to change how you receive these emails?
You can update your preferences or unsubscribe from this list

Email Marketing Powered by MailChimp

Permalink

PurelyFunctional.tv Newsletter 292: Composition, Modeling, Scope

Issue 292 – September 17, 2018 · Archives · Subscribe

Hi Clojurers,

Well, I’m looking forward to getting back into the swing of things after a few weeks of moderate work on PurelyFunctional.tv due to the birth of my daughter. Have I missed anything in the Clojure world while I was out? Let me know.

Rock on!
Eric Normand <eric@purelyfunctional.tv>

PS Want to get this in your email? Subscribe!


Hire Eric to train your team

I have a couple of spots open for client work. I’m currently offering two specialized services. These are open for remote or on-site:

Clojure Kickstart is training to get your team of programmers productive in Clojure. Get the skills, the tooling, and the workflow you need to make the most of Clojure.

After the Prototype is where I and your team workshop a powerful core abstraction and come up with a plan for refactoring it into your codebase.

If you or someone you know are interested, please reply to this email.


Java is still available at zero-cost

Stephen Colebourne explains what is known about Oracle’s new licensing and planned release cycles for their JVM and JDK. This article cleared up a lot about what JDK I should choose and recommend in the future. Daniel Compton linked to this in The REPL, but I thought it was important enough to share again.


The Power of Composition YouTube

Scott Wlaschin explains why functional programmers like composition, and some of the cool things we can do with it.


SWARMing: Scaling Without A Religious Methodology YouTube

Katherine Kirk & Dan North talk about company culture and systems thinking using asian philosophical ideas.


Datomic Ions YouTube

Stuart Halloway explains Datomic Ions. This one goes into depth about what it is and how it works. It’s over an hour long.


Domain Modeling Made Functional YouTube

Scott Wlaschin talks about modeling different business domain requirements using only the type system. It includes optional values, validation, and more complex requirements.


Techniques in Functional JavaScript YouTube

Jonathan Walters presents some cool patterns for doing functional programming in JavaScript, including writing his own function composition and currying.


How do things compose across domains? Podcast

One of the important parts of my theory of functional programming is that composing anything with an action results in a new action. That means that actions are easier to create than the other two domains (calculations and data). I talk about that in this episode of my podcast.


Clojure Scope Video Course

Scoping rules are a very important part of any language. Clojure has three types of scope, and knowing how to use each will help you write maintainable code. That’s what this short course (40 minutes) will teach you. I sometimes think that this is too basic of a course, but it fits well with my goal of making PurelyFunctional.tv the most comprehensive Clojure training available.

The post PurelyFunctional.tv Newsletter 292: Composition, Modeling, Scope appeared first on PurelyFunctional.tv.

Permalink

How does FP achieve reuse?

Functional programming gets its reuse by creating small components that are very well-defined. Small means they’re probably generally useful (like lists). Well-defined means they are easy to build on top of.

Transcript

Eric Normand: How does functional programming achieve reuse?

Hi, my name is Eric Normand. These are my thoughts on functional programming. I think that the way functional programming achieves reuse is a different way from how we would traditionally think of reuse. It’s that we are aiming for very small components.

I think what happens in a lot of programmers, this happens to me, I’ll frame it in the first person.

I really love it when I get to reuse a huge thing for some other purpose. It feels like making something more complex with more conditionals, more IF statements makes it more reusable. This function has so many parameters, I can use it for so many things.

It feels like you’re reusing all of this code, but the truth is that it becomes harder to reuse with more parameters, with more configuration, with more methods if it’s a class. It becomes harder to reuse because each one is adding complexity and each one is making the thing more fit for more and more specific purpose.

What functional programmers tend to do, this is totally possible in object-oriented programming, by the way. It just doesn’t seem to be the trend that people are following. Until FP start…We’ll talk about that in a second. What FP people do is, they make tiny little things.

I’m talking about super tiny, like the count cell. It’s got two fields and it’s got basically two operations, two getters. It just like the first and the rest. It has a constructor and that’s it. Now with that, you can create a whole host of data structures, or we’ll look at something like this sequence abstraction in Clojure.

That creates a whole…It’s very simple. It’s like a count cell except it’s more of an abstraction. That creates a whole ecosystem of things that abide by this tiny little protocol or the tuple. The tuple is so simple. It’s so reusable because it does almost nothing.

What happens in object-oriented programming, and this is…Here’s the thing, when I talk about object-oriented programming, in the back of my mind there’s a little avatar of Alan Kay. He says, “Wait, that’s not what I meant.” I’m not talking about what Alan Kay meant. I’m talking about how people practice it today.

What you see is things getting bigger and bigger. Classes have more parameters for configuration. They have more methods instead of getting smaller. You have these things like a giant-person class that your company uses.

This is supposed to be used across all these software at your company but it becomes harder to use, less reusable, less usable by the different parts of your software because it’s been customized to each department. The way you make something more reusable is actually by making it simpler. By making it have fewer moving parts.

This is why functional programming at the moment is having a big influence because we’re actually the ones who are showing, “Look, you don’t need this whole like abstract class hierarchy with all these single method classes or interfaces. All you need really is a function, right? A lambda.”

Java has added that in. If you look at a lot of the classes that came along with Java 8 were very simple. The interfaces that came with it were just so simple. They were just like, “This is what we need.” A function of two arguments, a string and a number. It returns a string. You just have this very granular approach. You start with these little pieces at the bottom and you can build them up.

That’s how it works. Here’s the thing. This is why it works. It feels really good. [laughs] This is why it doesn’t work, why the O-O approach doesn’t work. It feels so good to take this thing and reuse it. This giant thing.

It feels awesome when you found a new use for this big thing because you say, “Oh, I just use this whole thing.” The thing is, it’s more rare, it’s harder to do. It feels more like you’re a master of the thing and you’re getting more effectiveness out of it. Except, it happens more rarely.

Whereas, think of something like the tuple. You use that all the time or a simple HashMap, you can use that all the time. It doesn’t even feel special. It’s such a small little thing you don’t…but then, when you look at how much code you’re reusing, it is significant.

The reuse happens. The way it works is, it’s logarithmic. It’s a tree. You have these little tiny things on the leaves of the tree. You build something a little bit bigger out of these two simple parts and that thing is still pretty simple in itself. That’s reusable.

The thing at the top is reusing this huge tree of stuff. Whereas in object-oriented programming, you make one big class and hope to reuse it. It’s not made out of those little parts. What’s the likelihood that you’ve got that class right? It’s one level. It’s like one giant thing as opposed to thousands of little things that get put together.

What’s the probability that you’re going to be able to reuse that giant thing versus this very simple thing with one use with one purpose? It’s very unlikely that you’ll get to reuse this big thing. Even if it’s like at the top of your tree in a functional programming with this tree of stuff built out of simpler parts and simpler parts and simpler parts.

The top of the tree might be wrong because it’s so big. Look how big it is. If it’s wrong, you can just go down one level of the tree and all those things are a little bit more likely to be right for their purpose. You go down one level again, those things are even more likely to be right for their purpose.

Even if you get it wrong, you can just go down a level and you’re at a better place to find reusable parts.

I’ve been ranting too much about this. Let me know what you think. I’m @ericnormand on Twitter. You can also email me at eric@lispcast.com.

It’s all about the reuse. That’s how you gain efficiencies of scale. As your code base gets bigger, you want more stuff to be able to reuse. You don’t want to slow down.

Get in touch with me. Follow me on Twitter. Email me. I’ll see you later. Bye.

The post How does FP achieve reuse? appeared first on LispCast.

Permalink

Just Juxt #34: Read Roman numerals (4clojure #92)

Clock

Roman numerals are easy to recognize, but not everyone knows all the rules necessary to work with them. Write a function to parse a Roman-numeral string and return the number it represents.

You can assume that the input will be well-formed, in upper-case, and follow the subtractive principle. You don't need to handle any numbers greater than MMMCMXCIX (3999), the largest number representable with ordinary letters.

(ns live.test
  (:require [cljs.test :refer-macros [deftest is testing run-tests]]))
  
(defn roman-numeral [s]
    (let [symbol-values {\I 1 \V 5
                         \X 10 \L 50
                         \C 100 \D 500
                         \M 1000}]
      (->>
        (partition-by identity s)
        (map (juxt first count))
        (partition-all 2 1)
        (map
          (fn [[[symbol cardinality] [next-symbol _]]]
            (let [value (symbol-values symbol)]
              (* cardinality value
                (if (and next-symbol
                         (< value (symbol-values next-symbol)))
                  -1 1)))))
        (apply +))))

(deftest roman-numeral-test
  (is (= 14 (roman-numeral "XIV")))
  (is (= 827 (roman-numeral "DCCCXXVII")))
  (is (= 3999 (roman-numeral "MMMCMXCIX")))
  (is (= 48 (roman-numeral "XLVIII"))))
  
(run-tests)

Permalink

Hello Clojure

Introduction

For many years, Clojure was known as a language that is not beginner friendly. For sure, the unusual order of the parenthesis contributed to this reputation, but also the complexity of the required tools in order to achieve a simple task like running a “Hello World” program.

Rich Hickey (the inventor of Clojure) often says that good Programming Languages are like Musical instruments: they are designed for professionals. You need to learn (and sometimes suffer during the learning period) in order to become a professional. This might hurt at the beginning. And even for professionals, it require some effort to perform at their best. Look at the fact of John Coltrane when he plays the saxophone!

Coltrane

Anyway, the order of the parenthesis is not going to change: Clojure is and will always be a LISP dialect.

But since the release of Clojure 1.9.0 in December 2017, Clojure features command-line tools that make it very easy (and not only simple) to run Clojure programs.

CLI tool installation

On Mac and Linux, it is very straightforward to install the Clojure CLI (Command Line Interface) tools; on Windows, you need to first install WSL on Windows and then follow the same instructions as for Linux.

On Mac

brew install clojure

On Linux:

curl -O https://download.clojure.org/install/linux-install-1.9.0.394.sh
chmod +x linux-install-1.9.0.394.sh
sudo ./linux-install-1.9.0.394.sh

See Getting Started for more detailed instructions.

If you are not familiar at all with the Clojure syntax, you will not be able to enjoy this article before having being told that in Clojure, the function symbol appears inside the parenthesis and the function arguments are separated by whitespaces.

Evaluating Clojure expressions

Once the CLI tool is installed, a program named clj is available for you. And now, it is a real pleasure to evaluate small Clojure code snippets with the -e flag.

> clj -e '(+ 1 2)'
3

It is a good idea to use single quotes instead of double quotes so that you don’t need to escape strings quotes when concatanating strings (Clojure supports only strings with double quotes):

> clj -e '(str "1 + 2 is: " (+ 1 2))'
"1 + 2 is: 3"

When your code snippet contains several expressions, the CLI prints the value of all the expressions one after line:

> clj -e '(+ 1 2)(+ 2 3)'
3
5

Running a Clojure file

Sometimes, you don’t want to evaluate a series of expressions but you want to run a program for its side effects like printing a cool message in the console. What message? “Hello world” for instance!

For that noble purpose, you write your Clojure code in a Clojure file and you pass to the CLI the path of the Clojure file.

> echo '(def my-name "world") (println "hello" my-name)' > /tmp/hello.clj

And then, you execute your file like this:

clj /tmp/hello.clj

A note that Clojure beginners should skip:

— start of the note —

If you run the exact same code with the -e flag as exposed in the previous section, the CLI will display the evaluation of both expressions:

  1. (def my-name "world") evaluates to #'user/my-name - this is the fully qualified name of the variable created by def
  2. (println "hello" my-name) evaluates to hello world

Try it by yourself:

> clj -e '(def my-name "world") (println "hello" my-name)'
#'user/my-name
hello world

— end of the note —

The REPL

But the best of all is obviously the REPL, the famous Read Eval Print Loop, one of the heroes of the LISP tradition. The REPL is an interactive development environment that allows you to evaluate successive Clojure expressions interactively.

You launch a Clojure REPL, simply by typing clj with no arguments:

> clj
Clojure 1.9.0
user=> (+ 1 2)
3
user=> (+ 4 5)
9
user=>

Now it is your turn to perform and create the Clojure code of your dreams!

This is only the tip of the iceberg, you can do much more powerful tricks with Clojure CLI like running a full Clojure Program with many namespaces or manipulating aliases to setup the perfect Java Classpath. Unfortunately, it is far beyond the scope of this article. If you want to learn more about it, feel free to read the Clojure CLI official guide.

Permalink

100 Days Of Clojure Code - Day 2

Alright, so I realize that a big part of this challenge is to struggle... and do it publicly. So what I'm gonna do here is dig up one of my dark secrets. Something that I couldn't figure out and instead of asking for help, just decided to bury it down somewhere where no one would ever find it... and that way no one would ever know...

No one would ever know that I don't know how to handle exceptions.

What!?

Yes, I got stuck on something that should be so easy, and couldn't find the answer and was too embarassed to ask, and just moved on. So today, I'm gonna try to fix it. Or else struggle some more and completely embarrass myself. Then maybe someone will tell me how to do it and then I'll feel really silly.

The project is called ctrain. I wanted a way to run 4clojure problems from the terminal. Why? Well for one, I had built myself a tiny computer with a Raspberry Pi Zero:

Pi

It was inspired by Star Trek, and I made the case out of the box that the screen came in. But having just a single-core processor, and such a tiny screen, doing anything in the browser or even a graphical environment was super-awkward if not impossible. What I wanted was a cool little portable coding computer, just to do 4clojure problems and stuff. But I wanted to do it all from the command line, without even needing to start a desktop session.

First I needed a way to get the problems. There's over 100 of them and I was not about to start copying them manually. We're learning programming to make our lives easier, right? The code for the site is on Github, so I cloned the repository and got it running locally according to the instructions in the README. This is a fun thing to do, and a worthwhile exercise in dealing with databases because it uses MongoDB. This alone took me several attempts, and I think I gave up more than once. But it sure was rewarding when I finally got the db to connect.

Success! I had my very own copy of the 4clojure site running on my machine. Now to figure out how to get the problems out of the database.

I probably could have used a JSON parser, but instead I looked at the code and saw that it used CongoMongo, so I loaded it up in a REPL to try it out. It was actually pretty simple just to run a query to make it spit out all its guts, which I saved to a file.

I made myself a little convenience macro for that, by slightly modifying the pp macro:

user=> (source pp)
(defmacro pp                      
  "A convenience macro that pretty prints the last thing output. This is              
exactly equivalent to (pprint *1)."
  {:added "1.2"}
  [] `(pprint *1))
user=> (defmacro pf 
  "A convenience macro that pretty prints the last result to a file named pf.edn. "
  [] `(pprint *1 (clojure.java.io/writer "pf.edn")))

Then, when I was all done scraping the dataset, I noticed that there is already a file called data_set.clj with the problems in it!

It's OK to point and laugh at me right now.

So then I had a file with my problem data, which is a giant vector of maps:

[{:_id 1 :title "Nothing but the Truth"
    :tests ["(= __ true)"],
    :description "Complete the expression so it will evaluate to true."}
 
  {:_id 2 :title "Simple Math"
   :tests ["(= (- 10 (* 2 3)) __)"]
   :description "Innermost forms are evaluated first."} ...

So that means I could slurp it in and grab any problem by its index:

(def problems
  (read-string (slurp "problems")))
  
(defn problem [n]
  (println (str "\n#" n ": " (:title (nth problems (- n 1)))))
   (println (str "\n" (:description (nth problems (- n 1)))) "\n")
   (run! println (:tests (nth problems (- n 1))))
   (spit (str "ans-" n)(read-line)))

We print out the problem's title, description, and each of the unit tests. Then we get input from the user and save it to a file on disk.

Next, we need to replace the double-underscore (__) in the problem with the answer, evaluate it, and compare the answer to what it's supposed to be:

(defn final [results]
  (loop [coll results]
    (if (empty? coll)
      (do
        (spit "prob" (inc (read-string (slurp "prob"))))
        (println "\nGOOD JOB! Here's the next one:")
        (-main)))
      (if (= false (first coll))
            (do
              (println "\nNope... try again or Ctrl+C to quit")
              (-main)))
            (recur (rest coll))))

(defn evaluator [answers]
  (loop [totest answers results []]
    (if (empty? totest)
      (final results)
      (recur (rest totest) (conj results (eval (read-string (first totest))))))))

(defn replacer [n]
  (let [ans (slurp (str "ans-" n))]
    (loop [tests (:tests (problems (- n 1))) replaced []]
      (if (empty? tests)
        (evaluator replaced)
        (recur (rest tests) (conj replaced (s/replace (first tests) "__" ans)))))))

And it worked! But after playing with it for awhile, I noticed that something wasn't right. I tried entering a blank line, and it said it was correct! My program must be broken!

What could be causing this behavior? Eventually I figured it out.

The unit tests all ask whether the result of evaluating your answer is equal to something. And anything is equal to itself!

See:

(= 1)

So it turns out there was nothing wrong with my program, I just needed to make it reject a blank answer:

(if (= ans "")
        (do
          (println "Nice try, but blank answers are not allowed.")
          (-main)))

At this point the program totally works. That is... until you enter an undefined symbol! We are (rather dangerously) using eval on arbitrary input, so the program will crash if you forget to quote a list or something. And on a Pi Zero, firing up the JVM takes like a couple minutes. This can be avoided by running it from the REPL, of course, but this should be a simple case of catching that exception.

Unfortunately, every attempt I've made has failed, and since this is such a weird application, I don't know how to ask for help.

How do you evaluate something, without evaluating it?

4clojure uses Clojail, a library for sandboxing Clojure code. So I suppose I could use that, but for this purpose, I think that is a little bit much.

Shouldn't I be able to just use a try/catch block?

(defn safe-eval [exp]
  (try
    (eval (read-string exp))
    (catch Exception e
      (prn "Oops... Undefined symbol")
      (-main))))

Unfortunately this doesn't work, and I feel that my novice is showing real bad right now.

It catches the undefined symbol, but also the good answers!

What am I doing wrong?

See this video for an entertaining demo.

See you tomorrow!

Permalink

Data Science Screencasts from Lambda Island

Lambda Island has produced a high-quality Clojure-focused screencast every few weeks over the past couple of years. Arne Brasseur is the author and producer of these videos. To date he's generated diverse walk-throughs on language fundamentals, library APIs, architectural considerations, and development tooling. The most recent two screencasts, numbers 43 & 44, have signalled a move into the realm of data science. Both screencasts make heavy use of kixi.stats: a library of statistical reducing functions. As the author of kixi.stats, I'm relieved to say that I think that the topics are dealt with really well.

The screencasts are intended for a broad audience, and the material covered is beginner-friendly. The first video could more accurately be titled "Exploratory data analysis with Clojure's Transducers", but in any case this is worthy material for a video of this length. If you want to understand how Clojure's transducers can be used to create summary statistics in a single pass over the data, and how to render empirical distributions to histograms and interpret them, this is a great place to start.

The second video builds towards the creation of a simple linear model. Along the way, standard data science concerns regarding data scrubbing, feature selection and model evaluation are covered. The screencast also demonstrates how to make use of transducers and composite reducing functions to achieve non-trivial results. For example, at one point we are shown how to create a reducing function which returns a linear model expressed as a function of x and y. Chaining a post-complete function to a reducing function to return a function?! It's functions all the way down!

A free trial is available, and discounts are available for those unable to cover the full fee. Whether or not you decide to subscribe, the code for each episode is freely accessible on GitHub. For example, here's the code for episode 43.

For me, the graphs namespace is particularly noteworthy because Clojure currently lacks a flexible high-level charting library. As a complete tangent to the main content of his screencasts, Arne has done an excellent job of spiking out a set of functions which could form the basis of such a library. Watch this space...

Disclosure: Henry was granted free access to Lambda Island in exchange for advising on the content of the episodes linked above.

Permalink

Multivolume rendering in Jupyter with ipyvolume: cross-language 3d visualization

The Jupyter notebook is becoming the standard environment for data science in many fields, with it comes the requirements of visualization. Two-dimensional visualization is arguably the most important, and there is a rich set of libraries to choose from. Many of these build on top of the workhorse of visualization in the Python world: Matplotlib. Some other, such as plotly, bokeh and bqplot take better advantage of the browser by providing fully interactive plots, being able to zoom, pan and make selections with high framerates and smooth transitions.

Although 3D visualization is used less often, sometimes it is essential to understand intrinsic 3D datasets (e.g., a brain scan), or complex structures that are difficult to comprehend with 2D projections.

Transparency in 2d, not a problem. Here showing a million points in bqplot (an open PR).

However, 3D visualization is hard. While transparency in 2D is trivial; it requires just blending operations, is nearly impossible to do correctly in 3D at acceptable framerates. In fact, it is an area of active research.

For large datasets in 2D, it does not make sense to plot each individual point. Instead, it makes more sense to work with statistics of the data. An approach that datashader and vaex take is to reduce the data to a 2d histogram (or any other statistic) and visualize this as a heat map.

Showing 150 million taxi pickup locations in NYC by showing a heat map instead of individual points, using vaex with bqplot

Volume rendering

The same technique can be used in 3D, except the rendering technique of 3D cubes is more difficult: it uses volume rendering, more specifically volume ray casting. With this technique, a ray is cast through the scene for each pixel on the scene and accumulates RGB and alpha values for each pixel in the 3d volumetric cube, and blends them together.

(left) Big data: exploring 100+ million rows by volume rendering a 3d histogram. (middle) Astronomical data cube: Radio observations. (right): Medical data cube: Scan of a male head.

Some datasets are intrinsically volumetric cubes. In radio astronomy two of the axes are sky coordinates, and a third axis is the frequency axis, forming a 3d intensity cube of (usually) emission. In medical imaging, the three axes are most usually just the spatial dimensions.

Multivolume rendering

Sometimes, multiple large datasets or multiple data cubes need to be rendered in the same scene. They could be overlapping, fully, partially, or not at all. To make things even more complex, a different volume rendering technique called maximum intensity projection works in a different way that makes it different to combine them in the same scene. In fact, I would argue it makes multivolume rendering at least as difficult as transparency techniques in 3D.

Glue-Jupyter uses ipyvolume for 3D rendering and also provides a richer UI and higher level data bookkeeping.

Glue to Glue-Jupyter

Glue is a desktop application for multi-dimensional linked-data exploration and uses multivolume rendering in its 3D visualizations, for instance, to visualize 3D selections (as demonstrated in the above screencast). Glue-jupyter (a project I am working on) aims to bring glue to the Jupyter notebook. It, therefore, requires a solution for multivolume rendering in the Jupyter notebook. Glue-jupyter is still in early development, but already provides an easy modern way to explore (multiple) datasets interactively, while at the same time giving programmatic access from the notebook.

Ipyvolume

Ipyvolume is the best 3D visualization package for the Jupyter notebook (disclaimer: I am the main author ;). With ipyvolume, you can do scatter plots, quiver plots, and multivolume rendering in one scene, with just a few lines of code, it does not get any easier!

Simple example showing volshow, scatter and quiver.

Moreover, ipyvolume is built on top of ipywidgets (Jupyter widgets for the frontend part), giving you all the features that it provides out of the box. Examples render live in the documentation, plots are easy to embed on a static HTML file, the ability to easily link properties together on the frontend (browser) or kernel side (Python process), and listen to any change of a property (a byproduct of the bidirectional communication). Also, ipyvolume reuses pythreejs, exposing a large part of the threejs API for free! And using ipywebrtc, we can record movies, stream them to a remote computer or take snapshots.

Multivolume rendering: male head scan and dark matter particle simulation in one scene.

As of version 0.5, ipyvolume supports multi-volume rendering, allowing you to visualize two datasets on top of each other. In the example on the left, we show a scan of a human head and at the same time a 3D histogram of the particles in a dark matter simulation. Any number of data cubes can be put in the same scene, a big feat for visualization in the Jupyter ecosystem! As far as I know, it is the first package to do the multivolume rendering for Jupyter.

What about large data cubes?

In case you are using a remote Jupyter notebook, possible using Jupyter Hub, you will have the best of two worlds: your code is close to the data and the visualization is local on your laptop/desktop. However, that means that the data needs to be transferred to the client/browser, which for large data cubes can be considerable (yeah, anything to the power 3 is usually large). To solve this, ipyvolume will send a low resolution data cube by default to the browser. If you zoom in, a change in the coordinates of the bounding box is detected (yay ipywidgets!), and a zoomed in higher resolution cutout version will be sent to the browser.

After zooming in, a new high-resolution cutout will be sent to the browser.

Cross-Language

The joy does not stop here, since most of the code of ipyvolume is frontend code (JavaScript) that runs in the browser, and there is not much preventing us from using it for another language. The BeakerX team at Two Sigma have already shown it can be used from all the JVM languages (Java, Clojure, Groovy, Scala, …).

Now together with the QuantStack team, we are building xvolume, a C++ binding to the frontend code of ipyvolume (yeah, I should rename that to jupyter-volume right?).

This means that with a single code base (plus some per-language glue) we can have a serious 3D visualization package (ipyvolume) for many languages in the Jupyter notebook.

TL;DR version / Conclusion

As of ipyvolume 0.5, we have a serious 3D visualization package for the Jupyter ecosystem that can even do multivolume rendering with mixed rendering techniques, combined with regular meshes, scatter and quiver plots. Reusing the front-end code, and being a Jupyter widget, we can reuse the library in all the JVM languages and C++.

Special thanks to Casper van Leeuwen and all other contributors for this 0.5 release!


Multivolume rendering in Jupyter with ipyvolume: cross-language 3d visualization was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Permalink

Why are actions hard to test by definition?

Functional programming divides the world into actions, calculations, and data. Actions are hard to test by definition, and we explore why.

Transcript

Eric Normand: Why are actions hard to test by definition? Hi, my name is Eric Normand. These are my thoughts on functional programming.

I was thinking the other day that people often say that functional code is easier to test, especially when you’re talking about pure functions. I totally agree. Pure functions are way easier to test than side effecting code.

I thought, because of the new perspective I’m bringing to functional programming where I’ve got these three groups, these three domains of things…I thought it would be good to see what that new perspective brings to the discussion.

When I was thinking about it, lo and behold, the definition of actions is exactly what makes it hard to test. Let’s go over the definition.

Actions are anything that are bound up in time. This is usually talked about in the functional world as effects or side effects. These are things that are not part of the input or the output of a function that return value. The way I like to think about it is things that depend on when they are run or how many times they’re run.

I have a little mnemonic. I say time or times. The time it is run or the number of times it is run. If it depends on when it is run, it’s hard to test because it’s not being run at the time that you want to test it.

When you want to test it is before it’s time to actually run the thing. You want to test it, even on a different machine. You are testing it on your continuous integration server or on your development machine.

It’s definitely not the time that you want to be running that. As an example, if you want to test that your thing is going to send the right email at the right time, that’s really hard to do. You have to fudge time. You don’t actually want to send the email to your actual users when you’re testing. You want to send it to a fake email address, which is hard to set up.

Even harder to set up is we want this thing to go out on the third of the month. It’s not the third of the month today. How do you test that on the third of the month it will go out without waiting for the third of the month? You want to test it now.

You have to set it up. You have to fake time. You have to pretend to your software like it’s a different day. That is one part of being an action. That’s what makes it hard to test is, it depends on when it’s run.

If you’re talking about something more subtle like doing global mutable state…Sending an email, that is obviously an effect but mutable state, let’s say, reading from a global mutable variable, that is also an action that depends on when it is run.

You’re going to have to set up that mutable state in whatever situation, whatever configurations, whatever state it needs to be in to simulate enough of the possible states that you feel like you have a good test. That’s hard, too. You have to look at all the code and say, “What states could this be in?”

If you’re not thinking about it that way, you’re probably going to miss states. You’re probably going to only test the easy cases and not consider that if other things are writing to this at different times then we’re going to have a problem.

You’re also probably not going to test…If you’re not doing this right, a thing that you could easily mess up on is the multithreaded case where the thing is being changed as you’re reading it. If you’re in a multithreaded world and you’re using global mutable state…

Let’s say you have two variables you’re reading from, two global variables. Those things are being written to by other threads while you’re running. That could be a problem.

In your test, you would have to test for that. You would have to run enough tests to be sure, meaning enough iterations of your test, to be sure that you’re sussing out all of the race conditions so that you can test it in your build server instead of in production.

That’s time. From the definition, I’m literally trying to go for what the definition is telling us. The other thing is an action could depend on how many times it is run.

When I say that, you should be thinking, “Wait a second. I need to run my stuff a lot to be able to test it. I need to be able to run this for different test cases. I need to be able to run it on my development machine as many times as I want. Every time I make a change in the code, I want to rerun my tests. Every time I deploy something I need to rerun the tests.”

If it depends on how many times you run it, that’s a big conflict there. There are some functions that depend more on the number of times than others. That’s one of the things that we talk about when we’re talking about, “What does a functional programmer do with all these actions that depend on how many times?”

There’s an analysis you can do to figure out, “Hey, does this thing really depend on every single time is different, is a different effect? Is it maybe idempotent? The first time you call this has an effect but the second time doesn’t.”

There’s a bunch of stuff we can do, but still, that argument remains. One of the reasons why non-functional code is hard to test is because it’s full of actions, basically.

Those actions depend on when and how many times they’re run. How many times they’re run, if something depends on that it’s really hard to test. In your tests, you want to have multiple test cases. You want to have multiple…

…If you’re doing example based testing, you want at least a few for each thing, for each unit. If you’re doing property based testing, then of course you’re going to want to generate hundreds, if not thousands, of example cases.

You’re constrained here. It’s going to make it a lot harder to test. What a lot of people do is they’ll set up a little mock environment or something. All of that is a lot of work compared to if you had to test calculations, which, remember, don’t depend on when and they don’t depend on how many times they are run.

You can run them as many times as you want with different arguments. These are pure functions, basically. You run them as many times as you want with different arguments. No one’s going to care. You can test the return value. That’s all there is to test. That’s nice.

You don’t have to have it send an email and then check if you receive the email. How long do you wait because maybe email is slow? Sometimes it doesn’t get delivered right away.

Anyway, the other thing is when it is run. It doesn’t matter when they’re run. That means they could be run on a different machine. They can be run during development. They should always give the same answer.

All right, that was me riffing on this idea that by definition, actions are hard to test. Maybe later we’re going to talk about what we can do about that.

I’ve probably already talked about it in other episodes but not as explicitly in that context of testing. I’m @ericnormand on Twitter. You can also email me at eric@lispcast.com. Thanks so much for watching. I’ll see you later.

The post Why are actions hard to test by definition? appeared first on LispCast.

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.