Journal - RC5, illegal access, docs, clojure.test

Trying something new here … writing down some visible things I worked on this week (not everything is visible of course :) and some conversations I was in around the Clojure sphere. This will be pretty rambly without much editing. Maybe I’ll do this more, maybe not, let me know if it’s useful.

IllegalAccessException

I spent some time earlier this week digging into an IllegalAccessException that was reported in REBL from people trying to use Java 11 + latest JavaFX + REBL. While I have spent some time working on illegal access warnings due to reflective accesses across modules with Java 9+, this one was a little different as it involved an actual exception. It also seemed like this particular error should have been caught by my changes in CLJ-2066.

Anyhow that involved trying to make something reproducible and examining the stack trace in more detail, which really told the story. There are two different paths in the Clojure reflector and CLJ-2066 addressed one of those, but this was in the second code path, trying to invoke a field or method reflectively.

The general problem the reflector faces is having a target object, a method name, and some parameters and the goal of either finding or invoking the most appropriate method. The reflector starts from the type of the target object to look for a matching method (same name, same arity, and compatible parameter types taking into account boxing, widening, etc).

In the new Java module system, code inside a module is, by default, inaccessible by code outside the module. Modules declare which parts are visible by “exporting” packages. Reflection has full rein to examine everything, but invocations are gated by accessibility according to the module definition.

It’s relatively common for a module to export a public interface, and a factory to create instances of the interface. Usually, the implementations are private and not exported. In this case, Clojure will have a target object of the private implementation class, but the methods on that private class are inaccessible. Invoking them will yield an illegal access warning or exception. Instead, the Clojure reflector needs to look up the class hierarchy and find a public, accessible variant of the method (usually the public interface). The reflector had some of that logic already, but the additionally accessibility check needed to be added, and this is all captured in CLJ-2454.

We are in the final burn down at this point to Clojure 1.10, having released RC4 last week. This access problem is squarely in the middle of compatibility with the latest Java releases, which is one of the key targets for 1.10, so we decided this was important enough to include before we release. That resulted in the release of 1.10.0-RC5.

Docs

I’ve also spent a lot of time in the last week updating docs on the web site. There are new bits and pieces of docs for 1.10 in several places and some migrations and updates from other places:

Also, I migrated Contrib workflow, Creating Tickets, and Developing Patches from the old community wiki. We plan to do a bit more clarifying some of the stuff in this area, so this is just a start. Sean Corfield requested some clarification around some of the contrib library stuff and I agree it’s probably a good time to do so as I see a lot of confusion and false assumptions about that.

I also burned through a bunch of site PRs - thanks to everyone on those. I won’t list them all but you can look at the recently closed PRs for more detail. I will highlight the addition of a new equality guide from Andy Fingerhut that is a newly updated version from the guide he has maintained for many years. Glad to finally add it!

We have a private CI system that auto-builds the clojure.org and clojurescript.org web sites. That’s been chugging along pretty much unattended for 3 years, but unfortunately the product where we have it hosted is shutting down tomorrow. So this week I’ve spent a bunch of time doing research to decide the best replacement option. Still working on that, but it will be a bit more manual till that’s up and running.

clojure.test

I had a long and interesting conversation on Slack this week about clojure.test, originally triggered I think by a tweet from Nikita Prokopov about the error reporting in clojure.test. I think there were several independent suggestions embodied in his changes there - focusing on root cause in stack trace reporting (good), filtering stack head to remove test infrastructure noise (good, and already being done to some degree), filtering other parts of the stack (I’m more dubious about), demunging Java to Clojure in stacks. I have mixed feelings about the last, but maybe reasonable in the context of clojure.test reporting.

Sean Corfield broadened the conversation to talk about clojure.test as a general area of improvement (and why it has not seen much love in years). Certainly many people have written their own take on test libs (even Stuart Sierra who was largely responsible for clojure.test, wrote a second version!) or test runners. So clearly, there is no shortage of ideas in this area.

But clojure.test has the privilege / onus of being “the one in the box”. I think Stuart S can garner a great deal of credit for it being good enough that many people are happy enough to get by using nothing more than that, even without substantial follow-on improvements. And certainly there is no reason you have to use it - all these external libs exist, do good stuff, and are used. But probably there are a set of changes for clojure.test that would be useful to people.

One thing that we have talked about is splitting out libraries like clojure.test into detached libs, like we did with spec. These could still be pulled in by default, but making them independent would let us rev them independently and more frequently than Clojure itself.

Both Sean and Rick Moynihan seemed interested in trying to come up with a list of improvement areas and that would be a helpful thing. There are only a relatively small number of clojure.test tickets in jira and I don’t think they really reflect the issues people have with it. Having a vetted and prioritized list of problems that could be addressed would be a huge help in deciding what is useful to do in the future. I’d say one thing we’re NOT interested in is breaking stuff. So any changes to clojure.test should be largely additive or in extensibility rather than radical changes to the api. Certainly I think there are some things in the areas of reporting, fixtures, and test running that could be useful.

Other tidbits

  • Uploaded the Day of Datomic Cloud videos from Strange Loop
  • Doing prep for clj and the Clojure api docs in relation to 1.10 release
  • Updated many CLJ tickets, did some screening+prescreening for 1.11, cleaned up many tickets
  • Starting to look at some tools.deps stuff again

Non Clojure stuff I enjoyed this week…

New Vulfpeck album, Hill Climber! I’m moderately obsessed with Vulfpeck and their unrelenting funkiness. Everyone should have more Vulf in their life. Sleeper track: Disco Ulysses.

Really good podcast with Ezra Klein interviewing Hasan Minhaj. Although actually, I think Hasan had some of the best questions for Ezra. Hasan’s new show is really good, also check that out.

Permalink

Clojure eXchange 2018

A group of 5 Clojure enthusiasts from Solita visited the Clojure eXchange conference in London on December 3rd and 4th. Clojure eXchange, or ClojureX, was organized for the 8th time now and this year’s event was the largest in its history. The conference was single-track and spanned two full days of talks on all things Clojure.

the Solita crew

The conference was organized by Skills Matter at Codenode. Signal was kind enough to sponsor the conference to avoid lack of conference t-shirt. There weren’t any other sponsors, but the conference itself was organized in a very professional manner.

The talks were very pragmatic, with many first-hand experiences on solving real world problems with Clojure. Despite the pragmatic focus, it was neat to see the continued tradition of having some of the more abstract tar pits explored as well ;)

Day 1

The welcoming ceremony had everyone stand up, and gradually sit down based on how long they had been using Clojure. This revealed us the following facts:

  • More than half of the audience had over 3 years of Clojure experience
  • Christophe Grand didn’t understand the instructions, or claimed the seniority by being the last-man-standing at the 10+ year mark*

This continues to make the case for Clojure’s maturity and production viability.

[*] We are inclined to believe in the latter

Keynote: The Spirit of Clojure

Karsten Schmidt kicked off the conference with a keynote filled with visual programmatic art.

Visual art by Karsten

Karsten has no fear of diving into new languages while solving problems in unfamiliar areas, reading research papers, writing programs in literate style and learning new tooling along the way (Emacs!). This is quite a feat to do :) Clojure made him feel like a novice again, a feeling that resonates well with many in the community.

Karsten, the author of the Thi-ng family of libraries, showcased many illustrative projects created with it, using formulas like De Jong attractors, Hilbert curve, Gyroid formula and many others. Volumetric visualizations allow sampling the formulas to generate views in different resolution, which allows you to not think in polygons. Karsten had also done data visualization projects and it was neat to see what the visual presentation of the data tells even without labeling.

He ended with a couple of notes: One on how people in different language communities share same ideas but use different names (Clojurians seldom mention Sparql but talk about Datomic, GraphQL is known in Javascript land but not Datomic). Another point he made was on the difficulty of writing performant code in Clojurescript, when immutability is the default, since one has to write more code to “undo” immutability.

In the latter point, it might be better to write such tight loops in plain Javascript. A runtime that does this for you would probably be a worthy research subject.

The keynote was interesting but there was quite a big difference between what the title made it sound like it was and what the talk itself ended up being.

Polymorphism à la Clojure: Functional Object Oriented Programming is not an Oxymoron.

Yehonathan Sharvit, author of klipse (executable slides via klipse), presented the Oxymoron (e.g. “The painting is an original copy”) of OOP in functional style.

He did an exploration into Object Oriented Programming style first via plain maps and then using protocols. In the first example, he used a send-msg function to talk to the objects which are a map with data and operations and returning a new object on accepting the message since mutation is bad: “When a person drinks, he’s not the same person” :). In this style of intermediate results, threading macro is your friend.

The protocol example used defrecord for implementing a protocol, which get’s rid of a send-msg. He presented five commandmends as a view to OOP in Clojure.

Couple of takeaway points:

  • Beware that dissoc makes a record into a map.
  • Although records cannot be used as functions, one can implement the IFn on a record via extend-type.

Data Science in Languages I Don’t Understand

Rob Stanley gave a speech on an implementation of a data driven system for helping farmers all around the world to perform their jobs more efficiently.

A big part of the software was Natural Language Processing, namely, detecting the language and intent of questions sent by farmers using SMS. Implementation consisted of Clojure code for the mathematical models, and a custom EDN formatted DSL for defining patterns that are used for the actual intent prediction.

A general feeling from the speech; Data Scientists are capable of learning and using Clojure (who would have known?), but Data Science in Clojure still requires writing a lot of your own plumping.

Now you’re Speaking my Language! : Building, Maintaining and Using a Patient-Friendly Medical Ontology

Chloe Pont and Sam Oleary from Health Unlocked showed how they used Clojure in building a social health discussion website.

The domain deals with text processing (synonyms, misspellings, acronyms and abbreviations) and a medical hierarchy of treaments and conditions. The main part of the talk was the ontology used to power the website.

The ontology is stored in a Clojure map (presumably in an EDN file), updated via code scripts and the resulting map is diffed via editscript library, which yields changes that are applied to a relational database in which the result is stored in a table structure, served via an API (a Clojure app) for use by the site. They later made a dsl to limit and verify changes to apply in the ontology change scripts. The ontology is used on the site for suggesting tags for posts, autocompletion and widening search by term detection.

The last question from the audience was most revealing: What database was used? The answer, MySQL, triggered spontaneous laughs from the audience :). Apparently the company previously had a Neo4J based system that was bloated and instead of extending that fragile system, they decided to reimplement what they actually needed in a simpler way, which seems fair enough.

Lightning Talk: Building A Simulation-Based Text Adventure

Peter Westmacott showcased a text-based browser game he’s been writing with ClojureScript. Being highly systems-driven akin to titles like Dwarf Fortress and Prison Architect, the game runs a simulation of a village and its inhabitants with the aim of modelling social interactions and exploring the concepts of contagious violence and scapegoating. Peter found Clojure’s declarative nature, as in open maps and data type literals, a boon for this kind of exploratory simulation work. Other topics such as obscure French philosophical anthropology were also covered, as per usual in a FP conference. Peter also managed to cram a flashy demo in his 10-minute talk, which was much appreciated by the crowd.

Lightning Talk: Running without an API

In this lightning talk Gaivile Vezeviciute presented an application that showed whether a running event in a running community called ParkRun was cancelled. The premise was that the events were announced on the community website, but there was no way to be notified about cancellations other than checking the website listing all the events. To solve this problem, Gaivile developed a Clojure application to check the status of the events she was interested in. As there was no API to check this information, her application scraped the website and checked the status of the events given as a parameter. The app got its start at London Clojure Dojo, an event where people gather to level up their Clojure skills through practical exercises.

This talk was a good fit for the lightning talk format. The problem statement was easy to explain and understand and there was enough time to present the solution in proper detail.

Lightning Talk: Making Music With Alda & Edna

Another eloquent speaker, Tristram Oaten demoed Alda and Edna, music programming languages for musicians and Clojurians. Tristram started with a brief intro on music theory and moved on to building progressively more complex compositions using an edn syntax. What stood out the most was the clever usage of Clojure data structures to express musical concepts such as chords.

Learning How to Design Automatically Updating AI With Clojure, Kafka and Deeplearning4j

Jason Bell is an author and at least locally known speaker since, starting at the apology at the start of the talk, half of the audience would miss the jokes :).

Jason went through an example of piping events through kafka and training models with Deeplearning4j (neural network), Weka (decision tree) and Apache Commons Math (linear regression). The example project was quite event-based, with separate topics for predictions requests and responses, and a Kafka Streams application for branching messages by content (training data/build model command) from an event topic.

Some libraries spotted were a Deeplearning4j wrapper jutsu.ai, Weka wrapper clj-ml (although it is a bit dated) and kixi.stats.

Are You Writing Java in Clojure?

Erik Assum’s talk was one (of many) that we most waited for.

With a background on working on legacy code bases and seeing code which is hard to test or reason about, Erik offered stable advise on how to structure your code to maximize reasonability and testability.

Erik had many good references in his talk, one of which was Stratified Design from the SICP book, describing how such design doesn’t necessarily mean a 3-tier architecture. He also referred to the imperative shell/functional core idea from Gary Bernhardt’s Boundaries talk. Make sure to reserve time to go over the references, they are good quality and well summarized in the talk.

With the help of the Elements of Clojure book he descibed solid ways to structure code to separate pull/transform/push phases. He also showed bad code that his boss wrote, how often would you do that in a conference? :)

Speed Bumps Ahead

Alexander Yakushev, who among other things, maintains the Clojure goes fast! website, gave a good presentation on performance optimization.

He had an interesting example where he compared the way Clojure compiler uses reflection to plain direct reflection use. It turned out that in this example, reflection itself wasn’t the main offender, but the Clojure compiler implementation, which is forced to loop over all methods of a class to find one that matches the argument types at runtime.

Even more interestingly, he used clj-java-decompiler, a library which decompiles Clojure code into semi-readable Java, to narrow down to the Clojure compiler implementation that revealed the above fact.

Alexander is the author of many useful libraries found in the clojure-goes-fast Github organization, such as clj-memory-meter and clj-async-profiler. Another interesting take was writing code that requires more tight control over performance in Java and using virgil to still keep the REPL workflow (one doesn’t have to quit the repl to recompile/load the Java classes).

Lightning Talk: Beginners Introduction to Clara Rules

Charlotte Fereday from ThoughtWorks (our Apax sibling) presented clara-rules, an expert system/rule engine library for managing complex and volatile business logic. Clara takes in a set of forward chaining rules or conditions in form of if A then B and then evaluates facts against them. The purpose of a rule engine is to simplify writing business logic into code. While probably overkill for simple cases, the library might work wonders in untangling convoluted conditionals into simple self-contained rules. Unlike enterprise rule systems, Clara offers a lightweight developer-driven approach based on functional programming, with all rules and facts stated out in plain Clojure.

Lightning Talk: Monitoring Clojure Applications with Prometheus

Joachim Draeger from the conference’s sponsor Signal, demonstrated monitoring Clojure microservices via Prometheus, an open source monitoring toolkit originated at Soundcloud. Joachim started by stressing the importance of visibility and metrics in distributed systems and then went on to shed light on the approach they’re taking at Signal. The presented solution seemed relatively plug-and-play: pull in a Ring middleware from the Clojure Prometheus client library, configure it to expose a metrics API endpoint and point your Prometheus instance to it. Prometheus then takes care of scraping data from the endpoint and feeding it onto a neat Grafana dashboard. The client library supports all sorts of custom metrics but also includes pre-made solutions for exposing JVM and Ring handler metrics, which seemed very compelling.

Brewing CIDER: It Starts with an Orchard

Bozhidar, the community’s beloved Rockstar had a very entertaining presentation again :)

He continued to make it clear that Cider, although being a Emacs extension (that Rocks!), is backed by a wealth of tooling libraries made by the Clojure community itself. One not so often mentioned library was MrAnderson, which relocates dependencies of the tooling libraries to avoid conflicts with the project using the tooling.

nREPL, the library that enables tooling for Emacs, Vim, VS Code (Calva, named after Calvados, which is distilled from Cider :)) and many more clients has a new logo and documentation website. The design of nREPL seems solid and a lot can be done with middleware (Clojurescript support via Piggieback as an example). A transit middleware seems to be in the works too.

All this community work happens with marginal funding so help in this regard would be good.

Bozhidar was skeptic that socket repl and prepl would be just re-inventing the wheel instead of making a better unified whole.

The continued effort on Clojure tooling got new powers on the second day, when Bozhidar announced that Christophe Grand would be joining the nREPL team in bringing the work explored in Unrepl to Cider (elided collections, client-side injection and sideloading for starters). This could be a direction that unifies the powers of the REPL landscape, a Hegemony as Bozhidar himself put it :)

Day 2

Keynote: Zeno and the Tar Pit

Christophe Grand explored a way to escape the Tar Pit of the internal state handling in a non-trivial stateful application with the help of Greek philosophers. He coined the term “Maps Fatigue” to describe the tension of building, converting and maintaining hierarchical in-memory data presentation in programs while coping with changing requirements.

Maps fatigue

The realization is that while Clojure lifted us from imperative programming model of Java with immutable data structures, we still have iterative processing model with seqs, while iteration order is not always meaningful (nor is the intention of iteration order easy to deduce by reading code). The base of this realization is Christophe’s past few years work on stateful stream prosessing and client/server applications.

This set us on a path on exploring databases and Datalog as a solution for the Functional Relational Programming model of the Tar Pit paper. The talk pointed to a good reading resource, recommended by Rich Hickey himself :) Foundations of Databases. He covered Datalog syntax, how to make it more readable and how to introduce a time model to be able to make updates and not only queries.

There was a teaser announcement on the final slide of a joint venture with couple of Clojurists called Tensegritics Consulting.

Applying Clojure’s Design Principles to its Documentation

During Tom McTighe’s talk, one could ask if “Clojure has good documentation” is an Oxymoron :) There’s a gap between the clarity of the language and it’s general documentation. Although there’s fair amount of material, it’s not too well organized (i.e. https://clojuredocs.org vs http://clojure-doc.org).

Klipse on the https://clojure.org/ website got mentioned in discussion after the talk and Yehonathan asked for support for persuasion to make this happen, since having a quick way to try out the language on its main website would not be a bad thing.

Quick and Easy Testing for Busy Mums (and other Clojure developers)

In her talk, Eleanor Mann presented a practical and lightweight approach to testing. Having previously worked as a Java developer in an environment that valued 100% test coverage, her experience had been that even the smallest change would break a large number of tests it would take a long time to go through them and fix them or to actually see whether there was a problem.

One of the main takeaways from the talk was that the best tests are diagnostic tools. After all, one of the reasons why we write tests is to make sure others don’t break our code in the future. This means that it should be apparent why the test exists and what it means when it’s failing. What this means in practice is that if you are, for an example, writing a test for a function that returns a map and in the test you are only interested in the value of one key, you should only check that instead of the whole map. This way it is clear for everyone reading the test what is being tested and changes to other values in the map do not break unrelated tests (presumably there would then be other tests for them).

Eleanor presented the concept of test lifecycle. Part of it is that tests should be written to be maintainable. As mentioned above, the test should communicate their intent to the reader. Test code should also be treated as proper code and instead of becoming a huge mess over time, it should be improved in the same way as the application code itself is being treated. This way it stays usable in the future.

Part of the test lifecycle is also that if the test is no longer useful it should be deleted.

The latter section of the talk was dedicated to tackling a problematic test suite. This boiled down to a few points:

  • Don’t eat the whole elephant at once - you don’t need to make the whole test suite better all at once
  • Don’t continue bad practices - instead for the new test code, use better practices and improve the test code that way
  • Don’t be fooled by bad tests - there may be tests but that doesn’t mean they are good and that they should be kept
  • Uncertain is same as no test at all - test may actually not check anything useful or it might be unstable. In this case it’s better to just remove the test and add a new one.

The talk was a good summary of a practical way to writing and maintaining tests. In itself it wasn’t particularly tied to Clojure but seemed like good advice whatever your selected technologies may be.

Using Duct Framework in Production

Iván Perdomo gave an experience report on using the Duct framework. Duct is a modular framework for building server-side applications in Clojure. They moved a report generation application from Java to Clojure and while adding components to the application, realised the need to manage dependencies between components.

The presentation nicely told that Duct is actually a reference implementation on how to use Integrant. There is a tension between a library and a tool that creates a project template with a pre-defined layout set up for a particular set of libraries, since project templates are hard to upgrade. But even though Duct has a Leiningen template for starting off, it seems to have a good upgradeability via Modules, which are pure functions that can transform the project configuration.

On the hallway track, it was mentioned that Duct might be arriving to version 1.0.0 with better documentation so stay tuned :)

Lightning Talk: Some Thoughts about Immutability and its Limits

Karl Brodowsky shared a brief look into immutability, its limits and how to work around them in Clojure. Clojure’s data structures are by default immutable, in that each mutation results in a new copy instead of modifying the original data. This seems prohibitively expensive at first, but in actuality Clojure’s data structures perform some clever data sharing under covers to avoid a full copy on each mutation and are thus quite efficient. In some rare cases however it may be necessary to squeeze out those extra milliseconds and resort to mutable data. Clojure has tooling for this in form of transients, which essentially enable turning off immutability in an isolated and performance-critical part of your code. Karl advocates keeping your APIs immutable and resorting to transients internally, only when necessary and only when you know what you’re doing: keeping the rules, i.e. accepting and returning plain immutable data structures in your API means your code stays composable with other libraries and safe for multithreadeding.

Lightning Talk: Reagent and LeafletJS or “Fun with Maps”

In this lightning talk Oliver Godby spoke about using Reagent and LeafletJS to add a map component (not the data structure, the geographical map!) to a web page. Using JavaScript libraries that don’t offer existing React components to begin with may not always be easy with Reagent and ClojureScript. In case of LeafletJS, there are existing React components that could probably be used with Reagent, but even that may have its problems, especially if you are not familiar with JavaScript and React.

The gist of using LeafletJS with Reagent came down to using form-3 Reagent components. Most of the time with Reagent it is not needed to define React lifecycle methods yourself. In the case of using non-React component with state, like LeafletJS is in this case, you need to define them for the component to be initialized and updated correctly.

Due to the fact that this was a lightning talk, we didn’t get to hear further details how the implementation went. Perhaps a little more insight what was needed with LeafletJS in specific could have been brought up.

Lightning Talk: Fullstack Clojure in the Movie Business

In his lightning talk, Nigel Runnels-Moss first described their usage of Clojure for a system for predictive analysis of the movie business (e.g. calculating optimum release times for films).

However, a major portion of the talk focused on the difficulty of hiring for Clojure development positions. As a solution, Nigel suggested to “hire for empathy’, meaning to look for competent developers that are capable of communicating with you and your team, and letting them pick up Clojure on the job.

Lightning Talk: Levelling up Lacinia with Spec

Antony Woods sneaked in Clojure in his previous company but now at Functional Works, using Clojure requires less sneaky actions :)

He presented leona, a library that generates lacinia schemas from specs. This came out of the necessity of getting rid of maintaining Lacinia schemas and Specs, both of which were a fair amount of code. spec-tools provided good leverage in implementing the library.

They also use venia to generate GraphQL queries on the client side.

Clojure Embedded in Music Hardware

Chris McCormick’s talk is probably best experienced by watching and listening the sounds produced by the hardware (try out the links to mentioned later too).

Chris is fond of Clojure and uses it out of it’s normal domain, namely in embedded hardware. There are surprisingly many Clojure -like languages and Chris maintains a list of them in awesome-clojure-likes. It turns out that code that deals with common math is shareable even throughout the dialects.

Chris also demoed a to-be Kickstarter MIDI controller programmed via Ferret language, which is a Clojure dialect that compiles to C++ that runs on the microcontroller. Another demo was running lumo on Raspberry Pi, playing a mod track. The last demo was a 8-bit sequencer that’s live-coded in the browser, which was very cool!

Growing a Product with Clojure: Lessons Learned

Łukasz Korecki told a story of progressive use of Clojure in replacing parts of a product initially written in Ruby.

He touched many topics, such as debugging performance (hint, strip whitespace to speed up the regex implementation in Java) and learning to tune the JVM (although plain copy of Riemann JVM flags sounds a bit scary :)). The big win for them was the ecosystem in Java (availability of SDKs and libraries) and also in Clojure (nrepl, ring, component etc.) and the stability of Clojure itself, which makes for a good foundation for a business.

On hiring, they were lucky to get an experienced lead developer for the backend at the start and people coming in were very interested to learn the language.

REPtiLe - A Cold Blooded Collaborative REPL

Ray McDermott of defn podcast gave a very collaborative presentation that got the whole audience involved, although the REPL REPL sing-along might have worked better after a beer (or maybe two) :).

This served as an intro to the topic of collaborative coding tools that would serve better the apropos clojure screencast, where a set of panelists talk on Clojure and code in a repl. Tmux had not worked well for them so Ray had created a tool that allows multiple users share evaluation of Clojure code from a web browser.

At the end, Ray pulled in Bozhidar and Yehonathan to use REPtiLe in a demo, which turned out a bit cold, but was entertaining anyway.

Parallel All the Way

Renzo Borgatti, the author of the upcoming Clojure, The Essential Reference ended the presentations by live-coding demo of a library called parallel, which he has been developing.

Although Clojure has built-in functions for parallel computation (pmap, pvalues, pcalls, reducers/fold, core.async’s pipeline), the parallel library seems to have many useful parallelizing functions and forms like p/let, p/slurp and p/sort.

The examples presented showed very fitting use of the library, almost “just add the alias p/ to regular code”. There’s some prior art in libraries like claypoole, which allows to limit concurrency passing a threadpool for the supported functions. Nonetheless, there’s a fair amount of new useful work offered by the parallel library.

ClojureX Park Bench Panel

The conference ended with a panel discussion with Jon Pither (JUXT), Bozhidar Batsov (toptal), Charlotte Fereday (ThoughtWorks) and Joachim Draeger (Signal Media).

One topic the discussion touched was learning resources and we were happy that the Clojure MOOC got mentioned by the audience. Enhancing this course more could be a big win :) ClojureScript could also be an angle to get more people into the language since it’s availability in the browser for creating good interactive content.

Conclusion

All in all, the ClojureX event was a great one! None of us had been in ClojureX before and we can fully recommend the event. The talks were great and so was the hallway track, which is a sign of active community. All the talks are available on the conference website, so if a talk sounded interesting, you can still go watch them.

PS. One brief closing note: The Clojurians Slack #clj-commons channel seems quite active after the conference, so if you are interested in the continued maintenance of orphaned libraries, this is a good place to follow.

Permalink

Reinforcement Learning : Exploration vs Exploitation : Multi-Armed Bandits


;; Reinforcement Learning : Exploration vs Exploitation : Multi-Armed Bandits

;; I'm reading the excellent:

;; Reinforcement Learning: An Introduction
;; by Richard S. Sutton and Andrew G. Barto

;; The book's website, on which is available a complete pdf, is here:
;; http://www.incompleteideas.net/book/the-book.html

;; In Chapter 2, they introduce multi-armed bandits as a simplified model problem

;; On the basis that you don't understand anything you can't explain to a computer, I thought I'd code it up:

;; Here is a 2 armed bandit
(defn bandit [action]
(if (= action :arms?) [:right :left]
(case action
:right (if (< (rand) 0.5) 4 0)
:left (if (< (rand) 0.2) 5 0)
:oops!!)))

;; We can ask it how many arms it's got, and what they're called
(bandit :arms?) ; [:right :left]

;; And we can pull those arms. Rewards are variable.
(bandit :right) ; 4 ; 4 ; 4 ; 0 ; 0 ; 0 ; 0
(bandit :left) ; 5 ; 0 ; 0 ; 0 ; 5 ; 0 ; 5 ; 0


;; Since we can see the code for this particular bandit, we know that
;; the expected value of pulling the right arm is 2 (a half-chance of
;; a reward of 4) and the expected reward for the left arm is 0.2*5 = 1

;; So if we were seeking to maximize reward, we'd probably be best to pull the right arm all the time.

(map bandit (repeat :right)) ; (0 4 0 0 4 0 4 0 0 4 0 4 0 4 4 4 0 0 4 4 4 4 4 0 4 0 4 0 4 4 4 4 4 0 4 0 4 4 0 0 4 4 0 0 0 0 0 0 4 4 0 0 0 0 4 4 0 0 4 0 4 0 0 0 0 0 4 0 4 0 0 0 0 0 0 0 0 4 4 4 4 4 0 4 4 ...)
(map bandit (repeat :left)) ; (0 0 0 0 0 0 0 0 0 5 5 5 0 0 0 0 0 0 5 0 0 0 5 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0 0 0 0 5 0 0 0 5 5 0 5 0 0 5 0 0 5 0 0 0 0 5 0 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0 0 0 0 5 5 0 ...) ;


;; The interesting question is, if we don't know how the bandit works, how should we design an algorithm that gets the most reward?

;; One thing our algorithm is going to have to do is to record what happens:

;; At first, we know nothing, so we can set up a table to represent that we know nothing
(defn initial-state [bandit]
(into {} (for [k (bandit :arms?)] [k (list)])))

;; We haven't pulled either arm yet
(initial-state bandit) ; {:right (), :left ()}


;; Once we pull an arm, we'll have an action/reward pair
(bandit :right) ; 4
;; the pair would be:
[:right 4]

;; When we get a new action reward/pair, we'll update our state
(defn update-state [state [action reward]]
(update-in state [action] #(conj % reward)))

(update-state (initial-state bandit) [:right 4]) ; {:right (4), :left ()}

;; here are some examples of using update-state
(update-state {:right (), :left ()} [:right 2]) ; {:right (2), :left ()}
(reduce update-state {:right (), :left ()} [[:right 2] [:left 3] [:right 4] [:right 5]]) ; {:right (5 4 2), :left (3)}
(update-state (initial-state bandit) [(rand-nth (bandit :arms?)) 2]) ; {:right (), :left (2)}


;; Once we actually have some data, we can make estimates of the expected rewards

;; We'll use as our estimate of the value of an action the average value seen so far, or zero if we have no information

;; To help with this, a couple of utility functions:

;; average-list tells us the average value of a list of numbers, with a default value if the list is empty.
(defn average-list [lst default] (if (empty? lst) default (/ (reduce + lst) (count lst))))

(average-list (list 1 2 3 4 5) 0) ; 3
(average-list (list) 10) ; 10
(average-list (list 1) 2) ; 1

;; mapvals applies a function to every value in a map, returning a new map with the same keys
(defn mapvals [m f] (into {} (for [[k v] m] [k (f v)])))

;; examples
(mapvals {} inc) ; {}
(mapvals {:a 1} inc) ; {:a 2}
(mapvals {:a 1, :b 2} inc) ; {:a 2, :b 3}
(mapvals {:a 1, :b 2, :c 3} #(* % %)) ; {:a 1, :b 4, :c 9}


;; In the book, Q_t(a) is the current estimate (at time t)
;; Using the two functions, we can define our estimate so:

(defn Q [state] (mapvals state #(average-list % 0)))

;; examples
(Q '{:right (5 4 2), :left (3)}) ; {:right 11/3, :left 3}
(Q '{:right (5 4 2), :left ()}) ; {:right 11/3, :left 0}
(Q (initial-state bandit)) ; {:right 0, :left 0}
(Q (update-state (initial-state bandit) [(rand-nth (bandit :arms?)) 2])) ; {:right 0, :left 2}











;; If we have estimates of the value of each arm, then a good way to
;; use them is to pull the arm with the highest estimate.

;; This is called 'exploitation', as opposed to 'exploration', which
;; is when you try things you think may be suboptimal in order to get
;; information

;; The 'greedy' action is the one with the highest expected value

;; To help with this, another utility function:

;; max-keys finds the keys with the highest value in a map, and returns a map with just these keys
(defn max-keys [m]
(let [slist (reverse (sort-by second m))
[_ max] (first slist)]
(take-while #(= (second %) max) slist)))

;; examples
(max-keys {}) ; ()
(max-keys {1 0}) ; ([1 0])
(max-keys {1 0, 2 0}) ; ([2 0] [1 0])
(max-keys {1 0, 2 1}) ; ([2 1])
(max-keys {1 0, 2 1, 3 -1 , 4 -3, 5 2, 6 2}) ; ([6 2] [5 2])

;; if there is a tie for the greedy action, we choose at random between the candidates
;; And so we can go from estimates to greedy action like this:
(defn greedy-action [estimates]
(first (rand-nth (max-keys estimates))))

;; examples
(greedy-action '{:right 10, :left 3}) ; :right
(greedy-action '{:right 10, :left 3 :centre 20}) ; :centre
(greedy-action '{:right 10, :left 3 :centre 3}) ; :right
(greedy-action '{:right 3, :left 3 :centre 3}) ; :right ; :right ; :centre ; :right ;

(greedy-action (Q '{:right (5 4 2), :left (3)})) ; :right
(greedy-action (Q '{:right (), :left (3)})) ; :left
(greedy-action (Q (initial-state bandit))) ; :left

;; OK, so we have our stage set, a way of recording what's happened, and some helpful functions defined.


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; Our first try at a learning algorithm will be 'by hand', as it were.

;; We'll always make the 'greedy' choice.

;; At first, we have no records to go on
(initial-state bandit) ; {:right (), :left ()}

;; expected values for both levers are therefore zero
(Q (initial-state bandit)) ; {:right 0, :left 0}

;; so the greedy action will get chosen at random
(greedy-action (Q (initial-state bandit))) ; :left

;; in this case, we've chosen :left, and the bandit's response is
(bandit :left) ; 0

;; we record it
(update-state (initial-state bandit) [:left 0]) ; {:right (), :left (0)}

;; and we have a new state
'{:right (), :left (0)}

;; new estimates
(Q '{:right (), :left (0)}) ; {:right 0, :left 0}

;; and again, we choose at random
(greedy-action (Q '{:right (), :left (0)})) ; :left

;; the bandit is not feeling very generous
(bandit :left) ; 0

(update-state '{:right (), :left (0)} [:left 0]) ; {:right (), :left (0 0)}

;; new state:
'{:right (), :left (0 0)}

;; new estimates
(Q '{:right (), :left (0 0)}) ; {:right 0, :left 0}

;; this time we choose :right
(greedy-action (Q '{:right (), :left (0 0)})) ; :right

;; and the bandit pays out!
(bandit :right) ; 4

(update-state '{:right (), :left (0 0)} [:right 4]) ; {:right (4), :left (0 0)}

;; You get the idea......

;; Let's automate that....

;; Given a state and a bandit, we decide an action and the bandit
;; responds, producing an action/reward pair, and a new state

(defn greedy-algorithm [bandit state]
(let [action (greedy-action (Q state))
reward (bandit action)]
[[action reward] (update-state state [action reward])]))


(greedy-algorithm bandit (initial-state bandit)) ; [[:left 0] {:right (), :left (0)}]

;; To get something we can iterate:

(defn step [[[a r] state]]
(greedy-algorithm bandit state))

(iterate step [ [:dummy :dummy] (initial-state bandit)])

;; ([[:dummy :dummy] {:right (), :left ()}]
;; [[:left 5] {:right (), :left (5)}]
;; [[:left 0] {:right (), :left (0 5)}]
;; [[:left 0] {:right (), :left (0 0 5)}]
;; [[:left 0] {:right (), :left (0 0 0 5)}]
;; [[:left 0] {:right (), :left (0 0 0 0 5)}]
;; [[:left 0] {:right (), :left (0 0 0 0 0 5)}]
;; [[:left 0] {:right (), :left (0 0 0 0 0 0 5)}]
;; [[:left 0] {:right (), :left (0 0 0 0 0 0 0 5)}]
;; [[:left 0] {:right (), :left (0 0 0 0 0 0 0 0 5)}]
;; [[:left 0] {:right (), :left (0 0 0 0 0 0 0 0 0 5)}]
;; [[:left 0] {:right (), :left (0 0 0 0 0 0 0 0 0 0 5)}]
;; [[:left 5] {:right (), :left (5 0 0 0 0 0 0 0 0 0 0 5)}]
;; [[:left 0] {:right (), :left (0 5 0 0 0 0 0 0 0 0 0 0 5)}]
;; [[:left 0] {:right (), :left (0 0 5 0 0 0 0 0 0 0 0 0 0 5)}]
;; [[:left 0] {:right (), :left (0 0 0 5 0 0 0 0 0 0 0 0 0 0 5)}]
;; [[:left 0] {:right (), :left (0 0 0 0 5 0 0 0 0 0 0 0 0 0 0 5)}]

;; In this case, the greedy algorithm happens to get a payout on its
;; first try, and decides that it will pull that arm for ever. It
;; never even tries the other arm.

;; Try again:

(iterate step [ [:dummy :dummy] (initial-state bandit)])
;;([[:dummy :dummy] {:right (), :left ()}]
;; [[:right 0] {:right (0), :left ()}]
;; [[:right 0] {:right (0 0), :left ()}]
;; [[:left 0] {:right (0 0), :left (0)}]
;; [[:right 4] {:right (4 0 0), :left (0)}]
;; [[:right 4] {:right (4 4 0 0), :left (0)}]
;; [[:right 4] {:right (4 4 4 0 0), :left (0)}]
;; [[:right 4] {:right (4 4 4 4 0 0), :left (0)}]
;; [[:right 4] {:right (4 4 4 4 4 0 0), :left (0)}]
;; [[:right 4] {:right (4 4 4 4 4 4 0 0), :left (0)}]
;; [[:right 0] {:right (0 4 4 4 4 4 4 0 0), :left (0)}]
;; [[:right 0] {:right (0 0 4 4 4 4 4 4 0 0), :left (0)}]
;; [[:right 4] {:right (4 0 0 4 4 4 4 4 4 0 0), :left (0)}]
;; [[:right 0] {:right (0 4 0 0 4 4 4 4 4 4 0 0), :left (0)}]
;; [[:right 4] {:right (4 0 4 0 0 4 4 4 4 4 4 0 0), :left (0)}]
;; [[:right 0] {:right (0 4 0 4 0 0 4 4 4 4 4 4 0 0), :left (0)}]

;; In this case, it tried the right arm a couple of times, then had a
;; go with the left arm, then went back to the right arm, won a
;; payout, and then got hung up on pulling the right arm repeatedly.



;; We've got a couple of problems here!

;; First is that the algorithm has clearly got into a state where it
;; always pulls the left arm (in the first case), and the right
;; arm (in the second case).

;; It can't be doing the right thing in both cases.

;; Secondly the state is growing linearly, as the algorithm remembers
;; all previous results. That's giving us algorithmic complexity
;; problems and the calculation will get slower and slower, and
;; eventually run out of memory.













Permalink

Edging forward

Clojure is a great language for building systems. One of the reasons Clojure isn’t more well known is that it can be difficult and time-consuming for programmers to set up their own development environment where they experience just how effective the language can be.

Back in 2016, I wrote about our Edge project, which is designed to help make it easier for people to started with their own Clojure projects.

Over the past couple of years we’ve heard of a number of project teams that have successfully used Edge as the base-camp for their ascent of new unclimbed peaks. All the while we’ve been refining Edge, making gradual improvements to keep it up-to-date. So now let me update you on what is in Edge 'master' today.

Fast feedback

Clojure is a Lisp, and one of the key benefits of this family of programming languages is they are designed from the ground up to support 'live' programming, making incremental improvements to running software. This makes for a hugely productivity workflow.

We’ve kept the 'reloaded workflow' of hot-loading updated code when triggered with an explicit reset, powered by the clojure.tools.namespace library which automatically detects and reloads your code changes. But we’ve also extended this into the realm of web apps, with hot-loading of changes to HTML, CSS and ClojureScript. This is all thanks to one of our newer projects, kick, which provides an embeddable and extensible build process inside apps. More about kick below.

Modular architecture

We’ve retained the system-oriented component design, but made the decision to switch out Stuart Sierra’s 'component' library for James Reeves' Integrant replacement. We think Integrant is the best way of defining, and configuring, the modular architecture we’ve gone for in Edge.

Talking of configuration, we’ve retained the approach of a single configuration file which supports multiple different environments, using our Aero configuration library, but with seamless integration with Integrant.

Adoption of tools.deps

Edge originally used Leiningen to launch itself, but we eventually moved to boot. Now we’ve changed again, this time to Clojure’s officially-supported tools.deps.

Edge is designed to be easy to modify, by using open architectural principles and well understood idioms battle-tested by us and our colleagues in the professional Clojure community on countless projects. Yet our decision to adopt tools.deps has enabled further modularity, allowing us to break Edge into numerous modules each with their own dependencies. This has the added benefit of looser coupling between the Edge infrastructure and the applications built upon it, making it easier to upgrade one independently of the other.

Deployment

We’ve got some really exciting news to report about deploying Edge. We’ve added an integration with our pack tool, for building uberjars which are ready-to-go and runnable wherever you have a Java runtime.

But we’ve also built in support for running Edge in a raw unpacked mode, much like you run Edge when you’re developing. This means you still get advanced optimized ClojureScript and CSS, but with a unique ability to update the application while it’s running, with minimal downtime. This is how we’ve been running smaller projects such as our own juxt.pro website.

How does this work? Well, we believe "builds are just programs" and we’ve long suspected Clojure is the best programming language for writing them. Inside Edge we have some cool tech we call 'kick', which can be used both at 'compile' time when creating runnable uberjars and at runtime, for the system to build and rebuild 'itself' whenever necessary.

Developer experience

As well as improved documentation, we’ve worked hard on the overall 'developer experience', or 'DX'. We’ve added more help at the command line, adding integration with Bruce Hauman’s rebel-readline project and adding colors.

We’ve created scripts to generate new Edge modules (which can be developed as individual projects such as micro-services within your Edge fork), and started to create a series of example modules to demonstrate various features of our stack, such as REST APIs, GraphQL, re-frame, Datomic, time-processing, cloud deployments and much more.

Demo

Here’s a quick 3½ minute demo that should give you a good impression of where we are with Edge and how you can get started with it today.

More to come

I’m really excited about the direction Edge is taking. Watch this space, there’s more to come soon!

Permalink

How to upgrade your Clojure projects to use Java 11

JDK 11 is the latest release of the JDK (Java Development Kit). There are several changes since JDK 8 that will require projects to update their dependencies or build tools. Many open source projects have resolved these issues when adding support for JDK 9 and 10. For most projects that have been updating their dependencies regularly, the upgrade process to JDK 11 shouldn’t be too difficult.

Last year I wrote a guide on upgrading Clojure projects to Java 9. This guide can be read on it’s own, but you can find some more background and context in last year’s guide too.

JDK 11

JDK 11 was released on September 25, 2018. JDK 11 is the first long term support release since JDK 8. Due to the short support lifespan of JDK 9 and 10 (six months each), and the number of breaking changes in those releases, many businesses and individuals have continued to use JDK 8.

Free public updates for Oracle’s JDK 8 end in January 2019 for commercial users, and December 2020 for personal users. After those dates, Oracle will not be providing any more free updates for JDK 8. This means that many Clojure projects will be looking to upgrade to JDK 11 soon, or investigating other JDK providers which support JDK 8. Oracle has also created a guide for updating to JDK 11. Most of the notes in the upgrade guide are not relevant to the majority of Clojure programs, but it’s worth a quick scan as well.

Checking for library upgrades

Before beginning to upgrade to JDK 11, I would recommend checking for new versions of your dependencies. These may have fixes for incompatibilities introduced in new versions of the JDK, reducing the amount of breakage you need to resolve later. It is also much easier to upgrade your libraries first and then your JDK, rather than trying to upgrade both at the same time. You can check for updates to Leiningen projects with lein-ancient, Boot projects with boot-deps, and tools.deps projects with deps-ancient. If your project is publicly available on GitHub, you can use Deps Versions to add a badge to your README to show if your dependencies are up-to-date.

java.util.Collection toArray

If you have previously upgraded to support JDK 9 or JDK 10, the main issue you are likely to face when upgrading to JDK 11 is the addition of a method in the java.util.Collection interface. A new toArray method was added which overloads the existing 1-arity method. Java and other statically typed languages on the JVM have the type information to resolve the ambiguity, but Clojure is not able to resolve this without developers adding extra type hints to specify which method to use.

Without the type hints you would get an error like this:

Exception in thread "main" java.lang.IllegalArgumentException: Must hint overloaded method: toArray, compiling:(clojure/core/rrb_vector/rrbt.clj:282:1)

This change affected several projects directly including core.rrb-vector (CRRBV-18), org.flatland/ordered (#37), datascript (#273), and Clojure (CLJ-2374). In practice, this isn’t an issue for Clojure. Clojure is distributed as an AOT compiled JAR and is compiled against older versions of Java. You were only likely to run into this issue if you were working on Clojure itself.

RRB Vector and Ordered have a much larger impact however. Many projects have dependencies or transitive dependencies on one of these libraries, including Midje, Lacinia, Fipp, Puget, lein-monolith, clj-yaml, compojure-api, and ring-middleware-format.

There are new releases for RRB Vector and Ordered: [org.clojure/core.rrb-vector "0.0.13"], and [org.flatland/ordered "1.5.7"] respectively. Many of the downstream consumers of these libraries have been updated to use these versions, but some haven’t yet.

If you are using Leiningen, this is a great use-case for the :managed-dependencies feature. If you add a vector of :managed-dependencies to your project.clj, Leiningen will choose those versions if your project depended on any version of that dependency. boot-bundle provides a similar feature for Boot.

:managed-dependencies [[org.clojure/core.rrb-vector "0.0.13"]
                       [org.flatland/ordered "1.5.7"]]

If you have any Leiningen plugins with dependencies on rrb-vector or ordered, you can override the resolved dependency by adding an explicit dependency to the :plugins vector. For example:

:plugins [[lein-monolith "1.0.1"]
          ;; Overrides older version of rrb-vector that
          ;; doesn't work on JDK 11.
          [org.clojure/core.rrb-vector "0.0.13"]]

Deprecations and removals

JDK 9 deprecated several Java EE and CORBA modules. The modules were still in the JDK, but not resolved by default. To get access to those modules you needed to explicitly add them back with the command line flag --add-modules, e.g. --add-modules "java.xml.bind".

Without this flag, you would get errors like:

Caused by: java.lang.ClassNotFoundException: javax.xml.bind.DatatypeConverter

The most common use of these modules in the Clojure community was the Base64 converters in javax.xml.bind. Java 8 added a Base64 class to java.util which is a suitable replacement.

If you need to support JDK’s older than Java 8, http-kit shows a way of using macros to support both methods. Note that if you do this and AOT compile your applications, make sure that you compile with the same version of Java you’ll be running in production.

JDK 9 also removed several of the sun.* APIs, including sun.misc.BASE64Encoder and sun.misc.BASE64Decoder. Again, the suggested migration path is to use java.util.Base64.

JDK 11 removed the Java EE and CORBA modules. If you or your dependencies were still using these modules, you will need to add them explicitly as dependencies. JEP 320 has more information on the removed modules and where to get their replacements. If you’re using a library that expects these dependencies, you will need to add a dependency on the JAR to download it from Maven Central. For example:

[javax.xml.bind/jaxb-api "2.4.0-b180830.0359"]

Build Tools

Leiningen 2.8.0 or later is required to use JDK 11. You can get the latest version of Leiningen by running lein upgrade. At the time of writing, Leiningen 2.8.1 was the latest version available.

Boot has been updated to 2.8.2 to build under JDK 11. However, in my testing, it seems that Boot 2.6.0 and up can run under JDK 11. 2.8.2 was released on 14 September 2018. Update boot with boot -u to get the latest version. At the time of writing, Boot 2.8.2 was the latest version.

tools.deps appears to have no blocking issues running JDK 11. There is an open issue about JDK 9 and up adding spurious newlines to a POM, but this is unlikely to be an issue for most people.

If you run into any issues with any of these build tools, let me know, and I’ll update this post.

Clojure and ClojureScript versions

Clojure and ClojureScript both have fixes to improve compatibility with JDK 9 and up. ClojureScript 1.10.63 and up include CLJS-2377 which avoids depending on java.xml.bind.

Clojure 1.9-beta1 has support for running on the bootclasspath under Java 9. If you can’t upgrade Leiningen to 2.8.1, then upgrading Clojure to 1.9-beta1 or later should also work.

Keeping up to date

Java’s new six-monthly release cycle has increased the rate of change in the JDK, and the number of versions that library consumers may be using. The increased willingness to deprecate and extract parts of the JDK means that many developers should consider testing against early access versions of the JDK to ensure that they aren’t caught off-guard. If you’re using Travis CI, I’ve written a guide on testing against the latest JDK.

Permalink

PurelyFunctional.tv Newsletter 304: Back to the factory

Issue 304 – December 10, 2018 · Archives · Subscribe

Hi Clojurists,

More resources about Agile and how it came from the factory. In my experience, people turn to Agile without really understanding where it came from, what it’s about, and what the alternatives are. I’m exploring those myself, because in my experience Agile makes me feel like Charlie Chaplin working at the factory. GIF

Please enjoy the issue!

Rock on!
Eric Normand <eric@purelyfunctional.tv>

PS Want to get this in your email? Subscribe!


I will be speaking at IN/Clojure 2019

I will be speaking at IN/Clojure in January. Be sure to get your tickets while they last.


3 models of software development as a factory

Is software engineering like a factory? If it is, the how so? Is there more than one way to frame that metaphor?


The Dehumanisation of Agile and Objects YouTube

James Coplien at his best, deconstructing Agile.


Real Software Engineering YouTube

Glenn Vanderburg talks about whether programming is engineering, how we got here, and where we can go moving forward. He also has a pretty good breakdown of some of the practices of eXtreme Programming as ways of building feedback loops of various time lengths into your development.


On Storytelling

Evan Czaplicki talks about his process for building Elm. It’s slow and methodical, much like Clojure. People complain and claim that bugs go unfixed for a long time. But he’s optimizing for something more broad and global instead of getting tickets through as fast as possible.


Jason Fried Q&A YouTube

Jason Fried talks in-depth about the process they use at Basecamp, which is nothing like the traditional sprint cycle.


Hammock Driven Development YouTube

Rich Hickey’s classic talk contains a few critiques of Agile development, including calling every cycle a “sprint”. Where is the time for thoughtful reflection?


Unlocking data-driven systems YouTube

A different way to develop: think a lot, then build a solution. Does your company give you time to think? Or are you always busy doing work others have thought about?


Implementing Programmer Anarchy YouTube

Fred George on managing the managers and giving programmers the freedom to solve problems instead of completing tickets. It’s a combination of shielding programmers from over-management, establishing clear goals, and providing a good architecture and coding practices to enable programmers to experiment.


Hiccup id and class Shortcuts Free lesson

Next week’s free lesson is all about how best to use the id and class shortcuts in Reagent’s Hiccup.

I’m going through this course and every week, making one lesson free for the week. You can follow PurelyFunctional.tv on Twitter to be reminded.

The post PurelyFunctional.tv Newsletter 304: Back to the factory appeared first on PurelyFunctional.tv.

Permalink

Easy Clojure, Easy REPL

Have you ever tried to convince a friend that Clojure is a great language?

Have you ever tried to convince your boss to let your team on a side project with Clojure?

If that’s the case, you have probably used as a solid argument the fact that in Clojure, the REPL is very powerful and very helpful for beginners. Inside the REPL, you can do everything and it is so simple to hack with Clojure expressions in the REPL.

If you did a good job, your friend or your boss decided to give it a try. She installed the Clojure CLI on her machine as explained in this blog post and she launched the Clojure REPL with a simple 3-letter command line:

>clj
user=>

Fantastic! Now she had the ability to experiment with Clojure.

After having typed a couple of arithmetic expressions like (+ 2 3) and a few list manipulations expressions like (map inc [1 2 3]), she probably tried to define a variable with a form like the apparenlty innocent (def my-var 42):

user=> (def my-var 42)
#'user/my-var

Then, she asked you with naive expression on her face: “what is this weird dash quote user ?”

At this point, you had two options:

  1. “Oh! That’s the fully qualified name of the variable you created”
  2. “Forget about it, you will understand this part, when you are more experienced with Clojure”

No matter what was your answer, my guess is that she felt a bit confused…

Disapointed

And it probably got worse when you tried to explain her that this was something “simple but not easy”…

The point of this imaginary story is to illustrate the fact that the default Clojure REPL is not beginners friendly.

When I started to write my Get Programming with Clojure book, I had to find a way to let my readers enjoy the power of the REPL without being confused by some weird dash quote symbols too early in their Clojure journey.

The solution I came with was to create my own REPL with a single objective in mind: to be beginners friendly. I named it the Klipse REPL and it is available on github.

The way I am handling def forms in the Klipse REPL is to display the value of the variable instead of its fully-qualified name.

user=> (def my-var 42)
42

No more questions about “weird dash quote symbols”!

Similarly, for function definition with defn forms: I decided to display the name of the function and the arguments that the function expects:

user=> (defn foo [x]
    =>   (* 42 x))
Created function foo ([x])

I made another small improvement in the traditional doc form that the default REPL provides: The doc macro provided by Klipse REPL includes a link to the form page in clojuredocs.org. Clojuredocs is one of the most valuable resources for Clojure beginners as it provides examples of usage of the Clojure forms. (There are so many situations where the docstring is so cryptic.)

For instance, take a look at the last line of the output of (doc inc):

user=> (doc inc)
-------------------------
clojure.core/inc
([x])
  Returns a number one greater than num. Does not auto-promote
  longs, will throw on overflow. See also: inc
-------------------------
Online doc: https://clojuredocs.org/clojure.core/inc

The Klipse REPL also includes all of the great features of Bruce Hauman’s rebel-readline for the simple reason that the Klipse REPL is built on top of rebel-readline. Some of them are:

  1. autocompletion
  2. indentation of multi-line expressions
  3. coloring of forms

Now that my book is available for early access, you can use it as another solid argument to convince your friend and your boss about the value of Clojure.

Permalink

CLJS ❤️'s Lambda

What is Clojure and why do you care?

Because every post about Clojure or Clojurescript (CLJS) has to explain what Clojure(script) is and why you should use it:

But in all seriousness a bunch of people have done better introductions to Clojure then this post will:

🔥Serverless🔥

Unfortunately there isn't an amazing parody commercial for Serverless so instead let's just try to actually motivate why you should care about it. Serverless is just a shorthand for describing a way to run code without worrying about infrastructure. What does that mean practically? Give them and zip and they throw it on some unused hardware, that's it!

It supported on AWS, Google Cloud, and Azure and probably other clouds. Because the examples are built with AWS that is where we will focus.

Probably the biggest motivator for migrating to or build new services with serverless is cost. The the free tier is very generous and it's pay as you go so it's free to prototype!

Eyes popping

You also get ease of deployment/operation and native integration with your cloud provider making it super easy to tap into services like cloud provided DBs, metrics/analytics, queues, email services, etc.

CLJS + Lambda

While there exist ways for you to plug together Clojurescript and Lambda they are coupled to specific build tooling and are a bit too much like a black box for my liking. This also is a good excuse to learn how all the pieces plug together. This post is a distillation of how to do just that. Before finally getting to the meat of the post we have to introduce one more player to the stage.

Shadow-CLJS

Shadow-CLJS is an alternative frontend to the Clojurescript compiler, it is fills the same role as lein-cljsbuild or Figwheel-main if you are familiar with the space. The big advantage it has over the others tools is awesome npm support. We can use basically all of npm with CLJS via Shadow which is important if we want to leverage things like AppSync or the AWS Node SDK.

MAKE THE DAMN LAMBDA ALREADY

Setup

Okay to being lets make a lambda, eventually we will throw away everything inside of it, but for now just leave it be.

And with our basic lambda up we just need our tools and some example code, first the tools:
brew install yarn #Or your OS equiv, apt-get, yum, etc
And that's it!

Example Code

For the example code I have forked Chenyong's minimal Shadow + Node example to create a hello world lambda with Shadow:

royalaid / minimal-shadow-cljs-nodejs

shadow-cljs hot code swapping for Node.js

Node.js example for shadow-cljs

Develop

Watch compile with with hot reloading:

yarn
yarn shadow-cljs watch app

Start program:

node target/main.js

REPL

Start a REPL connected to current running program, app for the :build-id:

yarn shadow-cljs cljs-repl app

Build

shadow-cljs release app

Compiles to target/main.js.

You may find more configurations on http://doc.shadow-cljs.org/ .

Steps

  • add shadow-cljs.edn to config compilation
  • compile ClojureScript
  • run node target/main.js to start app and connect reload server

License

MIT


git clone https://github.com/royalaid/minimal-shadow-cljs-nodejs.git /tmp/lambda-example
cd /tmp/lambda-example
yarn

and now all of your deps are setup!

Code Breakdown

While the project should be pretty self explanatory I will given a high level overview of the important bits, the shadow-cljs.edn and src/server/main.cljs files.

shadow-cljs.edn

The shadow-cljs.edn is the equivalent of project.clj for lein or boot.clj for boot, its the build configuration file for Shadow. The most important part of that file for us is the :target :node-library line which tells shadow to setup module.exports for use by node, this is bridge between the CLJS code with write and the Javascript that Node reads. The second most important line is :exports {:handler server.main/handler}. This line tell Shadow what to call the export and then what CLJS function to bind to it, in our case the handler function in the Clojurescript namespace server.main. This function is what AWS will end up calling when we execute the lambda.

main.cljs

As for the src/server/main.cljs, this is what will eventually be run by AWS after Shadow transpiles it for us. The lone function inside of the namespace is what actually invoked. Important note, the function signature has to have 3 args (2 if using async/await, which we aren't) and must "return" its value by then invoking the cb arg and passing the result to the second arg of that callback (cb) invocation.

We need to do this because Lambda expects the result to come as a promise or from the callback passed in. This took me waaaaaaaaay too long too piece together and hopefully this won't trip up anyone else either. Additionally we need to be sure to return Javascript values and not Clojurescript values hence the use of #js for the array.

Build and release time

Note: I am going to skip the process of actually developing the code but check out the earlier linked Shadow + Node example for details and workflow for how to use Shadow with Nodejs and hot-reloading.
cd /tmp/lambda-example # Get you back to the correct directory
yarn shadow-cljs release app # This transpiles and optimizes your CLJS
zip -r archive.zip target # Prep your output for upload to Lambda

Then we just upload our new zip to our old Lambda and point the handler function at our CLJS function (which will be .handler, main.handler in the recording).

🎉🎉🎉

And there we have it! If you have any questions or feedback reach out on Twitter, Mastodon, or @royalaid on the Clojurians Slack or Zulip

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.