Clojure Setup, and History Briefly

Mon Apr 20, 2015

So it looks like Clojure is finally about to get mileage at work. Which is excellent, as far as I'm concerned, because I really like the language and have been looking for an excuse or two to get more familiar with it. Unfortunately, I've had precious little programming time in my off-hours lately. Various reasons: books and papers I need to read, books and chapters I need to write, and some unrelated personal stuff. But if it's for work, rather than my own playing around, I can justify some work hours, as well as bumping it to the top of my personal studying list.

Setting up an environment turns out to be fairly simple. About the same complexity level as setting up for Common Lisp. On the one hand, the build tool gets you a copy of the language runtime so you don't need to worry about it, on the other, there's no equivalent to quicklisp-slime-helper. Anyway, the steps are

And that's basically it. At that point, you can run cider-jack-in to get a SLIME-like interactive REPL running|1|. They're not exactly the same, notably the debugger and stack-trace in ciderisn't nearly as interactive or useful as the one in SLIME, but you still get minibuffer argument hints and a macroexpander. If it weren't for the fact that lein repl takes something on the order of 5 seconds to start up, switching over from CL would be completely painless|2|.

History

I mentioned a little while ago that I'm thinking about full-history data-stores. I've already kind of implemented one, though it is tightly bound to a particular data-structure, and I've done a bit of experimenting with a generalization. Having set up Clojure, I was inspired to do a bit more playing around.

(ns history.core
  (:require [clojure.java.io :as io]
            [clojure.edn :as edn]))

(defn make-archive
  ([apply-fn zero]
   {:into apply-fn :state zero :history () :zero zero})
  ([apply-fn zero fname]
   (let [f (io/as-file fname)]
     (when (not (.exists f))
       (with-open [w (io/writer fname)]
         (.write w (with-out-str (prn zero)))))
     {:into apply-fn :state zero :history () :zero zero :file f})))

(defn multiplex-archive [arc stream-vector]
  (assoc arc :streams stream-vector))

(defn commit-event [arc event]
  (let [ev-str (with-out-str (prn event))
        file (arc :file)
        streams (arc :streams)]
    (and file
         (with-open [w (io/writer file :append :true)]
           (.write w ev-str)))
    (and streams (doseq [s streams] (.write s ev-str)))))

(defn apply-event [arc event]
  (let [new-arc (assoc arc :state ((arc :into) (arc :state) event))]
    (if (arc :history)
      (assoc new-arc :history (cons event (arc :history)))
      new-arc)))

(defn new-event [arc event]
  (commit-event arc event)
  (apply-event arc event))

(defn load-archive [fname apply-fn]
  (with-open [in (java.io.PushbackReader. (io/reader fname))]
    (let [arc (make-archive apply-fn (edn/read in) fname)
          eof (gensym)]
      (reduce
       apply-event arc
       (take-while
        (partial not= eof)
        (repeatedly (partial edn/read {:eof eof} in)))))))

So, the idea is that an Archive is a thing with

You make a new Archive by initializing some of the above points. You add an event to it by committing the event, then calling the application function on the Archives' current state. Load an Archive by opening a file, reading the Zero from it, then reduceing the remainder of the records over said Zero with the application function. And that's that. If you wanted to see a previous state of the Archive, you'd just stop folding before you got to the last event.

The above implementation is a bit simpler than my earlier CL-based equivalent, in that it doesn't yet deal with partial loads, timestamps or reconciliation. All of those look like they'll be trivial changes, and they won't impact the existing machinery at all|5|. The minimal is still fairly useful though. Here's an example use:

(let [app (fn [arc ev]
            (case (get ev 0)
              :insert (let [[_ k v] ev] (assoc arc k v))
              :delete (let [[_ k] ev] (dissoc arc k))))
      zero {}]
  (defn make-history-table
    ([] (make-archive app zero))
    ([fname] (make-archive app zero fname)))
  (defn load-history-table [fname]
    (load-archive fname app)))

That's how you would go about declaring a history-aware table. You'd use it by making one, and running some events against it.

history.core> (make-history-table)
{:into #<core$eval7113$app__7114 history.core$eval7113$app__7114@58e793e4>,
 :state {}, :history (), :zero {}}

history.core> (new-event (make-history-table) [:insert :test "test"])
{:into #<core$eval7113$app__7114 history.core$eval7113$app__7114@58e793e4>,
 :state {:test "test"},
 :history ([:insert :test "test"]),
 :zero {}}

history.core> (new-event (new-event (make-history-table) [:insert :test "test"]) [:insert :another-test "bleargh"])
{:into #<core$eval7113$app__7114 history.core$eval7113$app__7114@58e793e4>,
 :state {:another-test "bleargh", :test "test"},
 :history ([:insert :another-test "bleargh"] [:insert :test "test"]),
 :zero {}}

history.core> (new-event (new-event (new-event (make-history-table) [:insert :test "test"]) [:insert :another-test "bleargh"]) [:delete :test])
{:into #<core$eval7113$app__7114 history.core$eval7113$app__7114@58e793e4>,
 :state {:another-test "bleargh"},
 :history ([:delete :test] [:insert :another-test "bleargh"] [:insert :test "test"]),
 :zero {}}

history.core>

This is a pretty stupid example, all things considered, but it illustrates how you'd go about putting together a minimally functional, history-aware data-structure. In reality, you'd declare your table to be an atom or agent, and declare a change function that updates its state with new-event. This is the main use-case I'm considering, so it might be prudent to just make that the default behavior of the library.

Minor Notes

First impressions are really, really good. As I've said before, I like Clojure. My impression of it is that it takes takes the best parts of Scheme and Common Lisp and runs with them. lein comes with basically all the stuff I like out of asdf, quicklisp and quickproject, with the added perks of good documentation and built-in consideration for unit testing. The only complaint I have is the annoying startup delay whenever I do lein something. In particular, the lein test command takes long enough that I'm not sure I'd want to pipe it through entr keyed on .clj file changes. I'll let you know how it goes once I've done some real work with it.

Footnotes

1 - |back| - If you run that inside a Clojure project directory, the REPL will also automatically load said project and enter its namespace.

2 - |back| - Incidentally, I thought this was to do with my Free Software bent, because the OpenJDK has the reputation of being slower than the Oracle equivalent. I asked a friend who uses the non-Free version, and he confirmed that the REPL just plain takes a while to start up. Not sure how to feel about that; M-x slime sets up an interactive REPL in about a second at the outside, and gives feedback on progress in the meanwhile.

3 - |back| - Optional because we may not need the full history in memory. Given my experiments with cl-notebook, I currently believe that you don't really need in-memory history unless you want to do real-time history manipulation or traversal. So, you do want it sometimes, but it potentially saves a lot of memory if you can do without.

4 - |back| - Again, optional because you might only want the in-memory representation without worrying about persisting it. I haven't come across this situation yet, but it might exist. It goes almost without saying that you want at least one of in-memory-history/tracking-file/tracking-streams, because if you have none, you don't really have a history-aware data structure. But depending on your use-case, you may need only one.

5 - |back| - Though they may require some changes to the storage format, which is why I haven't published this little library quite yet.

Creative Commons License

all articles at langnostic are licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License

Reprint, rehost and distribute freely (even for profit), but attribute the work and allow your readers the same freedoms. Here's a license widget you can use.

The menu background image is Jewel Wash, taken from Dan Zen's flickr stream and released under a CC-BY license