Full Disclojure

Partially Implemented Protocols

2010-08-12T13:28:00.000-07:00

Last time I discussed using a hash maps to improve code reuse with protocols. While we produced a set of good solutions, I couldn't help but feel like we aren't quite there. There still is a fair amount of chatter on the list about how to implement a default protocol, for example.

I started experimenting with an idea I'm calling a Partially Implemented Protocol (PIP). The idea is that you can use a PIP like you would a BaseClass in Java. After playing around for a little bit, I settled on the following solution:

The two major components is the PIP record and extend+ macro. Let me re-write yesterday's example with the new expand+ syntax.

(defprotocol UIProtocol
(render [])
(action [evt])
(button-down [evt])
(button-up [evt]))

(def BaseUIProtocol
(pip UIProtocol
{:action (fn [this evt] ...)
:button-down (fn ([this evt]...)
:button-up (fn ([this evt]...)})

(extend+ TerranUIType
BaseUIProtocol
{:render an-existing-fn})

(extend+ ProtossUIType
BaseUIProtocol
{:render a-different-fn})

What I really like about the use of a PIP is that it allows reference to the protocol & partial implementation with one record. You can get at the internal simply by using keyword lookup, and you can also derive a PIP from another PIP like so:

(def up-down-override
:button-down button-down-2
:button-up button-up-2})

(extend+ ZergUIType
(pip BaseUIProtocol up-down-override)
{:render a-buggy-fn})

I'm also working on versions of reify+ proxy+, etc. that all work with PIPs, assuming this idea is worth exploring.

Protocol Implementation Awesomeness

2010-08-11T08:03:00.000-07:00

I recently read a post about Racket & Clojure, and there was a question about protocols raised. I wanted to take the time to discuss composition of implementations with Protocols. Of course, you probably know this term better as inheritance.

Personally, my main driver for multiple inhertiance is being able to re-use different partial implementations of a class. There are a lot a times I would want to mix & match implementations of this protocol, such as UI code. Consider the following protocol

(defprotocol UIProtocol
(render [])
(action [evt])
(button-down [evt])
(button-up [evt]))

Now, we can extend the protocol as such

(extend UIType
UIProtocol
{:render an-existing-fn
   :action (fn [this evt] ...)
   :button-down (fn ([this evt]...)
   :button-up (fn ([this evt]...)})

So far so good. However, suppose we have another UI type we want to define

(extend AnotherUIType
UIProtocol
{:render a-different-fn
   :action (fn [this evt] ...)
   :button-down (fn ([this evt]...)
   :button-up (fn ([this evt]...)})

The only thing that is different between the implementations is the rendering logic (similar behavior, different skin? What sort of problem domain has that?). Our implementation works, but it's repetitive.

In order to clean it up, we'll need to use a defining feature of Lisp: code is data. Remember we are building a data structure to define our type. As such, we can use every DRY trick we know. In our specific example, we take advantage of the fact that the extend macro is expecting a hash-map, and it doesn't care how it gets there. So, here's a new take on the code.

(def base-impl
{:action (fn [this evt] ...)
   :button-down (fn ([this evt]...)
   :button-up (fn ([this evt]...)})

(extend TerranUIType
UIProtocol
(merge base-impl {:render an-existing-fn}))

(extend ProtossUIType
UIProtocol
(merge base-impl {:render a-different-fn})

BAM! I've overridden render with my specific implementation. We just simulated the abstract base class pattern. Now, let's see how flexible this really lets us be.

(def up-down-override
:button-down button-down-2
   :button-up button-up-2})

(extend ZergUIType
UIProtocol
(merge base-impl up-down-override {:render a-buggy-fn});-p

What this let me do is define my own inheritance rules a la carte. No implicit order of operations, no compiler limitations, no confusing precedence rules. If you can figure out the map merging, you can do it. Period. That's why I think Clojure Protocols are so awesome!

Steve Jobs just ruined the iPhone for Clojure

2010-04-08T21:06:00.000-07:00

Recently Apple released new terms of service with their iPhone OS. By now most of you have seen the following section. I've copied this from Daring Fireball

3.3.1 — Applications may only use Documented APIs in the manner prescribed by Apple and must not use or call any private APIs. Applications must be originally written in Objective-C, C, C++, or JavaScript as executed by the iPhone OS WebKit engine, and only code written in C, C++, and Objective-C may compile and directly link against the Documented APIs (e.g., Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited).

I'm pissed off. I believe the hype, and there's a lot of HCI experiments I'd like to do with the iPhone & iPad. There's a ton of opportunities & ideas, waiting to be discovered. Being able to do these experiments in Clojure would be some much fun, that it's worth the $500 investment.

If I'm reading the TOS, we can't have a version of Clojure on this platform. Any ideas I have - no, we have - now can't be shared with the world.

This is NOT thinking differently Apple.

New Getting started page

2010-04-06T10:02:00.000-07:00

Clojure's getting started information has moved. You can check it out here:

Clojure Getting Started

Too much Clojure when...

2010-03-26T08:12:00.000-07:00

I can no long spell the following words properly. This is really annoying when talking to people outside of this community :)

Closure/Clojure
Definitely/Defnitely
Disclosure/Disclojure (self inflicted)
Composure/Compojure

Thanks for nothing Rich :-p

What're your experiences?

Thoughts on namespace management

2010-02-18T07:34:00.000-08:00

Konrad Hinsen recently posted some ideas for handling namespaces

http://onclojure.com/2010/02/17/managing-namespaces

There are a lot of good ideas in there, and right now I'd like to talk specifically about the proposed :like & :clone clauses. I have a bit of a different approach.

Supposed we add a new project file, namespace_config.clj. It would be a basic rules engine for configuring namespaces. I'm thinking there would be a macro, extend-ns, that is roughly defined as follows. The exact implementation needs work.

This lets us set up a series of predicates & resulting actions that would be applied to each ns in the project. Everything would be done in one central location, so (hopefully) that would cut down on maintenance.

Code Kata: A data sifter

2010-01-24T12:16:00.000-08:00

Here's a problem inspired by a scheme example I discovered another day.

http://programming-musings.org/2006/02/07/scheme-code-kata/

Write a data sifter, sift, that partitions a string into a list of lists. Start with the case of using letters as a delimiter, and numbers as data. There can be any number of repetitions of numbers & letters.

user=>(sift "a1b2cd34")
(("a" ("1")) ("b" ("2")) ("c" ()) ("d" ("3" "4")))

Next, add the ability to your sift function to accept a list as input, as well as a string.

user=>(sift ("a" "1" "b" "2" "c" "d" "3" "4"))

(("a" ("1")) ("b" ("2")) ("c" ()) ("d" ("3" "4")))

After that, add the ability to take a vector/array as an input

user=>(sift ["a" "1" "b" "2" "c" "d" "3" "4"])

(("a" ("1")) ("b" ("2")) ("c" ()) ("d" ("3" "4")))

Finally, let your sift accept a collection of any object, and an arbitrary predicate. If the predicate is true, the object is a delimiter (e.g. a String). If the predicate is false, the object is data (e.g. a Number).

user=>(sift string? ["a" 1 "b" 2 "c" "d" 3 4])

(("a" (1)) ("b" (2)) ("c" ()) ("d" (3 4)))

I'll be posting my solution in about a week.

1.2 fn Proposal: same & multisame

2010-01-03T20:21:00.000-08:00

Hello Clojure Developers,
Writing software frequently follows the same process. Observing & understanding the processes and coming up with effective solutions is the task of library design. An API is judged by how well it fits into the process.

Application design is a slightly different process. For the purposes of this proposal, it involves three oversimplified steps.

Convert problem data to a form the API can understand.
Use the API to come up with a solved version of the problem.
Convert the API produced solution back to the problem domain solution.

One of features we all love about Clojure is the sequence abstraction. For many problems, it drastically reduces the time required to convert the specific problem to a form that the API understands. Once a problem is transformed to operating on a string, map, vector or set the sequence functions can take it from there. This brings steps 1 & 2 very close to each other.

However, there currently is not much work done to bring steps 2 & 3 closer to each other. This is evidenced by the fact that there are specialized namespaces in contrib for handling strings (str-utils2), functor application (generic.functor), and I was in the middle of proposing additions to the map-utils library.

All this code duplication started to smell. Here we all are writing specialized routines to AVOID using the sequence functions in our code. This is not right.

I have a proposal to eliminate this smell. I've written a higher order function called same. Here's the doc:

lib.sfd.same/same
([index? seq-fn & args])
"same is a mutlimethod that is designed to "undo" seq. It expects a seq-fn that returns a normal seq, and the appropraite args. By default it converts the resulting seq into the same type as the last argument. An optional leading integer, index, can be provided to specify the index of the argument that should be used to convert the seq. If it is a sorted seq, the comparator is preserved.

This operation is fundamentally eager, unless a lazy seq is detected. In this case no conversion is attempted, and laziness is preserved."

Please take a moment to review a fairly robust list of examples now:

http://github.com/francoisdevlin/devlinsf-clojure-utils/blob/master/test/lib/sfd/same_test.clj

Afterwards, you can peruse the code here:

http://github.com/francoisdevlin/devlinsf-clojure-utils/blob/master/src/lib/sfd/same.clj

This one function will provide the same functionality as the proposed map-utils, some of c.c.str-utils2, c.c.generic.functor, or any desired set & vector utils. It's based on a multimethod, so you are a simple defmethod addition away from keyword-utils or symbol-utils (assuming you'd want to treat them like strings).

I've also designed a method, multi-same, for functions that take a sequence in and split it into several sequences. Here's a quick example, as the uses for multi-same are still being developed.

user=>(multi-same partition 2 "abcd")
("ab" "cd")

One thing that I find VERY fascinating is the areas where same & multi-same do NOT allow str-utils2 to be replaced out of the box. Some of these can easily be explained. str-utils2/trim is a very string specific piece of code. However, others cannot easily be explained. Why is it that there is no way to split a sequence similar to a regular expression?

I think these areas where string processing is easier represent places we need to improve our sequence library. I've included some new functions in lib.sfd.seq-utils, and I would ask this group to consider adding them to c.c.seq-utils or core.

There also isn't a parser that works with predicates & sequences in core yet. I suspect fn-parse may be a start. I'd appreciate help from anyone that is good with parsers/monads.

So, here's a chance to simultaneously reduce the amount of code in contrib and add lots of functionality to Clojure. In summary, here's what I'm proposing

Add same to core
Add multi-same to core
Add new sequence fns to contrib or core
Add a new sequence parser to contrib or core

There is one more plea I would like to add on a personal note. People are always detracting from Lisp, asking us to provide examples of how it is more productive than other languages. I believe this is an opportunity to build something that replaces several libraries with one function. It would be an example of how Lisp *eliminates the need* for writing code, very concisely.

UPDATE 1/3: I re-wrote same & multi-same to work with a protocol, per Stuart Sierra's suggestion.

I look forward to the discussion,
Sean

How to write a Clojure reader macro, part 2

2009-12-25T20:59:00.001-08:00

This is a follow up to my earlier post about writing a reader macro. Here's the disclaimer again.

DISCLAIMER

I Completely agree with Rich's decision to NOT support reader macros. In an activity as simple as writing this post, I found many, many places to make a mistake the hard way. This is an extremely difficult activity to get right, and the end result is something that is not quite the same as normal Clojure. Use the following information at your own risk.

Okay, now that that's been said, let's get on with it. We're going to write a multi-character delimited reader macro this time. Since I'm a point free junkie, we're going to use partial for our example.

Here's what the final use case is going to be

user=>#[+ 1]
#< core$partial ...>

Let's start modifying LispReader.java again. The first thing we're going to do is insert a static symbol

//Inserted at line 40
static Symbol INTERPOLATE_S = Symbol.create("clojure.core", "partial");

Now, Let's take a look around line 84. You'll see the following entry in the array

macros['#'] = new DispatchReader();

The # character is bound to a DispatchReader. This is the object closure uses to implement multiple character reader macros (ever notice that they all start with #?). You'll also notice that there is a dispatchMacros array with several entires in it. Add the following entry

dispatchMacros['['] = new PartialReader();

We also need to define the PartialReader class. It is based on the VectorReader class, which can be found around line 994.

The heavy lifting is done by the readDelimitedList method. Note that the closing delimiter needs to be provided, and the recursive flag should be set to true. It returns an IPersistentList object. The only thing that needs to be done is to prepend a partial to the list. That is why the cons method is used to add a partial symbol (you still need classic macro-fu).

We've added everything we need to add to LispReader.java. All that's left to do is recompile clojure.jar and test the results

user=>(map #[+ 1] [1 2 3])
(2 3 4)

Of course, now that I think about it, comment might be a better symbol to use than partial...

That's how you add a delimited reader macro. Next time we'll look at creating a new dispatch character, and properly escaping everything.

Proposal for 1.2 : New Seq Utilities

2009-12-17T16:34:00.000-08:00

Time to brainstorm everyone!

I've been trying to come up with new sequence functions. Of course, there's nothing really new about these at all. They just aren't in core or c.c.seq-utils. Do any of these belong in contrib or core?

alternate [pred coll]

returns an alternating collection. Similar to a regex partition.

split [pred coll]

returns a collection of collection, split by matching the predicate.

take-last [n coll]

The mirror of drop-last

take-until [pred coll]

Returns a lazy sequence of successive items from coll while (pred item) returns false. pred must be free of side-effects.

drop-until [pred coll]

Returns a lazy sequence of the items in coll starting from the first item for which (pred item) returns true.

rotate [n coll]

Take a collection and left rotates it n steps. If n is negative, the collection is rotated right. Executes in O(n) time.

rotate-while [pred coll]

Rotates a collection left while (pred item) is true. Will return a unrotated sequence if (pred item) is never true. Executes in O(n) time.

rotate-until [pred coll]

Rotates a collection left while (pred item) is nil. Will return a unrotated sequence if (pred item) is always true. Executes in O(n) time.

Anyway, I define all of these functions and more here:

http://github.com/francoisdevlin/devlinsf-clojure-utils/blob/master/src/lib/sfd/seq_utils.clj

http://github.com/francoisdevlin/devlinsf-clojure-utils/blob/master/src/lib/sfd/patterns.clj

Do others have ideas for missing seq utilities?

How to write a Clojure reader macro

2009-12-15T19:51:00.000-08:00

This is an article on how to write a basic reader macro in Clojure.

DISCLAIMER

The first thing to identify is a behavior that you would like to have a reader macro for. For our example, I am going to use a modified form of Chas Emerick's amazing string interpolation macro. You can find his original article here. I took a modified version of his code, and placed it in core.clj (Be sure to create a new git branch). The code I used is below

Now that the desired functionality is in core, it is time to modify the reader. In this case we need to modify the file LispReader.java. I defined a static variable INTERPOLATE_S, and I am going to assign it the "|" reader macro.

//Inserted at line 40
static Symbol INTERPOLATE_S = Symbol.create("clojure.core", "interpolate-s");

Now, in order for this to be found we need to make an entry in the macros array. This can be done like so:

//Inserted at line 86
macros['|'] = new WrappingReader(INTERPOLATE_S);

The WrappingReader class takes a Symbol object, and wraps it around the next form that is read. Recall how the following form

@a-ref

is expanded to (deref a-ref). In our case

|"A string ~(+ 2 2)"

will be expanded to

(interpolate-s "A string ~(+ 2 2)")

Let's rebuild clojure.jar and try this out at a REPL.

As you can see this works just like Chas' macro. There are still a few things that need to be covered, such as:

* How to create a multiple character reader macro
* How to create a delimited reader macro

These will be topics for another day.

uses for take/drop-while

2009-12-08T22:10:00.000-08:00

I've found a use for take-while & drop-while in a map.

To start let's define the following

user=>(defn sort-map [m] (apply sorted-map (apply concat m)))

That's right. A function to cast hash-map to tree-maps.

user=>(sort-map {3 :a 1 :c 2 :b})
{1 :c, 2 :b, 3 :a}

Why would I do that? Because I now have a way of making subsets based on the keys.

user=>(keys-pred take-while #(< 2 %) (sort-map {3 :a 1 :c 2 :b}))
{1 :c}

user=>(keys-pred drop-while #(< 2 %) (sort-map {3 :a 1 :c 2 :b}))
{2 :b, 3 :a}

So, when would this have an application?

Let's define the following map

And there you go, a use for take/drop while with maps :)

Feedback for a visitor closure generator

2009-11-23T08:02:00.000-08:00

Multimethods are a great abstraction in Clojure. They are great for adapting a wide variety of inputs to a function. However, I often need to decouple the adaptation of data from the function that operates on the adapted data. Also, after the work is complete, I need to adapt the data back to the original format.

As such, I've found the need to create visitor closures in my code. Along the way I came up with a helper function to generate visitor closures. I'd like feedback on the visitor generator.

* Getting Motivated: String functions *

I write the following code a lot to deal with keywords & symbols:

(keyword (some-str-fn (name a-keyword) str-fn-args))
(symbol (some-str-fn (name a-symbol ) str-fn-args))

Each of these can be abstracted out as
(defn visit-keyword
[str-fn a-keyword & args]
(keyword
(apply str-fn (name a-keyword) args)))

(defn visit-symbol
[str-fn a-symbol & args]
(symbol
(apply str-fn (name a-symbol ) args)))

So, the code is now called as follows

(visit-keyword str-fn a-keyword args)
(visit-symbol str-fn a-symbol args)

This is one possible implementation of a visitor patter for symbols, keywords & strings.

* More Problems: Map Functions *

It's common to filter on values or keys when working with maps. I've written a lot of code like this:

;keys
(into {} (some-pred-fn (comp a-pred key) map-fn-args))
;values
(into {} (some-pred-fn (comp a-pred val) map-fn-args))

A first approach would be to write specific helper function for each case. However, this can be turned into a visitor as well.

(defn visitor-keys-pred
[pred-fn pred-arg & args]
(into {}
(apply pred-fn (#(comp % key) pred-arg)
args)))

(defn visitor-vals-pred
[pred-fn pred-arg & args]
(into {}
(apply pred-fn (#(comp % val) pred-arg)
args)))

The definition looks a little funky, but we'll see why in a minute. For now the functions can be called as follows:

(visitor-keys-pred pred-fn pred-arg args)
(visitor-vals-pred pred-fn pred-arg args)

* Putting it together: Visitor Closures *

As you can see, a pattern is starting to develop. Each visitor function has a similar signature:

[f first-arg & rest-args]

This obviously becomes

[f & args]

Each visitor follows a similar pattern, too.
1. Apply a visit-fn to get to a common base type
2. Do work on the base type
3. Apply a return-fn to get back to the original type.

We can turn this description into a closure generating function:

(defn visitor
"Used to implement visitor patterns. (first (args)) is modified by the visitor function, and the result is wrapped in a return-fn call."
[visit-fn return-fn]
(fn [f & args]
(return-fn
(apply f
(visit-fn (first args))
(rest args)))))

We now have a new way to define previous visitor functions. Here's what they look like using this new helper function:

(def visit-keyword (visitor name keyword))
(def visit-symbol (visitor name symbol ))

;Works w/ predicate functions
(def visit-keys-pred (visitor #(comp % key) (partial into {}))
(def visit-vals-pred (visitor #(comp % val) (partial into {}))

As you can see, the only thing we described is the visit-fn and return-fn. The rest of the behavior is defined by the visitor pattern itself.

* Second form *

The one catch is that visitor assumes that the dispatched data is the first argument to f. Sometimes it is the last argument to f, as in take & drop. It is easy enough to define visitor* that works on the last argument of a function. The definition is in the link.

http://gist.github.com/241144

* Back to Multimethods *

The power of the individual closures can be amplified when wrapped in a multimethod. Consider our String/Symbol/Keyword group.

(defmulti visit-string (fn [& args] (second args))

(defmethod visit-string clojure.lang.Symbol
[f & args]
(apply visit-symbol f args))

(defmethod visit-string clojure.lang.Keyword
[f & args]
(apply visit-keyword f args))

(defmethod visit-string :default
[f & args]
(apply f args))

We now have a visit-string function that can truly visit anything. Our function calls above become:

(visit-string str-fn a-keyword args)
(visit-string str-fn a-symbol args)

* Closing questions *

So now that you've how & why this works, I've got some questions for the group:

* Did I accidentally duplicate functionality in core?
* How can this be more flexible?
* What maintenance problems are there that I don't see?
* Will speed be an issue?
* Is the signature of visitor sensible? Is the signature of the generated function sensible? How could it be better?

Thanks in advance,
Sean

Short circuiting reductions

2009-09-30T09:18:00.000-07:00

Sometimes reduce needs to short circuit

(defn reducer
  "Returns a reduction closure the terminates when pred is false.  Applies f to the last value that returns true. f defaults to identity if not provided.  Behaves like reduce if pred is always true."
  ([pred] (reducer pred identity))
  ([pred f] (do-stuff:TBD))

Uses for juxt

2009-09-23T11:31:00.000-07:00

The juxtaposition operator can be used in parsing text.

Suppose we have the following text to process:

INSERT TEXT HERRE

Consider the following function which gets information from the clipboard

(defn bom-parser
  [part-num] 
    (map (& 
      (juxt last second (constantly part-num))
      (p split #"\s+")
      trim)
      ((& split-lines
        (p gsub #"\"" "")
        (p gsub #"NOT SHOWN" ""))
       (get-clip))))

In the mapping operation you can see a call to juxt. This helps turn a block of text into a list of vectors.