It’s All About State

The trouble with classes is that they intimately couple state and the functions that manipulate that state. Indeed, this is often heralded as a benefit of the paradigm.

It isn’t.

Managing state is a vital part of programming. But it’s the boring and nonproductive part.

A program is something that transforms data. It takes inputs and produces outputs. It responds to events by changing some state.

Relying on objects makes it more difficult to do this: the coupling of functions and instance data means that every object has two conflicting personalities, one focussed on maintaining the integrity of the state and the other driven to change that state.

Make Transformations First Class

Let’s look an some typical OO code. Given an account number, it finds the correctonding customer, and from there finds the active accounts, and sums their balances.

balance = Customer
          .find(account_number)
          .active_accounts()
          .sum_balance()

This is bread and butter code. But it’s bad code.

The fault lies in the . operator. It invokes the method on its right in the context of the object on its left. When you use it, you have to make sure that the object has that method, and that it does the correct thing. Typically this means that we load our classes with a bunch of convenience methods just to provide the correct object mappings: Customer.find presumably returns a Customer object. The Customer class must then have an active_accounts instance method that returns a new object of class AccountCollection that in turn has an instance method sum_balance that returns a Currency object.

The . operator encourages us to fill our classes with convenience methods simply so that we can write code that reads well.

But let’s forget about that for a minute, and think of our problem not in terms of classes but instead in terms of transformations.

Now our problem looks more like this:

The boxes represent some part of the state, and the arrows are transformations applied to the state. The account number is transformed into a customer record, which is transformed into a list of accounts, and so on.

In many languages, this process can be expressed directly as a pipeline of operations. Elixir might look like this:

account_number
|> Customer.lookup
|> extract_customer_accounts
|> filter(&account_is_active/1)
|> extract_field(:balance)
|> sum

In this code there are no objects; instead we have data structures containing state, and each data structure is processed by a function to produce a new data structure. And, just as important, each of these data structures is immutable.

If you come from an OO background, you’re probably thinking that this is bullshit. It looks like more code, and possibly even lower-level code.

But stop and think about what this means. The functional part of your code is no longer tied to the data. And functions are now independent of each other: no function relies implicitly on the state tucked away in an instance variable by some prior function. Just think how this changes how you can approach testing. And composing code. And sharing. And…

So, Give Up Objects and Write Functional Code?

No. I personally use a middle ground, and it is working really well for me. I still write classes and use objects. By they are no longer the logic of my programs.

I use objects to hold state. The only methods they contain deal directly with setting and validating that state. They may also have a method to return their representation as a string or as JSON, and they might have a copy constructor (see later).

But these objects are immutable: they only reveal state via accessor methods, and they have no interface that changes their state.

Embarassing Simplicity

db-staging and gql-test are simply shell scripts that set environment variables. In this case I’m using the fish shell, but this’ll likely work in bash and zsh.¹

#!/opt/homebrew/bin/fish

set -x DB_HOST localhost
set -x DB_PORT 3306
set -x DB_NAME pip
set -x DB_USERNAME dave
set -x DB_PASSWORD "secret"

exec $argv

#!/opt/homebrew/bin/fish

set -x GQL_SERVER  "https://my.server"
set -x GQL_ACCESS_TOKEN  .....

exec $argv

The only special thing is the last line. After the script sets the configuration into environment variables, it calls exec to invoke the next command on the command line. This command will inherit the environment variables just set. If it’s another configuration script, it can set more variables, and then chain to the next command.

For example, when you run

$ db-staging gql-test ruby ....

the db-staging script receives the arguments gql-test ruby .... When it calls exec, the gql-test script is run, receiving ruby ... as an argument. This is our application, and it runs with all the configuation we just set up.

And, when it finishes and returns us to our top-level prompt, the variables are no longer in the environment.

Software Minimalism

Of course, every developer knows that this applies to what we do. Everryone knows rgat we start by producing the simplest thing that could work. We never start projects by thinking about tools, frameworks, and libraries.

Do we…

Inner Tennis, Inner Ear

Learning is Forgetting

Civilization advances by extending the number of important operations which we can perform without thinking about them Alfred North Whitehead

Tennis and vestibular therapy are both great examples of what it means to learn. When we start out doing something, we have no way to make sense of what we’re seeing. We often have to rely on someone else to interpret it for us: “that message means it can’t find the file” or “you’ll find it easier to turn the wheel if you accelerate coming out of the turn.”

Gradually we internalize all these sensations: we start to build context. We then have a framework which we can use both to store new knowledge and to predict outcomes in novel situations.

After a while, we get to the point where we handle all the low-level stuff without consciously thinking about it: we don’t remember driving to the supermarket, we don’t know why we bounced the ball one extra time before serving, we don’t know why the code we just wrote feels wrong.

The Drefus model of skill acquisition provides a useful model of this process. It describes five stages of learning, starting at Novice where you have to be told what to do and process it using explicit concious thought. At the other end of the spectrum is Expert, where you have internalized the knowledge and can act without thinking about it. As you gain experience, you learn to trust your intuition.

The beginner doesn’t know. The expert doesn’t necessarily know why.

After my little episode, I had to treat my brain as a novice at balancing, showing it explicitly what things felt like while reciting out loud what I was doing. After about a week of this, I’m now at the point where I can do most daily activities without thinking about balance.

So much of what we do, both as developers and as people, relies on this intuition. And we can develop this intuition using conscious execises: doing a thing and telling ourselves what we’re doing.

Earmark

One of the things I do when faced with a new language is to try to implement a basic Markdown parser. Markdown is ugly enough that you have to explore lots of language features to get it right.

When I first learned Elixir, I wrote a Markdown parser that was actually usable, called Earmark. It was used early on to format the documentation for Elixir libraries.

After a while, I reached the point where I had to move on, so I asked if anyone would be interesting in taking over the project.

Robert Dober stepped up. This was one of the most fortunate things that have happened to me in my open source career. Robert has not just maintained Earmark; he’s extended it in ways I’d have never imagined. It’s really his project now, not mine. And I’m so grateful for that.

But to my shame, I’ve never publicly thanked Robert for his work. So, as he himself steps down from the project, I want to say “Thank you, Robert, for all the effort and imagination you’ve invested in this software. You’ve made a major difference to the Elixir world, and made things better for developers globally.”

Give a maintainer a hug today. They deserve it.

Video

My video solution is to create slides with a pure black background. I then combine my camera video and the window capture from the slide presentation inside OBS.

I then set up a video filter for the slide channel which basically masks out anything that is black: only the slide foreground appears in the final video. I tried using the green screen filter with a custom color, but I ended up getting better results with a luma key:

The next problem was that the slide content was hard to read on the video background. So I added a gray box around the content on the slide, and added a color key filter to OBS to match it and then reduce its opacity. The result (shown at the top of this post) is a semi-transparent gray background for the slide content.

And that’s it. I create slides with the black background and the content enclosed in the gray box. When merged with the camera video, the black disappears and the gray becomes semitransparent. When I don’t want any slide content on the feed, I simply include a blank black slide in the deck.

To start a talk, I bring up the slide presentation on a screen, make sure OBS is connected to both that screen and my camera feed, and then start the OBS virtual camera. I then configure the video source for whatever is being used to broadcast the talk (Zoom, Jitsi, or whatever) to use the OBS feed.

Audio

I’m still struggling a little with audio. Unfortunately, OBS doesn’t send sound out through its virtual camera device. But only OBS knows the delays it is introducing to the video feed, and so only it can properly synchronize the audio and video. I’m still experimenting with this: currently I’m using VB-Cable to intercept the incoming output, feed it through some processing, and then feed the result as audio input to the video conference software. The results have not been great. The audio sounds fine, but the timing drifts. If anyone has a good way of handling this, I’d love to hear it.

Recap: Dynamic Components

A dynamic component represents a server factory. You ask the factory for a new server, and then use that server until you’re done with it. Being a server, it can maintain its own state.

$ mix new game

def play_game() do
  game = %Hangman.State{} |> Hangman.choose_word_to_guess
  accept_guess(game)
end

def accept_guess(game) do
  IO.puts "The word so far: #{game.word_so_far}"
  IO.puts "Letters used: #{game.letters_guessed |> Enum.join(",")}"
  guess = get_next_letter()
  game  = Hangman.score_guess(game, guess)
  cond do
   game.won || game.lost ->
    handle_end_of_game(game)
   game.good_guess ->
    handle_good_guess(game, guess)
   else
    handle_bad_guess(game, guess)
   end
end

My Perspective

I’m really nervous about the client having unfettered access to the state of the game. I’m not worried about cheating. I’m concerned because the state is really to do with the implementation of the game. And the implementation of a module should be no ones business but the module’s.

In the code above, the client assumes that the game state contains a string, word_so_far representing the current state of the guess, and that it contains an enumerable, letters_guessed of the guesses so far.

Maybe when we first wrote it, both were true.

But maybe things changed in the game. We decided to change the way we record guesses—say we use a bitmap instead. And maybe we no longer keep a version of the word so far: we can always reconstruct it from the target word and the guesses.

Both of these things are internal implementation issues, but both changes break the client.

This is coupling, and on a larger scale this is a major reason changing code is difficult,

If processes in Elixir were expensive, I would 100% agree with this approach. But they’re not. Instead, developer time is a significant cost, and in particular the time spend maintaining and changing code. So I’m prepared to take the hit of having an extra process lying around if it makes the life of future-me easier.

So my approach would be to turn the game from being as library into being a server. It keeps its game state totally hidden from the outside world: all the rest of the world gets to see is a PID and an API.

The client might look something like this:

alias Hangman, as: H  # 'cos I'm lazy

def play_game() do
  game = H.create()
  accept_guess(game)
end

def accept_guess(game) do
  IO.puts "The word so far: #{H.word_so_far(game)}"
  IO.puts "Letters used: #{H.letters_guessed(game) |> Enum.join(",")}"
  guess = get_next_letter()
  result = Hangman.score_guess(game)
  cond do
   result.won || result.lost ->
    handle_end_of_game(game)
   result.good_guess ->
    handle_good_guess(game, guess)
   else
    handle_bad_guess(game, guess)
   end
end

Not much different, really. But now the internal state is opaque, and it’s only made available via API calls. If we change the internal implementation, we can keep the same API, and therefore not break anything.

So, because of this, I’m a big fan of encapsulating state inside processes, even if it isn’t technically necessary.

And that’s what I use Dynamic components for.

As always, I’m looking forward to some interesting comments and perspectives.

The Smoking Guns

Here’s the result of running mix new my_app --sup:

"my_app"
├── config
│   └── config.exs
├── lib
│   ├── "my_app"
│   │   └── application.ex
│   └── "my_app.ex"
├── mix.exs
├── README.md
└── test
    ├── "my_app_test.exs"
    └── test_helper.exs

Here’s a Ruby project created with bundle gem my_app:

"my_app/"
├── bin
│   ├── console
│   └── setup
├── Gemfile
├── lib
│   ├── "my_app"
│   │   └── version.rb
│   └── "my_app.rb"
├── "my_app.gemspec"
├── Rakefile
└── README.md

Here’s a JavaScript project:

├── dist
│   ├── app.js
│   └── index.html
├── node_modules
├── src
│   ├── lib
│   │   ├── login.js
│   │   └── user.js
│   ├── app.js
│   └── index.html
├── gulpfile.js
├── package.json
└── README

And here’s the recommended Go project tree:

"project-layout/"
├── api
│   └── README.md
├── assets
│   └── README.md
├── build
│   ├── ci
│   ├── package
│   └── README.md
├── cmd
│   ├── README.md
│   └── "_your_app_"
├── configs
│   └── README.md
├── deployments
│   └── README.md
├── docs
│   └── README.md
├── examples
│   └── README.md
├── githooks
│   └── README.md
├── init
│   └── README.md
├── internal
│   ├── app
│   │   └── "_your_app_"
│   ├── pkg
│   │   └── "_your_private_lib_"
│   └── README.md
├── LICENSE.md
├── Makefile
├── pkg
│   ├── README.md
│   └── "_your_public_lib_"
├── README.md
├── scripts
│   └── README.md
├── test
│   └── README.md
├── third_party
│   └── README.md
├── tools
│   └── README.md
├── "vendor"
│   └── README.md
└── web
    ├── app
    ├── README.md
    ├── static
    └── template

Let’s look at just the top level directory of each:

my_app
├── config
├── lib
├── mix.exs
├── README.md
└── test

"my_app/"
├── bin
├── Gemfile
├── lib
├── "my_app.gemspec"
├── Rakefile
└── README.md

├── dist
├── "node_modules"
├── src
├── gulpfile.js
├── package.json
└── README

"project-layout/"
├── api
├── assets
├── build
├── cmd
├── configs
├── deployments
├── docs
├── examples
├── githooks
├── init
├── internal
├── LICENSE.md
├── Makefile
├── pkg
├── README.md
├── scripts
├── test
├── third_party
├── tools
├── vendor
└── web

Now let’s imagine we can find an alien, unsullied by the preconceptions and practices of us developer folk. Let’s call this individual Normal Human Being.

We show these structures to Normal and ask “what’s going on here?” Normal thinks about this for a while.

“Well,” they say, “I imagine that you organize your thinking hierarchically (how quaint) and that you put the most important things at the top level. Doing things this way means that you can then split each top-level important thing into subthings at the next level in your hierarchy.”

“So, based on what I see across this selection of programming languages, you’re showing me four projects where the most important thing about the code is the boilerplate housekeeping. Clearly, you must be seriously advanced if this trumps the actual code that you write.”

Once they realized that they were wrong, and that, yes, we do worship housekeeping, the NHBs invaded the planet, conquering us without a shot being fired (largely because we were still waiting for NPM install to run on the fire control computers).

First, Choose The Correct Problem

I believe the reason for these baroque structures is simple: in the past we learned that when we write complex applications, things got out of hand. We ended up with directories with random names, some containing hundreds of files, each project different from the others.

So we fixed the wrong problem.

We said, “if we impose a strict structure on our projects, we can tame this complex katamari damacy of files.”

But what we should have said was, “let’s not write such big projects.”

Break It Down

Imagine if instead we could build our code using lots of individual components, and that each component could fit into a single source file of (at most) a couple of hundred lines.

What would the project directory tree look like then?

├── "my_app.ex"    <- "source of component".
├── meta.yml       <- "dependencies, build options, etc".
└── "tests/"

The README is simply the leading comment block in the source file. The build tool uses the metadata to find and resolve dependencies, compile the source, and construct an executable. For a component with no dependencies that followed the default conventions of the language, and which used doctests, the component could be as simple as:

└─ "my_app.ex"    <- all we need

You May Say I’m Bikeshedding

And I am. Project directory structure is definitely not a big problem.

But the fact that we had to invent these structures, which hide the actual code of our project two, three, or more levels deep in a directory tree, is a smoking gun. We are writing all our code in a single project; a single application. The lessons of the monorail seem to have been forgotten. Frameworks such as Phoenix encourage you to bundle application logic into the web serving codebase. Deployment tools make it simpler to release monolithic applications. And coupling between components only increases.

You May Say I’m A Dreamer…

…but I am experimenting with possible solutions. I’m working on a proof of concept library that makes it easier to write distributed, concurrent components in Elixir. I’m playing with a tool that makes it easier to assemble these components into deliverable solutions, and to deploy those assemblies to a dynamic cloud of machines. It’s way too early to say I have a solution to any of this. But I do have something I can experiment with.

I gave a talk about this at Empex last month.

Things have changed even since then. But I’m still convinced we need to rethink how we write code.

Separating Execution Strategy from Logic

Elixir and Erlang have an interesting execution module. You program using processes and message passing, but they abstract this into the concept of servers. You call a function (typically GenServer.call(pid, args)) and this in turn send a message to the server identified by pid. Inside that server, you write callback functions that are invoked in response to these messages.

In real life, no one wants to use a server whose API involves such convolution. So the convention arose that you’d provide an API layer to your server, written in the same module. Here’s an example, stolen from the Elixir guide, and cut down somewhat:

defmodule KV.Registry do
  use GenServer

  ##############
  # Client API #
  ##############

  def start_link do
    GenServer.start_link(__MODULE__, :ok, [])
  end

  def lookup(server, name) do
    GenServer.call(server, {:lookup, name})
  end

  def create(server, name, value) do
    GenServer.call(server, {:store, name, value})
  end

  ####################
  # Server Callbacks #
  ####################
  
  def init(:ok) do
    {:ok, %{}}
  end

  def handle_call({:lookup, name}, _from, names) do
    {:reply, Map.fetch(names, name), names}
  end

  def handle_call({:store, name, value}, _from, names) do
    {:reply, Map.put(names, name, value)}
  end
end

Here the top half of the module is the public facing API, and the lower half is the code that runs in a separate process and that implements the functionality.

I’ve never been comfortable with this. It seems to bury the important part—the actual implementation—in amongst all kinds of GenServer housekeeping. It also makes the development of the code a lot trickier—you have to write the server at the same time that you write the implementation.

So in the course I recommended a different approach—one that I’ve been using personally for a while. In it, I write the application functionality in its own module, under lib/. This has no GenServer support—after all, it’s just the app logic.

defmodule Kv.Impl do       # in lib/kv/impl.ex
  def new() do
    %{}
  end

  def lookup(names, name) do
    Map.fetch(names, name)
  end

  def store(names, name, value) do
    Map.put(names, name, value)
  end
end

That makes it easier to see what’s going on. I can also write tests directly against this logic.

Now, here’s the weird part. You might look at this and say that here’s the module that other applications should call. But I think that’s not the case. Let’s instead declare the API in the top-level kv.ex file:

lib
├── kv
│   └── impl.ex
└── kv.ex       ("the API belongs in here")

Right now, this API just calls directly down to the implementation.

defmodule Kv do
  defdelegate new(),                     to: Kv.Impl
  defdelegate lookup(names, name),       to: Kv.Impl
  defdelegate store(names, name, value), to: Kv.Impl
end

Also pretty clean, right?

So now we have a working application (aka library), and people can start using it.

defmodule Kv.Server do
  use GenServer
  alias Kv.Impl

  def init(store) do
    { :ok, store }
  end

  def handle_call({:lookup, name}, _, store) do
    result = Impl.lookup(store, name)
    { :reply, result, store }
  end

  def handle_cast({:store, name, value}, store) do
    { :noreply, Impl.store(store, name, value) }
  end
end

defmodule Kv do
  def new() do
    { :ok, names } = GenServer.start_link(Kv.Server, %{})
    names
  end
  
  def lookup(names, name) do
    GenServer.call(names, {:lookup, name})
  end
  
  def store(names, name, value) do
    GenServer.cast(names, { :store, name, value })
    names
  end
end

lib
├── kv
│   ├── impl.ex     ("Application implementation")
│   └── server.ex   ("GenServer implementation")
└── kv.ex           (API)

Bonus Section

lib
├── kv
│   ├── bucket
│   │   ├── impl.ex
│   │   └── "server.ex"
│   ├── bucket.ex
│   ├── impl.ex
│   └── "server.ex"
└── kv.ex

What is a private function?

A private function (or method) is one that can only be called from inside the module (or class) that defines it. It is invisible as far as the rest of the code is concerned.

Why do we need them?

A module is a collection of functions that share a common purpose: working out sales tax, interfacing to Twitter, creating a chord progression, and so on. The rest of the code in an application calls functions in a module when it needs that module’s expertise. For example, a Twitter module will have functions to send a tweet, read a tweet, and maybe subscribe to a timeline. The functions that do this are part of the modules external interface—its API.

The API functions are public. They are exposed to the rest of the app.

But an API function could be doing a complex job. So we would want to split it up into a number of subfunctions. These will typically be written in the same module. You can think of them as the implementation of the API.

Logically, they’re just functions. We can define them just as we define the API functions. But there’s a problem. Other people may read our code and see that, as well as the “official” API, we expose all these helper functions. And maybe one or two of these might be useful in their code. So they call them.

So what if someone calls my helper functions?

Imagine it’s a month later, and you realize that you could improve the implementation of the module you wrote. You get stuck in, rip out half the code and replace it with new stuff.

Now, you don’t want to change your API—other people depend on that. But you feel totally free to change any of the helpers. After all, they’re just there to implement the API. After to finish hacking, you make sure your API passes its tests, and publish your masterpiece.

Thirty seconds later the emails start arriving: “You broke my code.” People who (wrongly) relied on the internal implementation of your module suddenly found that the functions they use had disappeared, or had changed.

Their code was coupled to the internal implementation of yours.

{:.wide}

Why is coupling bad?

To my mind, there is only one rule when it comes to designing software:

Given the choice between two alternatives, choose the one that makes future change easier

Most of the principles of good design are just someone’s idea of how to codify some aspect of this.

Avoiding unnecessary coupling is one of those principles.

If thing Y depends on thing Z, then changes to thing Z affect thing Y. Even worse, if X depends on Y, then a change to Z might force a change to Y which then breaks X. And, to make it a total disaster, dependencies aren’t nice and linear like this. Instead, they form a complex mesh. In a bad (typical?) code base, these dependency chains can often interconnect the majority of the code.

The problem is that a change to any module in such a system has the potential to ripple through to every other part of the system. Change the calling sequence of a function, and potentially dozens of other modules will need to change, too. This is the software equivalent to the butterfly flapping its wings in Tokyo. It’s chaos. And it makes it hard (and stressful) to change code.

What’s this got to do with private functions?

Every time you code a call from module A to a function in module B, you set up a dependency between them. A becomes coupled to B.

That’s not a bad thing. The whole reason you write code is to have it be called. But you try to arrange things so that when you write a module you provide a public API, which you expect people to call, and a private implementation, which is none of their business.

In the old days, this was never actually enforced. You wrote comments saying one(), two(), and three() are the public API, and you’d have some kind of banner comment saying

/**************************************************/
/*  The low-level implementation follows...       */
/**************************************************/

Then language designers wised up, and added visibility modifiers. You could declare that the functions that formed your module’s API were publicly available, and the rest were private. Now you were free to change the internals, safe in the knowledge that, as long as you didn’t change the API, nothing would break as a result of your refactoring.

What’s this got to do with testing?

Nothing.

And that’s the point.

If I feel the need to test a piece of code, I want to isolate that code as much as I can. This lets my tests focus on just the thing they’re testing. Sure, I want to test my APIs. But I also want to isolate and test pieces of the implementation, too.

Doesn’t that mean your tests may fail if you change the implementation?

Of course.

But then you can’t refactor

Says who? Seriously.

Refactoring encourages you to change the implementation without changing the API. It suggests using tests to verify this. Those tests shouldn’t fail at the end of each step of refactoring. But they may well fail during the refactoring.

So if they can fail, then so can the tests that I write on my private implementation functions. The only difference is that the API tests act as proxies for the rest of the application. You shouldn’t change them—the same tests should run identically against the pre- and post-refactored code.

The tests of the implementation are volatile. If you break them, it could be because they are now testing the wrong thing. It’s perfectly OK to change these to reflect the changes to the implementation.

The point?

• It’s good to decompose complex functions into smaller ones. Ideally each function has just one responsibility.

• It’s good to differentiate the stuff that shouldn’t change (your public interface) from the stuff that may change (your internal implementation). Making the implementation private makes this easier.

• You should test code at both the API level and at the granular level. Visibility modifiers make the latter nearly impossible. Hence the Private library.

The People

Over these 20 years, I’ve been blessed to work with and learn from a vast number of amazing people. The dedication and skill of everyone at the Bookshelf—Janet, Susannah, the editors, and everyone else—continues to amaze me. The effort and enthusiasm of our authors is stunning. And the loyalty of our hundreds of thousands of readers is humbling.

Why the Change?

Deep in my soul, I’m a programmer. Given a free choice, I create code, write or speak about code, or sit chatting about code.

At the Bookshelf, I’ve written tons of code—the online systems, the book production toolchain, and a bunch of other stuff. But I also do a lot of grown-up stuff, like meetings, marketing, accounting, and so on. And it feels like personally I’m seeing diminishing returns from this.

This year I turn 60. I’m feeling like I probably have one, maybe two, new adventures in me. And I just knew that if I stayed where I was, I’d be comfortable, but I’d be thinking wistfully about what might have been.

I talked about this with Andy last year, and in January I finally reached the decision. He has been fantastic since then as we work to extract my fingers from the various pies that make up the Bookshelf.

I’m not abandoning the bookshelf—I’m still an owner, and will be working with Andy as needed. I’m just moving away from the day-to-day stuff.

What Next?

I honestly don’t know. That’s exciting.

I do know that I will be teaching a credit class on Elixir and real-world development at SMU this fall. I also want to start creating a different kind of content for some tutorial ideas I have.

I now have more time to attend and speak at conferences, workshops, and so on. One of the great joys in my life is the set of people who I considered to be good friends even though we only meet stroboscopically at conferences around the world. We meet in Amsterdam and continue a conversation we started in Bangalore three months earlier. I’m looking forward to seeing them more often.

I’m also looking forward to doing more advisory and consulting work. I really enjoy looking at problems from a different angle and coming up with surprising solutions (and I’m actually quite good at it).

In the meantime, I’m doing what I love to do—playing with new languages, tools, techniques, and ideas.

I’ll continue to be pragdave. I’m just pushing a new release or two….

Keep up with me at pragdave.me.

And remember to make it fun!

DirWalker—Some Background

I’m working on an app that helps organize large numbers of photos (about 3Tb of them). I needed to be able to traverse all the files in a set of directory trees, and do it lazily. I wrote a GenServer where the state is a list of the paths and files still be be traversed, and the main API returns the next n paths found by traversing the input paths. The code that returns the next path looks something like this:

defp next_path([ path | rest ], result) do
  stat = File.stat!(path)
  case stat.type do
  :directory ->
    next_path([files_in(path) | rest], result)
  :regular ->
    next_path(rest, [ path | result ])
  true ->
    next_path(rest, result)
  end
end

So, if the next file in the list of paths to scan is a directory, we replace it with the list of files in that directory and call ourselves. Otherwise if it is a regular file, we add it to the result and call ourselves on the remaining paths. (The actual code is more complex, as it unfolds the nested path lists, and knows how to return individual paths, but this code isn’t the point of this post.)

The Real World Intrudes

Having added my DirWalker library to Hex.pm, I got a feature request—could it be made to return the File.Stat structure along with the path to the file?

I wanted to add this capability, but also to make it optional, so I started coding using what felt like the obvious approach:

defp next_path([ path | rest ], opts, result) do
  stat = File.stat!(path)
  case stat.type do
  :directory ->
    next_path([files_in(path) | rest], result)
  :regular ->
    return_value = if opts.include_stat do
      {path, stat}
    else
      path
    end
    next_path(rest, [ return_value | result ])
  true ->
    next_path(rest, result)
  end
end

So, the function now has nested conditionals—never a good sign—but it is livable-with.

Then I thought, “while I’m making this change, let’s also add an option to return directory paths along with file paths.” And my code explodes in terms of complexity:

defp next_path([ path | rest ], opts, result) do
  stat = File.stat!(path)
  case stat.type do
  :directory ->
    if opts.include_dir_names do
      return_value = if opts.include_stat do
        {path, stat}
      else
        path
      end
      next_path([files_in(path) | rest], [return_value | result])
    else
      next_path([files_in(path) | rest], result)
    end
  :regular ->
    return_value = if opts.include_stat do
      {path, stat}
    else
      path
    end
    next_path(rest, [ return_value | result ])
  true ->
    next_path(rest, result)
  end
end

Moose Lends a Paw

So, lots of duplication, and the code is pretty much unreadable. Time to put down the keyboard and take Moose for a walk.

As it stands, the options map represents some state—the values of the two options passed to the API. But we really want to think in terms of transformations. So what happens if we instead think of the options as transformers?

Let’s look at the include_stat option first. If set, we want to return a tuple containing a path and a stat structure; otherwise we return just a path. The first case is a function that looks like this:

fn path, stat -> { path, stat } end

and the second case looks like this:

fn path, _stat -> path end

So, if the include_stat value in our options was one of these two functions, rather than a boolean value, our main code becomes simpler:

defp next_path([ path | rest ], opts, result) do
  stat = File.stat!(path)
  case stat.type do
  :directory ->
    if opts.include_dir_names do
      return_value = opts.include_stat.(path, stat)
      next_path([files_in(path) | rest], [return_value | result])
    else
      next_path([files_in(path) | rest], result)
    end
  :regular ->
    return_value = opts.include_stat.(path, stat)
    next_path(rest, [ return_value | result ])
  true ->
    next_path(rest, result)
  end
end

We can do the same thing with include_dir_names. Here the two functions are

fn (path, result)  -> [ path | result ] end)

and

fn (_path, result) -> result end

and now our main function becomes:

defp next_path([ path | rest ], opts, result) do
  stat = File.stat!(path)
  case stat.type do
  :directory ->
    return_value = opts.include_stat.(path, stat)
                |> opts.include_dir_names.(result)
    next_path([files_in(path) | rest], return_value)
  :regular ->
    next_path(rest, [ opts.include_stat.(path, stat) | result ])
  true ->
    next_path(rest, result)
  end
end

Changing the options from being simple state into things that transform values according the the meaning of each option has tamed the complexity of the next_path function.

But we don’t want the users of our API to have to set up transforming functions—that would force them to know our internal implementation details. So on the way in, we want to map their options (which are booleans) into our functions.

defp setup_mappers(opts) do
  %{
    include_stat:
      one_of(opts[:include_stat],
             fn (path, _stat) -> path         end
             fn (path, stat)  -> {path, stat} end),
    include_dir_names:
      one_of(opts[:include_dir_names],
             fn (_path, result) -> result            end, 
             fn (path, result)  -> [ path | result ] end)
  }
end

defp one_of(bool, if_false, if_true) do
  if bool, do: if_true, else: if_false
end

If you’re interested in all the gritty details, the code is in Github.

My Takeaway

I wrote my first OO program (in Simula) back in 1974 (which is probably before most Elixir programmers were born—sigh). During the intervening years, I’ve developed many reflexes that made object-oriented development easier. And now I’m having to rethink that tacit knowledge.

Programming in Elixir encourages me to move away from state and to think about transformations. As I force myself to apply this change in thinking at all levels of my code, I discover interesting and delightful new patterns of development.

And that’s why I’m still having a blast, hacking out code, after all these years.

Move to the Right

Let’s look again at the four values:

Individuals and Interactions over Processes and Tools
Working Software over Comprehensive Documentation
Customer Collaboration over Contract Negotiation, and
Responding to Change over Following a Plan

The phrases on the left represent an ideal—given the choice between left and right, those who develop software with agility will favor the left.

Now look at the consultants and vendors who say they’ll get you started with “Agile.” Ask yourself where they are positioned on the left-right axis. My guess is that you’ll find them process and tool heavy, with many suggested work products (consultant-speak for documents to keep managers happy) and considerably more planning than the contents of a whiteboard and some sticky notes.

If you see this too, then it’s more evidence of the corruption and devaluation of the word “agile.”

(Of course, some of these consultants may well have paid for a two-day training course. I haven’t, so they are masters and I am not, which means I’m probably wrong.)

Back to the Basics

Here is how to do something in an agile fashion:

What to do:

• Find out where you are
• Take a small step towards your goal
• Adjust your understanding based on what you learned
• Repeat

How to do it:

When faced with two or more alternatives that deliver roughly the same value, take the path that makes future change easier.

And that’s it. Those four lines and one practice encompass everything there is to know about effective software development. Of course, this involves a fair amount of thinking, and the basic loop is nested fractally inside itself many times as you focus on everything from variable naming to long-term delivery, but anyone who comes up with something bigger or more complex is just trying to sell you something.

All of these sentences are imperative—they are based on verbs telling us what to do and how to do it.

And that leads me to my suggestion.

Let’s abandon the word agile to the people who don’t do things.

Instead, let’s use a word that describes what we do.