Hillel Wayne

Toolbox languages

Mon, 15 Jul 2024 00:00:00 +0000

A toolbox language is a programming language that’s good at solving problems without requiring third party packages. My default toolbox languages are Python and shell scripts, which you probably already know about. Here are some of my more obscure ones.

AutoHotKey

Had to show up! Autohotkey is basically “shell scripting for GUIs”. Just a fantastic tool to smooth over using unprogrammable applications. It’s Windows-only but similar things exist for Mac and Linux.

Useful features:

You can configure shortcuts that are only active for certain programs, if a global flag is set, if certain text appears on screen, etc.
Simple access to lots of basic win32 functionality. Opening the file selection dialog is just f := FileSelect().
The GUI framework is really, really good. Honestly the best of any language I’ve used, at least for small things.

Example problems:

Audacity doesn’t let you configure mouse shortcuts, so I used AutoHotKey to map the middle-mouse to a keyboard shortcut anyway.

#HotIf WinActive("ahk_exe audacity.exe")
  MButton::Send "^l" ; silence selection
#HotIf

I made typing `;iso` fill in the current date.

:R:;iso:: 
{
    Send(FormatTime(,"yyyy-MM-dd"))
}

This is a tool I use to take timestamped notes.

; right-ctrl + d
>^d::
{
  TimeString := FormatTime(,"MM/dd hh:mm tt")
  t_msg := InputBox(,TimeString,"w200 h100")
  if t_msg.Result = "OK" {
    timestampfile := A_WorkingDir . "\Config\timestamps.txt"
    FileAppend(TimeString . "`t" . t_msg.Value . "`r`n", timestampfile)
  }
}

Other uses: launch REPLs for toolbox languages. Input the 100-keypress sequence required to beat one videogame (if you know, you know).

J

An array language, like APL. Really good at doing arithmetic on arrays, hair-pullingly frustrating at doing anything with strings or structured data. I used to use it a lot but I’ve mostly switched to other tools, like Excel and Raku. But it’s still amazing for its niches.

Useful features:

It is insanely terse. Things that would take a several lines in most languages take a few characters in J, so I like it for quickly doing a bunch of math.
First-class multidimensional arrays. + can add two numbers together, two arrays elementwise, a single number to every element of an array, or an array to every row (or column) of an higher-dimension array.
There are lots of top-level primitives that do special case mathematical things, like decompose a number into its prime factors.

Example problems:

Get all of the prime factors of a number:

    q: 2520
2 2 2 3 3 5 7

Given two processes, each running a four step algorithm, how many possible interleavings are there?

    ni =: !@:(+/) % */@:!
    ni 4 4
70

What if I wanted a table of interleavings for each value of 1 to 3 processors and 1 to 3 steps?

   (ni@:$)"0/~ >: i. 3
1  1    1
2  6   20
6 90 1680

Frink

Possibly the most obscure language on this list. Frink is designed for dimensional analysis (math with units), but it’s also got a bunch of features for covering whatever the developer thinks is interesting. Which is quite a lot of things! It’s probably the closest to “a better calculator” of any programming language I’ve seen: easy to get started with, powerful, and doesn’t have the unfamiliar syntax of J or Raku.

Useful features:

Lots of builtin units and unit modifiers. calendaryear is exactly 365 days, tropicalyear is 365.24, and half nanocentury is about 1.6 seconds.
Date literal notation: # 2000-01-01 # - # 200 BC # is 2199.01 years.
There’s a builtin interval type for working with uncertainties. It’s a little clunky but it works.

Example problems:

If someone was born at midnight on Jan 1st 1970, when do they become a billion seconds old?

   # 1970 # + 1 billion seconds
AD 2001-09-09 AM 02:46:40.000 (Sun) Central Daylight Time

If I run a certain distance in a certain time, what’s my average pace?

   // In miles per hour
   2.5 miles / (27 minutes + 16 seconds) -> mph
5.5012224938875305623

   // In meters per hour
   2.5 miles / (27 minutes + 16 seconds) -> meters / hour
8853.3594132029339854

   // In (minutes, seconds) per mile
   1 / (4.5 miles / hour) -> [minutes/mile, seconds/mile, 0]
13, 19

What’s (6 ± 2) * (8 ± 1)?

   x = new interval [2, -2]
   (6 + x) * (8 + x/2)
[28, 72] // range is between 28 and 72

Raku

Raku (née Perl 6) is a really weird language filled to the brim with dark magic. It’s very powerful and also very easy to screw up. I’m not yet comfortable running it for a production program. But for personal scripting and toolkits, it’s incredible.

Useful features

You can define your own infix operators! And postfix operators. And circumfix operators.
Lots and lots of syntactic sugar, to a level that worries me. Like instead of [1, 2] you can write <1 2>. And instead of ["a", "bc"] you can write <a bc>. Raku Just Knows™ what to do.
If you define a MAIN function then its parameters are turned into CLI arguments.
Multimethods with multiple dispatch, based on runtime values. Combining this with MAIN makes small CLI tooling really easy.
Many of the mathematical operators have unicode equivalents (like ∈ for `(elem)`), which synergizes well with all of my AutoHotKey hotstrings.

Example problems

Generate three random 10-character lowercase strings.

> for ^3 {say ('a'..'z').roll(10).join}
fflqymxapa
znyxehaqvo
qwqxusudqw

Parse unusual structured data formats with grammars (see link).

Copy a bunch of SVG ids over into inkscape labels.

use XML;

my $xml = from-xml-file("file.svg");

for $xml.elements(:NEST, :RECURSE<99>) -> $e {
  with $e<id> ~~ /k\w/ {
    say $_.target;
    $e.set("inkscape:label", $_.target);
  }
}
$xml.save()

Write a CLI with a few fiddly combinations of options (example).

Picat

My newest toolbox language, and the language that got me thinking about toolboxes in general. A heady mix of logic programming, constraint solving, and imperative escape hatches. I first picked it up as a Better Constraint Solver and kept finding new uses for it.

Useful features:

Assignment to variables. Shockingly useful in a logic language. Lots of problems felt almost right for logic programming, but there’d always be one small part of the algorithm I couldn’t figure out how to represent 100% logically. Imperative provided the escape hatch I needed.
The planner module. I love the planner module. It is my best friend. Give it a goal and a list of possible actions, Picat will find a sequence of actions that reaches the goal. It is extremely cool.

Example problems:

If I run at 4.5 miles/hour for X minutes and 5.1 for Y minutes, what should X and Y be to run 3.1 miles in 38 minutes?

import cp.
[X, Y] :: 1..60, 45*X + 51*Y #= 31*60,  X+Y #= 38, solve([X,Y]).
X = 13
Y = 25
yes

Given a bunch of activities, time constraints, and incompatibilities, figure out a vacation plan.

Checking if a logic puzzle has multiple solutions. Checking if the clues of a logic puzzle are redundant, or if one could be removed and preserve the unique solution.

Mocking up a quick Petri net reachability solver.

What makes a good toolbox language?

Most of the good toolbox languages I’ve seen are for computation and calculation. I think toolbox languages for effects and automation are possible (like AutoHotKey) but that space is less explored.

A toolbox language should be really, REALLY fast to write. At the very least, faster than Python. Compare “ten pairs of random numbers”:

# python

from random import randint
[(randint(10), randint(10)) for _ in range(10)]

# Raku
^10 .roll(2) xx 10

# J
10 2 ?@$ 10

A few things lead to this: a terse syntax means typing less. Lots of builtins means less writing basic stuff myself. Importing from a standard library is less than ideal, but acceptable. Having to install a third-party package bothers me. Raku does something cool here; the Rakudo Star Bundle comes with a bunch of useful community packages preinstalled.

If you can do something in a single line, you can throw it in a REPL. So you want a good REPL. Most of the languages I use have good repls, though I imagine my lisp and Smalltalk readers will have words about what “good REPL” means.

Ideally the language has a smooth on-ramp. Raku has a lot of complexity but you can learn just a little bit and still be useful, while J’s learning curve is too steep to recommend to most people. This tends to conflict with being “fast to write”, though.

Other tools I want in my toolbox

jq for json processing
Javascript, so I can modify other people’s websites via the dev console
Some kind of APL that offers the benefits of J but without the same frustrations I keep having
A concatenative PL if I ever find out what small problems a CPL is really good for
Something that makes webscraping and parsing as easy as calculation. Requests and bs4 ain’t it.

Thanks to Saul Pwanson for feedback. If you liked this post, come join my newsletter! I write new essays there every week.

I train companies in formal methods, making software development faster, cheaper, and safer. Learn more here.

My new book, Logic for Programmers, is now in early access! Find it here.

Composing TLA+ Specifications with State Machines

Mon, 17 Jun 2024 00:00:00 +0000

Last year a client asked me to solve a problem: they wanted to be able to compose two large TLA+ specs as part of a larger system. Normally you’re not supposed to do this and instead write one large spec with both systems hardcoded in, but these specs were enormous and had many internal invariants of their own. They needed a way to develop the two specs independently and then integrate them with minimal overhead.

This is what I came up with. Warning: this is a complex solution is aimed at advanced TLA+ users. For a much (much) gentler introduction, check out my website learntla.

The example

Let’s start by giving a motivating system: a Worker sends an authentication request to a Server. If the password matches the server’s internal password, the server responds “valid”, otherwise it responds “invalid”. If the worker receives “invalid” as a response, it goes into an error state. The worker can retry from that state and submit a new authentication request.

The worker and server have shared state via the request/response. As an additional complication, we’ll add internal state to the server, in the form of a request log that is hidden from the worker.

We can use this example to show the problems of composition and my solution (though I’ll say the example is a little too simple to make it worthwhile).

The problem with composition

What we want is for the composition to be as simple and painless as possible. If our specs are WorkerSpec and ServerSpec, the easiest composition would just be

CombinedSpec == WorkerSpec /\ ServerSpec

I talk about the problems we have in-depth here, but the gist is that if ServerSpec and WorkerSpec are “normal” specs, they’ll place contradictory constraints on the shared variables.

For example, WorkerSpec will likely read the server response, but not modify it. So to run WorkerSpec independently of the composition, we have to say the response never changes, which is equivalent to saying we can’t change it, which makes it impossible for ServerSpec to send a response!

The normal way around this is to break apart both WorkerSpec and ServerSpec into collections of actions, and then carefully stitch them together in non-contradictory ways. Which is about as complex as it sounds: composing two specs can be as much work as writing them in the first place.

This is why I’m trying to find a better way.

The big idea

What we need to do is write specs intended to represent part of the world and then incorporate them into a “whole world” main spec. To do this, we’ll use one of TLA+’s most powerful features: we can use x' to both assign the next value to x and constrain what the next value can be. Say we have

VARIABLE x, y 

Foo == 
  /\ x' \in {0, 1}
  /\ y' \in {0, 1}

Bar == x' < y'

Next == Foo /\ Bar

When TLC evaluates Next, it reads x' and y' in Foo as assignments. There are four possible assignments, so the model checker evaluates them all.

Then, since x' and y' are already chosen, TLC reads the statement in Bar as a constraint. Three of the possible assignments break that constraint, so TLC eliminates those possibilities, leaving us with a unique next state.

This means that a Spec Q can take two specs, X and Y, and constrain them against each other. X can have an action Inc that increments x_log, and then Q says that Inc can only happen if y_flag is true. Similarly, Q can make one assignment trigger another:

Inc == 
  /\ x_log' = x_log + 1

Sync ==
  IF ~y_flag /\ y_flag' 
  THEN Inc 
  ELSE TRUE

Next == (X!Next \/ Y!Next) /\ Sync

Now, changing y_flag to true forces an increment in x_log. Q is using one spec to drive side-effects in the system.

This is just a state machine! Constraints on transitions are just guard clauses and assignments on transitions are just effects. These can be enforced on other specifications that the state machine doesn’t know about.

Here’s how to make this idea work in practice:

Write an abstract state machine for all of the high-level state transitions of the core component
Write the other components as “open” specs that don’t fully describe their next states.
Refine the core component into the main spec, with a Sync action that adds guards and side-effects to the state transitions.

The Solution

We’ll model this with three separate specs: workerSM.tla, server.tla, and system.tla.

The state machine

Since the whole system is based around the worker’s state machine, let’s start with workerSM.tla. This won’t represent what happens when the worker transitions states, just what the transitions are.

(source)

------------------- MODULE workerSM -------------------
EXTENDS Integers

VARIABLES state

Transitions == {
    [from |-> "init", to |-> "ready"]
    , [from |-> "ready", to |-> "requesting"]
    , [from |-> "requesting", to |-> "done"]
    , [from |-> "requesting", to |-> "error"]
    , [from |-> "error", to |-> "ready"]
}

Init == state = "init"

Valid(t) == state = t.from
ValidTransitions == {t \in Transitions: Valid(t)}
ValidOutcomes == {t.to : t \in ValidTransitions}

Done ==
    state = "done"

Next ==
    \E t \in ValidOutcomes:
        state' = t

Fairness == 
    /\ WF_state(Next)
    /\ SF_state(state' = "done")

Spec == Init /\ [][Next]_state /\ Fairness
Liveness == <>Done

=========================================================

Transitions represents the set of valid transitions. The three Valid- operators are just helpers. Adding them is good practice; most TLA+ specs don’t have enough helpers.

The Fairness constraint is a little complex, but all it’s saying is that we can’t always transition from requesting to error. If we get there often enough, we’ll eventually transition to done instead.

Otherwise this spec doesn’t put any conditions on the transfers: we don’t need to “do anything” to go to done. That’s what system.tla is for.

show cfg

I used this to directly test the worker.

SPECIFICATION Spec

PROPERTY Liveness
CHECK_DEADLOCK FALSE

It passes, meaning this spec guarantees <>Done.

Next, the outside component we’ll integrate with the worker.

The server

------------------- MODULE server -------------------
CONSTANTS Password, NULL

VARIABLE req, resp, log

internal == <<log>>             \* (a)
external == <<req, resp>>       \* (a)
vars == <<external, internal>>  \* (a)

Init == 
    /\ req = NULL
    /\ resp = NULL
    /\ log = {}

CheckRequest ==
    /\ req # NULL
    /\ log' = log \union {req}
    /\ IF req = Password THEN
         resp' = "valid"
       ELSE
         resp' = "invalid"
    /\ req' = NULL

Next == \* (b) 
  CheckRequest

Spec == Init /\ [][Next]_internal \* (c)
====

We’re doing three unusual things in this spec. First is that we split the variables into internal and external vars (a). External vars represent state that’s shared with other parts of the composed specification, while internal only pertains to the server’s properties. Here we represent requests to the server with req and responses with resp.

Second, the spec will trivially deadlock (b). Next only contains the CheckRequest action, CheckRequest is only enabled if req isn’t null, but nothing in the spec makes req not-null. This spec isn’t “self-contained”, and it needs to be used by another specification to be meaningful.

Third, Spec is only stuttering invariant with respect to internal variables. This is done by writing [][Next]_internals instead of [][Next]_vars (c). This is the easiest part to miss but will make both spec composition and spec refinement easier down the line.

Now let’s put the server and the worker together.

The system

This is by far the most complex part. Let’s start with the whole spec, and then cover a breakdown of the most important sections.

system spec

---- MODULE system ----
EXTENDS TLC, Integers, Sequences
CONSTANT NULL
VARIABLES ws \* worker state
 , server_req \* requests to server
 , server_resp \* server response
 , server_log \* log (internal to server)

Strings == {"a", "b", "c"}

comm_vars == <<server_req, server_resp>>
vars == <<ws, server_log, comm_vars>>

Worker == INSTANCE workerSM WITH state <- ws

Server == INSTANCE server WITH 
  req <- server_req,
  resp <- server_resp,
  log <- server_log,
  Password <- "a"

Sync ==
    LET i == <<ws, ws'>> IN
    /\ CASE
            i = <<"ready", "requesting">> ->
            /\ \E x \in Strings:
                /\ server_req' = x
            /\ server_resp' = NULL
        []  i = <<"requesting", "error">> ->
            /\ server_resp = "invalid"
            /\ UNCHANGED comm_vars
        []  i = <<"requesting", "done">> ->
            /\ server_resp = "valid"
            /\ UNCHANGED comm_vars
        [] OTHER -> UNCHANGED comm_vars

Init ==
    /\ Worker!Init
    /\ Server!Init

Done == \* No deadlock on finish
    /\ Worker!Done
    /\ UNCHANGED vars

ServerNext ==
    /\ Server!Next
    /\ UNCHANGED ws

WorkerNext ==
    /\ Worker!Next
    /\ Sync
    /\ UNCHANGED server_log

Next == 
    \/ WorkerNext
    \/ ServerNext
    \/ Done

Fairness == 
    /\ WF_vars(Next) 

Spec == Init /\ [][Next]_vars /\ Fairness 

RefinesServer == Server!Spec
RefinesWorker == Worker!Spec

====

VARIABLES ws \* worker state
 , server_req \* requests to server
 , server_resp \* server response
 , server_log \* log (internal to server)

Strings == {"a", "b", "c"}

comm_vars == <<server_req, server_resp>>
vars == <<ws, server_log, comm_vars>>

I like to group my vars by purpose and spec. Since both the server and the worker use server_req and _resp, I put them in a separate comm_vars grouping.

Worker == INSTANCE workerSM WITH state <- ws

Server == INSTANCE server WITH 
  req <- server_req,
  resp <- server_resp,
  log <- server_log,
  Password <- "a"

I hard-coded the server’s Password constant for pedagogical convenience. We don’t have to instantiate the NULL constant because it’s the same name in both system.tla and server.tla, so it propagates automatically.

Sync ==
  \* See next section

Init ==
    /\ Worker!Init
    /\ Server!Init

ServerNext ==
    /\ Server!Next
    /\ UNCHANGED ws

WorkerNext ==
    /\ Worker!Next
    /\ Sync
    /\ UNCHANGED server_log

Done == \* No deadlock on finish
    /\ Worker!Done
    /\ UNCHANGED vars

Next == 
    \/ WorkerNext
    \/ ServerNext
    \/ Done

Init just shells out to both Worker!Init and Server!Init to set up their respective variables. It’s easy here because they don’t share any variables. If there was something that they both used or if system needed any extra booking variables, I’d handle them specially here.
ServerNext just says “the server can handle its own state updates”, but accounts for ws stuttering (because every variable needs to be assigned a value on every step). The server is independently active in the system and its behavior doesn’t synchronously depend on the worker. Keeping the components separate isn’t always possible when composing, but it makes things more convenient when applicable.
WorkerNext is the same: it behaves independently of ServerNext. But here’s where the “system” starts to play a role, in Sync.

Sync

Sync is where things get interesting.

Sync ==
    LET i == <<ws, ws'>> IN
    CASE
          i = <<"ready", "requesting">> ->
          /\ \E x \in Strings:
              server_req' = x
          /\ server_resp' = NULL
      []  i = <<"requesting", "error">> ->
          /\ server_resp = "invalid"
          /\ UNCHANGED comm_vars
      []  i = <<"requesting", "done">> ->
          /\ server_resp = "valid"
          /\ UNCHANGED comm_vars
      [] OTHER -> UNCHANGED comm_vars

To recap the earlier explanation, primed variables can be used in other expressions after being assigned. This means we can attempt a transition in one action and then check if the transition is valid in a later action. This is the key to making this all work. Without this, we’d have to put the guards and side effects in the same action as the transitions. For example, in this line:

      []  i = <<"requesting", "error">> ->
          /\ server_resp = "invalid"
          /\ UNCHANGED comm_vars

We are constraining the requesting -> error transition to only be possible if the server response was “invalid”. That’s only possible if Server!Next rejected the password. But our worker doesn’t need to know this. We can develop them independently and rely on the Sync to correctly compose their behaviors.

We can also use this to drive effects on the system:

      []  i = <<"ready", "requesting">> ->
          /\ \E x \in Strings:
              server_req' = x
          /\ server_resp' = NULL

This makes a ready -> requesting transition also trigger a request (and clear any outstanding response). This then enables Server!CheckRequest, which enables ServerNext, so the server can react to the system.

(source)

Remember, we added all of this machinery in first place in order to compose two complex specifications together. We need to make sure we’re composing them in a way that doesn’t violate either of their own properties. That’s handled by these lines:

RefinesServer == Server!Spec
RefinesWorker == Worker!Spec

These are refinement properties. At a high level, these check that system.tla doesn’t make the Server or Worker do anything they’re not normally able to do. For example, this property would fail if we removed something from log, since that’s not possible in Server!Spec. ¹

In TLA+, refinements are transitive. workerSM.tla had the liveness property <>Done. Since I already verified it in workerSM.cfg, I don’t need to test it in system.tla. If RefinesWorker passes, then Worker!Liveness is guaranteed to pass, too!

Why refinement is transitive

When we test RefinesWorker, what we’re verifying is

Spec => Worker!Spec 
(S)     (WS)

We already know from model checking workerSM.tla that

Worker!Spec => Worker!Liveness (2)
(WS)           (WL)

Implication is transitive: if S => WS and WS => WL, then S => WL.

And currently it fails:

show cfg

SPECIFICATION  Spec

CONSTANTS
  NULL = NULL

PROPERTY
    RefinesServer
    RefinesWorker

The error trace in VSCode. (source)

The problem is that the worker can keep picking the wrong password, so the server keeps rejecting it and the worker never completes. One fix is to say the system never retries the same password:

Sync ==
    \* ...
            /\ \E x \in Strings:
+               /\ x \notin server_log
                /\ server_req' = x

This makes the spec pass. If you’re concerned about leaking the server’s internal log to the system, you can add a log to the worker, too, either in system.tla or in a separate component that you include in the Sync.² An alternative fix uses a strong fairness constraint:

Sync ==
    \* ...
            /\ \E x \in Strings:
-               /\ x \notin server_log
                /\ server_req' = x

Fairness == 
    /\ WF_vars(Next) 
+   /\ SF_vars(WorkerNext /\ server_req' = "a")

This says that if it’s always-eventually possible to execute WorkerNext in a way that sends the right password, the spec eventually will. This makes the spec pass without changing the essential logic of the system.

Closing the open world

One last thing we need to do: make server.tla model-checkable. We can’t test it directly because it’s not a “closed” specification: it relies on some other spec to send requests. For more complex specs we want to be able to test that spec’s properties independently of the composition.

Fortunately, this is an “already solved problem” in the TLA+ community: refine the open spec into a closed one. We do this with a new file MCserver.tla:

---- MODULE MCserver ----
EXTENDS TLC, server

Strings == {"a", "b", "c"}
ASSUME Password \in Strings

MCInit == Init

WorldNext ==
    /\ \E x \in Strings:
        req' = x
    /\ UNCHANGED <<log, resp>>

MCNext ==
    \/ Next
    \/ WorldNext

MCSpec == MCInit /\ [][MCNext]_vars
====

This augments the server with an external World that can send requests. Now we can test the behavior of server.tla by model checking MCserver.tla.

show cfg

SPECIFICATION MCSpec
CONSTANT 
    NULL = NULL
    Password = "b"

If you’re on the VSCode TLA+ extension nightly, running the “Check model with TLC” command on server.tla will automatically run the model checker with MCserver.tla. It also comes with a TLA+ debugger. The VSCode extension is great.

Advantages and Drawbacks

This is a lot of work! So why do this instead writing one large spec?

The point of composing specs at all is so we can work on them independently. It’s not a big deal when each spec is ~30 lines, but when they’re each 200+ lines, you can’t just rewrite them as one spec.

So let’s compare it to a more conventional approach to composition. I’ll use the MongoDB Raft composition, which Murat Demirbas analyzes here. The independent specs are MongoStaticRaft and MongoLoglessDynamicRaft, which composed together in MongoRaftReconfig. The Next for MongoStaticRaft looks like this:

\* MongoStaticRaft

Next == 
    \* (a)
    \/ \E s \in Server : ClientRequest(s)
    \* etc
    \* (b)
    \/ \E s \in Server : \E Q \in Quorums(config[s]) : BecomeLeader(s, Q)
    \* etc

While the compositional Next looks like this:

\* MongoRaftReconfig
Next == 
    \/ OSMNext /\ UNCHANGED csmVars
    \/ CSMNext /\ UNCHANGED osmVars
    \/ JointNext

OSMNext == 
    \* (a)
    \/ \E s \in Server : OSM!ClientRequest(s)
    \* etc

JointNext == 
    \* (b)
    \/ \E i \in Server : \E Q \in Quorums(config[i]) : 
        /\ OSM!BecomeLeader(i, Q)
        /\ CSM!BecomeLeader(i, Q)
    \* etc

The actions under (a) must be manually repeated in OSMNext, while the actions under (b) must be be carefully interlaced with the other spec under JointNext. This is the standard way of composing two specs, which gets progressively more difficult the more specs you need to integrate. It’s also hard to check refinements in this paradigm.

I’m trying to avoid both issues with my Sync-based approach. Each component already has its own Next that we can use without changes. Anything that affects shared variables is handled by the additional Sync operator.

Would this spec benefit from my approach? I don’t know. I designed the approach for a primary “machine” interacting with external components, where here the two specs are on equal footing. I have a sketch for what the conversion would look like, but I don’t know if it’s necessarily better.

Sketch of changes

This is all without adding an extra state machine. Pick one “primary” spec, say CSM. Spec then becomes

Next == 
    \/ OSMNext
    \/ CSMNext

OSMNext ==
    /\ OSM!Next
    /\ UNCHANGED <<csmVars>>
    /\ UNCHANGED <<sharedVars>> \* new op

CSMNext ==
    /\ CSM!Next
    /\ Sync
    /\ UNCHANGED <<osmVars>>

We need to add sharedVars to OSMNext so it doesn’t call OSM!BecomeLeader on its own— that needs to happen through Sync. sharedVars will share some variables in common with with csmVars and osmVars.

Sync would look similar to JointNext:

Sync ==
    \/ \E i \in Server : \E Q \in Quorums(config[i]) : 
        /\ OSM!BecomeLeader(i, Q)
        /\ CSM!BecomeLeader(i, Q)
    \/ \* ...
    \/ UNCHANGED <<sharedVars>>

It annoys me that we’re repeating CSM!BecomeLeader in both CSMNext and Sync, but it’s the most straightforward way to ensure that both OSM and CSM use the same values for i and Q. I figured out some ways to deduplicate this but they all rely on creative misuse of TLA+ semantics.

My prediction is that conventional composition works for a wider variety of cases, but the Sync is more maintainable and scales better in the cases where it does work.

Neither approach handles the “one-to-many” case well. That’s where you have a spec for a single worker and try to use it to add N workers, where N is a model parameter. I discuss why this is so difficult in my article Using Abstract Data Types in TLA+.

Conclusion

This is a powerful technique, but also one that takes experience and adds complexity. It’s good if you need to write a very large specification or need a library of reusable components. For smaller specifications, I’d recommend the standard technique of putting all of the components in one spec.

This was developed for a consulting client and worked beautifully. I’m excited to share it with all of you! If you liked this, you can read other advanced TLA+ techniques here, like how to make model-checking faster..

Thanks to Murat Demirbas and Andrew Helwer for feedback. If you liked this post, come join my newsletter! I write new essays there every week.

I train companies in formal methods, making software development faster, cheaper, and safer. Learn more here.

Appendix: `Sync` without primes

Some company styleguides forbid using primed variables as an expression in later actions. In that case, you can get the same effect like this:

-Sync ==
-   LET i == <<ws, ws'>> IN
+Sync(t) ==
+   LET i == <<t.from, t.to>> IN

+Do(t) ==
+    /\ ws = t.from
+    /\ ws' = t.to  

WorkerNext ==
-   /\ Worker!Next
-   /\ Sync
+   /\ \E t \in Worker!ValidTransitions:
+       /\ Do(t)
+       /\ Sync(t)
    /\ UNCHANGED server_log

Do(t) is effectively just emulating the behavior of Worker!Next, except it lets us “save” the transition we use and pass it into Sync.

This is also useful for preserving parameters passed into an action, which is sometimes necessary for composition.

This is why we needed to write Server!Spec as [][Next]_internal and not [][Next]_vars. Server!Next only needs to hold for the internal variables, not the shared ones! ^[return]
I had a long section on composition via multilayer refinement: system.tla refines worker.tla refines workerSM.tla. But it ended up being too complex to thoroughly explain. Maybe that’ll be a part 2! ^[return]

What We Know We Don't Know: Empirical Software Engineering

Wed, 29 May 2024 00:00:00 +0000

This version of the talk was given at DDD Europe, 2024.

Technology is a multitrillion dollar industry, but we know almost nothing about how it’s best practiced. Empirical Software Engineering, or ESE, is the study of what works in software and why. Instead of trusting our instincts we collect data, run studies, and peer-review our results. This talk is all about how we empirically find the facts in software and some of the challenges we face, concluding with a guide on how to find existing research and an overview on some things we’ve learned about DDD.

Slides are here.

Sources

I referenced a bunch of papers in my talk. These are links so you can read them yourself:

Intro

Big Data is slower than laptops: Scalability! But at what COST?

Why we care

Section references

The Pragmatics of TDD

TDD is dead. Long live testing.

Methods

Controlled Trials

Comparing syntax highlightings and their effects on code comprehension

Natural Experiments

Simple Testing Can Prevent Most Critical Failures

Natural Experiments Gone Rogue

A Large Scale Study of Programming Languages and Code Quality in Github (original faulty study)

On the Impact of Programming Languages on Code Quality (replication study)

My 6,000 word writeup of the whole fiasco

Observational Studies

Fixing Faults in C and Java Source Code: Abbreviated vs. Full-Word Identifier Names (preprint)

DDD

Survey Paper

Domain-Driven Design in Software Development: A Systematic Literature Review on Implementation, Challenges, and Effectiveness

Interesting Papers

Design, Monitoring, and Testing of Microservices Systems: The Practitioners’ Perspective

Practitioner Views on the Interrelation of Microservice APIs and Domain-Driven Design: A Grey Literature Study Based on Grounded Theory

Refactoring with domain-driven design in an industrial context

Tackling Consistency-related Design Challenges of Distributed Data-Intensive Systems - An Action Research Study

Note that there are other interesting papers in the survey paper, these are just the ones I brought up in the talk.

Additional Sources

Recommended Reading

Teaching tech together

Leprechauns of Software Engineering

The Programmer’s Brain

Making Software

Free research

It Will Never Work In Theory

ACM digital library

ArXiv

Comment Section: Software Friction

Tue, 14 May 2024 00:00:00 +0000

These are some of the responses to Software Friction.

Blogs on a similar topic

Laurie Tratt wrote What Factors Explain the Nature of Software? which touches on the topic of friction, too.

Emails and Comments

I’m an Infantry Officer in the US Marine Corps, we talk quite a bit about friction. One book you may be interested in if you’re reading Clausewitz is Marine Corps Doctrinal Publication 1 (MCDP 1) Warfighting (link to PDF at bottom). It’s a pretty quick read.

We approach it largely in the ways you just laid out, we train people early in their careers about friction and that helps prime people to identify ways to solve it as they gain experience.

One tactic we use quite a bit that I don’t see you mention but that I think would work really well for software teams as well is using “hot washes”. Basically, immediately after an operation, whether in training or real-life, sitting down with all the key players to go over what went well, what went poorly, and whether or not there should be updates to Standard Operating Procedures for better future performance. Doing these regularly also helps take the “ego sting” out of it, for lack of a better phrase, where it can be hard to get called out about a mistake you made in a public setting with peers but like anything else it gets easier with repetition. Regular hot washes also build shared understanding between teammates which helps communication and reduces friction.

I asked how “hot washes” compare to postmortems. The response:

The main difference I would say (and maybe this is common in agile circles and I just don’t know about it) is that typically there will be a “scribe” that will record the points in a specific format (typically Topic-Discussion-Recommendation) and then these points are saved for posterity and stored in libraries at the unit, or frequently at the service level. This lets you reference mistakes you previously made, or even reference mistakes made years ago by units facing similar problems that you are about to face. The Marine Corps Center for Lessons Learned maintains the service-level library of these.

One of the challenges that I think is a bit more unique to us relative to a large software firm is that we have higher turnover, with people rotating out of a unit typically after around two years. So a persistent store of knowledge is hard to maintain in the heads of staff, it needs to get written down to be effective.

Software Friction

Wed, 01 May 2024 00:00:00 +0000

In his book On War, Clausewitz defines friction as the difference between military theory and reality:

Thus, then, in strategy everything is very simple, but not on that account very easy. Everything is very simple in war, but the simplest thing is difficult. These difficulties accumulate and produce a friction, which no man can imagine exactly who has not seen war.

As an instance of [friction], take the weather. Here, the fog prevents the enemy from being discovered in time, a battery from firing at the right moment, a report from reaching the general; there, the rain prevents a battalion from arriving, another from reaching in right time, because, instead of three, it had to march perhaps eight hours; the cavalry from charging effectively because it is stuck fast in heavy ground.

Ever since reading this, I’ve been seeing “friction” everywhere in software development:

A vendor’s API doesn’t work quite as you thought it did, or it did and then they changed it.
Bugs. Security alerts. A dependency upgrade breaks something.
Someone gets sick. Someone’s kid gets sick. Someone leaves the company. Someone leaves for Burning Man.
The requirements are unclear, or a client changes what they want during development. A client changes what they want after development.
A laptop breaks or gets stolen. Slack goes down for the day.
Tooling breaks. Word changes every font to wingdings. (This is a real thing)

This list is non-exhaustive and it’s not possible to catalogue all possible sources of friction.

Some Properties of Friction

Friction matters more over large time horizons and large scopes, simply because more things can go wrong.

Friction compounds with itself: two setbacks are more than twice as bad as one setback. This is because most systems are at least somewhat resilient and can adjust itself around some problem, but that makes the next issue harder to deal with.

(This is a factor in the controversial idea of “don’t deploy on Fridays”. The friction caused by a mistake during deployment, or of needing to doing a rollback, would be made much worse by the friction of people going offline for the weekends. The controversy is between people saying “don’t do this” and people advocating for systemic changes to the process. Either way the goal is to make sure friction doesn’t cause problems, it’s a debate over how exactly to do this.)

Addressing friction can also create other sources of friction, like if you upgrade a dependency to fix a security alert but the new version is subtly backwards incompatible. And then if you’re trying to fix this with a teammate who lives in a different timezone…

Addressing Friction

Friction is inevitable and impossible to fully remove. I don’t think it’s possible to even fully anticipate. But there are things that can be done to reduce it, and plans can be made more resilient to it. I don’t have insight into how military planners reduce friction. This is stuff I’ve seen in software:

Smaller scopes and shorter iterations

This is the justification for “agile” over “waterfall”. When you have short timelines then there’s less room for friction to compound. The more you’re doing and the longer your timeline the more uncertainty you have and the more places things can go wrong. You still have room for friction if you’re doing lots of small sprints back to back, though. Then you’re just running an inefficient marathon.

More autonomy

Friction is the difference between the model and the world, and at a high level you can only see models. If people have enough autonomy to make locally smart decisions, then they can recover from friction more easily. But if people get so much autonomy they isolate, they can make things much worse. I once saw an engineer with a lot of autonomy delete a database that was “too slow”.

Redundancy

This could be spare equipment in storage, high bus factors, or adding slack to a schedule. Then if something goes wrong you can fix it more quickly, leaving less room for another problem to compound. This comes at the cost of efficiency under normal circumstances, which is why projects naturally drift towards less redundancy.

Better planning

Good planning won’t identify all sources of friction, but planning will identify more sources, and that’s a big benefit. For example, writing formal specifications can expose problems in the design or turn unknown-unknowns into known-unknowns (which you can then study into more detail). This can be the difference between being blindsided by 5 things and being blindsided by 15 things. This is why I’m so bullish on formal methods.

Automation

This is a double-edged sword. On one hand, automating processes leaves less room for people to make mistakes. On the other, automated processes can have their own bugs, which creates their own sources of friction. Also, if the automation runs long enough people will forget how it works or the full scope of what it does, leaving everyone completely unprepared for if it breaks. Automation can come at the cost of experience.

Experience

The more problems you’ve encountered, the more problems you will see coming, and the more experience you’ll have recovering from problems. Unfortunately this is something you mostly have to learn the hard way. But one shortcut is…

Gaming

One interesting book on this is the Naval War College Fundamentals of War Gaming. In it they argue that there’s two purpose to wargaming: gathering information on how situations can develop, and giving commanders (some) experience in a safe environment. If a trainee learns that “weather can disrupt your plan” in a wargame, they don’t have to learn it with real human lives. Similarly, I’d rather practice how to retrieve database backups when I’m not desperately trying to restore a dropped table. To my understanding, both security and operations teams use gaming for this reason.

(At the same time, people have to devote time to running participate in games, which is inefficient in the same way adding redundancy is.)

Checklists and runbooks

Ways of formalizing tacit knowledge of dealing with particular problems.

Questions I have about friction

Is it useful to subcategorize sources of friction? Does calling a tooling problem “technical” as opposed to “social” friction do anything useful to us?

How do other fields handle friction? I asked some people in the construction industry about friction and they recognized the idea but didn’t have a word for it. What about event planners, nurses, military officers?

How do we find the right balance between “doing X reduces the effect of friction” and “not doing X is more efficient right now”?

Is friction important to individuals? Do I benefit from thinking about friction on a project, even if nobody else on my team does?

Thanks for Jimmy Koppel for feedback. If you liked this post, come join my newsletter! I write new essays there every week.

I train companies in formal methods, making software development faster, cheaper, and safer. Learn more here.

Update 2024-05-30

I’ve collected some of the comments I received on this post here.

Don't let Alloy facts make your specs a fiction

Wed, 10 Apr 2024 00:00:00 +0000

I’ve recently done a lot of work in Alloy and it’s got me thinking about a common specification pitfall. Everything in the main post applies to all formal specifications, everything in dropdowns is for experienced Alloy users.

Consider a simple model of a dependency tree. We have a set of top-level dependencies for our program, which have their own dependencies, etc. We can model it this way in Alloy:

sig Package {
  , depends_on: set Package
}

run {some depends_on}

Show Alloy tip

I’m going to use a slightly different model for the next example:

abstract sig Package {
  , depends_on: set Package
}

lone sig A, B extends Package {}

run {some depends_on}

I do things this way because it gives visualizations with A and B instead of Package$0 and Package$1. Alloy has built-in enums but they don’t play nice with the rest of the language (you can’t extend them or give them fields).

If we look through some of the generated examples, we see something odd: a package can depend on itself!

These kinds of nonsensical situations arise often when we’re specifying, because we have an intent of what the system should be but don’t explicitly encode it. When this happens, we need to add additional constraints to prevent it. For this reason, Alloy has a special “fact” keyword:¹

fact no_self_deps {
    all p: Package {
       p not in p.depends_on 
    }
}

Show Alloy tip

You can write the same fact purely relationally:

fact {
    no depends_on & iden
}

In general, the model checker evaluates purely-relational expressions much faster than quantified expressions. This can make a big difference in large models!

Alloy will not generate any models that violate a fact, nor will it look for invariant violations in them. It’s a “fact of reality” and doesn’t need to be explored at all.

The pitfall of facts

“No self-deps” is a great example of a fact. It’s also an example of a terrible fact. Beginners often make this mistake where they use facts to model the system, which quickly leads to problems.

Consider the real system for a second, not the spec. Where do the package manager dependencies come from? Usually a plain text file like package.json or Cargo.toml. What if someone puts manually a self-dependency in that file? Presumably, you want the package manager to detect the self-dep and reject the input as an error. How do you know the error-handling works? By having the checker verify that it accepts valid manifests and rejects ones with self-loops.

Except it can’t test the rejection because we told it not to generate any self-dependencies. Our fact made the self-deps unrepresentable.

Normally in programming languages, “making illegal states unrepresentable” (MISU) is a good thing (1 2 3 4). But specification covers both the software you are writing and the environment the software is running in, the machine and the world. If you cannot represent the illegal state, you cannot represent the world creating an illegal state that your software needs to handle.

Show Alloy tip

There is a technique to make invalid states representable in the world but not in the machine: refinement. The link goes to a TLA+ explanation but the same principle works in Alloy too: write an abstract spec without MISU, write an implementation spec with it, then show that the implementation refines the abstract spec. But you’ll do this with signatures and predicates, not with facts.

Instead of facts, you want predicates. Then you can test for the predicate being true or check that it’s necessary to get other properties. Make the constraint explicit instead of implicit.

// instead of

fact no_self_deps {/*body*/}

run {some_case}
check {some_property}

// do

pred no_cycles {/*body*/}

run {
  no_cycles and some_case
}

check {no_cycles implies some_property}

Predicates have the additional benefit of being “locally scoped”: if you have three facts and want to check a model with only two of them, you have to comment the third fact out.

When to use facts

So where should we use facts? When does it make sense to universally enforce constraints, when doing so could potentially weaken our model?

First, facts are useful for narrowing the scope of a problem. “No self-deps” is a perfectly reasonable fact if we’re only specifying the package installer and something else is responsible for validating the manifests. Writing the as a fact makes it clear to the reader that we’re not supposed to validate the manifest. This means we don’t make any guarantees if the assumption is false.

Second, facts rule out fundamentally uninteresting cases. Say I’m modeling linked lists:

sig Node {
  next: lone Node //lone: 0 or 1
}

This generates regular lists and lists with cycles, which are interesting to me. I don’t want to constrain away either case. But it also generates models with two disjoint lists. If I only care about single linked lists, I can eliminate extra lists with a fact:

fact one_list {
    some root: Node | root.*next = Node
}

Show Alloy tip

one_list also rules out “two link lists that merge into one”. If that’s something you want to keep, use the graph module:

open util/graph[Node]

fact {
    weaklyConnected[next]
}

Similarly, you can use facts to eliminate extraneous detail. If I’m modeling users and groups, I don’t want any empty groups. I’d add a fact like Users.groups = Group.

Third, you can use constraints to optimize a slow model. This is usually through symmetry breaking.

Finally, you can use facts to define necessary relationships that Alloy can’t can’t express natively. In the project I worked on, we had Red and Blue nodes in our graph. Red nodes had at least one edge to another node, Blue nodes had at most one. We wrote this as

abstract sig Node {
}

sig Red extends Node {
  edge: some Node
}

sig Blue extends Node {
  edge: lone Node
}

But then we couldn’t write generic predicates on nodes that use edge, because Alloy treated it as a type error. Instead we wrote it with a fact:

abstract sig Node {
  edge: set Node
}

sig Blue, Red extends Node {}

fact {
  all r: Red  | some r.edge
  all r: Blue | lone r.edge
}

Show Alloy tip

Okay, one more (rather niche) use case. Say you have a temporal model and a canonical spec predicate for system behavior. Then a lot of your assertions look like

module main

// spec

check {spec => always (prop1)}
check {spec => always (prop2)}

// etc

You can clean this up a lot by exporting all of properties to properties.als and writing it like this:

open main

fact {spec}

check {always (prop1)}
check {always (prop2)}
// etc

Conclusion

Constraints are dangerous because you need error states in order to check that your program avoids error states.

If you’re interested in learning more about Alloy, there’s a good book here and I maintain some reference documentation. I’m also working on a new Alloy workshop. I ran an alpha test last month and plan to run a beta test later this summer. Sign up for my newsletter to stay updated!

Thanks for Jay Parlar and Lorin Hochstein for feedback. If you liked this post, come join my newsletter! I write new essays there every week.

I train companies in formal methods, making software development faster, cheaper, and safer. Learn more here.

In other specification languages this is usually either a runtime constraint (like in TLA+) or a direct modification to the system spec itself. ^[return]

The Tale of Daniel

Mon, 01 Apr 2024 00:00:00 +0000

It’s April Cools! It’s like April Fools, except instead of cringe comedy you make genuine content that’s different what you usually do. For example, last year I talked about the strangest markets on the internet. This year I went down a different rabbit hole.

Daniel has been a top 5 baby name twice since 2000. There are over 1.5 million Daniels in the United States (and almost 100,000 last-name Danielses). There are who knows how many Dans, Dannies, and Daniellas out there. “Daniel” is, by any way you slice it, a very modern name.

It may be the oldest attested name in common use.

Last year was Chicago’s last Newberry Library book sale. Part of my haul was several collections of ancient literature:

I’m interested in ancient history, which shouldn’t be a surprise to anybody who reads this blog. And I love how their literature reflects the modern human experience. Take this lament carved in a pyramid from the Egyptian Middle Kingdom:

I have heard the words of Imohotep and Hardedef \ Whose sayings are recited whole. \ What of their places?

Their walls have crumbled, \ Their places are gone, \ As though they had never been! (pg 196)

This anonymous poet, writing 4500 years ago, shares our fears about losing our past and of our memories being forgotten. Beautiful.

But I found something far more relevant in the Mesopotamian literature.

Straightway Daniel the Raphaman … gives oblations to the gods to eat (pg 118)

Daniel the Raphaman

DANIEL

The story comes from the Tale of Aqhat, pulled out of the ruins of a forgotten city, estimated to approximately 1350 BCE. That’s a hundred years older than the oldest known Chinese writing, one thousand years older than the book of Daniel, and thirty-four hundred years before today. I needed to know if this was the same name, transmitted across cultures, stretching 3400 years of unbroken “Daniels” to our modern day. Is this where “Daniel” comes from, or is just a strange coincidence?

A historian of the modern world has to learn how to sip from a firehose of evidence, while the historian of the ancient world must learn how to find water in the desert. — Bret Devereaux

I’ve done some historical work before, but all of it was in software, a “history of the modern world”. I decided to try anyway. This is what I’ve put together from reading books and trawling JSTOR.¹

This is the tale of Daniel.

Ugarit

Obviously, finding an unbroken lineage of specific Daniels across three millennia is impossible. Instead, I want to learn if the name was in the “cultural consciousness”. If so, we can be modestly confident that real people are getting named Daniel, like how real people are being named Khaleesi today.

So let’s start with where the name was first found. Ugarit was a port city of the shores of Ras Shamra, unknown until 1928. Scholars believe that it was a city of roughly 8,000 people and the capitol of a kingdom of 20,000 (ref). While never a great power, Ugarit was a fairly “literate” society and had extensive trade relations with all of the regional powers.

Along with a trove of archaeological material (pots and doodads), excavators also found large collections of clay tablets. Like with many Bronze Age civilizations, most of these recovered documents are either religious, diplomatic, or administrative. That’s because clay tablets are expensive and only reserved for important matters. Most of these were dated to very approximately 1500-1200 BCE.

But also in that collection, we found three piece of literature. The first, the Baal Cycle, tells of how Baal Hadad became one of the great gods of the Ugaritic pantheon. The second, the Legend of Keret, probably matters a lot to another person chasing a completely different white whale, but doesn’t matter to me. The final one, the one that started this obsession of mine, is the Tale of Aqhat.

One of three recovered tablets that make up the tale. (source)

I found this picture in the InscriptiFact collection. Sadly no digital transcription! It tells of the story of the great hero Aqhat, who was loved and murdered by the goddess Anath. But I don’t actually care about any of that, I’m only after the Daniel. I’m pretty sure this is Daniel’s name:

I have a few friends who can read cuneiform, so I asked if any understood this text. I found out 1) that’s like asking a person who “can read ASCII” if they can understand Spanish, and 2) this is not actually cuneiform. It’s just an alphabet that looks like cuneiform.

The Ugaritic alphabet. (source)

Aqhat’s father is 𐎄𐎐𐎛𐎍. Left to right: D, N, ‘I, L. “Judged by El”, El being the chief God in the Ugaritic pantheon.

The name could be older than this text, since most early written myths were first oral traditions. This is just the first time the name “Daniel” is attested. The name could also be a lot younger, at least the modern name. The old name could just be an unusual coincidence, like how people in both England and China have the last name “Lee”.² For “Daniel” to be that old, it’s not enough for the Ugarits to have had the name, it has to come from them, in a chain of Daniels reaching into present.

And here we run into a problem. We know roughly how old the tablets are because we know roughly when the city collapsed. By cross-referencing names of Egyptian rulers that appear in Ugarit tablets with the much better preserved Egyptian corpus, historians estimate that the city collapsed around 1190 BCE. This corresponds with the “Bronze Age Collapse”, a cataclysmic era where dozens of cities and kingdoms in the region are turned to ruins. Scholars still don’t have a clear picture why.

Regardless of the why, the Ugarites disappeared from history. The name Daniel would have to be carried by a new generation.

But this is also where the chain breaks.

Phoenicia

The Ras Shamra find was a one-in-a-million miracle, an enormous corpus of about 40,000 words, roughly as much as one Harlequin romance novel. After that? Well, here’s the Hadad Statue:

The Hadad Statue (source)

If you look closely, you can see the bottom is covered in faint writing. The language is “Samalian”, one of the many languages in the region. The engraving is roughly 400 words long and represents over half of all Samalian. This essay is four times longer than the Samalian corpus.

So what happened? It’s easy to imagine the Bronze Age Collapse turning the broader region into a Stone Age wasteland, but the archaeological evidence suggests the opposite. Rather, it points to a rapid economic recovery of the region, with new kingdoms soon establishing a large number of seafaring trade routes. Eventually they’d come in contact with the Greeks, who’d call all of the many distinct groups “Phoenicians”.

But we know so little of what the “Phoenicians” (in all their languages) wrote because of how they wrote it. Instead of Ugarit’s clay tablets, the Phoenicians used papyrus. Papyrus is lighter and cheaper, but also rots after a few decades. The fragments of text we do have— with one big exception— are engravings.³ Inscriptions on tombs, statues, weapons, etc. This is a major problem for historians, but an even bigger problem for me, because it makes the question of Daniel impossible to answer.

Nonetheless, we can look for clues that the same stories persisted, even if we don’t have concrete evidence. Let’s start with the first two lines of the Hadad statue:

𐤀𐤍𐤊 𐤐𐤍𐤌𐤅 · 𐤁𐤓 · 𐤒𐤓𐤋 · 𐤌𐤋𐤊 · 𐤉𐤀𐤃𐤉 · 𐤆𐤉 · 𐤄𐤒𐤌𐤕 · 𐤍𐤑𐤁 · 𐤆𐤍 · 𐤋𐤄𐤃𐤃 · 𐤁𐤏𐤋𐤌𐤉‎· 𐤒𐤌𐤅 · 𐤏𐤌𐤉̇ ·· 𐤀̇𐤋𐤄𐤅 ·· 𐤄𐤃𐤃 · 𐤅𐤀𐤋 · 𐤅𐤓𐤔𐤐 · 𐤅𐤓𐤊𐤁𐤀𐤋 · 𐤅𐤔𐤌𐤔 · 𐤅𐤍𐤕𐤍 · 𐤁𐤉𐤃𐤉

This is the “Phoenician alphabet”—the same one that inspired the Greek alphabet— which many distinct cultures in the region used for their own languages. Unlike Ugaritic, it’s read right-to-left. Scholars don’t think this alphabet descended from Ugaritic, but the two writing systems are closely related: the same sounds appear in roughly the same places in the alphabet.⁴ Here’s the translation (from wiki):

I am Panamuwa, son of Qarli, king of Y’DY, who have erected this statue for Hadad in my eternal abode (burial chamber). The gods Hadad and El and Rašap and Rākib-El and Šamaš supported me.

The same gods, El and Baal Hadad, are important gods in the Ugarit corpus! And this is coming four hundred years after the fall of Ugarit. It suggests the Baal Hadad and El weren’t local gods to the Ugarits, but regional gods, part of a shared culture. The tale of Aqhat could have been preserved as a literary tradition, just on sources we don’t have anymore. And with Aqhat comes his dad(niel).

To get out of the Daniel dark ages, we have to look at a different inscription found 600 miles away.

The Misha Stele (source)

Like the Hadad statue, the Misha Stele is one of our only sources on a long-dead language. In this case, Moabite. It’s a distinct language from Samalian but this uses the same Phoenician alphabet. Here’s a sample (also from wiki):

I have built its gates and I have built its towers. I have built the palace of the king, and I made the prisons for the criminals within the wall. And there were no wells in the interior of the wall in Karchah. And I said to all the people, ‘Make you every man a well in his house.’ And I dug the ditch for Karchah with the chosen men of Israel.

Israel?

Yes, Israel.

The Greeks called this region Phoenicia, but the locals always called it Canaan. The same “Canaan” from the Bible. The Israelites were a Canaanite population with Canaanite religions: Israel literally means One who fought El. Slowly, over years and centuries, they developed their own ethnic identity. At some point, there were two Israelite kingdoms: the kingdom of Israel/Samaria in the north and the kingdom of Judah (where we get “Jew”) in the south.

Moab is here, too (source)

Israel was conquered in 720 BCE while the kingdom of Judah limps along, in one form or another, until 130 CE.⁵ And it’s through the people of Judah that we have the best corpus of that region’s writing.

"Présentation de la Loi" (source)

The Tanakh⁶, or “Old Testament”, is a 300,000 word corpus of ancient Hebrew, chronicling the stories, religious practices, and history (mythic and attested) of the Jews.⁷ It’s the reason we can decipher the scant inscriptions of other Canaanite languages like Moabite, Samalian, and even Ugaritic itself!

And it’s where the tale of Daniel continues.

Judah

Quick recap on Daniel so far: we know the Ugarites wrote about a son of Daniel, the Canaanites (nee Phoenicians) shared a culture with the Ugarites, and the Israelites shared a culture with the Canaanites.

In 600 BCE, the Neo-Babylonian empire conquered Judah, which leads to the sixty year “Babylonian exile” of Jews from Canaan. It’s allegedly during this time that we get the collection of prophecies known as the “Book of Ezekiel”.⁸ Mostly it’s about how the Jews lost the favor of God and will one day earn it back, and with it redemption.

Far, far more importantly, it marks the triumphant return of Daniel.

“Son of man, if a land sins against Me by trespassing grievously, I shall stretch forth My hand upon it and break its staff of bread, and I shall send famine upon it and cut off from it both man and beast. Now should these three men be in its midst- Noah, Daniel, and Job-they would save themselves with their righteousness, says the Lord God.⁹ — Ezekiel 14:13-14

Daniel is written (דנאל). As a distant descendent of Phoenician, Hebrew reads right-to-left: Daled, Nun, Aleph, Lamed. “God judged me.” Same meaning as the Ugaritic name, same letters as 𐎄𐎐𐎛𐎍.

I’m amazed that the same meaning can be preserved across so many centuries. But I also want to be as thorough as possible. Two names, eight hundred years apart, with the same meaning, is still not enough for me. I still worry that it could just be an enormous coincidence that two cultures in the same region, speaking sister languages, came up with the same name-meaning.

Ezekiel provides the missing link. In the Tanakh, both Noah and Job are treated as pious men, but critically, neither are Israelites.¹⁰ Making Daniel part of the trio implies he was also a righteous non-Jew.

Ezekiel would expect people would know who “Daniel” was if he used that name. This would make sense if Daniel was, say, a righteous figure in the broader Canaanite folklore. At least the Tale of Aqhat implies he was a righteous and just man who was worthy of El’s blessing. He prayed for a son and his wish was granted. He’d be a distinct character, a known righteous non-Israelite, which Ezekiel used as a point of reference.

A tenuous chain, I know, but enough to persuade me at least. Enough to persuade other scholars too. It may be the best we’ll ever do.

We can finally use the Bible to establish that Daniel wasn’t just a folkloric name, but a name people actually used. From the Book of Ezra:

And these are the heads of their father’s houses and the lineage of those who ascended with me in the kingdom of King Artaxerxes from Babylon. Of the sons of Phinehas, Gershom: of the sons of Ithamar, Daniel; of the sons of David, Hattush. — Ezra 8:1-2

And these were the sons of David who were born to him in Hebron: the firstborn, Amnon, to Ahinoam the Jezreelitess; the second, Daniel, to Abigail the Carmelitess. — Chronicles 3:1

If they’re using Daniel as the name of a minor son of a minor figure, it’s probably a real name.

The False Daniel

But now we have a problem.

The Ezekiel Daniel has four letters. But all of the later Daniels have five: Daled, Nun, Yud, Aleph, Lamed (דניאל) .

It’s still possible to read the four-letter version as “Daniel” but it’s more likely to be “Danel” or “Danil”. And, Ezekiel’s Daniel being closest to the Ugaritic DN’IL, suggests that DN’IL was also “Danel”. That’s how most modern scholars now transliterate the name.

So yes, I may have been chasing a false Daniel this whole time. Theophanu is not Tiffany, after all. Maybe “Daniel” is at best attested from 600 BCE, putting it in contention with things like Debrah and Zhou. There is a tradition to pronounce the Ezekiel DN’IL as “Daniel”, so it could be the actual pronunciation and the Yud was added later. I don’t know when the tradition started, though. At the very least, this strengthens the connection between Ezekiel’s דנאל and the Ugarite’s 𐎄𐎐𐎛𐎍: Ezekiel was working off a different name than the rest of the Tanakh authors.

Then again, maybe this is all just quibbling over specifics. The Latin alphabet wasn’t meant for Canaanite names, and the Hebrew alphabet wasn’t meant for Ugaritic ones. The Biblical שרה could be Romanized as “Sarah” or “Sara”, חנכה could be Hannukah, Channuka, or Chanukah. Maybe it doesn’t matter whether 𐎄𐎐𐎛𐎍 is דניאל or Danel or דנאל or Daniel. It’s still the same name.

Regardless, we’re following a chain, and there’s still one more link.

From Daniel to Today

There’s one last question here: lots of people use Tanakh names, like Sarah and Benjamin and David. But many Tanakh names, like Tamar and Yehuda and Hillel, are only used by Jews. Why is Daniel in the former group and not the latter?

I’d speculate that the difference is how likely it is for a Christian reader to encounter the name. “Adam” appears in the first book of Genesis, while Shamgar is a minor figure in the book of Judges. Christians would most likely encounter names that appear either really early, have entertaining stories, or are directly relevant to the New Testament.

And here’s where we finally get to the last link between then and now: the Book of Daniel. Like the Book of Ezekiel, this recounts the story of someone living during the Babylonian exile. Unlike Ezekiel, it was pretty clearly written four hundred years after the exile. Daniel contains an awful lot of prophecies that seem very relevant to the Greek occupation (the one that leads to the story of Channukah). The most famous of them is the prophecy of 70 weeks:

And you shall know and understand that from the emergence of the word to restore and to rebuild Jerusalem until the anointed king [shall be] seven weeks, and [for] sixty-two weeks it will return and be built street and moat, but in troubled times. — Daniel 9:25

Later writers took this to be a prophecy about Jesus. I don’t admit to understand how it’s supposed to be about Jesus, but it is. So the book of Daniel is really important in Christian thought. This kept the name “Daniel” in the Christian mainstream for the next 2000 years, carrying it across the world to Europe and India and East Asia and South America, all the way to the present daytoday. The end of a 3400 year chain.

So that’s the story of the name of Daniel, from an ancient clay tablet to the millions of Daniels alive now.

I want to reiterate that I’m not a trained historian and almost certainly got a lot of details wrong. Also, this isn’t and wouldn’t pass peer review. Don’t cite this for a class paper. Happy April Cools!

Thanks to Predrag Gruevski for feedback. If you enjoyed this, check out the April Cools website for all the other pieces this year!

I accessed JSTOR as a UChicago Alumnus benefit, but I also found out that anyone with a Chicago public library card has access too! Hundreds of public libraries do this! ^[return]
Note though that the Chinese “Lee” is just one possible Romanization of 李. “Li” is another. ^[return]
We have a much larger Egyptian corpus because papyrus lasts longer in Egypt’s hot and arid climate. We have large Greek and Roman corpuses because monks tirelessly copied old texts into new books for centuries. Volcanoes helped. ^[return]
We know the order because we’ve recovered Ugarit abedecaries (inscriptions of all an alphabet’s letters in order). ^[return]
At least, that’s when Jews were permanently expelled from the region. Growing up I was taught that the important event was the burning of the Second Temple, which happened in 70 CE. ^[return]
I originally wrote “the Torah” here, but as one commenter pointed out, the Torah is just the first five books of Moses. The Tanakh also includes the Nevi’im and Ketuvim, both of which I rely on later in the essay. I know all my old Rebbes are very disappointed in me. ^[return]
I grew up an Orthodox Jew, and it’s extremely weird to study the actual history of your religion. Like vivisecting holy books. ^[return]
From my research it seems that it’s generally accepted that most of Ezekiel is written contemporaneously with the Babylonian exile, though it may have been smoothed out later. ^[return]
For Old Testament verses I’m using the Chabad translations. My Hebrew never got past a first grade level. ^[return]
Noah is obvious: his story takes place before Abraham’s. Job is tougher, as there’s nothing in the story that makes it clear he’s not Jewish. I’m basing this off the claims of many different Rabbis and modern scholars. ^[return]

Comment Section: The Hunt For The Missing Data Type

Mon, 18 Mar 2024 00:00:00 +0000

I got a lot of responses to The Hunt For the Missing Data Type. I’ve included some of the most interesting ones below.

Response Blogs

The “missing” graph datatype already exists. It was invented in the ‘70s, about datalog.

Emails and Comments

Everything in quotes is verbatim.

GraphBLAS

My name is Michel Pelletier, I’m one of the contributors to the GraphBLAS API standard and python bindings.

Congrats on your blog going up to the front page on HN. I replied with some information that I think you might find useful to your question, but it’s pretty buried under a lot of the discussion there. I noticed that you invite people to email you, so I decided to send you the information directly.

The graph data type you’re missing is the one you already mentioned, a matrix. You mention adjacency matrices, but in the context of your blog post, they’re considered only a storage format, not the graph itself. But graphs and matrices are conceptually and algebraically isomorphic. All graphs, and thus all composite data structures, are mathematically matrices, and all matrices are graphs. Hypergraphs and Multigraphs are represented by “Incidence Matrices” which are node-to-edge to edge-to-node adjacencies using two rectangular matrices.

A really excellent introductory paper by a large group of researchers, spearheaded by Dr. Jeremy Kepner, Director of the Supercomputing Center and MITs Lincoln Laboratory is:

https://arxiv.org/pdf/1606.05790.pdf

The issue with computers and thinking of graphs as matrices is that most graphs are sparse, and most matrix libraries (like numpy) are dense. This makes using adjacency matrices very expensive, since most of the “space” in a dense matrix ends up being zero. This is extremely inefficient and defeats cache hierarchies typical of von Neumann architectures. These two worlds have not quite converged yet.

There is however a lot of research and development around efficient sparse matrix computation, and thus, sparse graph analysis. While these may seem different, they are actually one in the same: matrix multiplication is a breadth first step across a graph. This is part of the isomorphic nature between them. A lot of ML and AI research involves both sparse matrices and graphs, and this research is very much unified in improving the performance of both paradigms.

A big question I get about this is “why”. Why write a linear algebraic formula instead of a function that traverses the nodes and edges? And one of the most important answers is parallelization over huge graphs. When graphs get really, really big, billions or trillions of edges, you need to divide the work of your algorithm efficiently. How are you going to do that? Fork every edge? Use a thread pool? How to schedule the work and partition the graph efficiently? Now do that on CUDA… the problem becomes almost impossible for even the smartest programmers to tackle.

With the GraphBLAS, a graph operation is a linear algebraic formula, decomposed into a series of matrix multiplications, you just say something like Ax = b, and the underlying library figures out how to do the work most efficiently on a specific target architecture. Run it on a chromebook or a supercomputer, the code doesn’t change, just the capacity of the machine for bigger graphs. You can think of the GraphBLAS as a language that can be “JIT” compiled based not only on the underlying architecture but also the shape and type of problem you feed it. Since LA is the universal language of math, science and engineering, this technique has natural application to a lot of existing work.

So I just wanted to send that your way, good luck on your further exploration of the subject. I’m happy to chat with you anytime if you’d like to find out more, as a member of the C API Committee, evangelizing is part of my job and I enjoy introducing the subject.

Thanks!

Gremlin

In earlier drafts of the essay I talked about TinkerPop, Apache’s graph computing framework, and Gremlin, the query language. I removed it from the final version, and several people noticed it was missing. Here’s one response.

You mentioned Cypher, but didn’t talk about Gremlin, which is an expressive graph query language for Neo4j/TinkerPop:

https://tinkerpop.apache.org/

It was used to power the Joern SAST tool that was acquired by ShiftLeft, and I think is used a lot in the finance world.

I last used it to make a graph of all of the software packages in Maven and their interdependencies.

It’s got nice bindings into programming languages - I used Python Gremlin so I could drive it from a familiar scripting language, since the default REPL/script is Groovy that I’m not so handy with.

You can interchange between querying the graph with Gremlin and doing “normal” imperative scripting in Python, then do further querying from the results. It felt quite natural.

I don’t know about the state of whole graph algorithms for it - I was always interested in traversal-based queries vs whole graph statistics like centrality.

Rubik’s Cubes

I ran across your article about missing software libraries for graphs because it was linked from the CodeProject newsletter. It was a wonderful article. I’m not of the same caliber as the people you consulted when you were writing the article, but nevertheless I thought I would offer you one more example of the difficulties in creating generic software libraries for graphs. Namely, I’m a part of a very informal group of people who have been using computers for years to investigate the mathematics of Rubik’s Cube.

The math underlying Rubik’s Cube is called group theory and one of the things you can do with groups is to map them out with graphs called Cayley graphs. That know-it-all called Google describes it as follows: “Cayley graphs are frequently used to render the abstract structure of a group easily visible by way of representing this structure in graph form. Properties of a group G, such as its size or number of generators, become much easier to examine when G is rendered as a Cayley graph.” In particular, each possible Rubik’s Cube position can be represented as a node in a Cayley graph and adjacent nodes are those which can be reached in exactly one move. You mentioned the 15 puzzle your article. It turns out that one of the Rubik’s Cube investigators has written a complete and very fast solver for the 15 puzzle. And it also turns out that the size of the 15 puzzle is a drop in the bucket as compared to the size of the Cayley graph for Rubik’s Cube.

In any case, I have been writing code since 1985 to investigate Rubik’s Cube. This is not just to “solve” Rubik’s Cube, which is actually quite easy. Rather, it’s to determine for each of the nearly 10^20 positions the least number of moves that are required to solve position. The problem remains unsolved simply because it is so large. It has been determined that any position can be solved in either 20 moves or in 26 moves, depending on what you count as one move. But that’s not the same thing as finding the solution for each possible position that requires the least possible number of moves.

In any case, I know from much experience that I have had to develop my own data structures that are quite specific to Rubik’s Cube in order to address the problem in any practical manner. The key issue (and here I’m quoting from your article) is that Performance Is Too Important. No libraries I could find did what I needed, so I rolled my own.

Thanks for a wonderful article,

Jerry

I asked for more information on what “20 or 26” moves meant and whether he could share more about the data structures he uses. His reply:

As you guessed, whether you count a single half rotation as one move or two is the primary example of what counts as one move. If you only count quarter rotations as one move, then that’s called the quarter turn metric. Any position can be solved in 26 moves in this metric. If you count either quarter rotations or half rotations as one move, then that’s called the face turn metric. Any position can be solved in 20 moves in this metric. But there are other ways to choose what counts as one move. A Rubik’s Cube has three layers in any direction. Typically you only count moves of the outer layers such as the top and bottom but not the layer in between the top and bottom or the outer layers such as the right and left but not the layer between the right and left. But it is sometimes interesting to count moves of those middle layers as one move. Another variation is the stuck axle problem where you pretend one of the axles is stuck. For example, you don’t move the top face and only move the layers on the other five faces of the cube. With this variation, you can still reach all the possible positions but the Cayley graph does not have the same symmetry as it would have if none of the axles were stuck. Also, many more moves can be required to solve a cube with a stuck axle than the standard 20 moves or 26 moves.

I don’t think there is any standard data structure for Rubik’s Cube. Each person working on the cube has their own, except that there is an obvious sense that any data structure that faithfully represents the cube has to be somewhat isomorphic to any other data structure that faithfully represents the cube. A big distinction is whether the data structure consists only of positions or whether it consists both of positions and moves. For example, two consecutive clockwise quarter turns of the front face results in the same position as two consecutive counter-clockwise quarter turns of the front face. In a Cayley graph, these move sequences become a loop if continued. The loop is a four move cycle (a 4-cycle). So do you store moves and positions, or do you just store positions?

I really don’t know how anybody else’s data structures handles these issues. For myself, I don’t explicitly store the entire Cayley graph. Instead, I store positions and with each position I store single bit for each possible move from that position that indicates whether the move takes you further from the solved position or closer to the solved position. There are 12 such bits for each position in the quarter turn metric and 18 such bits for each position in the face turn metric. Those bits implicitly define a Cayley graph, but I do not store the graph explicitly. Other people working on the problem talk about using canonical sequences of moves, for example you can make two successive clockwise moves of the front face but not two successive counter-clockwise moves of the front face. I do something similar but not exactly the same using my bits.

The other issue is that I needed a tree structure and a tree can be thought of as a special case of a graph. Which is to say, a tree is just a graph where one node is declared to be a root node and where there are no loops in the graph. I had to roll my own tree structure. The tree structure I needed arises as follows. There are 54 color tabs on a standard Rubik’s Cube. In the standard mathematical model, the center color tab on each face does not move which leaves 48 color tabs which move. Of the 48 color tabs, 24 are on the corners of 3x3 squares and 24 are on the edges of 3x3 squares. The corner color tabs and edge color tabs are disjoint, so I represent a cube position by labeling each corner color tab with a letter from A to X and by labeling each edge color tab with a letter from A to X. Each position is then an ordered pair of words where each word consists of 24 letters and where each letter appears exactly one time in each word.

So why do I need a tree? Well, I need to be able to find these words very quickly. It’s like finding words very quickly in a spell check dictionary. Nominally, each node in the tree needs 24 pointers to other nodes in the tree. Except that unlike real words in a real spell check dictionary, each letter can appear only one time in each word. So as I get towards the leaf nodes of the tree, each node is going to consist mostly of null pointers, a huge waste of very valuable memory. So I had to create a tree structure to accommodate the storage of the positions for fast retrieval. No standard library routines were fast enough and conservative enough of memory.

So essentially I have two data structures layered together. One of them uses bit switches to define the Cayley graph for the cube, and the other of them uses a spell check dictionary style tree to locate specific positions very quickly. And trees are just special cases of graphs.

I don’t know if that answers your questions, but I hope it helps.

Jerry

Graphs and knowledge databases

Hillel — Thank you for sharing your exploration of graph representations with clear explanations as to why there is no clear winner. I’m wondering if the problems you cite aren’t so large as to preclude viable alternatives.

I come to this question first as a Smalltalk programmer and then as the creator of Wiki. Both have a graph like structure assembled in bits and persistent in use. The small community around my most recent wiki implementation is highly motivated to add graphs to pages along with paragraphs, images, outline, and tables. We have partially met this with Graphviz using Dot as a markup. But this does not yield to the computation and sharing as we get with files of yaml or csv.

We’ve recently adopted the practice of representing graphs (in javascript, for example) as an object with arrays for nodes and relations. The empty graph would be:
{nodes:[], rels:[]}
Nodes and relations are themselves objects with a string for type and an additional object for properties and some managed indices that hook them together. This then serializes conveniently into (loop free) json, a format widespread and adequate for graphs measured in kilobytes.

These graphs converts easily into Neo4j objects for which we have considerable experience. More often we avoid maintaining a database beyond wiki itself. Although we have also built a modest Cypher interpreter we have not found that useful. We have made the surprising discovery that we are more likely to merge many small graphs into the one that solves the problem at hand rather than build one large Neo4j graph and query out the graph needed to solve the same problem.

More recently we have come to separate problems into “aspects” based on cross-cutting concerns in the problem space. We might have tens of these graphs or even hundreds. We browse these as one might browse a wiki where the equivalent of a wiki-link comes from recognizing the same node appearing in yet-to-be-included graphs. This is a “recommending” process and query is replaced by choosing and un-choosing recommendations with work-in-progress rendered immediately with Graphviz running in the browser.

I started this email with a more complete description of our graph abstraction but stepped back from that as I was unsure you would find our experience interesting. I could also describe applications where this has proven useful usually involving some collaboration problem within a community.

I would love to correspond if you think there is any overlap in interests.

Thank you and best regards — Ward

His code for this is avaiable here, along with docs and examples. Followup email:

Hillel — We are on the hunt for small graphs representing aspects of larger problems. Here is a case where when asked to read a paper headed to the European Patterns conference I chose to read it very carefully and map every “relational” sentence in their patterns. Here is a picture where the an unexpected overlap between aspects is shown in yellow.

This particular graph viewer has had a circuitous life: started as scripts on wiki pages; extracted as a stand-alone web app on top of Croquet for online collaboration; then here as a single-user “solo” application heading back into wiki. It is not yet self-explanatory. But you can bring it up with the “open” link in the last paragraph of the wiki page where I coordinated with a co-reviewer who shares an interest in this sort of exploration. http://ward.dojo.fed.wiki/aspects-of-pattern-relations.html

Two more similar projects that show off different “aspect” possibilities: Chopping up a year of recent changes, and, annotating a search engine source code with node-relation comments and extracting them into graph files with GitHub actions.

The Hunt for the Missing Data Type

Sat, 02 Mar 2024 00:00:00 +0000

A (directed) graph is a set of nodes, connected by arrows (edges). The nodes and edges may contain data. Here are some graphs:

All graphs made with graphviz (source)

Graphs are ubiquitous in software engineering:

Package dependencies form directed graphs, as do module imports.
The internet is a graph of links between webpages.
Model checkers analyze software by exploring the “state space” of all possible configurations. Nodes are states, edges are valid transitions between states.
Relational databases are graphs where the nodes are records and the edges are foreign keys.
Graphs are a generalization of linked lists, binary trees, and hash tables.¹

Graphs are also widespread in business logic. Whitepapers with references form graphs of citations. Transportation networks are graphs of routes. Social networks are graphs of connections. If you work in software development long enough, you will end up encountering graphs somewhere.

I see graphs everywhere and use them to analyze all sorts of systems. At the same time, I dread actually using graphs in my code. There is almost no graph support in any mainstream language. None have it as a built-in type, very few have them in the standard library, and many don’t have a robust third-party library in the ecosystem. Most of the time, I have to roll graphs from scratch. There’s a gap between how often software engineers could use graphs and how little our programming ecosystems support them. Where are all the graph types?

As I ran into more and more graphs in my work, this question became more and more intriguing to me. So late last year I finally looked for an answer. I put a call out on my newsletter asking for people with relevant expertise— graph algorithm inventors, language committee members, graph library maintainers— to reach out. I expected to interview a dozen people, but in the end I only needed to talk to four:

Zayenz: Former core developer of the Gecode constraint solver, and who has “implemented every graph algorithm there is”
Bradford: Author of the Nosey Parker security library and inventor of several new graph algorithms
Nicole: Former graph database engineer
Kelly: Maintainer on the NetworkX python graph library and compiler developer.

After these four people all gave similar answers, I stopped interviewing and start writing.

The reasons

There are too many design choices

So far I’ve been describing directed graphs. There are also undirected graphs, where edges don’t have a direction. Both directed and undirected graphs can either be simple graphs, where there is a maximum of one edge between two nodes, or multigraphs, where there can be many edges. And then for each of those types we have hypergraphs, where an edge can connect three or more nodes, and ubergraphs, where edges can point to other edges. For each possible variation you have more choices to make: do you assign ids to edges or just to nodes? What data can be stored in a node, and what can be stored in an edge? That’s a lot of decisions for a library to make!

But wait, do these distinctions matter at all? A simple graph is just a degenerate multigraph, and and undirected edge can be losslessly transformed into two directed edges. A language could just provide directed hyperubermultigraphs and let users restrict it however they want.

There are two problems with this. First of all, it changes the interface, like whether various operations return single values or lists. Second, as I’ll discuss later, graph algorithm performance is a serious consideration and the special cases really matter. Kelly raised the example of maximum weight matching. If you know that your graph is “bipartite”, you can use a particular fast algorithm to find a matching, while for other graphs you need to use a slow, more general algorithm.

A bipartite graph (source)

[It] ties back to the “algorithm dispatch problem.” Given a Problem P, a Graph G, and Algorithms A, B, C to solve P on G… which one do you run? If we don’t know that G is bipartite, and Algorithm C only works on bipartite graphs, how much time can we afford to determine whether or not G is bipartite? — Kelly

The perfect graph library would support a lot of different kinds of graphs. But that takes time away from supporting what people want to do with graphs. Graph algorithms are notoriously hard to get right. In this essay, the inventor of Python implemented his own find_shortest_path algorithm. It had to be updated with corrections five times!

Every single implementation of pagerank that I compared to was wrong. — Nicole

So which algorithms should come with the library? “The amount of things people want to do with graphs is absurd,” Kelly told me. That matches my experience, and the experiences of all my interviewees. It sometimes seems like graphs are too powerful, that all their possibilities are beyond my understanding. “The question is,” Kelly said, “where do you draw the line?”

For NetworkX, “the line” is approximately 500 distinct graph algorithms, by themselves making up almost 60,000 lines of code. By comparison, the entire Python standard library, composed of 300 packages, is just under 600,000 lines.²

With all that, it’s unsurprising that you don’t see graphs in standard libraries. The language maintainers would have to decide which types of graphs to support, what topologies to special-case, and what algorithms to include. It makes sense to push this maintenance work onto third parties. This is already the mainstream trend in language development; even Python, famous for being “batteries included”, is removing 20 batteries.

Third parties can make opinionated decisions on how to design graphs and what algorithms to include. But then they’re faced with the next problem: once you have a graph interface, how do you represent it?

There are too many implementation choices

Let’s imagine we’re supporting only barebones simple directed graphs: nodes have identities, edges do not, neither has any associated data. How do we encode this graph?

(source)

Here are four possible ways a programming language could internally store it:

Edge list: [[a, b], [b, c], [c, a], [c, b]]
Adjacency list: [[b], [c], [a, b]]
Adjacency matrix: [0 1 0; 0 0 1; 1 1 0]
A set of three structs with references to each other

Different graph operations have different performance characteristics on different representations. Take a directed graph with 100 nodes and 200 edges. If we use an adjacency matrix representation, we need a 100×100 matrix containing 200 ones and 9,800 zeros. If we instead use an edge list we need only 200 pairs of nodes. Depending on your PL and level of optimizations that could be a memory difference of 20x or more.

Now instead take a graph with 100 nodes and 8,000 edges and try to find whether an edge exists between node 0 and node 93. In the matrix representation, that’s an O(1) lookup on graph[0][93]. In the edge list representation, that’s an O(|edge|) iteration through all 8,000 edges.³

Graphs with only a few edges are sparse and graphs with almost all edges are dense. The same program may need to do both operations on both kinds of graph topologies: if you’re constructing a graph from external data, you could start out with a sparse graph and later have a dense one. There’s no “good option” for the internal graph representation.

And all this trouble is just for the most barebones directed graph! What about implementing node data? Edge data? Different types of nodes and edges? Most third party libraries roughly fall in one of two categories:

Offer a single rich datatype that covers all use-cases at the cost of efficiency. NetworkX stores graph as a dict of dicts of dicts, so that both nodes and edges can have arbitrary data.⁴
Offer separate graph types for each representation, and rely on the user to store node and edge data separately from the graph type.

An example of the second case would be Petgraph, the most popular graph library for Rust. Petgraph has graph, graphmap, and matrix_graph for different use-cases. Bradford used Petgraph for Nosey Parker, a security tool that scans for secrets across an entire history of a git repo. His benchmarking graph is CPython, which has 250k commits and 1.3M objects but only a few edges per commit node. He went with an adjacency list.

Supporting many representations has a serious downside: you have to do a lot more work to add algorithms. If you write a separate version of the algorithm for each graph representation, you’re tripling or quadrupling the maintenance burden. If you instead write a generic abstraction over polymorphic types, then your library is less performant. One programmer I talked to estimated that a hand-rolled graph algorithm can be 20x faster or more than a generic algorithm.

And this gets into every interviewee’s major complaint.

Performance is too important

A “generic” graph implementation often doesn’t cut it. — Bradford

This is the big one.

Many, many graph algorithms are NP-complete or harder.⁵ While NP-complete is often tractable for large problems, graphs can be enormous problems. The choice of representation plays a big role in how fast you can complete it, as do the specifics of your algorithm implementation.

Everyone I talked to had stories about this. In Nosey Parker, Bradford needed to reconstruct a snapshot of the filesystem for each commit, which meant traversing the object graph. None of the four provided graph walkers scaled to his use case. Instead he had to design a “semi-novel” graph traversal algorithm on the fly, which reduced the memory footprint by a factor of a thousand.

I was able to get working a proof of concept pretty quickly with [petgraph], but then… this is one of those cases where the performance constraints end up meeting reality. — Bradford

Zayenz raised a different problem: what if the graph is simply too big to work with? He gave the example of finding a solution to the 15 puzzle. This is done by running a A* search on the state space. A state space with over 20 trillion states.

If you generate all the nodes, you’ve lost already. — Zayenz

Zayenz oversaw one research project to add graphs to the Gecode constraint solver. They eventually found that a generic graph type simply couldn’t compete with handpicking the representation for the problem.

Even graph databases, designed entirely around running complex graph algorithms, struggle with this problem. Nicole, the graph database engineer, told me about some of the challenges with optimizing even basic graph operations.

If you’re doing a traversal, you either have to limit your depth or accept you’re going to visit the entire graph. When you do a depth search, like “go out three steps from this and find the path if it exists”, then you’re just committing to visiting quite a bit of data. — Nicole

After leaving that job, she worked as a graph query performance consultant. This usually meant migrating off the graph database. She told me about one such project: to speed the graph queries up, she left one computation as-is and rewrote the rest as MapReduce procedures. “Which was a lot harder to understand,” she said, “But would actually finish overnight.”

All of this means that if you have graph problems you want to solve, you need a lot of control over the specifics of your data representation and algorithm. You simply cannot afford to leave performance on the table.

It was unanimous

So, the reasons we don’t have widespread graph support:

There are many different kinds of graphs
There are many different representations of each kind of graph
There are many different graph algorithms
Graph algorithm performance is very sensitive to graph representation and implementation details
People run very expensive algorithms on very big graphs.

This explains why languages don’t support graphs in their standard libraries: too many design decisions, too many tradeoffs, and too much maintenance burden. It explains why programmers might avoid third party graph libraries, because they’re either too limited or too slow. And it explains why programmers might not want to think about things in terms of graphs except in extreme circumstances: it’s just too hard to work with them.

Since starting this research, I’ve run into several new graph problems in my job. I still appreciate analyzing systems as graphs and dread implementing them. But now I know why everybody else dreads them, too. Thank you for reading!

Thanks to Predrag Gruevski for research help, Lars Hupel, Predrag Gruevski, Dan Luu, and Marianne Bellotti for feedback, and to all of the people who agreed to do interviews. If you liked this post, come join my newsletter! I write new essays there every week.

I train companies in formal methods, making software development faster, cheaper, and safer. Learn more here.

Appendix: Languages with Graph Types

Graph Querying Languages

Graph querying languages (GQLs)⁶ are to graph databases what SQL is to relational databases. There is no widely-used standard, but two of the most popular are SPARQL for querying RDF triples and Neo4j’s cypher. Ironically, GraphQL is not a graph querying language, instead being named for its connection to the Facebook Graph Search. I considered graph databases themselves mostly distinct from graphs in programming languages, but their query languages show how graphs could work in a PL.

The main difference between all GQLs and SQL is that the “joins” (relationships) are first-class entities. Imagine a dataset of movies and people, where people act in, direct, or produce movies. In SQL you’d implement each relationship as a many-to-many tables, which makes it easy to query “who acted in movie X” but hard to query “who had any role in movie Y, and what was that role”. In SPARQL relationships are just edges, making the same query easy.

PREFIX mv: <your_movie_ontology_URL>
SELECT ?person ?role
WHERE {
    ?person ?role mv:casablanca.
}

Cypher has a similar construct. GQLs can also manipulate edges: reverse them, compose them together, take the transitive closure, etc. If we wanted to find all actors with some degree of separation from Kevin Bacon, we could write

PREFIX mv: <your_movie_ontology_URL>
SELECT ?a
WHERE {
    mv:kbacon (:acted_in/^:acted_in)+ ?a.
    # a/b = join two lookups
    # ^a = reverse a
    # a+ = transitive closure
}

SPARQL cannot give the length of the path nor do computation along the path, like collecting the chain of movies linking two actors. GQLs that support this are significantly more complicated.

My main takeaway from looking at GQLs is that there’s a set of useful traversal primitives that a PL with graph support would need to provide. Interestingly, the formal specification language Alloy has all of these primitives for its “relation” datatype. For this reason I find working with a graph representation in Alloy much easier than in a proper programming language. That said, these all work with labeled edges and may not work for other graph representations.

Mainstream Languages with Graphs in the Standard Library

Python added a graphlib in 2020. Based on the discussion here, it was because topological sorting is a “fundamental algorithm” and it would be useful for “pure Python implementations of MRO [Method Resolution Order] logic”. Graphlib has no other methods besides TopologicalSorter, which only takes graphs represented as node dicts. Unusually, the direction of the node dict is reversed: the graph a -> b is represented as {b: [a]}.

As of 2023, nothing in CPython uses graphlib and there are fewer than 900 files referencing it on Github. By comparison, another package added in 2020, zoneinfo, appears in over 6,000 files, and the term def topological_sort( appears in 4,000. I’d guess a lot of these are from before 2020, though. Some skimming suggests that all of these custom topological sorts take different graph representations than graphlib, so they wouldn’t be convertable regardless. Graph representation matters.

There are two other languages I found with graph types: Erlang and SWI-Prolog. I don’t know either language and cannot tell when they were added; with Erlang, at least, it was before 2008. I reached out to a person on the Erlang core language committee but did not hear back.

Graph languages

Programming languages where “everything is a graph” in the same way that everything in bash a string and everything in lisp is a list. Some examples include GP2 and Grape. Based on some correspondence with people in the field, right now this is still highly academic.

Mathematics Software Languages

Mathematica, MATLAB, Maple, etc all have graph libraries of some form or another. I am not paying the thousands of dollars in licensing needed to learn more.

Update 2024-03-18

I’ve collected some of the comments I received on this post here.

No, really. Hash tables are bipartite graphs. This was used to prove performance of cuckoo hashing operations. ^[return]
I derived both computations with cloc 1.96. I ran cloc in networkx/networkx/algorithms (56989) and in cpython/Lib (588167). The whole networkX library is ~90,000 lines of code. ^[return]
You can make this more efficient by keeping the edge list sorted and doing an O(log(|e|)) binary search, at the cost of making edge insertions more expensive. ^[return]
NetworkX has functions to convert graphs into other representations but not for working with those representations directly. ^[return]
14 of the 21 canonical NP-complete problems are graph problems. ^[return]
Not to be confused with the GQL language, a proposed GQL standard that’s still under development. ^[return]

Planner programming blows my mind

Tue, 20 Feb 2024 00:00:00 +0000

Picat is a research language intended to combine logic programming, imperative programming, and constraint solving. I originally learned it to help with vacation scheduling but soon discovered its planner module, which is one of the most fascinating programming models I’ve ever seen.

First, a brief explanation of logic programming (LP). In imperative and functional programming, we take inputs and write algorithms that produce outputs. In LP and constraint solving, we instead provide a set of equations and find assignments that satisfy those relationships. For example:

main =>
  Arr = [a, b, c, a],
  X = a,
  member(X, Arr),
  member(Y, Arr),
  X != Y,
  println([X, Y]).

Non-function identifiers that start with lowercase letters are “atoms”, or unique tokens. Identifiers that start with uppercase letters are variables. So [a, b, c, a] is a list of four atoms, while Arr and X are variables. So Member(X, Arr) returns true as you’d expect.

The interesting thing is Member(Y, Arr). Y wasn’t defined yet! So Picat finds a value for Y that makes the equation true. Y could be any of a, b, or c. Then the line after that makes it impossible for Y to be a, so this prints either ab or ac. Picat can even handle expressions like member(a, Z), instantiating Z as a list!

Planning pushes this all one step further: instead of finding variable assignments that satisfy equations, we find variable mutations that reach a certain end state. And this opens up some really cool possibilities.

To showcase this, we’ll use Picat to solve a pathing problem.

The problem

We place a marker on the grid, starting at the origin (0, 0), and pick another coordinate as the goal. At each step we can move one step in any cardinal direction, but cannot go off the boundaries of the grid. The program is successful when the marker is at the goal coordinate. As a small example:

+---+
|   |
| G |
|O  |
+---+

One solution would be to move to (1, 0) and then to (1, 1).

To solve this with planning, we need to provide three things:

A starting state Start, which contains both the origin and goal coordinates.
A set of action functions that represent state transitions. In Picat these functions must all be named action and take four parameters: a current state, a next state, an action name, and a cost. We’ll see that all below.
A function named final(S) that determines if S is a final state.

Once we define all of these, we can call the builtin best_plan(Start, Plan) which will assign Plan to the shortest sequence of steps needed to reach a final state.¹

Our first implementation

import planner.
import util.

main =>
   Origin = {0, 0}
   , Goal = {2, 2}
   , Start = {Origin, Goal}
   , best_plan(Start, Plan)
   , println(Plan)
   .

Explanation

main is the default entry point into a Picat program. Here we’re just setting up the initial state, calling best_plan, and printing Plan. {a, b} is the syntax for a Picat array, which is basically a tuple.

Every expression in a Picat body must be followed by a comma except the last clause, which must be followed with a period. This makes moving lines around really annoying. Writing it in that “bullet point” style helps a little.

Since final takes just one argument, we’ll need to store both the current position and the goal into said argument. Picat has great pattern matching so we can just write it like this:

final({Pos, Goal}) => Pos = Goal.

Explanation

Without the pattern matching, we’d have to write it like this:

final(S) =>
  S = {Pos, Goal}
  , Pos = Goal
  .

If we write a second final predicate, the plan succeeds if either final returns true.

Finally, we need to define the actions which the planner can take. We only need one action here.

action(From, To, Action, Cost) ?=>
  From = {{Fx, Fy}, Goal}
  , Dir = [{-1, 0}, {1, 0}, {0, -1}, {0, 1}] 
  , member({Dx, Dy}, Dir)                    % (a)
  , Tx = Fx + Dx
  , Ty = Fy + Dy
  , member(Tx, 0..10)                        % (b)
  , member(Ty, 0..10)                        % (b)
  , To = {{Tx, Ty}, Goal}
  , Action = {move, To[1]}
  , Cost = 1
  .

Explanation

From is the initial state, To is the next state, Action is the name of the action— in this case, move.¹ You can store metadata in the action, which we use to store the new coordinates.

Writing action ?=> instead of action => makes action backtrackable, which I’ll admit I don’t fully understand? I’m pretty sure it means that if this definition of action pattern-matches but doesn’t lead to a viable plan, then Picat can try other definitions of action. This’ll matter more for later versions.

As with the introductory example up top, we’re using member to both find values (on line (a)) and test values (on lines (b)). Picat also has a non-assigning predicate, membchk, which just does testing. If I wasn’t trying to showcase Picat I could instead have use membchk for the testing part, which cannot assign.

Cost is the “cost” of the action. best_plan tries to minimize the total cost. Leaving it at 1 means the cost of a plan is the total number of steps.

After writing this I realize I should have instead used a structure for the action, writing it instead as $move(T[1]). I didn’t want to rewrite all of my code though so I’m leaving it like this. ^[return]

And that’s it, we’re done with the program. Here’s the output:

> picat planner1.pi

[{move,{1,0}},{move,{2,0}},{move,{2,1}},{move,{2,2}}]

That’s a little tough to read, so I had Picat output structured data that I could process into a picture.

main =>
  Origin = {0, 0}
  , Goal = {2, 2}
  , Start = {Origin, Goal}
  , best_plan(Start, Path)
- , println(Plan)
+ , printf("Origin: %w\n", Origin)
+ , printf("Goal: %w\n", Goal)
+ , printf("Bounds: {10, 10}\n")
+ , printf("Path: ")
+ , println(join([to_string(A[2]): A in Plan], ", "))
  .

I used a Raku script to visualize it.² Here’s what we now get:

> raku format_path.raku -bf planner1.pi

+-----+
|     |
|     |
|  G  |
|  •  |
|O••  |
+-----+

To show that the planner can route around an “obstacle”, I’ll add a rule that the state cannot be a certain value:

  , Tx = Fx + Dx
  , Ty = Fy + Dy
+ , {Tx, Ty} != {2, 1}

+-----+
|     |
|     |
| •G  |
| •   |
|O•   |
+-----+

Let’s comment that out for now, leaving this as our current version of the code:

Code

import planner.
import util.

main =>
  Origin = {0, 0}
  , Goal = {2, 2}
  , Start = {Origin, Goal}
  , best_plan(Start, Plan)
  % , println(Plan) 
  , printf("Origin: %w\n", Origin)
  , printf("Goal: %w\n", Goal)
  , printf("Bounds: {10, 10}\n")
  , printf("Path: ")
  , println(join([to_string(A[2]):  A in Plan], ", "))
  .

final(S) =>
  S = {Pos, Goal},
  Pos = Goal.

action(From, To, Action, Cost) ?=>
  From = {{Fx, Fy}, Goal}
  , Dir = [{-1, 0}, {1, 0}, {0, -1}, {0, 1}]
  , member({Dx, Dy}, Dir)
  , Tx = Fx + Dx
  , Ty = Fy + Dy
  % , {Tx, Ty} != {2, 1}
  , member(Tx, 0..10)
  , member(Ty, 0..10)
  , To = {{Tx, Ty}, Goal}
  , Action = {move, To[1]}
  , Cost = 1
  .

Adding multiple goals

Next I’ll add multiple goals. In order to succeed, the planner needs to reach every single goal in order. We start with one change to main:

main =>
  Origin = {0, 0}
- , Goal = {2, 2}
+ , Goal = [{2, 2}, {3, 4}]

Goal now represents a “queue” of goals to reach, in order. Then we add a new action which removes a goal from our queue once we’ve reached it.

action(From, To, Action, Cost) ?=>
  From = {Pos, Goal}
  , Goal = [Pos|Rest]
  , To = {Pos, Rest}
  , Action = {mark, From[1]}
  , Cost = 1
  .

Explanation

[Head|Tail] splits a list into the first element and the rest. Since Pos was defined in the line before, Goal = [Pos|Rest] is only true if the first goal on the list is equal to Pos. Then we drop that goal from our new state by declaring the new goal state to just be Rest.

(This is where backtracking with ?=> becomes important: if we didn’t make the actions backtrackable, Picat would match on the move first and never mark.)

Since we’re now destructively removing goals from our list when we reach them, final needs to be adjusted:

final({Pos, Goal}) =>
- Pos = Goal.
+ Goal = [].

And that’s it. We didn’t even have to update our first action!

+-----+
|   G |
|   • |
|  G• |
|  •  |
|O••  |
+-----+

Code

import planner.
import util.

main =>
  Origin = {0, 0}
  , Goal = [{2, 2}, {3, 4}]
  %, Goal = [{9, 2}, {0, 4}, {9, 6}, {0, 9}]
  , Start = {Origin, Goal}
  , best_plan(Start, Plan)
  , printf("Origin: %w\n", Origin)
  , printf("Goal: %w\n", Goal)
  , printf("Bounds: {10, 10}\n")
  , printf("Path: ")
  , println(join([to_string(A[2]):  A in Plan], ", "))
  .

final({Pos, Goal}) =>
  Goal = [].

action(From, To, Action, Cost) ?=>
  From = {{Fx, Fy}, Goal}
  , Dir = [{-1, 0}, {1, 0}, {0, -1}, {0, 1}]
  , member({Dx, Dy}, Dir)
  , Tx = Fx + Dx
  , Ty = Fy + Dy
  , member(Tx, 0..10)
  , member(Ty, 0..10)
  , To = {{Tx, Ty}, Goal}
  , Action = {move, To[1]}
  , Cost = 1
  .

action(From, To, Action, Cost) ?=>
  From = {Pos, Goal}
  , Goal = [Pos|Rest]
  , To = {Pos, Rest}
  , Action = {mark, From[1]}
  , Cost = 1
  .

Cost minimization

Going through the goals in order doesn’t always lead to the shortest total path.

main =>
  Origin = {0, 0}
- , Goal = [{2, 2}, {3, 4}]
+ , Goal = [{9, 2}, {0, 4}, {9, 6}, {0, 9}]

+----------+
|G         |
|•         |
|•         |
|•••••••••G|
|         •|
|G•••••••••|
|•         |
|•••••••••G|
|         •|
|O•••••••••|
+----------+

What if we didn’t care about the order of the goals and just wanted to find the shortest path? Then we only need to change two lines:

action(From, T, Action, Cost) ?=>
  From = {Pos, Goal}
- , Goal = [Pos|Rest]
- , T = {Pos, Rest}
+ , member(Pos, Goal)
+ , T = {Pos, delete(Goal, Pos)}
  , Action = {mark, From[1]}
  , Cost = 1
  .

Now the planner can delete any goal it’s passing over regardless of where it is in the Goal list. So Picat can “choose” which goal it moves to next so as to minimize the overall path length.

+----------+
|G•••••••••|
|•        •|
|•        •|
|•        G|
|•        •|
|G        •|
|•        •|
|•        G|
|•         |
|O         |
+----------+

Final code:

Code

import planner.
import util.

main =>
  Origin = {0, 0}
  , Goal = [{9, 2}, {0, 4}, {9, 6}, {0, 9}]
  , Start = {Origin, Goal}
  , best_plan(Start, Plan)
  , printf("Origin: %w\n", Origin)
  , printf("Goal: %w\n", Goal)
  , printf("Bounds: {10, 10}\n")
  , printf("Path: ")
  , println(join([to_string(A[2]):  A in Plan], ", "))
  .

final({Pos, Goal}) =>
  Goal = [].

action(From, To, Action, Cost) ?=>
  From = {{Fx, Fy}, Goal}
  , Dir = [{-1, 0}, {1, 0}, {0, -1}, {0, 1}]
  , member({Dx, Dy}, Dir)
  , Tx = Fx + Dx
  , Ty = Fy + Dy
  , member(Tx, 0..10)
  , member(Ty, 0..10)
  , To = {{Tx, Ty}, Goal}
  , Action = {move, To[1]}
  , Cost = 1
  .

action(From, To, Action, Cost) ?=>
  From = {Pos, Goal}
  , member(Pos, Goal)
  , To = {Pos, delete(Goal, Pos)}
  , Action = {mark, From[1]}
  , Cost = 1 
  .

Other variations

Picat supports a lot more variations on planning:

best_plan(S, Limit, Plan) caps the maximum cost at Limit— good for failing early.
For each best_plan, there’s a best_plan_nondet that finds every possible best plan.
sequence(P, Action) restricts the possible actions based on the current partial plan, so we can add restrictions like “you have to move twice before you turn”.

The coolest thing to me is that the planning integrates with all the other Picat features. I whipped up a quick demo that combines planning and constraint solving. The partition problem is an NP-complete problem where you partition a list of numbers into two equal sums. This program takes a list of numbers and finds the sublist with the largest possible equal partitioning:³

Code

% got help from http://www.hakank.org/picat/set_partition.pi 
import planner.
import util.
import cp.

main =>
Numbers = [32, 122, 77, 86, 59, 47, 154, 141, 172, 49, 5, 62, 99, 109, 17, 30, 977]
  , if final(Numbers) then
      println("Input already has a partition!")
      , explain_solution(Numbers)
    else
        best_plan(Numbers, Plan)
      , printf("Removed: %w%n",[R: $remove(R, _) in Plan])
      , $remove(Last, FinalState) = Plan[Plan.length]
      , printf("Final: %w%n", FinalState)
      , explain_solution(FinalState)
    end
  .

final(Numbers) => get_solutions(Numbers) != [].

get_solutions(Numbers) = S =>
  X = new_list(Numbers.length)
  , X :: 0..1
  , X[1] #= 0 % symmetry breaking
  , sum(Numbers) #= 2*sum([Numbers[I]*X[I]: I in 1..Numbers.length])
  , S = solve_all([$limit(1)], X) 
  .
  
action(From, To, Action, Cost) =>
   member(Element, From)
   , To = delete(From, Element)
   , Action = $remove(Element, To)
   , Cost = Element
   .

explain_solution(Numbers) =>
  [Sol] = get_solutions(Numbers)
  , Left =  [Numbers[I]: I in 1..Numbers.length, Sol[I] = 0]
  , Right = [Numbers[I]: I in 1..Numbers.length, Sol[I] = 1]
  , printf("%s=%d%n", join([to_string(N): N in Left], "+"), sum(Left))
  , printf("%s=%d%n", join([to_string(N): N in Right], "+"), sum(Right))
  .

Removed: [5,17]
Final: [32,122,77,86,59,47,154,141,172,49,62,99,109,30,977]
32+99+977=1108
122+77+86+59+47+154+141+172+49+62+109+30=1108

This is all so mindblowing to me. It’s almost like a metaconstraint solver, allowing me to express constraints on the valid constraints.

Should I use Picat?

Depends?

I would not recommend using Picat in production. It’s a research language and doesn’t have a lot of affordances, like good documentation or clear error messages. Here’s what you get when there’s no plan that solves the problem:

*** error(failed,main/0)

But hey it runs on Windows, which is better than 99% of research languages.

Picat seems more useful as a “toolkit” language, one you learn to solve a specific class of computational problems, and where you’re not expecting to maintain or share the code afterwards. But it’s really good in that niche! There’s a handful of problems I struggled to do with regular programming languages and constraint solvers. Picat solves a lot of them quite elegantly.

Appendix: Other planning languages

While originally pioneered for robotics and AI, “planning” is most-often used for video game AIs, where it’s called “Goal Oriented Action Planning” (GOAP). Usually it’s built as libraries on top of other languages, or implemented as a custom search strategy. You can read more about GOAP here.

There is also PDDL, a planning description language that independent planners take as input, in the same way that DIMACS is a description format for SAT.

Thanks to Predrag Gruevski for feedback. I first shared my thoughts on Picat on my newsletter. I write new newsletter posts weekly.

If you want just any plan, regardless of how long it is, you can call plan(Start, Plan) instead. ^[return]
The commented version of the formatter was originally up on my Patreon. You can now see it here. ^[return]
“Find the most elements you can remove without getting you a valid partition” is a little more convoluted, but you can see it here. ^[return]

An RNG that runs in your brain

Mon, 22 Jan 2024 00:00:00 +0000

Humans are notoriously bad at coming up with random numbers. I wanted to be able to quickly generate “random enough” numbers. I’m not looking for anything that great, I just want to be able to come up with the random digits in half a minute. Some looking around brought me to an old usenet post by George Marsaglia:

Choose a 2-digit number, say 23, your “seed”.

Form a new 2-digit number: the 10’s digit plus 6 times the units digit.

The example sequence is 23 –> 20 –> 02 –> 12 –> 13 –> 19 –> 55 –> 35 –> …

and its period is the order of the multiplier, 6, in the group of residues relatively prime to the modulus, 10. (59 in this case).

The “random digits” are the units digits of the 2-digit numbers, ie, 3,0,2,2,3,9,5,… the sequence mod 10.

Marsaglia is most famous for the diehard suite of RNG tests, so he knows his stuff.¹ I’m curious why this works and why he chose 6.

We’re going to use Raku, the language for gremlins.² I’ll be explaining all the weird features I use in dropdowns in case you’re a bit of a gremlin, too.

Intro

The sequence is periodic, meaning that if we iteratively apply it we’ll eventually get the same element. Let’s start with a function (“subroutine”) that produces the whole sequence:

my sub next-rng(Int $start, Int $unitmult = 6, --> List) {
    my @out;
    my $next = $start;
    repeat while $next !(elem) @out {
        @out.append($next);
        $next = sum($next.polymod(10) Z* $unitmult,1);
    };
    return @out;
}

Explanation

Raku is an extremely weird language but I’ll keep it as straightforward as I can.

@ and $ are sigils for “positional” (listlike) and “scalar” respectively. Defining a positional without assigning anything defaults it to the empty array.
(elem) checks for membership, and ! can be applied to negate any infix operator. (elem) is the ASCII version— Raku also accepts ∈.
polymod splits a number into a remainder and dividend, ie 346.polymod(10) = (6 34). It takes multiple parameters, so you can do things like num.polymod(60, 60, 24) to get hours-minutes-seconds.
Z is the “zip” metaoperator, applying a infix op elementwise between two lists. (4, 6) Z* (6, 1) = (4*6, 6*1).

Once we have a sequence we can print it with the put or say commands, which have subtly different behaviors I’m not going to get into.

put next-rng(1);

01 06 36 39 57 47 46 40 04 24 26 38 51 11 07 42 16 
37 45 34 27 44 28 50 05 30 03 18 49 58 53 23 20 02 
12 13 19 55 35 33 21 08 48 52 17 43 22 14 25 32 15 
31 09 54 29 56 41 10 01

Remember, the random numbers are the last digit. So the RNG goes 1 -> 6 -> 6 -> 9 -> 7 -> 7 -> …

Investigating Properties

If the RNG is uniform then each digit should appear in the sequence the same number of times. We can check this by casting the last digits to a multiset, or “bag”.

say bag(next-rng(1) <<%>> 10);

Explanation

<<op>> is the hyper metaoperator and “maps” the inside operator across both lists, recursively going into list of lists too. IE ((1, 2), 3) <<+>> 10 is ((11, 12), 13)! Hyperoperators have a lot of other weird properties that make them both useful and confusing.
Bags count the number of elements in something. bag((4, 5, 4)){4} = 2. Confusingly though they can only contain scalars, not arrays or lists or the like.

Bag(0(5) 1(6) 2(6) 3(6) 4(6) 5(6) 6(6) 7(6) 8(6) 9(5))

That seems to be a uniform-enough distribution, though I’m a bit less likely to get a 0 or a 9.

My next idea comes from the diehard tests. From the wiki:

Overlapping permutations: Analyze sequences of five consecutive random numbers. The 120 possible orderings should occur with statistically equal probability.

There are only 54 5-number sequences in the dataset, so I’ll instead apply this to 2-number “transitions”. I’ll do this by outputting a 10-by-10 grid where the (i, j)th index (from 0) is the number of transitions from last-digit i to j. For example, the sequence includes the transition 28 -> 50, and no other transitions of form X8 -> Y0, so cell (8, 0) should be a 1.

sub successions-grid(@orbit) {
  my @pairs = (|@orbit , @orbit[0]).map(* % 10).rotor(2 => -1);
  for ^10 -> $x {put ($x X ^10).map({@pairs.grep($_).elems})}
}

Explanation

| @f, $x concats $x directly onto @f. Without the | it’d be a two-element list instead.
The * in `* % 10` is a whatever, a weird little operator that does a lot of things in a lot of different contexts, but usually in this case lifts the expression into a closure. Usually. It’s the same as writing map({$_ % 10}).¹
rotor(2 => -1) gets two elements, then goes one element back, then gets two more, etc. [1, 2, 3, 4].rotor(2 => -1) is [(1, 2), (2, 3), (3, 4)]. You could also do rotor(2) to get [(1, 2), (3, 4)], or rotor(1 => 1) to get [1, 3]. Rotor is really cool.
^10 is just 0..9. For once something easy!
X is the cross product metaoperator. So if $x = 2, then $x X ^4 would be ((2 0), (2 1), (2 2), (2 3)). And yes, the operator can get much, much stranger.
grep(foo) returns a list of all elements smart-matching foo, and .elems is the number of elements in a list. So @pairs.grep($_).elems is the number of elements of the list matching $_. This took me way too long to figure out

Actually -> $x {$x % 10} but close enough ^[return]

> successions-grid(next-rng(1, 6))

0 1 1 1 1 1 0 0 0 0
1 1 0 0 0 0 1 1 1 1
0 0 1 1 1 1 1 1 0 0
1 1 1 1 0 0 0 0 1 1
0 0 0 0 1 1 1 1 1 1
1 1 1 1 1 1 0 0 0 0
1 1 0 0 0 0 1 1 1 1
0 0 1 1 1 1 1 1 0 0
1 1 1 1 0 0 0 0 1 1
0 0 0 0 1 1 1 1 1 0

We can see from this table that some transitions are impossible. If I generate a 0, I can’t get a 6 right after. Obviously not a great RNG, but my expectations were pretty low anyway.

Why 6?

What if instead of multiplying the last digit by 6, I multiply by 4?

> say next-rng(1, 4);
01 04 16 25 22 10

I dunno, I kinda like an RNG that never gives me 3. The distinct sequences are called orbits and their lengths are called periods. Let’s see all the possible orbits we can get by using 4 as the multiplier:

sub orbits-for-mod(int $mult, $top = 20) {
  my &f = &next-rng.assuming(*, $mult);
  (1..$top).map(&f).unique(as => &set)
}

Explanation

& is the sigil for “callable” or ~~function~~ subroutine. The .assuming method does a partial function application, and passing a * makes it partially apply the second parameter.¹
The map returns a sequence of lists, which we pass to unique. as => &set converts every sequence in the map to a set and compares those for uniqueness, instead of the original lists. But the final result uses the elements prior to conversion.

If that’s confusing, a simpler example is that [-1, 1].unique(as => &abs) returns [-1], while [1, -1].unique(as => &abs) is [1].

Daniel Sockwell (aka codesections) kindly agreed to read a first draft of this post, and he told me about assume. Thanks Daniel! ^[return]

> say orbits-for-mod(4, 38).map(*.gist).join("\n");
[1 4 16 25 22 10]
[2 8 32 11 5 20]
[3 12 9 36 27 30]
[6 24 18 33 15 21]
[7 28 34 19 37 31]
[13]
[14 17 29 38 35 23]
[26]

Explanation

Quoting Daniel Sockwell:

The .map(*.gist).join("\n") is just there to prettify the output. cycles-for-mod returns a Seq of Arrays; mapping over each Array with .gist converts it into a string surrounded by square brackets and .join("\n") puts a newline between each of these strings.

If you picked 13 as your starting value, your random digits would be 3, 3, 3, 3, 3, 3.

Preempting 50,000 emails(source)

For obvious reasons, 4 should never be our multiplier. In fact for a multiplier to give a “good” RNG, it needs to have exactly one orbit. As we’ll see later, this guarantees a(n almost) uniform distribution.

> say (1..30).grep(*.&orbits-for-mod == 1)
(3 6 11 15 18 23 27)

Explanation

.& applies a top-level routine as a method. grep(*.&f) is the equivalent of grep({f($_)}).
&orbits-for-mod returns a list. == coerces both inputs to numbers, and coercing a list to a number returns the number of elements. So we’re testing if the returned list has one element, ie there’s exactly one orbit. (If you don’t want to compare without coercion, use either === or eqv.)

This way of doing things is pretty slow and also only looks for orbits that start with a number up to 20. So it would miss the 26 -> 26 orbit for x=4. We’ll fix both of these issues later.

So some “good” choices for n are 6, 11, and 18.

Note that if you end up with a three digit number, you treat the first two digits as a single number. For n=11, 162 leads to 16 + 22, not 6 + 22 (or 6 + 1 + 22).

Why does this work?

Here’s a part of the explanation that really confused me:

and its period is the order of the multiplier, 6, in the group of residues relatively prime to the modulus, 10. (59 in this case).

After talking with some friends and a lot of reading Wiki articles, it started making more sense. I’m mentally computing a “multiply with carry” RNG with constants a=x and c=10. This choice has a cool property: if MWC(x) = y, then 10y mod (10mult-1) = x!

MWC:    01 -> 06 -> 36 -> ... -> 41 -> 10 -> 01
10y%59: 01 -> 10 -> 41 -> ... -> 36 -> 06 -> 01

That’s pretty neat! It’s easier for me to mathematically reason about 10y mod 59 than “multiply the last digit by six and add the first digit”. For example, it’s clear why the RNG generates 0 and 9 slightly less often than the other digits: no matter which multiplier we pick, the generated sequence will go from 1 to 10n-2, “leaving out” 10n-1 (which ends with 9) and 10n (ends with 0).

“Multiply and modulo” is also known as the Lehmer RNG.

Finding better RNGs

So what other numbers work? We already know that a good multiplier will produce only one orbit, and I showed some code above for calculating that. Unfortunately, it’s an O(n²) worst-case algorithm.³ Thinking about the MWC algorithm as “Lehmer in reverse” gives us a better method: if n is a good multiplier, then the period of the orbit starting from 1 should be 10n-2.

The Lehmer approach also gives us a faster way of computing the orbit:

sub oneorbit(\x) {
  10, * *10% (10*x - 1) … 1
}

Explanation

(source)

Real Explanation

Writing \x instead of $x as a param lets use use x instead of $x in the body.
... is the sequence operator. It can do a lot of different things, but the important one for us is that if you write 10, &f … 1, it will start with 10 and then keep applying &f until it eventually generates 1.
In * *10%[etc], the first * is a Whatever and the second * is regular multiply. This then lifts into the function -> $a {$a * 10 % (10*x - 1)}.

This actually produces the orbit in reverse but we’re only interested in the period so nbd.

Then we check the period using the same “== coerces lists to lengths” trick as before.

> say (1..100).grep({oneorbit($_) == 10*$_-2});

(2 3 6 11 15 18 23 27 38 39 42 50 51 62 66 71)

I can see why Marsaglia chose 6: most programmers know their 6 times-table and it never returns a 3-digit number, so the addition step is real easy. The orbit has only 58 numbers and you won’t get some digit sequences, but if you need to pull out a few random digits quickly it’s totally fine.

If you want more randomness, I see a couple of candidates. 50 has a period of 498 and is incredibly easy to compute. If the final digit is even then you don’t need to do any carries: 238 -> 423!

That said, the 50-sequence doesn’t seem as random as other sequences. There’s a point where it generates 9 even numbers followed by 8 odd ones. Don’t use it to simulate coin flips.

The last interesting number is 18. It has a respectable period of 178 and has every possible digit transition:

> successions-grid(next-rng(1, 18))

1 2 2 2 2 2 2 2 1 1
2 2 2 2 2 2 1 1 2 2
2 2 2 2 1 1 2 2 2 2
2 2 1 1 2 2 2 2 2 2
1 1 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 1 1
2 2 2 2 2 2 1 1 2 2
2 2 2 2 1 1 2 2 2 2
2 2 1 1 2 2 2 2 2 2
1 1 2 2 2 2 2 2 2 1

The downside is that you have to learn the 18 times-table. This isn’t too bad: I internalized it with maybe 10 minutes of practice. I’m still not great at doing the whole MWC step but I can consistently produce another random digit every five seconds or so. That’s good enough for me.

You can see the Raku code I used to research this here. It’s set up as a CLI so you can use it in a few different ways; see the file for more info.

Thanks to Codesections for feedback and Quinn Wilton and Jeremy Kun for helping me understand the math.

Assume that every time I say “RNG” I mean “PRNG”. ^[return]
Also full disclosure the code I’m showing is less gremliny than the code I originally wrote. So just know it can be more gremlins than this. ^[return]
if multiplier n has a single orbit, then we’ll run next-rng on ~10n-2 numbers, and the function will iterate 10n-2 times (since it has to go through every number in the orbit). If I bothered to skip numbers I’d already seen in an orbit then the runtime would collapse to O(n). ^[return]

The World and the Machine

Thu, 04 Jan 2024 00:00:00 +0000

This is just a way of thinking about formal specification that I find really useful. The terms originally come from Michael Jackson’s Software Requirements and Specifications.

In specification, the machine is the part of the system you have direct control over and the world is all the parts that you don’t. Take a simple transfer spec:

---- MODULE transfer ----
EXTENDS TLC, Integers
CONSTANTS People, Money, NumTransfers

(* --algorithm transfer
variables
  acct \in [People -> Money];

define
  \* Invariant
  NoOverdrafts ==
    \A p \in People:
      acct[p] >= 0
end define;

process cheque \in 1..NumTransfers
variable
  amnt \in 1..5;
  from \in People;
  to \in People
begin
  Check:
    if acct[from] >= amnt then
      Withdraw:
        acct[from] := acct[from] - amnt;
      Deposit:
        acct[to] := acct[to] + amnt;
    end if;
end process;
end algorithm; *)
====

The code that handles the transfer (represented by the cheque process) is the machine. It currently has a race condition that can break the invariant NoOverdrafts, where someone with $5 can submit two checks for $4 each and get them both past the guard clause.

One way you could solve this is by adding a lock so that the banking service only resolves one cheque at a time. One way you can’t solve it is by forcing people to deposit one cheque at a time. You don’t have any control over what people do with their chequebook! The people and their behavior is part of the world.

Whether something belongs to the world or the machine depends on your scope in the system. If you maintain one service and the other teams aren’t returning your calls, then their components are part of the world. If they’re sending you bad data, you need to faithfully model receiving that bad data in your spec as part of designing your machine.

Some notes on this model

While you need to model the whole system, you’re only designing the machine. So the implementation details of the machine matter, but you don’t need to implement the world. It can be abstracted away, except for how it affects the machine. In the above example, we don’t need to model a person deciding to write a cheque or the person depositing it, just the transfer entering the system.

Observability and observable properties

Like in OOP, some system state is restricted to the world or machine.

The world can both read and write the from and to variables, but the machine can only read them. In most specifications these restrictions are implicit; you could write the spec so that the machine changes from, but your boss wouldn’t let you build it.
The “program counter”, or line each process is currently executing, isn’t readable or writable by the world. It’s an implementation detail of the machine.
acct can be written by the machine and read by the world. I call these observable.

We can divide properties into internal properties that concern just the machine and external (observable) properties that can be seen by the outside world. NoOverdrafts is observable: if it’s violated, someone will be able to see that they have a negative bank balance. By contrast, “at most one process can be in the withdraw step” (OnlyOneWithdraw) is internal. The world doesn’t have access to your transaction logs, nobody can tell whether OnlyOneWithdraw is satisfied or not.

Internal properties are useful but they’re less important than observable properties. An OnlyOneWithdraw violation might be the root cause of a NoOverdrafts violation, but NoOverdrafts is what actually matters to people. If a property isn’t observable, it doesn’t have any connection to the broader world, so nobody is actually affected by it breaking.

Misc

If both the world and machine can write to a variable, generally the world should be able to do more with the variable than the machine can. IE the machine can only modify acct by processing transfers, while the world can also do deposits and withdrawals. It’s exceedingly hard to enforce MISU on state the world can modify.
If the world can break an invariant, it’s not an invariant. Instead you want “resilience”, the ability to restore some system property after it’s been broken. See here for more on modeling resilience.
It’s not uncommon for a spec to break not because the machine has a bug, but because the surrounding world has changed.

Thanks to Andrew Helwer and Lars Hupel for feedback. If you liked this, come join my newsletter! I write new essays there every week.

Notes on Every Strangeloop 2023 Talk I Attended

Tue, 05 Dec 2023 00:00:00 +0000

This is my writeup of all the talks I saw at Strangeloop, written on the train ride back, while the talks were still fresh in my mind. Now that all the talks are online I can share it. This should have gone up like a month ago but I was busy and then sick. Enjoy!

How to build a meaningful career

Topic: How to define what “success” means to you in your career and then be successful. Mostly focused on psychological maxims, like “put in the work” and “embrace the unknown”.

I feel like I wasn’t the appropriate audience for this; it seemed intended for people early in their career. I like that they said it’s okay to be in it for the money. Between the “hurr dur you must be in it for the passion” people and the “hurr durr smash capitalism” people, it’s nice to hear some acknowledgement that money makes your life nice.

Playing with Engineering

Topic: the value of “play” (as in “play make believe”, or “play with blocks”) in engineering. Some examples of how play leads to innovation, collaboration, and cool new things.

Most of the talk is about the unexpected directions her “play” went in, like how her work in education eventually lead to a series of collaborations with OK Go. I think it was more inspirational than informative, to try to get people to “play” rather than to provide deep insight into the nature of the world. Still pretty fun.

Is my Large Language Model a Strange Loop?

(Disclosure, I didn’t actually see this talk live, I watched Zac Hatfield-Dodds rehearse and gave feedback. Also Zac is a friend and we’ve collaborated before on Hypothesis stuff.)

Topic: Some of the unexpected things we observe in working LLMs, and some of the unexpected ways they’re able to self-reference themselves.

Zac was asked to give the talk at the last minute due to a sickness cancellation by another speaker. Given the time crunch, I think he pulled it together pretty well. Even so it was a bit too technical for me; I don’t know if he was able to simplify it in time for the final presentation.

Like most practical talks on AI, intentionally or not he slipped in a few tricks to eke more performance out of an LLM. Like if you ask them to answer a question, and then rate the confidence of the question they asked, they tend to be decently accurate at their confidence.

Zac’s also a manager at Anthropic, which gave the whole talk some neat “forbidden knowledge” vibes.

Concatenative Languages Talk

(Disclaimer: Douglas is a friend, too.)

Topic: Why stack-based languages are an interesting computational model, how they can be Turing-complete, and some of the unusual features you get from stack programming.

The first time I’ve seen a stack-based language talk that wasn’t about Forth. Instead, it used his own homegrown stack language so he could just focus on the computer science aspects. The two properties that stick out to me are:

Stack programs don’t need to start from an empty stack, which means entire programs will naturally compose. Like you can theoretically pipe the output of a stack program into another stack program, since they’re all effectively functions of type Stack -> Stack.
Stack ops are associative: if you chop a stack program into subprograms and pipe them into each other, it doesn’t matter where you make the cuts, you still get the same final stack. That’s really, really cool.

My only experience with stack machines is golfscript. Maybe I’ll try to pick up uiua or something.

Comedy Writing With Small Generative Models

Topic: “Small” generative AI models, like “taking all one-star amazon reviews for the statue of liberty and throwing them into a Markov chain”.

This was my favorite session of the conference. The technical aspects are pretty basic, and it’s explained simply enough that even layfolk should be able to follow. His approach generates dreamy nonsense that should be familiar to anyone who’s played with Markov chains before.

And then he pulls out a guitar.

The high point was his “Weird Algorithm”, which was like a karaoke machine which replaced the lyrics of songs with corpus selections that matched the same meter and syllables. Like replacing “oh I believe in yesterday” with “this is a really nice Hyundai”.

I don’t know how funny it’ll be in the video, it might be one of those “you had to be there” things.

An approach to computing and sustainability inspired from permaculture

Topic: The modern pace of tech leaves a lot of software, hardware, and people behind. How can we make software more sustainable, drawn from the author’s experiences living on a boat.

Lots of thoughts on this one. The talk was a crash course on all the different kinds of sustainability: making software run on old devices, getting software guaranteed to run on future devices, computing under significant power/space/internet constraints, and everything in between. I think it’s intended as a call to arms for us to think about doing better.

I’m sympathetic to the goals of permacomputing; what do I do with the five old phones in my closet? That’s a huge amount of computing power just gone to waste. The tension I always have is how this scales. Devine Lu Levinga is an artistic savant (they made Orca!) and the kind of person who can live on a 200-watt boat for seven years. I’m not willing to give up my creature comforts of central heating and Geforce gaming. Obviously there’s a huge spectrum between “uses less electricity than a good cyclist” and “buys the newest iPhone every year”, the question is what’s the right balance between sustainability and achievability.

There’s also the whole aesthetic/cultural aspect to permacomputing. Devine used images in dithered black/white. AFAICT this is because Hypercard was black/white, lots of retrocomputing fans love hypercard, and there’s a huge overlap between retrocomputing and permacomputing. But Kid Pix is just as old as Hypercard and does full color. It’s just not part of the culture.

Nit: at the end Devine discussed how they were making software preservation easier by writing a special VM. This was interesting but the discussion on how it worked ended up going way over time and I had to run to the next session.

Can a Programming Language Reason About Systems?

(Disclaimer: Jesus I’m friends with way too many people on the conference circuit now)

Topic: Formal methods is useful to reason about existing legacy systems, but has too high a barrier to entry. Marianne made a new FM language called “Fault” with a higher levels of abstraction. Some discussion of how it’s implemented.

This might just be the friendship talking, but Fault looks like one of the more interesting FM languages to come out recently. I’m painfully aware of just how hard grokking FM can be, and anything that makes it more accessible is good. I’ll have to check it out.

When she said that the hardest part is output formatting I felt it in my soul.

Making Hard Things Easy

Topic: Lots of “simple” things take years to learn, like Bash or DNS. How can we make it easier for people to learn these things? Four difficult technologies, and different approaches to making them tractable.

I consider myself a very good teacher. This talk made me a better one.

Best line was “behind every best practice is a gruesome story.” That’ll stick with me.

Drawing Comics at Work

Topic: Randal Munroe (the xkcd guy)’s closing keynote. No deep engineering lessons, just a lot of fun.

Postmortem

Before Julia Evans’ talk, Alex did A Long Strange Loop, how it went from an idea to the monolith it is today. Strange Loop was his vision, an eclectic mix of academia, industry, art, and activism. And it drew a diverse crowd because of that. I’ve made many friends at Strangeloop, people like Marianne and Felienne. I don’t know if I’ll run into them at any other conferences, because I don’t think other conferences will capture that lightning in a bottle. I’ll miss them.

I also owe my career to Strangeloop. Eight years ago they accepted Tackling concurrency bugs with TLA+, which got me started both speaking and writing formal methods professionally.

There’s been some talk about running a successor conference (someone came up with the name “estranged loop”) but I don’t know if it will ever be the same. There are lots of people can run a good conference, but there’s only one person who can run Alex’s conference. Whatever comes next will be fundamentally different. Still good, I’m sure, but different.

A better explanation of the Liskov Substitution Principle

Tue, 07 Nov 2023 00:00:00 +0000

Short version: If X inherits from Y, then X should pass all of Y’s black box tests.

I first encountered this idea at SPLASH 2021.

The longer explanation

A bit of background

In A Behavioral Notion of Subtyping Liskov originally defined subtyping in inherited objects as follows:

Subtype Requirement: Let P(x) be a property provable about objects x of type T. Then P(y) should be true for objects y of type S where S is a subtype of T.

Later, Robert Martin named the Liskov Substitution Principle:

Functions that use pointers or references to base classes must be able to use objects of derived classes without knowing it.

If you had fly_to(Bird, location), then you should be able to call fly_to on any subtype of Bird. This means that Penguin and Ostrich couldn’t be subtypes of Bird, since you cannot call fly_to on a flightless bird!

Liskov would later “approve” of the rule in her book Program Development in Java.

The Devil is in the Details

The LSP looks simple; the problem is applying it. How do you check that a function works on all subclasses of a class? Functions can do a lot of things! So we have a set of enforceable rules, where if you follow all the rules, you (presumably) satisfy LSP. The most well known of these rules is that inherited methods of a class must not strengthen preconditions or weaken postconditions.¹ That means it must accept a superset of the values the parent method accepts and output a subset of the parent’s possible outputs. Given

class Parent {
   run(x: int): out {
    require {x >= 4}
    # some code that computes out
    ensures {out % 2 == 0}
   }
}

class Child extends Parent {
# stuff
}

Child.run cannot be overloaded to take only even integers, or to also output odd numbers. Preconditions and postconditions together are called contracts.

This is just the first rule and already we have three problems. First, it’s not super clear why this follows from the LSP. Second, it’s hard to remember whether it’s weaken preconditions and strengthen postconditions or the other way around. Third, most languages don’t have dedicated pre/postcondition syntax. For these languages the rule is taught as a restriction on the method’s type signature: the parameters must be contravariant and the return values covariant. This makes things even more arbitrary and confusing and isn’t as accurate as the original version.

Long tangent on types and contracts

The above is historically inaccurate: the original Liskov paper lists both the contract rule and the type rule as separate requirements for subtyping. But IMHO the type rule is equivalent to the contract rule. Saying “parameters must be contravariant” is the same as saying “preconditions must not be strengthened.” Consider this case of contravariant parameters:

Parent.foo(x: Cat) {
    # code 
}

Child.foo(x: Animal) {
    # code
}

We can rewrite it purely as contracts:

Parent.foo(x) {
    requires typeof(x) == Cat;
    # code
}

Child.foo(x) {
    requires typeof(x) == Animal;
    # code
}

It’s less intuitive that we can rewrite contracts as types, but that’s also possible:

foo(x: Int): out {
    require {x >= 4}
    # code
    ensure {out % 2 == 0}
}

# becomes

foo(x: AtLeastFour): (out: EvenNumber) {
    # code
}

It’s true that most languages can’t statically typecheck this, but that’s a limitation on our current typecheckers, not an essential problem. Some languages, like Liquid Haskell, handle it just fine.

That said, even if the two rules are theoretically equivalent, in practice your programming language will favor checking some things as contracts and other things as types.

We can’t ensure LSP with function contracts alone. Consider!²

@dataclass
class WrappedCounter:
    val: int # assume that 0 <= val < 1000

    def add(self, x):
        assert x >= 0 # Precondition
        self.val = (self.val + x) % 1000

class Counter(WrappedCounter):
    def add(self, x):
        assert x >= 0 # Precondition
        self.val += x

Counter is obviously not a subtype of WrappedCounter, but it has the same method contracts! The difference is that WrappedCounter has an extra invariant, a property that holds true of an object at all times. Counter doesn’t have that property, so it can’t be a child class. That leads to rule two, subtypes must not weaken the class invariants.

The method rule and the invariant rule look somewhat related, but the next one comes out of left field.

@dataclass
class WrappedCounter:
    val: Nat
    limit: Nat

    def add(self, x):
        assert 0 <= x < self.limit # Precondition
        self.val = (self.val + x) % self.limit

class EvilCounter(WrappedCounter):
    def add(self, x):
        assert 0 <= x < self.limit # Precondition
        self.limit += x
        self.val = (self.val + x) % self.limit

Let’s look at our two rules:

Does it pass the method test? Yup, both have the exact same type signatures and preconditions.
Does it pass the invariant test? Yup, they both guarantee that self.val ∈ [0, limit).

But EvilCounter is clearly not a subtype of WrappedCounter. It doesn’t wrap!

We need a third rule, the history rule: the subtype cannot change in a way prohibited by the supertype. In this case, this means that limit must remain unchanged. WrappedCounter follows this rule and EvilCounter violates it, so it’s not a subtyping relation.³

The history rule always felt like a tack-on. It makes me worry that we’re just patching up holes as we find them, and that somebody will come up with a counterexample that follows all three rules but still isn’t a subtype. Like what happens when we add in exceptions or generics? We might have to add another arbitrary rule. It all just makes me lose trust in the applicability of LSP.

The Fix

In their college classes, Elisa Baniassad and Alexander J Summers found a different approach:

For a class to be substitutable for its supertype, it must pass all the supertype’s black box tests.⁴

To check substitutability, we come up with a test:

def test_add_five():
    c = WrappedCounter(10)
    c.add(5)
    assert c.val == 15

This passes, so we’d expect the equivalent test to pass for Counter:

def test_add_five():
    c = Counter(10)
    c.add(5)
    assert c.val == 15

This also passes. If every test we write passes, then Counter is a subtype of WrappedCounter.

So let’s write a second test:

def test_add_million():
    c = WrappedCounter(10)
    c.add(1_000_000) 
    assert c.val == 10

And this passes for WrappedCounter but fails for Counter. Therefore Counter is not a subtype of WrappedCounter.

Show limitation

Note that passing tests doesn’t guarantee you’ve got a subtype, since there might be a failing test you just haven’t considered writing. It’s similar to how, even if you have a rule violation, you might never actually run into a subtype violation in production. This is more a pedagogical technique to help people internalize LSP.

This explains a lot

I like how the testing approach to LSP explains all the rules. If you give me a case where an LSP rule violation causes a problem, I can give you back a test case which fails for the same reason. Let’s Pythonize my earlier example of a precondition rule violation:

class Parent:
    def run(self, x: Int):
        assert x >= 4
        pass
        # return something

class Child(Parent):
    def run(self, x):
        assert x % 2 == 0
        super()
        pass

Here’s my test:

def test_parent():
    c = Parent()
    c.run(5)

def test_child():
    c = Child()
    c.run(5)

We don’t need to do anything with the output of run, just call it. The first test (with c = Parent()) will do nothing while the second test (with c = Child()) will throw an error, failing the test. Child is not a subtype.

So that’s one way the testing approach makes things easier. It also shows why the history rule is necessary and also why it originally seemed like a crude hack to me.

def test_history_rule():
    c = WrappedCounter(0, 10)
    c.add(6)
    c.add(6)
    assert c.val == 2

We originally needed the history rule because contracts and invariants only span the lifetime of one method. But now we can just call a sequence of methods in one test!

Does this work?

By “work”, I mean “does it help students internalize the LSP.” I found the paper a little confusing here: it says they “observed that students were able to better understand the responsibilities of the subtype in a general sense”, but also that “the same proportion of students” understood the rules, which kinda sounds like ‘no impact’? I emailed the researchers for a clarification and here’s their response:

It was a big improvement. Students went from not demonstrating any intuition around the rule, to “getting it”. “Oh - subclass has to pass all superclass black box tests! I get it!”. And then you can delve into ways tests can break (narrowing preconditions, etc). But at the root of it all - it’s about the tests. They did way better on LSP questions, and I believe there was one question in particular that we asked before and after, and it was way better done (like from a failing grade to a good passing grade on average) with the testing approach.

They also told me about an indirect benefit: it also helped students understand the difference between black- and white-box tests! Once you get a sense for LSP in the original

It helped clearly delineate whether tests were about “the packaging” (like what’s written on the box) or the “how” you’ve done it (which is specific to the implementation, and can change if you make different choices). You have to test both, and make sure both the packaging is okay, and check that the underlying implementation is not having unwanted side effects, or introducing errors — and it then became clear to students why we needed both kinds of tests, and why differentiating was useful.

Pretty cool!

Thanks to Liz Denhup for feedback. If you liked this, come join my newsletter! I write new essays there every week.

How did we go from functions to methods? Think of the method obj.run(x: int) as being syntactic sugar for the function run(obj: Class, x: int), kind of like how Python methods have a required self parameter. ^[return]
If I wanted to actually guarantee that val >= 0, I could add a post_init method to the dataclass. ^[return]
In this specific case you can catch this violation with the postcondition self.limit == old.limit, but this is just a simple illustrative example. ^[return]
Where “black box” means “public methods and accesses only”. ^[return]

Learn AutoHotKey by stealing my scripts

Mon, 21 Aug 2023 00:00:00 +0000

tl;dr annotated AHK scripts here.

Anybody who’s spent time with me knows how much I love AutoHotKey, the flat-out best Windows automation tool in the world. Anybody’s who’s tried to use AutoHotKey knows how intimidating it can be. So to help with that, I’m sharing (almost) all of my scripts along with extensive explanations. There’s fourteen files in total, covering (among other things):

Fast open specific folders on your computer
Fast insertion of the current date, em-dashes, and ¯\_(ツ)_/¯s
How to extend any program with new hotkeys
A modal hotkey system if you’re a vim fan like me
A simple GUI demo
A script to convert any timestamp into UTC and your local time:

This is also an example of an educational codebase, a codebase designed specifically from people to learn from. So everything is heavily commented with “what” it’s doing and sometimes why I’m doing it that specific way. I wrote a bit about the theory of educational codebases over at my newsletter. Feel free to file an issue if you’d like to see something explained better!

My Problem With the Four-Document Model

Wed, 05 Jul 2023 00:00:00 +0000

Two universal facts about user documentation:

Documentation is really important.
We are really bad at writing it.

We don’t have good theories on what makes for useful documentation. That is except for the four document model, or Diátaxis.¹ I’m glad that people use it. I’m also a little frustrated that people use it even when its inappropriate. My problem is that it’s not a universal or comprehensive model: there’s a lot of documentation people need to write that doesn’t neatly fit the model.

What is the 4doc model?

The 4doc model says that user-facing documentation should fall into four categories:

Tutorials describe how to get up and running as fast as possible.
How-tos describe how to do specific things.
Explanations teach a topic you already know in-depth.
References describe exactly what things are and what they do.

Each category serves a different purpose, targets a different audience, and is written in a special style. Tutorials are friendly and simplified, while references are dry and comprehensive.

4doc was originally started by Daniele Procida at Divio to improve their documentation, and has since become popular in the broader tooling ecosystem.

My problems

I think of the 4doc model like the five-paragraph essay. In the US, kids are taught to write essays in three parts:

A starting paragraph the context and the main thesis,
Three body paragraphs that each independently support the thesis
A concluding paragraph with summarizing the arguments and thesis.

It’s a good starting point to learn essay writing. The problem is that students think it’s the only way to write essays, and then they cram everything they write into a format straitjacket.

4doc is much better than 5para: I’ve never read a halfway-decent 5-paragraph essay but I’ve read plenty of good 4doc pages. Even so, it still leads to the same kind of “this is the only way” thinking. People use the model even when it’s not the right choice and they don’t vary up the format even when they should.

My specific issue is that the model is not universal and it’s not comprehensive.

It’s not universal

From the structure page:

Good examples of the scheme in substantial projects include:

the Divio Developer Handbook

Django’s documentation

django CMS’s documentation

Divio and Django CMS are tools. Tools and libraries are different than frameworks and languages. Tools have simple conceptual models, the set of interrelated ideas you need to internalize to use them properly. You just need to know what they do and how to use them. The 4doc model, developed at Divio, most reflects the documentation needs of tools.

Look at django. Frameworks are one step up from tools and are still mostly about “what and how”, but they also have a lot of interrelated concepts you need to understand. While it still ostensibly follows the 4doc model, the “explanation” section is triple the size of the tutorial and how-to sections combined.² The explanation section also teaches the user, which isn’t supposed to happen:

It’s not the place of an explanation to instruct the user in how to do something. Nor should it provide technical description. These functions of documentation are already taken care of in other sections. — divio

It’s only possible to separate explanation and instruction for things if you don’t need a good mental model. Things like tools.

If we zoom out to programming languages (PLs), the 4doc model breaks down entirely. If you look at the Python docs, the tutorial section needs to also explain and reference, and the references are both explanation and how-to. Language concept maps are just too dense to separate.

I’ve seen some languages try to strictly follow the 4doc model and it doesn’t work. 4doc is oriented around random-access, languages are taught in a sequence-of-lessons. The two don’t mix well.

In short, the 4doc model is not universal. It was designed for documenting tools and doesn’t fully address the needs of frameworks and programming languages.

It’s not comprehensive

4doc would be comprehensive if all useful forms of documentation fell into one of the four types. This isn’t true for the same reason 4doc is nonuniversal: it only covers documentation forms that are necessary for tools.

If you’re documenting a complex topic like a programming language, there’s at least (at least) two more kinds of docs you’ll want:

1. Conceptual Overviews

If I was presented with 4doc-following documentation and had to add just one thing to it, I’d write a conceptual overview.

Conceptual overviews cover the concepts of the tool/framework/language. It presents what the big picture of what this is all for and previews the conceptual model users will learn. The denser the model, the more you need a conceptual overview.

The 4doc model describes documentation types in terms of cooking, so here’s one for conceptual overviews:

All soups work in the same basic way: prepare ingredients like vegetables and meat and boil them in water. This softens the ingredients and flavors the water (“broth”). We can make higher-quality soups by first cooking some ingredients in oil (which develops more flavors), adding some ingredients later in the boil (so they don’t get too soft), using a boiling different liquid (like chicken stock or beer), or using spices.

For a more in-depth example of an overview, you can see one I wrote for TLA+. It first explains the problem TLA+ is trying to solve (designing complex concurrent systems), introduces the concepts involved (specs, behaviors, properties, models), and only then shows any code. And even then it doesn’t explain how to run the code, just how various aspects of the code relate to the concepts.

(“Aren’t these just a kind of explanations?” Explanations are intended for people who already know, or are in the process of learning, your subject. It comes after the tutorial. Conceptual overviews come before the tutorial. They are for people who are getting ready to learn the subject, or for outsiders evaluating if they want to learn it.)

2. Snippets and Examples

Examples are complete code intended to be studied. Snippets are samples of code intended to be used. As much as we make fun of Stackoverflow-based development, we copy and paste from websites because that works and makes us productive. If you provide them from the start, developers don’t have to find them from sketchy third parties.

Examples also show how something is done, which is subtly-but-fundamentally different from how-to do something. Imagine I present you an example of a code, and then an example of the same code, optimized. I’m not giving you a rote how-to on optimizing code, I’m demonstrating a way of thinking about optimizing code. It all comes back to the mental model.

Or take Julia Evans’ debugging mysteries, a set of Choose Your Own Adventures where the reader debugs a problem. It looks like a how-to, but it’s a branching path, not a linear set of steps.

The cooking analogy would be a video of a chef making a dish and explaining what she’s thinking while she makes it. Or maybe doing a taste-test of undersalted dishes. It’s not a perfect analogy. Written examples don’t make a whole lot of sense in tradecraft.

Good is the enemy of better

I want to be clear: I don’t think 4doc is bad. I think it’s quite good! But I think you have to think about how you’re using it. Like any other tool, you have to evaluate what it does for you. The model says at much:

Diátaxis is based on sound theoretical principles and has been proven in practice, but it’s not the final word in documentation. … the structure it proposes is not intended to be a plan, something you must complete in your documentation. It’s a guide, a map to help you check that you’re in the right place and going in the right directions. — How to use Diátaxis

I’m unhappy that the original Divio page doesn’t have this disclaimer and instead has big quote calling it the “The Grand Unified Theory of Documentation”. Procida doesn’t seem to be believe this and I wish divio would stop pitching 4doc as the one-and-only documentation system. It might be good for getting people on board but it’s bad for broader documentation innovation.

Thanks to Lito Nicolai for feedback. If you liked this, come join my newsletter! I write essays once a week.

Some people see the two sites and assume Divio plagiarized the 4doc model from Diátaxis. I looked into it and can happily say it’s not true. Daniele Procida came up with it while working at Divio and then got their permission to spin it off into Diátaxis. ^[return]
In the pdf build the tutorial is 70 pages, the how-to is 90, and the “using Django” section is 500. ^[return]

Is Software Engineering Real Engineering?

Wed, 24 May 2023 00:00:00 +0000

Description: What makes software engineering different from “traditional” engineering? To find out, I interviewed 17 “crossovers”: people who have worked professionally as both a software and a traditional engineer. In aggregate, we learn three things: we are in fact engineers, we’re not actually that different as a field, and there’s a lot we can both teach and learn.

Slides are here. Video is here.

FAQ

These are some of the questions I remember people asking me after the talk. I’ll also put up questions emailed too.

Why does it even matter if we’re engineers or not? Why can’t we just be software developers?

Two reasons. First, lots of people do have strong opinions on the “engineering” question and use it to make strong value judgements on software as a field. I can’t tell people to stop caring about this question, but I can try to make sure that if they compare “software” to “engineering”, they don’t have misconceptions about what “engineering” is.

Second, I deeply, deeply believe in the value of interdisciplinary dialogue. If software is like engineering, then we can learn a lot about software by studying engineering. Like, a lot of devs talk about how incredibly unpredictable software projects are, but every engineer I talked to said the same thing was true about their old jobs. So how do they deal with unpredictability? Is it anything we could do, too?

Did you talk to people about job satisfaction?

A little. Most of the people strongly preferred their jobs in software than their old jobs in classical engineering.

Did you talk about meetings?

I did not. There’s still a lot to learn about software and trad engineering!

Why didn’t you talk to biomedical/audio/environmental/etc engineers?

While I think there’s insights to learn from these people, too, I wanted to define “engineering” as conservatively as possible. I was worried that if I used a more liberal definition, software folk would be more skeptical of what I learned. By sticking to “stereotypical” engineering fields I can avoid that specific objection.

Why do other fields think software is “easy”?

Every field seems easy until you actually get into it.

Waterfall was originally agile from the start.

Winston Royce originally introduced “Waterfall” in Managing The Development of Large Software Systems. It’s a common misconception that he was originally rejecting waterfall and advocating something more Agile-like. However, a close reading of the paper shows that while he recommends some changes to the basic waterfall approach, they remain highly waterfall and his result is still “highly waterfall” him support waterfall and most of his recommendations are to keep in the waterfall style. See Leprechauns of Software Engineering for a more in-depth discussion.

Isn’t it too early to start collecting information about software “snapfits”? How do we know current solutions to EX plugin systems aren’t all fundamentally flawed?

I discuss this question in depth in this essay.

So what is the most important videogame ever made?

Space Traveler.

References

Certification regulations

Books

Software Craftsmanship: https://www.oreilly.com/library/view/software-craftsmanship-the/0201733862/
The Handbook of Industrial Engineering: https://onlinelibrary.wiley.com/doi/book/10.1002/9780470172339
Engineering a Safer World: https://direct.mit.edu/books/oa-monograph/2908/Engineering-a-Safer-WorldSystems-Thinking-Applied
The first snap-fit handbook: https://www.sciencedirect.com/book/9783446227538/the-first-snap-fit-handbook

Quotations

Jeff Atwood: https://blog.codinghorror.com/bridges-software-engineering-and-god/
Systems guy: No direct quote, but see https://quoteinvestigator.com/2019/09/19/woodpecker/

Misc

The formal methods sample code is the Isabelle proof from let’s prove leftpad.
A preventable coding error knocked out 911 service for millions
Capacitor specsheet: https://store.comet.bg/download-file.php?id=10107

Somehow AutoHotKey is kinda good now

Wed, 03 May 2023 00:00:00 +0000

I love Autohotkey so much that it keeps me on Windows. It’s the best GUI automation tool out there. Here’s a shortcut that opens my current browser tab in the Wayback Machine:

#HotIf WinActive("ahk_exe firefox.exe")
>!^s::
{
    Keywait("RControl")
    Keywait("RAlt")
    SendEvent("^l")
    SendInput("{left}https://web.archive.org/web/*/{enter}")
}
#HotIf

By comparison, the official extension takes four files to do the same thing. Four files!¹

But I come here to bury AHK, not to praise it. In January AutoHotKey v2 got released and it’s wildly backwards incompatible with v1. I had to change maybe 20% of my code to get it working again. And I got off easy, I don’t have any third party scripts!

The trauma of Python replacing print x with print(x) is legendary. Couldn’t they have updated AHK without breaking everything?

No.

You see, AHK v1 was very, very bad.

Why AHK was bad

Implicit Strings

So imagine you just got started with AHK and want to show a message box. You boot up v1 and write

MsgBox Hello world!

In v1 you don’t have to surround strings with quotes. It means less syntax— easier for normies— up until the point you want to include a variable. In most languages you’d write something like "Hello " . name but AHK will take that a little too literally:

Fortunately you can still %interpolate% the variable:

name = Jeff
MsgBox Hello %name%!

Now what if you want to show the name in ALLCAPS? Easy, just do this:

name = Jeff
tmp := StringUpper(name)
MsgBox Hello %tmp%!

Surprise! The hacked on interpolation syntax isn’t real interpolation and anything besides a single variable name is a syntax error. This implicit-stringness plagued all the commands, many of which took variables and strings. In WinWait y, y, y the first two y’s are the string “y” while the last y is the value of the variable y.

v2 got rid of implicit strings, so now you instead do this:

name := "Jeff"
MsgBox(Format("Hello {1}!", StrUpper(name)))

I removed six different tmp hacks.

Inconsistent function/command syntax

My sample above isn’t actually valid AHKv1. It calls StringUpper like a function, but it’s actually a command. Since it’s a command, you instead write:

name = Jeff
StringUpper, name, tmp
MsgBox Hello %tmp%!

Yes, StringUpper is implicitly defining a new variable, yes you have to assign the output if you want to do anything with it, yes v1 had regular functions too and the whole distinction is pointless.² It’s just that functions were added circa 2005, so all of the existing commands stayed commands for backwards compatibility.

In v2 everything’s just a function. tmp := StrUpper(name) works fine. There’s still functions like MouseGetPos that assign output to its arguments but at least you now have to explicitly pass in references, like MouseGetPos &xpos, &ypos.

Other command/function problems

How do you check if x is equal to 10? IfEquals x, 10. How do you check if a keypress k isn’t in “aeiou”? IfNotInString k, aeiou.³

There were nineteen special if commands.

v1 got a generalized function if in 2009, but beginners would regularly use the command ifs instead of the function if. Thirteen years later v2 disarmed this footgun once and for all.

Error level

v1 had a global value called ErrorLevel, which roughly mimics return codes from shell programs. If you tried calling SoundPlay but it couldn’t play the sound, AHK would set ErrorLevel to 1.

Now there’s an obvious problem with this, which is that it was global state changed by everything. You could never be 100% sure that the ErrorLevel you read came from the command you expected it. v1 did keep a separate ErrorLevel for each thread so at least it was mostly thread safe. Mostly but not completely: thread interrupts would truncate ErrorLevel strings longer than 127 characters.

You may be asking why ErrorLevel could even be a string, and it’s because v1 was terrible. While SoundPlay would only set 0 or 1, SoundGet would set ErrorLevel to

“Invalid Control Type or Component Type”

“Can’t Open Specified Mixer”

“Mixer Doesn’t Support This Component Type”

[etc]

In fact you can stuff whatever arbitrary thing you want in ErrorLevel, and at some point v1 abandoned the pretence that it was for “errors”. For example, calling StringReplace would set the ErrorLevel to the number of substrings it replaced.

v2 just uses try/catch blocks.

The goddamn object hash tables

This one has a special place in hell. v1 didn’t have proper hash tables; you instead defined a new object where the fields were your “keys”. In AHK, then and now, all identifiers are case insensitive. So the “hash table” {a: 1, A: 2} is equivalent to {a: 2}.

While small in comparison to the other problems, this one had the extra twist of being completely undocumented. You were just supposed to know that because commands weren’t case sensitive, hash keys weren’t either. I wasted so much time debugging this.

v2 still doesn’t have case sensitive identifiers but it does have a dedicated Map class which behaves as it should.

Miscellaneous terriblenesses

Most errors would create anomalous behavior: invalid array lookups returned empty values, as would referencing missing variables.
Instead of taking function callbacks, commands like SetTimer and OnClick took labels. Then they’d invoke a goto. This was exactly as bad as you think.
Just kidding, it was worse. goto could take dynamically computed labels and it could jump from inside a function to outside.
Multiline hotkeys had to end with a return statement, while single-line ones didn’t. Not a deal-breaker, just annoying.
v1 ignored any script setup or global assignments after the first defined hotkey, even if that hotkey was in an included file.

AHKv2 is worth it

So with all those issues, you’d probably wonder why anybody would bother. The language was terrible, now it’s passable. But passable is a very low bar, and there’s plenty of good languages out there already.

The thing is, even when it was a terrible language, AHK was still worth using. Nothing else filled or even cared about its niche: modifying the behavior of other programs. Normally, if I want to extend a program, I need to use its plugin system. That means learning:

The provided APIs
The capabilities, limits, and idiosyncrasies of the system
The entire development environment needed to write plugins in whatever language
How to actually deploy the plugin, which always takes way more time than I’d hope.

And that’s if there’s a plugin system in the first place; there is no way to extend the desktop Spotify app short of writing a new client from scratch.

That’s why AutoHotKey is so useful. It can target windows, find pictures, and send input, which is enough to jury rig my own “plugins” regardless of what the program actually supports. v1 may have sucked but the alternative was not being able to do that. And now the language is actually okay! Life is great!

v2 in action

I run formal methods workshops. FM tools have unfamiliar syntax and semantics to programmers, and many have terrible error messages. To keep classes from getting bogged down in people having syntax errors, I go through a list of “checkpoints” for each spec. We start with a blank file and modify it to checkpoint 1, then modify it to checkpoint 2, etc. If someone’s spec won’t compile, they can just run a diff to see their mistake.

I like this approach but had two problems I needed to fix. First, people need to know which checkpoint we’re on, and they must be able to find out without disrupting the class. Second, if I don’t follow the checkpoints exactly, the diffs won’t show people what they got wrong.

I solved both of these with AHK. The script shows the current checkpoint on my shared screen, the checkpoint content on my private screen, and switches to the next checkpoint when I press 6 on the numpad. Here’s what it looks like:

And here’s the whole program:

Show code

#SingleInstance Force

Controller := Gui("AlwaysOnTop Resize","Workshop Spec Controller")
Display := Gui("ToolWindow AlwaysOnTop +Owner" Controller.Hwnd,"Workshop Spec",)
Display.SetFont("s12")
Display.MarginY := 5
Display.Add("Text", "section", "Current Spec:")
Display.SetFont("s16")
SpecNum := Display.Add("Text", "W230 vSpecNum", "")

SpecNameCtrl := Controller.AddComboBox("vName", ["example1", "example2", "etc"])
                Controller.AddEdit("vSpecNumber yp")
SpecNumberCtrl := Controller.AddUpDown("Range1-20", 1)

cSpecText := Controller.AddEdit("r40 vSpecText xm w300 -WantReturn")
cSpecText.SetFont("s12", "consolas")
UpdateSpecBtn := Controller.Add("Button", "Default w0", "")

SpecNumberCtrl.OnEvent("Change", UpdateSpecTracker)
UpdateSpecBtn.OnEvent("Click", UpdateSpecTracker)
Display.OnEvent("Close", CloseAll)
Controller.OnEvent("Close", CloseAll)

Display.Show("W230 x60 y1200")
Controller.Show("x2700 y500")

set_text(text, num) {
    specname := Format("{1}__{2:02}.tla", text, num)
    SpecNum.Text := specname
    file := "path\to\specs"  text  "\" specname
    if FileExist(file) {
        spec := FileRead(file)
        cSpecText.Value := spec
        Controller.Show("AutoSize")
    } else {
        cSpecText.Value := file
    }
}

UpdateSpecTracker(ctrl, info) {
    set_text(SpecNameCtrl.text, SpecNumberCtrl.value)
}

CloseAll(*) {
    ExitApp
}

NumpadRight::ControlSend "{up}", "Edit2", "Workshop Spec Controller"
NumpadLeft::ControlSend "{down}", "Edit2", "Workshop Spec Controller"

Two GUIs, communicating state, two global hotkeys, less than 50 lines of code. AHK is wonderful.

If you’re on Windows you can download AHKv2 here or via winget install AutoHotkey.AutoHotkey. Mac users may be able to get similar benefits with a mix of Hammerspoon and Shortcuts (which I’m very jealous of). Feel free to email me if you have questions about AHK— I love helping other devs get started.

Thanks to Predrag Gruevski for feedback. If you liked this, come join my newsletter! I write essays once a week.

I mean it’s doing a lot of other stuff too, but it’s stuff I don’t need. My five-minute script solves my needs just fine. ^[return]
No, you can’t use normal functions in a variable interpolation, either. ^[return]
And of course, if you wanted to check that the string "k" wasn’t in the variable aeiou, it’d be tmp = k, ifNotInString tmp, %aeiou%. ^[return]

Breaking the limits of TLA+ model checking

Mon, 17 Apr 2023 00:00:00 +0000

I haven’t written much about TLA+ on the blog since the new learntla went up. Any examples or special topics I think of go there instead. But I recently thought of a cool demo that doesn’t quite fit the theme of that book: there are some things you can’t easily check with the model checker but can check if you work with the state space as a directed graph.

Take for example observational determinism (OD): does an algorithm always converge on the same answer? Let’s use the example of four threads, each nonatomically incrementing a counter once. To show that the final value of x isn’t always 4, you just have find one trace where it’s not 4. But to show that the final value is inconsistent, you have to find two traces that get different answers! The TLA+ model checker (TLC) can do the first but not the second.

Show spec

------------------------------ MODULE threads ------------------------------

EXTENDS Integers
Threads == 1..4 \* four threads!
(* --algorithm threads

variables 
  x = 0;

process thread \in Threads
variables tmp = 0;
begin
  Get:
    tmp := x;

  Inc:
    x := tmp + 1;
end process;

end algorithm; *)
====

If you wanted to test for observational determinism, you’d normally have to mimic it with some hairy “hypermodeling” tricks I wrote about here. Not only is this difficult and fragile, it’s also really inefficient. The regular model has only 755 distinct states, while the hypermodel has over 100,000! It’d be much easier if we could “step outside” the model checker and just examine the state space directly.

So I started looking for a way. And I got a lot further than I expected, well past checking OD and all the way into finding probabilities! If you want to poke around with any of the components, you can download the whole project here.

Steps

1. Get the state space dump

This is easy, just model check with -dump dot,actionlabels file. TLC will output a graphviz file.

strict digraph DiskGraph {
nodesep=0.35;
subgraph cluster_graph {
color="white";
2665600213816166005 [label="/\\ x = 0\n/\\ pc = <<\"Get\", \"Get\", \"Get\", \"Get\">>\n/\\ tmp = <<0, 0, 0, 0>>",style = filled]
2665600213816166005 -> -1245546546836852765 [label="Get",color="black",fontcolor="black"];
// 2000 more lines

That’s… going to be hard to work with.

2. Clean up the file

Let’s replace the node ids with something a little more readable. I wrote a script to replace each node id with a short string. The first node in the file is renamed to “a”, the second to “b”, the twenty-seventh to “ab”, and the 5796th to “aaa”. I took this as an opportunity to learn some Rust.¹

Show code

fn fn fn mod

style="background: #272822">

//use std::io; #![allow(unused_imports)] #![allow(dead_code)] #![allow(unused_variables)] use std::fmt; use std::fs; use std::path::Path; use std::env; use regex::Regex; // 1.7.1 use std::collections::HashMap; num_to_id_str(id: u32) -> String { let mut out: String = String::with_capacity(5); let mut id: u32 = id; while id != 0 { id = id - 1; out.push(((id % 26) + 97) as u8 as char); id = id / 26; }; out.chars().rev().collect::<String>() } next_id_str(id: &mut u32) -> String { *id = *id + 1; num_to_id_str(*id) } main() { let mut id: u32 = 0; let mut replacement_table: HashMap<&str, String> = HashMap::new(); let args: Vec<String> = env::args().collect(); let file_path: &str = &args[1]; let out_path: &str = &args[2]; let graph = fs::read_to_string(file_path) .expect("Should have been able to read the file"); let mut out = String::with_capacity((&graph).len());  let re = Regex::new(r"-?[[:digit:]]{10,}").unwrap(); let unchanged_chunks = re.split(&graph); let mut text_words = re.find_iter(&graph); //needs to be mut for iteration for chunk in unchanged_chunks { out.push_str(chunk); match text_words.next() { Some(mat) => { let key = mat.as_str(); if !replacement_table.contains_key(key) { replacement_table.insert(key, next_id_str(&mut id)); } out.push_str(replacement_table.get(&key).unwrap()); } None => continue }; } fs::write(out_path, out).expect("Unable to write file"); } #[cfg(test)] tests { use super::*; #[test] fn it_increments() { let mut id: u32 = 0; next_id_str(&mut id); next_id_str(&mut id); assert_eq!(id, 2);  } #[test] fn str_generator() { let results: Vec<String> = vec![1, 2, 25, 26, 27, 28, 52, 53].iter().map(|&x| num_to_id_str(x)).collect(); assert_eq!(results, vec!["a", "b", "y", "z", "aa", "ab", "az", "ba"]); } #[test] fn incrementor_and_generator() { let mut id: u32 = 0; next_id_str(&mut id); assert_eq!(next_id_str(&mut id), "b"); } } all



3. Try Gephi

Gephi is a graph analysis tool. I used it because it was the first tool I found. There’s probably better ones out there.



Uhhhhhh that’s a mess. Let’s run a layout algorithm to make it at least tractable.



That’s a little better. Those little dots on the bottom of the graph are graphviz options that got accidentally turned into nodes. I went through the data table and manually deleted them.

4. Discover that Gephi is terrible

Gephi can make nice looking graphs and not much else. After fighting through the terrible documentation and UX I managed to filter for initial and terminal states. You can see the queries on the right-hand side of the window. Once I got the filters working I added a “Type” data column and labelled the corresponding states. That way I could assign special colors to root and terminal nodes.



After that, I could manually look through the terminal nodes and see that they have different values of x.



Okay, so OD is refuted. I’m interested in checking more than just observational determinism, though, and I wanted to see what else Gephi could do. To start with, I wanted to extract the x value from the label and put it in its own column. After that, I’d see which values are most common in terminal states.

This is where I learned that Gephi hates me.

The only way to extract x is to duplicate the column and do a regex find/replace on the whole string. Gephi defaults to running the find/replace on every single column and resets that default every time you reopen the find/replace modal. If you forget to re-enable it just once then it’s time to reload the graph and start over, because there’s no way to undo changes.

In fact nothing in Gephi is undoable, and nothing is well documented, which together make the system inscrutable. You can’t learn what buttons do and you can’t safely try them out, especially when some buttons do things like “wipe all your layout and customizations and go back to grey blobs.”

…In retrospect, I should’ve guessed Gephi was a dead end.² I don’t want to repeat painstaking manual labor every time I change the spec. What I really want is a software library.

(Unrelated, it’s interesting that most languages don’t have a graph type in their standard library. I’m guessing it’s because graphs have lots of different representations, which also have very different performance and API considerations.)

5. Start over with NetworkX

NetworkX is a graph analysis library. I used it because I already know Python. There’s probably better ones out there.

NetworkX represents graphs as a dict of dicts of dicts.³ As far as I could figure out, you have to implement graph operations imperatively. For example, to extract x from the label, I had to iterate through every node and add x as an attribute.
for node in G:
    label = G.nodes[node]["label"]
    x = re.search(r'x\s*=\s*(\d+)', label).group(1)
    G.nodes[node]["x"] = int(x)


To find the terminal states, I first removed all the self-loops, and then filtered for all the nodes with zero outbound degree (no outbound edges).
G.remove_edges_from(nx.selfloop_edges(G))
terminals = [n for n, deg in G.out_degree() if deg == 0]


After that, I can pull all their x attributes to see the spread.
from collections import Counter
Counter([G.nodes[t]["x"] for t in terminals])

# output
Counter({4: 24, 3: 36, 2: 26, 1: 23})


So there are some end states where x=4 and some where x=3, proving that our system is not observationally deterministic.

6. Go mad with power (and probability)

Something I wasn’t expecting: x != 4 in three quarters of the end states. That seems really high for a race condition! I implemented the spec in C++ and it only had the race condition 0.1% of the time.⁴






  
    Show C++ code
  
  #include <iostream>
#include <thread>
#include <vector>

const int NUMRUNS = 30000;
const int NUMTHREADS = 4;
int x = 0;
bool wait = true;

void *increment_x()
{
    while (wait) {;}
    int tmp = x;
    tmp = tmp + 1;
    x = tmp;
    return NULL;
}

int main()
{
    int results[NUMTHREADS] {0};
    std::vector<std::thread> threads(NUMTHREADS);

    for(int i = 0; i < NUMRUNS; i++) {
      x = 0;
      wait = true;

      for(int j=0; j < NUMTHREADS; j++) threads[j] = std::thread(increment_x);
      
      wait = false;
      for (auto& th : threads) th.join();
      results[x-1] += 1; 
    }
    for(int i = 0; i < NUMTHREADS; i++) std::cout <<  results[i] << " ";
}









There are two reasons why “count the end states” misestimates the likelihood. First, it assumes there’s only one path per end state. To see why this is a problem, here’s the state space for two threads:



The space has three end states, and x = 2 in two of those. But there are four paths to those three end states; two of them end in x = 1. That means it’s twice as likely to happen as either of the other states, assuming all transitions are equally likely.

And that’s the second issue: all transitions are not equally likely. In the spec, a Get followed by a second Get represents one thread interrupting another. Threads (on Windows) get 20ms slices at a time, and both assignment and increment operations both happen on the nanosecond scale. Each thread will more often complete both operations than complete one and then get interrupted. So two consecutive Gets should happen pretty rarely.

“Pretty rarely” isn’t a concept in TLA+: either a state is possible or it isn’t.⁵ Figuring out the state’s likelihood is out of scope for the language. But with graph analysis, we’ve expanded the scope, and maybe that’ll be enough to get something interesting. I feel really out of my depth here and am learning this all as I go, but let’s give it a shot anyway.

So let’s say that if a thread takes an action, like Get, it’s more likely that thread will Inc than that the other thread will interrupt it and Get. As a very rough substitute, we’ll say that Inc transitions have more “weight” than Get transitions do.

 
   
  (TikZ source)


In the first two images, we have only Gets and only Incs, so each transition is equally likely to happen. In the final image, we have one of each. If I weight each Inc as 3 times heavier than a Get, then it’ll happen 75% of the time. If we instead had 2 Incs and a Get, each Inc would happen 3/7ths the time and the Get would happen 1/7th.

That gives us the probability, given a state, of which edge we take. Now what’s the likelihood of reaching a given state? Let’s walk through an example state space:

 
   
  (TikZ source)


I marked the weight of each edge. Now let’s walk through the likelihoods of each node:


R is the single root node, so it has likelihood 1.
R is equally likely of going to A or B, so they each have likelihood 0.5.
A goes to C 50% of the time. But we have to get to A to get to C, and that only happens 50% of the time! So the likelihood of C is 0.5*0.5 = 0.25.
Similar logic shows that the likelihood of E is 0.5*0.75 = 0.375.
The most interesting case is D. We can reach it from either A or B. So its likelihood is the sum of the two paths, or 0.375.


(This only works because there are no cycles in the state space. IIRC the technique broadly works with cycles, but you have to keep doing iterations over the graph until the likelihoods converge. This is basically the idea behind representing probabilistic systems as a stochastic matrix.)

Now we have our algorithm for calculating each state likelihood. Then if we take all the terminal states with x={k} and sum their likelihoods, we’ll have the proper probability that we end with x={k}. Repeat for every {k} and we have the distribution of end values!

I implemented this algorithm, but explaining the code took 1,000 words and this essay is already long enough as it is. So I hosted the code elsewhere.

7. Compile this all into a github repo

Link here. It contains all of the work I put into this experiment:


A copy of the TLA+ spec, which you can use to generate new state space graphs
Five premade state space graphs, so you don’t have to learn how to use TLA+
A (python) script to make the graph nodes a little more readable
A python script that calculates the probabilistic property, along with notes and commentary on the code
A Gephi workspace with all the hard cleanup work already done for you
A PRISM probabilistic model of the same problem, just because.


I wrote it to be easy to study, so you can experiment with it too. I hope you find it useful!

To close out the experiment, let’s find some likelihoods:
python likelihoods.py graphs/2_threads.dot --weight 1
Counter({2: 0.5, 1: 0.5})

python likelihoods.py graphs/2_threads.dot --weight 10
Counter({2: 0.91, 1: 0.09})

python likelihoods.py graphs/2_threads.dot --weight 100
Counter({2: 0.9901, 1: 0.0099})


Thoughts

I’m excited about what this opens up! I’m hopeful the approach scales up to larger state spaces, too; it seems the majority of the script runtime is spent on importing the graphviz into NetworkX, the actual graph computations finish much faster. Even if it NetworkX doesn’t scale, maybe a more ergonomic graph analysis library would.

And man it’s feels good doing a blog post on advanced TLA+ use again. It’s been over a year since the last time. I enjoy writing material for beginners but it doesn’t scratch the same itch.

Anyway if you enjoyed this, I’m running a 1 day TLA+ workshop in May and have a newsletter I update weekly.

Thanks to Jay Parlar, Lorin Hochstein, and Andrew Helwer for feedback.





The project repo uses a python script instead to cut down on the number of dependencies.
 ^[return]
Recently the Gephi team announced Gephi Lite, which among other things supports custom javascript node attributes. It doesn’t yet support graphviz import, though it’s still a work in progress.
 ^[return]
Nodes have node neighbors, node neighbors have connecting edges, connecting edges have attributes.
 ^[return]
I couldn’t get shared memory working in Rust and I was suspicious Python’s Global Interpreter Lock would artificially depress the racecond rate. Shoutout to GPT4 for helping me bumble my way through C++
 ^[return]
In contrast, PRISM is a probabilistic model checker that can calculate state likelihoods without you having to dump state spaces into another program. Then again PRISM doesn’t have functions or arrays so writing the spec in the first place is a big pain in the butt. I ended up modeling the system in PRISM for fun; you can see it here.
 ^[return]



OCR Everywhere
Sat, 15 Apr 2023 00:00:00 +0000
Powertoys is “a set of utilities for power users” that Microsoft keeps separate from Windows so they can get away with less testing. My favorite one is “Text Extractor”, which lets you OCR capture anything on the screen. Win-Shift-T:



Text in the selected field is stored to your clipboard.



It doesn’t work well with photographs or non-computer fonts, though.

Hillel Wayne

Toolbox languages

Useful features:

Example problems:

Useful features:

Example problems:

Useful features:

Example problems:

Further reading:

Useful features

Example problems

Useful features:

Example problems:

Further reading:

What makes a good toolbox language?

Other tools I want in my toolbox

Composing TLA+ Specifications with State Machines

The example

The problem with composition

The big idea

The Solution

The state machine

The server

The system

Sync

Refinements

Closing the open world

Advantages and Drawbacks

Conclusion

Appendix: Sync without primes

What We Know We Don't Know: Empirical Software Engineering

Sources

Intro

Why we care

Methods

DDD

Additional Sources

Comment Section: Software Friction

Blogs on a similar topic

Emails and Comments

Software Friction

Some Properties of Friction

Addressing Friction

Questions I have about friction

Update 2024-05-30

Don't let Alloy facts make your specs a fiction

The pitfall of facts

When to use facts

Conclusion

The Tale of Daniel

Ugarit

Phoenicia

Israel?

Judah

The False Daniel

From Daniel to Today

Comment Section: The Hunt For The Missing Data Type

Response Blogs

Emails and Comments

GraphBLAS

Gremlin

Rubik’s Cubes

Graphs and knowledge databases

The Hunt for the Missing Data Type

The reasons

There are too many design choices

There are too many implementation choices

Performance is too important

It was unanimous

Appendix: Languages with Graph Types

Graph Querying Languages

Mainstream Languages with Graphs in the Standard Library

Graph languages

Mathematics Software Languages

Update 2024-03-18

Planner programming blows my mind

The problem

Our first implementation

Adding multiple goals

Cost minimization

Appendix: `Sync` without primes