11/24/2025

I’m someone who likes to work a little slower, explore a subject in-depth, and always go back to basics. If you’re reading this, you probably appreciate a similar craftsmen-like approach, and often find good ideas, overlooked by popular trends.

Craftsmen correctly identify that true principles and skills that don’t change. This helps them see past the latest craze, tune out noise, and focus on long-term impact. But success with this approach can grow into a temptation to ignore time entirely. We can come to believe that our work is time-less. If we put in the hard work to make a great project, it will be appreciated, no matter when it’s done or what comes after.

On a personal level, this is obviously not true. We go through seasons of life where time, skills, energy, and interests vary, Seasons for projects are limited, and the nature of the work we can do in each season also changes. I unfortunately, have had to leave projects behind, not because they are not good, but because I am no longer the person who had the vision and excitement that started them.

I recently saw an interview with John Carmack, that prompted this topic:

I used to be known for the catch line, “It’ll be done when it’s done.”. You know, “When will Doom ship?”, “When it’s done.”

With a little bit more perspective now. If you’re talking slipping a quarter, slipping six months? Yeah great that’s definitely fine. But, when you’re talking about slipping years? When years go by, the world changes around you. Time has a physicality that you may not appreciate.

We spent six years on that game [Rage]. It was using flashy new technology. We had an E3 where we were game of show. But it didn’t quite ship. And by the time it got out, the world had changed around us. The technology decisions that were made for some earlier systems weren’t necessarily the right thing for the latest ones. We now had Call of Duty and Battlefield as these juggernauts we were competing with. I look back at it as one of those real decisions. I think we should have done whatever it would have taken to ship that two years earlier. Be less ambitious with some of the technologies and get it out earlier.

Changing technology, and competition were specifically identified but we can list all kinds of things that change for-the-worse over a long project:

• The vision becomes less inspiring, as contributors and customers acclimate to the message.

• The insights behind the project become realized by others, and incorporated into other works.

• Marketing which rang culturally true, now sounds out of place.

• Knowledge about project details become lost to turnover and memory.

Every project makes assumptions about what the world will look like when it’s finished (what people will like, use, want, etc). The longer that prediction must extend into the future, the harder it is to get right, and the greater the cost to update, and maintain those parts that expire.

And those changing seasons of life? They happen to every single person contributing to a project. You cannot expect anyone to operate or contribute in the exact same way they do now, even if they are committed to seeing it through.

John follows up with a related experience, from a different project:

Quake was the first really traumatic game to ship, internally. We’re still only talking like two-year developments but at the time it felt really long. We had all sorts of internal strife because we were trying to do so many things: six degree of freedom rendering, modding, internet-based game servers, and it was a lot of stuff. I later looked back and said, we could have done half of those things in a “Super Doom” and shipped it earlier and then done the other half even better on a game coming later. I still roll that over in my mind.

I find this quote striking. Because unlike Rage, Quake was an out-of-the park success! The game was well received. It made a lot of money. The technology was incredible, influenced the whole industry, and set them up for different projects. But he is still “rolling over” decisions that might have saved 6-12 months, nearly 30 years later! ¹

–

Recently, I was working at a startup that shared a lot of my interests in craftsmanship. They wanted to build the right technology, had a big long-term plan, and a great market opportunity. It probably goes without saying, but this is a rare gig to find.

I also was able to join at a magical time. The product had been prototyped, all the funding was secured, they had smart people, and all we had to do was execute.

But strangely that time never came, not because of many major roadblocks. We just kept on doing what craftsmen do well; finding better ways to do things, coming across new technologies, growing our feature set. It became momentum for the whole organization. Months went by, then years, and somehow we weren’t any closer to ship. All of the learning that would come from pausing development, polishing, and engaging with real customer problems, kept being pushed down the road.

I was growing impatient, but otherwise occupied with interesting technical problems. Then I heard Carmack’s interview and realized my fun and comfortable job was actually becoming an urgent issue. I left for another opprotunity, which turned out to be exactly what I needed. It taught me all I originally wanted to learn about startups, and allowed more of my skills to be appreciated, I can tell you about it another time. But what I know is my life would be very different had I stayed working on that same project for a few more years.

08/02/25

I’m back with another house maintenance project (and not talking about computer technology).

My irrigation system has been making awful water hammer noises. It’s so bad, that I haven’t felt comfortable watering my lawn, due to fear of damaging the plumbing. Here are the symptoms of my problem:

1. Only occurs when a valve opens, not when it closes. (fast moving water slamming into a valve is a common problem).
2. Occurs regardless of how long the sprinkler has been on. I can close a valve, and reopen it 10 seconds later and it’s the same.
3. Severity varies across different zones. Zones closer to the manifold seem worse.

These rule out most of the quick water hammer fixes, for example adding a delay between zone activation doesn’t help. So I determined the best solution was to install a water hammer arrestor. Here is how that went.

Disclaimer: I am an occasional DIYer, not a professional plumber. My intent is to share my exprience with the hope that it might be helpful for you.

Plan

I want to insert the arrestor as close as possible to the manifold, so it provides benefit for all zones, and protects as much of the plumbing as possible. Unfortunately, the sprinkler box is out of space, so I needed to dig up and insert on to the incoming line.

I also want make enough space for a new box to house the arrestor, so I can access it later.

There are quite a few positions and orientations to consider. Vertical orientation seems natural, but the arrestor is actually too long, and so I chose to lay it horizontally (OK for this model). I could replace the elbow with a tee to get shock absorption at the corner, but I don’t want to rebuild the manifold connection.

Attempt 1

So I cut the pipe, attached the tee, and put it all together. The elbow with the threaded manifold connection gives me an advantage. I can connect this piece last and avoid the challenge of connecting two pieces of PVC. (This is classic plumbing problem where there is insufficient space to slide a coupling on to both sides.).

My expectation was that the threaded piece would haves some give and allow me to easily connect back to the manifold. But that was not the case. With everything assembled, the pipe and the manifold were offset from each other by about 1/8", and there was absolutely no give.

I was completely out of luck. Not anticipating this problem was an amateur mistake.

Attempt 2

I needed to try again. I researched alternative ways to join the pipe (and avoid the same problem), but none looked better than my initial idea.

So I cut the tee and redid it all. This time I was extra careful, and measured everything with the elbow piece attached to the manifold. On the final assembly step, I connected the elbow and threaded it to the manifold with the cement wet. This was tricky because I had about 5 seconds to insert the piece, rotate it into position through the box, and get the threads aligned. One trick that helped was inserting a piece of plastic underneath the connection to keep them out of the mud.

It all worked out!

Arrestor assembly

Although not strictly necessary, I attached a PVC 40 nipple rather than threading the arrestor directly into the tee. Brass into PVC connections are less reliable, and I want to protect the tee’s threads if I need to replace in the future.

I filled the arrestor box with gravel and cut a block for the weight to rest on.

Results

I was pleasantly surprised with how well this little part works. 3 of my zones are completely quiet, and 1 has small bump when it starts.

(Yes, I spilled PVC primer all over my lid.)

Thanks to ChatGPT for all the plumbing advice!

06/21/2025

This summer I replaced part of the siding on my 1970s house in Boulder. The project turned out great, better than I hoped!

While planning, I had a hard time finding good information about the techniques and materials used for this style of home. This video gave me confidence that I could do it, but lacked a lot of detail, especially in regards to possible replacement materials. So I wanted to share what I learned in the project, in case it can help anyone else.

Disclaimer: I am an occasional DIY'er not a construction professional.

Siding materials

The original siding on the house is a wood composite material referred to as hardboard, or by the brand name “Masonite”. It’s basically glorified cardboard, and has long been defunct as it doesn’t hold up well to moisture. Very dry climates like Colorado may be the only places where it’s still surviving.

Here you can see some moderate bubbling indicating moisture damage:

About half of my house was actually in good shape. But the parts that are heavily exposed to sun and rain were in disrepair.

The original board is 12" wide and 7/16" thick.

Notice the water damage on the edge:

I was unable to find anyone who supplies the original hardboard, so I had to substitute. I chose LP Smartside smooth finish, which is 12" wide and 3/8" thick. This is a much higher quality material made from OSB, but has similar dimensions and appearance. The manufacturer also provided recommendations for install and materials like nails, which were helpful to refer to.

Removing old boards

The previous stuff was face nailed so it’s pretty easy to get underneath with a pry bar. After a little effort, the nail typically separates from the siding, making it easy to pull out with a hammer. Since it’s being removed, you can be pretty rough with every board except the last.

You’ll notice that my sheathing is not modern OSB but instead fiberboard, another defunct product.

I don’t know too much about it. Part of my motivation for the project was to determine if any places needed to be replaced. I was pleasantly surprised to find it in good condition and didn’t need to touch it.

You’ll notice that there is no vapor barrier/house wrap on the outside. This is more common for older homes, but my research also indicated it’s more acceptable for fiberboard constructions.

Assembly

The house is 24 ft wide, so a single row requires a 16ft and 8ft board. The long and short side alternate with each row to protect and hide seams.

One piece of siding was ripped into 2" strips for the base. This is installed below the bottom row along the foundation. The first row of siding then rests upon this.

Once the first row is done, it gets much easier. You just lay the next board above at the proper height and nail it in. I chose to pre-drill holes in the siding to prevent splitting on the thin side.

We did hammer the whole thing by hand. I considered purchasing a pneumatic nailer, but they are really only available at professional price points. Note that siding requires a special kind of nailer as the nails are different than the ones you can use with a framing or roofing nailer. It’s worth it if you do the whole house.

Painting

The boards come primed on the outside side. So it’s ready to paint with 2 coats. Large rollers worked well for these large boards. The color match turned out great and look just like the old boards!

The new seams and edges must be caulked before painting. I also took time to cleanup up the old boards and seams were not replaced.

All done! Thanks to my family for all their help!

04/05/2025

As a kid, I loved getting books from the library about combat robots. I was always interested in tools and building things, but the idea of making a robot was just a little more exciting than making furniture.

Learning about robots introduced me to new building techniques, but unfortunately actually making a combat robot, was just not very accessible at the time. Welding? Car batteries? Wheel-chair motors? It’s a tough sell for parents and suburban neighbors.

After 20 years, the hobby has gotten a lot more accessible, thanks to 3D printers, but also improvements in batteries, motors, and radio. The whole process is smaller, cheaper, and easier to assemble, so that even local schools have robot making programs.

Recently a friend (who is a 3D printer enthusiast) invited me to make a 150g robot for a small community event, and I had a great time. We didn’t do anything extraordinary, but I wanted to share a little bit about the project. My takeaway, is this can be a great hobby for learning about electronics and mechanics, especially for kids.

Design

At 150g most robots use a body that’s 3D-printed or adapted from an RC cars. To keep weight low, they typically only have 2 wheels. The main design decision to make is how your robot will attack. Most of your weight and body will arrange itself around that.

Spinning weapons are the most popular. It’s a simple and effective design that’s really hard to beat. But we wanted to try something a little different. We found an idea online about a spring loaded flipper. The spring is loaded by lifting an arm using a “snail cam” which can be rotated continuously.

We had high hopes of having multiple weapons, but as these things tend to go, you must simplify and cut scope as you go.

Cam

The main part I worked on is the flipping arm and snail cam. It’s designed with an increasing, but decelerating slope, so that the work required by the motor stays constant.

My first attempt was to cut one with a hand saw from UHMW polyethylene. They may not look excellent, but should work, after a little sanding.

We found the 3D printing was sturdy and smooth enough, so we chose that instead. I wrote a python script to generate the basic curve in Blender, and then used simple boolean cuts to get all the parts we need.

Electronics

N20 is a popular style of motor for this size. Apparently they vary greatly in quality and specification. We just got a few cheap ones on Amazon. I like that you can order them with a small gearbox targeted for a specific RPM, so it’s easy to get the torque/speed you need. For example, the cam uses a much lower RPM than wheel motors.

The radio receiver, and motor controllers are available as a single board. Just wire the motors to the appropriate terminals. The model we used is called “Malenki Nano” which is very popular. Connecting the receiver to a controller was as easy too.

Body

The body was drawn in a traditional CAD program. It went through many iterations and printings as we tried to fit everything in. Here are a few techniques we found helpful:

1. Attach components with zip ties. It makes it easy to move parts between bodies, and is very sturdy. Just add holes for them in your CAD design.

2. Prototype parts in cardboard first. If you have a simple base, it’s easy to tape on pieces to see how they will fit.

3. Use fast and cheap printer settings until the final revision (and it’s always one more than you think).

Battle

The event was very casual and had a positive atmosphere. People were open about showing their robots and learning from each other. Ages varied greatly, with some younger kids, teenagers, and a few Dad’s joining their sons.

A pro stopped by to show off his full-size robot!

In our first (and only) battle we took a few big hits from a spinning blade. Luckily none of our major parts were damaged. So we should be ready to try again after printing a new body.

At a previous startup, I invented an Augmented Reality (AR) system for scanning buildings, and automatically extracting floor plans from scans. Recently a few of the patents I wrote were published, and I wanted to share those, with a brief summary.

Capturing Environmental Features Using 2D and 3D Scans

US-11989895-B2

This describes an augmented reality workflow for capturing a 3D mesh of an environment, and recording points of interest. A key idea is to capture all the user interactions and raw 3D data first. Later in a post-process step, we determine user intent and relative positioning using complete global information.

Floor Plan Extraction

US-12299784-B2

Once we have a 3D scan of an environment, how do we go about turning this into a 2D floor plan? Floor plans are conceptual, rather than objective, and so are guided by user input. This invention describes both an augmented reality workflow and an algorithm for processing that input. It turns out, doing this perfectly is likely NP-complete, but this proper understanding guides effective heuristics.

Surface Animation During Dynamic Floor Plan Generation

US-12229859-B2

It’s important to give feedback to the user during the scanning process. One way we do that is through the augmented reality visualization that shows which parts of a room are captured, and is programmed through shaders. The biggest challenge is to show accurate estimates quickly, despite the scan being incomplete.

Aligning Polygon-like Representations With Inaccuracies

US-12651104-B2

Scan measurements contain error, and the real world is imperfect. How can we correct the alignment of complex floor plan shapes with minimal distortion? One answer is to simultaneously move all points of the floor plan in a multi-variable optimization approach.

Door and Window Detection in an AR Environment

US-12423828-B2

Along with the shape of the building, we are also interested in the apertures of each room. The polygonal shape tells us where the walls are in 3D. We can then render the portion of the 3D mesh corresponding to each wall, and detect cutouts in the rendered image. This transforms a complex 3D detection problem into a traditional 2D computer vision problem. These 2D detections are then projected back into 3D features on the wall.

11/18/2022

A surprising lesson I learned from graduate school is that math is just very different than everything else I have tried learning in the past. It has it’s own style of thinking that has become my primary method for solving hard problems. But, coming from a background in professional software development for years before, I expected a lot more of my core engineering skills to carry over. Here I want to discuss some differences I noticed and their implications.

Debugging and Proving

Debugging programs taught me to check my work with a radically empirical approach. When something isn’t working as expected, I “turn off” that part of my brain that makes assumptions about what it should be doing, and directly examine the behavior of the program. Usually it starts with, “does this line ever get called?”, and grows increasingly more skeptical, “does 2+2 really equal 4?”. When the problem is tricky, no question is too stupid to ask. The goal is to quickly find the gap in my mental model and accommodate it. It’s a simplified version of the scientific method, with a bit of a bias towards experimenting over hypothesizing.

Beginners struggle with debugging because they aren’t as skeptical. They feel the computer is doing impossible or magical things outside their control. Perhaps it’s uncomfortable or unintuitive for them to question their understanding when it’s already shaky.

However, this skill of “turning off” your judgment and observing gets you almost nowhere in math. Proving a theorem is all about building up new information from facts you know. You have to look at it a conceptual object and assert with confidence, “This fits into X category. I know X category has this property. Therefore… ”. It’s an active thinking process that takes constant energy and work. While debugging is an exercise in humility, proving is all about making bold claims.

Even though it takes a lot of work, then end result is very satisfying. In the process of proving you demonstrate that you know something in the truest way possible. You don’t just happen to hold a true opinion, but can explain, at increasing levels of detail, why it’s true.

There is still an experimental aspect to math (some mathematicians prefer more experimentation than others), it just isn’t like interacting with a computer and passively observing output. When you don’t know where to go, you study examples and try to discern patterns. This is very engaging work as you must be the one to construct the examples. The most insight comes from exotic examples, that push things to the extreme, and take some creativity to come up with.

Engineering and Discovery

Debugging is just one small part of programming, and is usually an easy part. What about the engineering skills that go into designing programs, like determining goals, evaluating trade-offs, creating plans, prototyping solutions step by step? In my experience, there just isn’t a whole lot of these that are applicable either.

Occasionally it’s helpful in math to plan out a strategy for approaching a larger theorem, but you have no idea what will be true or false, so you can’t get into much detail, or make many assumptions that rely on prior steps. There aren’t really trade-offs to speak of, except where you want to spend your time and effort.

Engineers solve problems by iteratively improving solutions to overcome key technical challenges in the project. In each iteration, they determine components that worked and components that didn’t, and it’s a matter of keeping what worked and fortifying or replacing the broken stuff.

Mathematicians obviously try a lot of potential solutions, but rarely is it componentized in the same way. Either the whole approach works or not. The solution usually comes in the form of a key discovery, or a sudden flash of insight, after a lot of thought and experimentation. This insight reveals the inter workings of the problem.

There is an alternative theory that programming is more like a craft than engineering, like woodworking, writing, or painting. A craftsmen starts with a rough outline or plan, and then gradually improves individual portions of the work, occasionally making sweeping corrections across the entire piece. Once again, I see very few parallels with math. There is little hope to transform a false result into a true one, or to discover new insights with highly local refinements.

Why is there such a big gap in these experiences? It’s hard to say. Math is certainly more focused on discovery and questioning what is true, as opposed to constructing something. This suggests a more direct encounter with constraints imposed by the world. But, even this distinction isn’t very clear, as engineering also requires dealing with natural trade-offs.

Meta-learning skills

All of this has changed how I think about meta-learning. There is an idea that all fields are pretty much the same, except for details, and you can just transfer skills between them. Perhaps you have heard the goal of university is to teach you “how to learn”. That just doesn’t appear completely true. Some things are similar enough for a transfer, but some are just different.

Despite differences I want to call out a meta-learning skill, which is used in math, that is hugely underrated in engineering and especially software. That is deliberate practice combined with feedback or coaching. Of course, it gets plenty of attention in sports and music. But engineers argue that this doesn’t matter for the kind of work they do. Their low-level implementation skills could optimized, but they aren’t the bottleneck. Most of their effort is spent on tasks that are not focused on efficiency, and not even repeatable. Consider tasks like conceptualizing the problem, gathering requirements, etc.

Dan Luu has responded to this, arguing that increased velocity significantly aids the other tasks, including simply being able to try more iterations in the same amount of time. Read his article to get all the details. I just want to add that math has been built up almost entirely of unique breakthroughs and discoveries, which rarely are the same as what’s come before. In other words, the solutions are even less repurposable than what engineers deal with. Yet, mathematicians place significant weight in memorization, practice, coaching, and studying a variety of areas.

There are a few rationales for this. Of course, practicing solving problems emulates the experience of actually making a mathematical discovery. It’s a pretty good emulation too, except that you have the assurance there is a solution and that the teachers have tailored it to your skill level.

Another reason is making new breakthroughs is just easier if you are fluent with lower level concepts. It’s hard to do calculus, if you struggle with algebra. Remember the discussion about how proving is difficult because you have to actively make new claims? How do you get better at this process? Either your learn the material more thoroughly so you can assert with more confidence from memory, or learn adjacent material so you can perform logical consistency checks as you go. Both of these are essentially just fluency.

Mathematicians also believe that the techniques used for old problems will become useful in the future. Even if you never reproduce a proof exactly, it becomes a tool in your toolbox and a source of inspiration. You may have noticed in college that math professors tend to have a “nack” for knowing what to try on a problem. They seem to get right to the core issue while you are left thinking “I would never have thought of that”. Although it may appear that they must have known the answer, it’s actually their trained intuition, built up from being required to solve a lot of problems in a short amount of time.

What would it like to adopt similar practices in programming? Of course, we should look at speed of writing code, following problem solving procedures, workspace improvements, etc. Coaching suggests a whole set of interesting opportunities to be explored.

But, I think at the most basic level it starts with mindset. Do we want to work hard to get better? There seems to be an attitude of avoiding learning at all costs, because it doesn’t directly count as productivity. Why don’t we embrace curiosity and practice as a separate and worthwhile activity? Why don’t we make a systematic study of the programs that have come before?

10/27/22

Suppose we need to iterate over the neighbors of a cell in a 2D grid:

* * *
* - *
* * *

This can be done with the following well known pattern:

void all_neighbors() {
    for (int i = -1; i <= 1; ++i) {
        for (int j = -1; j <= 1; ++j) {
            if (i == 0 && j == 0) continue;
            std::cout << i << ", " << j << std::endl;
        }
    }
}

But what if we just want to iterate the horizontal and vertical neighbors, skipping the diagonal?

   *
 * - *
   *

Most programs explicilty repeat the operation 4 times, or use a table. But there is a trick to do it in a loop. Think of the neighbor as a vector offset, and then rotate it 90 degrees, 4 times.

template <typename I>
void rotate_2d(I& a, I& b) {
    std::swap(a, b);
    b = -b;
}

void direct_neighbors() {
    int i = 1;
    int j = 0;
    for (int k = 0; k < 4; ++k) {
        std::cout << i << ", " << j << std::endl;
        rotate_2d(i, j);
    }
}

Try it out!

int main() {
    std::cout << "diagonal" << std::endl;
    all_neighbors();
    std::cout << "square" << std::endl;
    direct_neighbors();
}

By: Justin Meiners

07/20/2022

The Quake engine provided the foundation for many early 3D games, including contributions to hit titles like Half-Life, Call of Duty, and Star Wars Jedi Knight II: Jedi Outcast. But its influence on 3D technology runs much deeper than source code. An entire generation of developers and modders was introduced to making 3D games through Quake, and the concepts it introduced can be found throughout game technology today.

Most of what’s been written about the Quake engine focuses on 3D graphics or multiplayer. And it’s hard to overstate just how many brilliant innovations the team came up with during its relatively brief development. (See Michael Abrash’s first-hand account, Fabien Sanglard’s code reviews, and, more recently, the excellent videos from Matt’s Ramblings.)

But much overlooked is the innovative Quake entity system which enables designers to craft interactive levels without writing custom code. Of particular interest is its similarity in design to UNIX. Both systems provide simple computational building blocks and a language (known as a shell) for combining them in complex ways. This system is modular for extension and is also emergent, enabling new gameplay not anticipated by the original engineers.

This article provides a brief introduction to how the system works, identifies several creative case studies from Quake, and discusses the parallels with UNIX design.

Entity basics

We will start by describing the interface for the engine, the .map file format. This is a text format describing a level. It will undergo several offline compilation steps before being loaded into the game runtime, but ultimately represents the features the engine provides to designers.

Every object in the Quake world is called an entity. An entity is a list of key-value properties. All entities have a property, classname, which specifies the entity’s kind.

The following specifies where the player will spawn in 3D space:

{
    "spawnflags" "0"
    "classname" "info_player_start"
    "origin" "32 32 24"
}

The meaning of a key, and whether it is applicable, is determined entirely by the classname. Many keys are common across a family of entity types, like spawnflags. Some keys are only useful for a particular class. For example, the light key configures brightness and applies only to the light class:

{
    "light" "250"
    "origin" "480 96 168"
    "classname" "light"
}

A .map file consists of one or more entities concatenated together. All aspects of a level, including lighting, interactive objects, and scenery, are described as entities. Thus a map editor is basically a tool for placing entities and modifying their properties.

Point entities

Let’s introduce the common entity structures. Point entities have a 3D position, specified by the origin property. For example, a monster:

or a weapon pickup:

Point entities are typically visualized as a 3D mesh. Some, like light, are nonvisual at runtime and are depicted in the editor as icons.

Solid entities

Solid entities are represented by a 3D volume. This volume is constructed from convex brushes which will be discussed later.

A moving platform is an example of a solid entity:

and so is a button:

Solid entities do not share a visual representation. The shape and size of each instance is customized for each use case.

Runtime representation

Entities that are loaded into the game all use the same entity data structure¹. This provides a uniform interface for communicating between entities and for systems to inspect them.

targetname basics

So far, .map is elegant, but doesn’t appear very special. Every game has objects that need to be placed in the world, and it’s helpful to make their properties configurable. What else would you do?! And many games have been successfully built by creating different entity types for all interactive objects in the game.

Instead, Quake adds a powerful way to combine entities. It does this through a property called targetname. This gives an entity a name that other entities can reference in one of their *target properties. For example, target means:

“When I am activated, find all the entities with a targetname matching my target and activate them.”

A targetname is not unique, so multiple entities can respond to the same action.

“Activate” is used loosely, as the exact meaning is determined by the type of entity. What’s essential is that relationships between entities can be created in the map editor. Using this system, a designer can set up novel interactive gameplay in each level without custom programming.

E1M1 Jump instructions

Let’s look at a few introductory examples.

The trigger on the left fires when the player touches it, with a wait time of five seconds before reactivating. When activated, it displays a message and plays a sound:

{
    "classname" "trigger_multiple"
    "message" "You can jump across..."
    "sounds" "2"
    "targetname" "t32"
    "wait" "5"
}

Across the gap is another trigger which fires and is removed when the player touches it. This uses killtarget to also delete the previous entity.

{
    "classname" "trigger_once"
    "killtarget" "t32"
}

The intent is to teach the player how to make their first jump. After their ability to do so has been demonstrated, the message is removed.

E1M1 Patrol path

Let’s look at a very different example where targetname is used to represent a logical linkage rather than an activation or event. Here, a path is specified for a monster to patrol until it is aggravated by the player:

A path is constructed from a sequence of path_corner entities. Each has a targetname and a target referring to the next node in the path. The target of the monster is set to the first node in the path.

{
    "classname" "path_corner"
    "origin" "-560 2352 40"
    "targetname" "t23"
    "target" "t24"
}

{
    "classname" "path_corner"
    "origin" "-104 2352 40"
    "targetname" "t24"
    "target" "t23"
}

In this screenshot, the two nodes are linked to each other in a cycle. The monster therefore repeats its walk along the path until aggravated.

E1M1 Ramp

This example shows more entities combined to create a larger system. As the player moves down the ramp, several interactions occur.

The large trigger_once on the path activates a light farther down the ramp:

{
    "classname" "trigger_once"
    "sounds" "3"
    "target" "t11"
}

{
    "classname" "light"
    "light" "400"
    "origin" "752 2000 -88"
    "spawnflags" "1"
    "targetname" "t11"
}

A func_door is used to pop up a depiction of the light source. (A “door” is just something that moves back and forth.)

{
    "classname" "func_door"
    "angle" "-1"
    "sounds" "1"
    "spawnflags" "1"
    "targetname" "t11"
}

The light reveals a nearby button.

{
    "classname" "func_button"
    "spawnflags" "2048"
    "target" "t9"
    "wait" "-1"
    "angle" "270"
}

The buttons are wired to a trigger_counter. After all three buttons are pressed, a door opens below.

{
    "classname" "trigger_counter"
    "count" "3"
    "target" "t10"
    "targetname" "t9"
}

What is an entity?

The targetname system is very simple but powerful. It transforms what would be a property editor into a primitive scripting system, by providing a means of combination², one of the essential capabilities of any programming system.

But just linking two objects together, like a door to the switch that opens it, is not enough to constitute scripting.

The breakthrough in Quake is recognizing that the ability to combine behaviors also changes the behaviors that you need to build. Instead of a suite of complex gameplay objects, you really need very simple building blocks that perform fundamental operations. These entities compose well and become reusable, enabling more applications with less code.

For example, func_door isn’t really a door. It’s an abstract conception of a door in a 3D game. It’s an object that moves from its resting position to a neighboring position and perhaps back again. Its meaning as a door depends on visuals and how it’s configured in a system. (We’ve already seen a completely different usage above.)

This line of thinking also leads to abstraction. Logical entities like gates, timers, and counters are introduced that enable machines to be built around the simple components.

One of Quake’s contributions is to invent this set of fundamental building blocks for 3D games.

Abstract entities

An odd consequence of the targetname scripting system is that many abstract entities live in 3D space, but that information is of no use to the game, and is entirely incidental. trigger_counter is one example of such an abstract entity.

But 3D placement does turn out to have some interesting benefits. Designers naturally place the entities that work together next to each other. So the entities that work together are visually associated. But their layout can also suggest how the machine operates, for example, when entities are placed in a directional sequence or arranged to lead toward an objective.

This provides a mnemonic benefit to designers who can often tell how something works just by looking at it. Similarly, debug tools can show entity systems running in-game.

UNIX philosophy

Here we start to see an analogy with UNIX. The shell is a text-based language that enables interaction with a computer through a command line and allows those same commands to be automated in scripts. The key feature of the shell is that the input and output of entire programs can be routed via pipes. One program can pass its output to be transformed by another.

How the shell works is not similar to Quake, nor is the data it interacts with. But its role is very similar. The introduction of composable programs changed how engineers conceived of programs.³ Instead of a monolithic script for solving a business problem, a program became a single logical operation, usually with just a few hundred lines of code, behind a well-defined input/output interface.

Iterating on this idea produced a core set of programs that are generally useful across many computer tasks. These programs do so little that they appear trivial and useless on first encounter, without an appreciation of their usage in larger systems. For example, head returns the first few lines of its input and discards the rest. tr replaces one character with another.

Combining these and a few more, we can find the 10 most common words in a book⁴:

tr -cs '[:alpha:]' '\n' < book.txt \
    | tr '[:upper:]' '[:lower:]' \
    | sort \
    | uniq -c \
    | sort -nr \
    | head

In a not dissimilar manner, a file downloader, an HTML converter, and a text editor can be combined to make a web browser.

Both the shell and targetname face a similar critique: their interfaces are too unstructured, leading to bugs and excessive reliance on convention.⁵ Quake lets you link entities together, but it does not guarantee compatibility, and the semantics of those links are loose.

Modularization

The UNIX philosophy enables all kinds of economic benefits through modularization. Programs can be thrown away and replaced while the system is preserved. They can be tested and qualified in isolation.

But the primary benefit is emergent applications. Programs can be useful in solving problems never anticipated by their designers⁶. And old programs that have been used for decades can suddenly be complemented by a new program that gives them new meaning and applications.

The variety and ingenuity of entity creations in Quake and the projects built upon it demonstrate a similar story.

Monism

We mentioned that at runtime all entities share the same data structure. UNIX has a similar focus on the concept of a file. It takes what were traditionally very distinct computer systems like data storage, networking, displays, and input devices, and unifies them under one read/write interface.

The benefit both systems enjoy is an elimination of special cases. Programs can be written to manipulate files and can then apply to many use cases. Similarly, Quake enables uniform operations on entities regardless of their type and despite their many behaviors.

How faithful is Quake to UNIX?

UNIX went through decades of iteration to get its core functionality. The entities in Quake are much closer to a first draft of the idea that future games attempt to refine.

One indication of this immaturity is that obviously unrelated functionality is provided by the same entity. For example, trigger_once can show a message and play a sound. A hypothetical fix would be to have separate func_sound and func_message entities, each of which could be activated by the same trigger.

An obvious way in which this increases flexibility and reuse is that a trigger can now play multiple sounds, and each entity that plays sound doesn’t duplicate that work. This specific correction was made in Valve’s Source Engine.⁷

Combining multiple functions is tempting for convenience, but weakens the modularization benefits of the core design. This flaw can also be seen in immature UNIX systems. Programs gradually expand with flags to modify program functionality, often with viral impact.⁸ Eventually, this leads to a clarification of responsibilities and a split into separate parts.

The game world

So far we’ve focused on the interactive and logical capabilities of entities. Let’s now look at how entities construct the levels themselves, including the structure and scenery for the gameplay.

worldspawn

Each map contains exactly one instance of the worldspawn class, a solid entity that describes the 3D shape of the level. worldspawn is also a “catch-all” for configuring the general setting and appearance of a level, such as the ambient level of light.

As established before, a .map file consists only of entities, and they follow the same basic schema. But the engine is free to treat them specially.

worldspawn is such an entity with a special role. The level geometry needs to be preprocessed into an efficient representation for visibility testing, rendering, and collision. light entities are also closely related to worldspawn. High-quality lighting data is precomputed for the level geometry, so most light entities are removed prior to game runtime.

All of this results in static scenery, a tradeoff necessary to meet performance requirements. This special treatment is an implementation detail that is hidden from the .map interface.

Brushes

The following explains in detail how the 3D shape of a solid entity is specified. A brush is the enclosed space of four or more planes (“oriented half-spaces” for math folks). For example, a cube is a brush constructed from six planes.⁹

Each plane is defined by three points lying on it, along with a few texture parameters:

( 256 64 16 ) ( 256 64 0 ) ( 256 0 16 ) mmetal1_2 0 0 0 1 1

Below are a variety of brush shapes from Quake:

Brushes have a few nice properties:

• They can be converted into a 3D triangle mesh for rendering.¹⁰
• Texture coordinates can be generated for them easily.
• They simplify physics calculations, such as collision detection.

Brushes can be drawn right in the level editor, so a designer can quickly tailor the shape of an entity or sketch a space.

Brushwork

Brush-based modeling is much less common in games¹¹. So let’s review a few examples from Quake to understand how worldspawn works.

E1M4

The archway is constructed from wedge slices that linearly approximate a curve. Note the irregular cut to give the ground around the door an organic appearance.

E1M5

This building makes heavy use of rounded columns and archways. Notice the many shim pieces needed to fill the gaps. The trim pieces around the column give a surface for an alternative texture.

E4M3

This outdoor area shows brushes being used to model small objects, rather than architecture. The grave bump is also a nice effect that disguises the flat shape of the grass.

Separation of view and model

Video games rely heavily on visuals, sound, and atmosphere to provide context and meaning for player interactions. The simulation itself can be very crude. What the player experiences as a medieval dungeon filled with knights is understood by the computer as a few boxes in space.

The Quake entity system leverages this idea by separating the behavior of an entity from its visual depiction, like the classical view/model distinction in programming. This enables more reuse of entities and flexibility for designers, rather than combining the behavior and visual representation in one game object.

Solid entity customization

The first way it does this is through the brush system. Solid entities like doors, buttons, and trains are drawn for each use case with no programmatic functionality changes. So the doors all look different and can take on new shapes.

E4M7 doors

A few creative entity depictions were already shown. Here is one more where a func_door pulls apart interlocked bars:

Invisible entities

Another way this is done is by simply placing an invisible entity over a visual object, effectively granting it that behavior. Let’s look at a few of these.

START difficulty selection

The game difficulty is selected by jumping through a portal¹².

The portal itself is static geometry with an animated center texture.

Placed on top of the model are an invisible trigger to set the difficulty and an invisible teleporter to move the player into the next room. These rest in front of the model.

E4M4 Healing pool

Near a small pool of water, the game announces, “You found a healing pool!”

The pool is constructed as a plane in the floor with a water texture. But underneath the surface are many health packs that cannot be seen.

An invisible trigger that displays the message and plays an ambient sound completes the effect.

Here we have a familiar entity, but by making it invisible we can give it a new visual that gives the player a fresh experience. And most players are completely unaware the pool is only finite!

Advanced entities

With an understanding of the entity system, let’s look at a few case studies from Quake that leverage all parts of the system to create memorable experiences.

E1M7 Lightning puzzle

Chthon, the boss, can be defeated by zapping him between two lightning rods.

Each rod is tied to a nearby button, with a third button used to activate the lightning.

(Notice that there are actually four buttons, but only two are ever selected, depending on the difficulty level.) The lightning effect itself requires some in-game hard-coding to position itself, but otherwise is an example of the entity system creating new gameplay mechanics.

E1M3 platforms

The player is locked in a room with the ceiling descending. Previously, they encountered similar rooms that could squash them. At the last second, the ceiling separates into two pieces, and the floor lifts the player up to the end of the level.

The ceiling is constructed from two separate func_train entities. A path positioned at the corners of each entity guides it through the movement.

The floor is actually a func_door. Its lifting motion is activated by a trigger_once timer with a wait time designed to coincide with the train movement.

E1M7 fireworks celebration

After Chthon is defeated, the level ends, and in the background, gibs rain down on the arena. This effect is built entirely with the entity system.

Completing the level activates a network of entities in an offscreen room.

A sequence of timers starts counting down, wired to activate an accompanying set of teleporters. These continue to fire as the world simulation continues independently of the player’s existence.

On top of each teleporter are two monsters that will be destroyed immediately after being transported. (Destruction is a consequence of two entities trying to teleport into the same place.)

This example represents a machine or program made with Quake. It combines the logic of the targetname system with the physical consequences of the world simulation. And this room exists entirely offscreen! It’s literally a black box whose internal operation and 3D existence are incidental to its function.

06/25/2022

Do you need a mesh adjacency structure? Are you considering implementing half edge or bmesh? Here is a trick to consider first, that may save you some energy.

Take your mesh’s indices array and reverse it:

// f -> v
uint32_t *index_to_vert = indices;

// v -> f:
multimap<uint32_t, uint32_t> vert_to_index;
for (size_t i = 0; i < index_count; ++i)
{
    vert_to_index.insert(make_pair(index_to_vert[i], i));
}

This multimap can handle most mesh traversal queries efficiently, such as:

// return number of faces a vertex belongs to
size_t degree(uint32_t vert)
{
    auto range = vert_to_index.equal_range(vert);
    return (size_t)(range.second - range.first);
}

Details

So what’s going on here? The original indices can be thought of as a map from faces to vertices (technically 3 indices for each triangle, but that’s the idea).

This new multimap goes in the opposite direction. It maps vertices to faces. Because a single vertex may be shared between faces, the map does not have unique keys (hence multi-).

It is a little more cumbersome to write certain operations. But this is mostly a matter of writing a few good helper functions. Let’s look at one more:

vector<uint32_t> neighboring_faces(uint32_t face_index)
{
    vector<uint32_t> neighbors;
    for (uint32_t f = 0; f < 3; ++f)
    {
        // visit vertex in this face
        uint32_t vert_index = indices[face_index * 3 + f];

        // find attached faces
        auto range = verts_to_faces.equal_range(vert_index);
        for_each(range.begin(), range.end(), [](auto i) {
            uint32_t neighbor_face_index = i / 3;
            if (neighbor_face_index == face_index) return;

            neighbors.push_back(neighbor_face_index);
        });
    }

    return neighbors;
    // TODO: remove duplicate results
    // TODO: you want to allocate a new vector for this. It's for illustrative purposes.
}

What are its limitations?

This data structure is an easy to implement solution for traversing a mesh. But if you need to edit the mesh, it usually won’t help. Addding or remove a mesh element typically requires rebuilding part of the multimap, which is inefficient, and tedious. As Fabien describes, it’s hard enough to maintain the invariants of a 3D mesh during editing, without this extra work.

Some editing algorithms may skim by. One technique is to use the multimap to visit the mesh, marking which faces and vertices to delete, and then throwing away the multimap and applying the changes.

It’s probably less efficient than a half-edge mesh, which maintains direct pointers. But the algorithmic complexity should be the same.

No manifold requirements

This technique has another advantageous property that’s worth calling out. Adjacency structures typically impose additional mathematical requirements on the mesh structure. These can be tedious to verify, and hard to fix if the mesh wasn’t constructed with them in mind. For example half-edges only work if the mesh is a 2-manifold with boundary, consequently edges have at most 2 faces attached.

But the multimap has no such requirement! It can be constructed for any indexed triangle soup!

One caveat to is that several mesh processing algorithms still assume similar requirements, so watch out for that.

What if you can’t use multimap?

The C++ std::multimap is a pretty complex data structure on its own, and most standard libraries don’t have an equivalent. So is it just hiding all the complexity? No! You can implement a cheap version using a sorted array and binary search.

The construction is very similar, but we sort at the end:

uint32_t *index_to_vert = indices;

using ReverseIndex = pair<uint32_t, uint32_t>;
ReverseIndex* vert_to_index = new ReverseIndex[index_count];

for (uint32_t i = 0; i < index_count; ++i) {
    vert_to_index[i] = make_pair(index_to_vert[i], i);
}

sort(vert_to_face, vert_to_face + index_count);

Note that index_to_vert and vert_to_index have the same size.

The purpose of sorting is to enable binary search (see lower_bound, upper_bound, and equal_range). The cost of a binary is O(log(n)) which in practice is not much different than O(1) (you can binary search a billion records in 32 steps).

Here is an example:

// compute a smooth vertex normal
vec3 average_normal(uint32_t v0)
{
    auto key = make_pair(v0, 0);
    auto i = lower_bound(vert_to_face, vert_to_face + n, key);

    vec3 n;

    while (i != vert_to_face + n && i->first == v0)
    {
        uint32_t face_index = (i->second) / 3;
        n += face_normals[face_index];
        ++i;
    }
    return n.normalized();
 }

Hope that helps!

I have one other reason for liking this simple array representation. In mathematical terms functions are sets of pairs. Indicies is a subset of the cartresion product F x V. The reverse index is build by swapping pairs from this set construction. This new set is a subset of V x F and weakens from function to a relation.

Literate programming is much more than commenting code

03/07/22

Most programmers haven’t heard of literate programming, and the few that have tend to think it means writing a lot of comments around your code. However, simply explaining code with prose, does not constitute literate programming. Consider the following quote from it’s inventor:

Some of my major programs could not have been written with any other methodology that I’ve ever heard of. The complexity was simply too daunting for my limited brain to handle; without literate programming, the whole enterprise would have flopped miserably. … Literate programming is what you need to rise above the ordinary level of achievement – Donald Knuth

Does this extraordinary leap in effectiveness sound like something that comes from re-explaining what code does? I didn’t think so.

The real advantages of literate programming come from two principles:

1. Code should be written for humans not machines.

2. Program designers utilize many types of information, not just code.

Literate programming systems help programmers apply these principles by providing two standard operations tangle and weave which manipulate and organize source code.

Tangle

Literate programs are written in plain text documents, but between the traditional prose, blocks of code (in any language) are interleaved. For example:

console.log( subsets([1, 2, 3, 4]) );

You can think of them like markdown documents, but with an important addition. In a literate program, blocks of code have identifiers, and they can reference each other. Each reference modifies the contents of a prior block, or includes it at the specificied location.

The operation tangle resolves all these references and produces normal source code to be read by the compiler.

Example

Let’s try it out. In the block above we introduced a subsets function, also known as the powerset:

Implementing it can be a little bit tricky. If you have never done it before, I recommend you write out some examples on paper and see if you can figure it out.

The key insight, is that if we take an element out of the original set, and all the subsets which contain it, we can get back the full subsets, by making a copy of each other subset and inserting the element back in. In other words, the number of subsets doubles with each element in the set:

Here is rough outline in code:

function subsets(set) {
    @{base case}
    @{remove element and compute subsets of smaller set}
    return @{subsets without element}
        .concat( @{subsets with element} );
}

@{test subsets}

This a recursive function, so we need a base case. If the input set is empty, its only subset is the empty set:

if (set.length == 0) { return [[]]; }

We use recursion by removing one element and computing the subsets of that array:

const element = set.pop();
const smaller = subsets(set);

All the smaller subsets are still subsets of our original set, so we want to include those.

smaller

But, we are missing all the subsets involving element. We simply add element in to each set.

smaller.map(s => s.concat([element]))

That’s all! Applying tangle to the literate file produces a single source file subsets.js.

A typical literate file produces many source files. Block types aren’t limited to a single language either. Here is how to run it in your command line:

node subsets.js

Discussion

Of course, the example above goes into more detail, and is separated into more blocks, than is helpful for most production code. It is just meant to illustrate how a little explanation, analysis, and division into logical parts, can make code significantly more readable.

At the micro-level we can overcome a lot of limitations imposed by the language syntax, kind of like a powerful macro system. In the example we break out small sub-expressions, even from the middle of statement, to explain further, or to make things more readable.

Some other use cases include:

• Separating inline SQL queries from string literals.
• Avoiding forward declarations in C.
• Breaking up large switch cases.
• Separating error handling from main logic.

More importantly, at the macro level we can control the overall presentation of the program. This is difficult to convey in a brief article, but think of it like writing code as a book or article. You can introduce your code with an overview, take time to develop the most important ideas and characters, gradually build up to the big picture, and then maybe fill in some extra details.

Consider the last time you started working on a large code base someone else wrote. It was probably overwhelming, especially considering how much of modern code is boiler plate, and interfacing between systems. You probably wanted to know:

• What code is most important and should be read first?
• What are they key data structures and algorithms?
• How is data stored?
• Which files are boiler plate that I can ignore?

A well-written README can point you to some files to get started, but it won’t isolate the core concepts from all the other code details surrounding them. Literate programs allow you to answer these questions naturally.

Weave

The weave operation generates documentation files from the literate file. This is a nicely formatted document like the one you are reading right now! This presents the code in a readable form,

If you are like me, you probably maintain some kind of design document recording decisions make during the project and data to provide justifications. Not, all of this content is appropriate to include in the source code, but a lot of it is, and the literate weave operation let’s you do just that.

Here are some examples of media you can include, as well as a specific use case I have desired in the past:

• Diagrams and images (instead of the ASCII kind). Illustrate a suite of test cases.
• Graphs and plots. Illustrate what a function looks like.
• Math formulas and equations. Analyze the complexity of an algorithm.
• Tables of data. Record survey data that justifies a decision made in code.
• Screenshots. Show a bug which is fixed by a particular section of code.
• Example inputs and outputs. Show how to use a pieces of code.

It’s important to note how this is different from tools which generate interface “documentation” from source code. Literate programs usually include the entire source, not just interfaces. Since it focuses on prose, it’s much more natural to include other media formats. You aren’t littering comments with awkward media references.

Getting started

I have told you a bit about why I like Literate programming, but to really understand how it benefits your code read some examples, or better yet, try it on your next project. To get started you will need a literate programming tool of which there are many.

srcweave

I recently wrote srcweave which was used to write this article you are reading now. See my article “Write your own Proof-of-Work Blockchain” for another example.

Literate

Literate is the system I used for several years, before writing my own. It is self hosting, which I find pretty cool, so you can read it’s own source code.

web

Donald Knuth wrote the original literate programming tools web and cweb. Of course, they output TeX instead of HTML for documentation. I don’t know anyone else who actually uses his tools, since TeX is more involved than most people need, but they are still worth learning from. The source for TeX is written in it, and is famous for having very few bugs and being extremely reliable.

3/05/22

Do you want a Common Lisp workflow like Python or Perl? No dumping images, or complex deployments. Just install a single lisp interpreter on your computer (/usr/local/bin/sbcl) and write, run, and share source files. I’ve done a lot of research and I’m here to explain how to get exactly that.

A step in the right direction is a .lisp file with an appropriate shebang:

#!/usr/bin/sbcl --script
(write-string "Hello, World!")

It can be run with

$ ./hello.lisp

But you’ll quickly run into roadblocks after “Hello, World!”. How do you include other source files without knowing their installation path? How do you load libraries?

The solution is a Quicklisp command for downloading libraries and exporting them called ql:bundle-systems. It creates a single folder with an index file called bundle.lisp. After running (load bundle.lisp), all of your dependencies are available to be load via asdf:load-system. The result is entirely self-contained.

Example using ql:bundle-system

1. Create a bundle containing your Quicklisp dependencies:

    (ql:bundle-systems (list "alexandria" "cl-ppcre" ...) :to "bundle/")
   

   This bundle/ folder will contain alexandria and cl-ppcre, along with their transitive dependencies. Each package will be organized as its own an asdf system.

2. Create an .asd file describing your project and its dependencies (see defsystem docs for details).

    (asdf:defsystem "my-lisp-program"
        :depends-on (:alexandria :cl-ppcre ...)
        :build-pathname "my-lisp-program"
        :components (
            (:file "...")
            ...))
   

   The :depends-on list must only reference systems which were downloaded by ql:bundle-systems.

3. Copy your application source into the bundle:

    $ mkdir -p bundle/local-projects/my-lisp-program
    $ cp *.lisp bundle/local-projects/my-lisp-program
    $ cp *.asd bundle/local-projects/my-lisp-program
   

   Now your code is just another asdf:system, alongside the others.

4. Install the bundle in a global system path. I prefer /usr/local/lib:

    $ cp -r bundle/ /usr/local/lib/my-lisp-program
   

5. Create a brief script which loads the bundle.lisp and the asdf system. For example my-lisp-program

    #!/usr/bin/sbcl --script
    (load "/usr/local/lib/my-lisp-program/bundle.lisp")
    (asdf:load-system "myapp")
    (my-lisp-program:do-stuff)
   

   asdf:load-system will examine your projects .asd file and load its dependencies, transitively.

6. Install the launch script in a global system path:

    $ mv my-lisp-program /usr/local/bin/
   

7. Run your script!

    $ my-lisp-program
   

That’s all! Consider automating these steps for your project, such as with make install.

The first time you run the script you may see a long log as sbcl compiles the code into .fasl files, for faster subsequent runs.

Further guidance

I use this approach in srcweave. Take a look at the make file for a real-world example.

Why other solutions didn’t work for me

There are a myriad of articles that claim to solve this problem. But after much research and trial and error, I found none of them to be satisfactory. Let’s review them:

Dumping an image

The popular advice for deploying Common Lisp is to dump an image of the compiler (save-lisp-and-die),

This makes a copy of the entire Lisp compiler with your source code included. This is a good solution for large applications. For example, if you want to deploy a web application to server, it’s pretty convenient to scp up an image file and be done.

However, it’s quite cumbersome if you have more than a handful to keep track of. Your computer becomes littered with a bunch of independent copies of Lisp. If you want to update sbcl you need to track down all your old images, delete them, and rebuild new ones. Images files are also notoriously large (~50 MB for sbcl).

Using Quicklisp in scripts

Can you make a shebang script that loads code with Quicklisp? You certainly can, but first note that sbcl --script will skip loading your system configuration (.sbclrc), so you need to hard code a path to load Quicklisp.

But now imaging if calling import in a Python started downloading code from the internet! That’s a security and reliability nightmare. But that’s how Quicklisp works!

That’s because Quicklisp isn’t intended to be a library loader. It’s a tool for downloading and discovering libraries. But it shouldn’t be distributed in your source code.

You should be using asdf for that instead. It’s the standard way to describe and load libraries, and it’s already included in most distributions.

Note that Quicklisp is also organized as a rolling release like your operating system. Rather than picking out individual library version tags, you pick a Quicklisp version which is a snapshot in time of all the libraries that aims to be compatible.

To reiterate, I will share an explanation from Reddit user eayse:

I advocate the habit of installing things with ql:quickload, but after initial installation, using asdf:load-system to actually bring the systems into memory. After all missing dependencies have been satisfied by network installations from the distributions configured in Quicklisp, ql:quickload just thunks down to ASDF.

Roswell?

Roswell is advertised as a solution, but it’s possibly the most un-Lispy tool in the ecosystem. It doesn’t download dependencies, or help you build them. What does it do? Help you download sbcl? Why in the world does it use a config file syntax that isn’t s-expressions? Why is part of it written in C? sigh.

Busybox style

The overhead of dumping an image can be shared by putting many scripts into a single image, instead of making an image for each. I call this “Busybox style” because it was popularized in C by the Busybox project. For a Common Lisp tutorial, see Steve’s article.

This certainly solves some of the disk usage problems. But it’s kind of a crutch, and the workflow is just not as good as Python.

By: Justin Meiners

The halting problem is to devise a method that can determine whether a given Turing machine terminates in a finite amount of time. It is well-known to be undecidable; no such method exists. Naturally, if you ever write a proof that implies how to solve it, there is reason for concern.

In doing research for my master’s thesis, unfortunately, I did just that. It took quite a bit of reading and discussion to detect what went wrong. It turns out I didn’t make any major mistakes, but I learned that the difference between computability and incomputability is quite subtle. I wanted to share this insight into what it means for a function to be incomputable.

My thesis focused on braid group algebra, and reasoning about how a particular property of braids can be computed. Since this is a specialized topic and the details aren’t important, I will explain a similar problem using Turing Machines directly.

Turing machine

A Turing machine consists of:

• An infinitely long tape, divided into cells. Each cell has a symbol written on it. This symbol comes from a finite alphabet S (often { 0, 1 } where 0 can be called “blank”).
• A read/write head positioned over a particular cell on the tape.
• A controller that moves the tape and reads and writes symbols. The controller is a finite state machine Q.

As a Turing machine runs, the read/write head sends the current symbol to the controller. The controller can then:

1. replace the symbol.
2. transition to a new state.
3. shift the tape by a finite amount.

Which symbol to write and which transition to make is determined entirely by the controller’s state and the symbol that was read.

For answering questions about computability and complexity, it doesn’t matter how many symbols the alphabet has or how complex the finite state machine is.

Estimating the number of operations

How do we go about determining whether a Turing machine will halt? Here is the basic idea: run the machine for a large number of steps. If it runs for longer than we expected, it probably won’t terminate!

Ok, that’s obvious and not very helpful. “probably won’t” isn’t good enough. The actual difficulty is that the machine might run for longer than we expect and still terminate. We might just need to run it a little bit longer. But, what if we could determine exactly what that threshold is? In other words, what if there was a specific number of steps that any terminating machine would not exceed? It might seem like there is no way to determine such a bound, but it is actually pretty straightforward.

We can estimate how long a Turing machine will run based on its complexity. In computer science, we often think longer programs are more complex. Similarly, a Turing machine with a lot of information on the tape has a “long program” to run and work with.

A Turing machine has only finitely many non-blank squares. For a fixed state machine Q and alphabet S, define A(n, Q, S) to be the set of Turing machines in Q and S that:

• halt
• have exactly n non-blank squares. (n can be thought of as an estimate of program length).

There are at most |S|^n machines of this kind; therefore, A(n, Q, S) is a finite set. If we then let l(T) be the number of steps machine T runs for before termination, { l(T) : T in A(n, Q, S) } is finite and thus has a maximum M.

So given any Turing machine with this particular controller and configuration, we can run it for M steps, and if it goes longer, then we know it will never terminate! We can even define a function f_{S, Q}(n) that is the approximate M value for our inputs! In fact, we don’t care about f in particular; ANY function that is strictly larger than f will do.

Incomputability

So why can’t we do this? Well, to know when to stop running, we would actually have to be able to compute the bound f in the real world. In other words, we need an effective procedure or algorithm for computing f. Since there is no way to solve the halting problem, the only conclusion is that f grows faster than any function that can be computed in the real world. Even though we can write programs to compute functions such as polynomials and exponentials, none of those grow fast enough. Perhaps we could compute a lookup table to store this bound, but that would also require infinite space.

This shows us that uncomputability is primarily a problem of growth rates, not any kind of tricky function definition. Some functions just grow too quickly to be computed by a machine.

Note that the same limitation occurs for less sophisticated models of computation. The Ackermann function is a classic example of a function that requires the full power of a Turing machine to compute. This is due to its fast rate of growth.

References

Minsky. Computation: Finite and Infinite Machines. Chapter 8.

By: Justin Meiners 06/14/2021

Introduction

Swift’s reduce can be used to accumulate elements in a sequence. For example:

[1.0, 2.0, 3.0, 4.0].reduce(0.0, +)

If we visualize the expression tree for this operation it’s essentially linear. You can think of it as a binary tree which is completely unbalanced.

 -----3.0----6.0---10.0
/     /      /    /
1.0   2.0   3.0  4.0

Assuming the binary operation is associative then we can choose which order to evalulate arguments in. For example, we could image a function balancedReduce which rearranges the order of evaluation to the following:

      10.0
    /      \
   /         \
  3.0       7.0
 /  \      /    \
1.0  2.0  3.0  4.0

We haven’t changed the result at all, just the order it’s evaluated in. It’s mathematically equivalent. However, there are lot’s of good computational reasons why one might want to do this.

Example Applications

These are just a few to illustrate it is a general problem:

1. Adding up many floating point results (as in my example above). Adding small numbers to larger numbers produced poor numerical results. Given a long enough list of values of similar magnitutdes this is guarenteed to happen.

   In contrast a balanced evaluation of such values will only add similarly sized values. (Intermediate results which are the result of adding equal numbers of values together).

2. Merge sort can be written by applying the binary operation of “merge sorted list” to a sequence of lists:

   [ [a1], [a2], ... [an] ]
   

   If you do this with reduce it is essentially insertion sort, one element is inserted into the growing list at a time. If you use a balancedReduce it will only merge lists that are roughly the same size, giving a proper O(n log(n)) merge.

3. It can be used to solve the problem of finding the smallest element, and second smallest element, in a sequence in the optimal number of comparisons. (Described in 5.3.3 of Volume 3 of Art of Computer Programming).

Proposed changes.

This capability can be added with a simple extension to Sequences of Equatable types.

extension Sequence where Element: Equatable {
    func balancedReduce(
        zero: Element,
        operation op: (Element, Element) -> Element
    ) -> Element{
        var counter = Array(repeating: zero, count: 32)

        for x in self {
            let carry = BinaryCounter.add(to: &counter,
                                          x: x,
                                          zero: zero,
                                          operation: op)

            assert(carry == zero, "reduced too many things")
        }

        return BinaryCounter.reduce(counter: &counter,
                                    zero: zero,
                                    operation: op)
    }
}

struct BinaryCounter {
    private init() {}

    static func add<C: MutableCollection>(
        to counter: inout C,
        x: C.Element,
        zero: C.Element,
        operation op: (C.Element, C.Element) -> C.Element
    ) -> C.Element where C.Element: Equatable {

        var i = counter.startIndex
        let end = counter.endIndex

        var carry = x
        while i != end {
            if counter[i] == zero {
                counter[i] = carry
                return zero
            }

            carry = op(counter[i], carry)
            counter[i] = zero
            i = counter.index(after: i)
        }

        return carry
    }

    static func reduce<C: MutableCollection>(
        counter: inout C,
        zero: C.Element,
        operation op: (C.Element, C.Element) -> C.Element
    ) -> C.Element where C.Element: Equatable {

        var i = counter.startIndex
        let end = counter.endIndex

        // find first non-zero
        while i != end && counter[i] == zero {
            i = counter.index(after: i)
        }

        if i == end {
            return zero
        }

        var x = counter[i]
        i = counter.index(after: i)
        assert(x != zero)

        while i != end {
            if counter[i] != zero {
                x = op(x, counter[i])
            }
            i = counter.index(after: i)
        }

        return x
    }
}

The implentation is very efficent, adding very few additional computaitons and using an additional piece of memory, roughly of size log(n).

Application examples

This implementation of merge sort is just for fun to show how it works.

func mergeSorted<T>(
    _ a: [T],
    _ b: [T]
) -> [T] where T: Comparable  {
    var i = a.startIndex
    var j = b.startIndex

    var out: [T] = []

    while i != a.endIndex && j != b.endIndex {
        if b[j] < a[i] {
            out.append(b[j])
            j = a.index(after: j)
        } else {
            out.append(a[i])
            i = a.index(after: i)
        }
    }

    if j != a.endIndex {
        out.append(contentsOf: b.suffix(from: j))
    } else {
        out.append(contentsOf: a.suffix(from: i))
    }

    return out
}

func mergeSort<S: Sequence>(
    _ items: S
) -> [S.Element] where S.Element: Comparable {

    let singletons: [[S.Element]] = items.map { [$0] }
    let zero: [S.Element] = []

    return singletons.balancedFold(zero: zero, operation: mergeSorted)
}

Questions/Concerns

It’s too theoretical

All of the theory is hidden inside, outside it looks like just fold/reduce. If you don’t know what it does, it still gives expected results.

Furthmore, it can be benefecial even if you don’t copmletely understand it. For example, advice could be given to “use this one to add up large lists of float” and it would benefit them.

The zero parameter is weird. Should it be nil instead?

I am unsure. My assumption was that introducing nil in the intermediate computation adds a lot of boxing/unboxing.

The name is bad/wrong

I am not attached to the name and happy to hear recommendations.

Can we make the 32/64 parameter less arbitary?

Of course 2^64 is the max of a 64 bit integer. Arrays of more than 2^64 elements are probably already disallowed. That’s assuming every element is a byte, so 2^32 seems like a reasonable upper bound assumption for the size of a sequence.

It could be a parameter, although this seems like a messy detail to expose.

References

I learned about this techinque from Alex Stepanov, the author of C++ STL. It is described in the book “Elements of Programming” chapter 11.2.

There is a related concept in C++. std::accumlate does a normal left fold, std::reduce is allowed to associate its arguments in any order it prefers. It additionally requires the operation is commutative, which I do not think is necessary for this. It appears std::reduce is primarily to facilitate parallel computation, but it demonstrates two similar operations whose results only differ on their association order.

5/3/2021

Stock markets are at tremendous highs, not only in absolute price, but also in terms of earnings yield. At the same time we see unsophisticated participants taking incredible risks in Robinhood, Gamestop, Dogecoin, and Tesla, indicating a public excitement alongside the financial metrics. So is the top in?

Seasoned passive investor should be amused by this question. It’s merely commotion for financial news. It matters for day traders and speculators. But, by passively holding broad market index funds for a long time horizon, we don’t need to worry about such things. Someone will always insist “this time is different”, but we just need to stay the course and ignore the market swings. Right?

Broadly, this message is correct. But, safe activities become dangerous when we become too comfortable. Just as we see recklessness among traders, a different brand of recklessness appears to be sneaking into passive investing.

For example, in the market decline around March 2020, due to the spread of COVID-19 and the associated lock downs, most Vanguard investors purchased additional equities rather than selling or acquiring less volatile assets. Buying when others are fearful is the adage, but if everyone is buying, the consensus is actually optimistic. Either:

1. Investors didn’t really believe economic effects would be that bad.
2. Investors discounted the negative effects years into the future and rebalanced out of cash.
3. Investors believed stocks would go up, regardless of negative economic effects.

Even among skeptics 1. wasn’t common during the first few weeks of uncertainty. Many investors held belief 2., but considering savings rates and that younger people need capital in coming years (such as for down payments on homes) its hard to believe the majority are prepared for their portfolios to be below par for 5 to 10 years.

This is concerning because those in camp 3 believe they can’t lose. No matter what happens they will make money in a time frame usable for them. This isn’t what passive investing teaches, and is instead based on passive myths.

Myth #1: Asset prices don’t matter.

New traders learn that “buy high” “sell low” is a lot harder than it sounds. It’s not just hard to find opprotunities, psychologically people seem biased towards losing money. Passive investors observe teach “time in the market” tends to beat “timing the market”. Instead, the passive investor focuses on accumulation of assets over the long term rather than trading price trends.

This is sound investment advice, but many investors have concluded that this means price doesn’t matter. We should pay any price, because no matter what the index will go up and we will eventually get our returns. (If you believe this, can I sell you some stock?)

Let’s make an axiomatic observation. The gross yield of an investment is:

(price sold + dividends recieved) / price purchased

(Inflation and time discounting proportional are proportional modifications). Thus, increasing purchase price, lowers returns. Therefore, price purchased determines your returns, so it cannot be ignored. Consequently we should always try to pay the lowest prices possible.

So is it time to start reading charts? No. Markets tend to do well at pricing assets (especially on public exchanges), and its difficult to predict where price will go. Purchasing at regular intervals (of any duration) at market prices, such as dollar-cost averaging, is an effective strategy to pay fair prices, and avoid extremes which may occur for short periods.

Rebalancing between assets at regular periods is another way to “buy high” and “sell low”. Adjusting asset allocation in response to extreme price moves is not day trading.

Revision: Always get the best prices you can and don’t overpay. Dollar-cost averaging is a strategy for getting good prices.

Myth #2: Buying broad market indices is the optimal investment strategy.

Passive investing is based on the idea that large markets typically price assets efficiently. Modern portfolio theory attempts to describe how to earn returns in such an environment, and is supportive of passive investing. It argues that assets are primarily distinguished by their level of risk and time horizons. Long term risky investments are cheaper than short term sure bets. Since the market understands how to price levels of risk, there is no free lunch.

Picture each ETF or mutual fund (such as an S&P index) lying on a graph of risk to return. Given this model, its absurd to think any one investment gives the greatest possible returns. Unless its the riskiest asset available, one can always shift down the risk curve and find something with higher returns. If S&P is a sure thing, wouldn’t leveraged S&P be even better?

The real question is how an index performs relative to other assets with the same level of risk. The effectiveness of popular indices like S&P as benchmarks for other assets, suggests it performs well. But, you it would still be hard to argue that it is the best for all time.

An enormous advantage to buying an index is that is easy and cheap. That’s worth a lot, even if a complex combination exists that is slightly better.

Revision: Passive investing can offer good returns relative to risk levels and is easy.

Myth #3: It’s impossible for an investor or fund to beat broad market indices.

When a Boglehead is shown a fund that outperforms a popular index, they immediately conclude it must be fraudulent, or soon headed for disaster. We just learned that one possible explanation is that they are simply taking on more risk. But another is that the fund is run by people with expertise, putting in time to squeeze out extra risk/return efficiency.. After all, we believe markets are efficient because there are experts of varying beliefs scrutinizing all kinds of information.

Let’s imagine five of your friends are indepdendtly starting restaurants and seek your investment. You are going to invest in a few of them. They offer to present their finances and business plan to you. Do you think it would be worth consider that information, or would you be better off just picking them randomly?

You can probably think of some obvious information that would be much better than random. Have any of them run a business before? Were they successful? How much money do they have? Is anyone else helping them? It may turn out your picks are unlucky, or others are successful underdogs, but its obvious at the small scale that its possible for you to make some judgment about their investment quality.

Professionals can ask the same kinds of questions about large companies. They can study business and corporate leadership, accounting practices, and perhaps having run businesses of their own. So they should be able to identify good businesses from bad, or at least eliminate the worst contenders.

This is made complicated by the context of a global economy, with so much uncertainty, and interdependence. But, the principle is no different. It’s possible for a person to reason about whether a business will succeed or not. If you accept that principle, then you can expect some professionals might make investments better than random, even if it takes a lot of effort and skill to obtain much of an edge,

Whether these kinds of people work at any of the high-fee funds in your company 401k is a separate question. As a matter of practicality, it may not be worth looking for them. But good investors almost certainly exist. There will be funds that outperform passive funds. That is entirely OK. The idea of passive investing is that its easy and cheap. You can beat most investors just by purchasing a broad market index. Isn’t that fantastic?

Revision: Professional investors participate in pricing assets. They can often make additional money through this activity and expertise. Passive funds are easier and cheaper than most publicly available investment products.

One commonality among these myths is that they imply you no longer have to think. Prices are always correct. Stop thinking about whether you’re over paying and buy. Stop thinking about whether an investment is worthwhile or if you should consider, better alternatives. The market is just unknowable, so you might as well not try. That doesn’t sound anything like shrewd investing, it sounds like marketing.

If we accept the revised statements there is endless opportunity for learning and improvement. What are good strategies for purchasing and rebalancing? Are there other funds I should consider in my portfolio? Are there investment opportunities in my local community? All of these questions are worth considering slowly and methodically.

12/27/20

Part 2 of AOC Day 18 asks you to evaluate some math expressions, but with a twist. + has a greater operator precedence than *. (This is opposite of how we typically read math):

2 * 3 + (4 * 5)
2 * 3 + 20
2 * 23
46

The standard method for expression evaluation is the shunting yard algorithm. This is the right and efficient way to solve this problem. But, it’s also not very obvious or natural. It’s hard to imagine how you would think if it yourself, and doesn’t really match how people actually see mathematical expressions.

I want to show off an alternative solution demonstrating how you could think of it yourself. My code is written in Common Lisp. For this particular problem this choice goes a bit deeper than just syntax as the solution makes heavy use of a symbolic expression manipulation which is harder to do in most languages.

Thinking about Precedence

What does operator precedence actually mean? It tells us the order in which to evaluate operators. So naively we should just do all the additions first, and then the multiplications. Can we program that? We certainly can, but doing just one operation at a time, means at each step we produce an intermediate result which is itself a math expression, just simplified a bit. For example given the expression:

 (1 + 2 * 3 + 4)

We first evaluate all the + to produce:

(3 * 7)

And then we evaluate all the ‘*’ to produce:

(21)

After evaluating every operator, we should get a list containing a single number.

Below is some code which does just that. The input chain is an expression (list of numbers and operators) and allowed is the operator we want to evaluate.

(defun simplify-chain (chain allowed)
  (prog ((result '())
         (op nil)
         (l nil)
         (r nil))

        (setf l (car chain))
        (setf chain (cdr chain))

        loop
        (setf op (car chain))
        (setf chain (cdr chain))

        (setf r (car chain))
        (setf chain (cdr chain))

        (if (eq op allowed)

            ; evaluate the operation.
            ; put the result back in the expression

            (setf l (funcall op l r))

            ; not an operation we want to evaluate right now.
            ; put it's symbols back in an the expression
            (progn (push l result)
                   (push op result)
                   (setf l r)))

        (if (consp chain)
            (go loop))

        (push l result)
        (return (reverse result))))

Handling Parenthesis

That is quite straightforward! All the trickiness is fiddling with infix. But, there is one problem. Due to parenthesis the operands may not be “ready” to evaluate. If we have:

3 + (2 * 4)

We can’t apply the + operation until the (2 * 4) is taken care of. But evaluating (2 * 4) is something we could do directly. There are only finitely many nested statements, so eventually following parenthesis must end with a list containing only numbers and operators. This is clearly a recursive problem! We need to evaluate all the parenthesized statements in a list, before we can evaluate the list itself.

The algorithm for evaluation looks like the following:

• To evaluate a number or operator symbol, just return it. (base case)

• To evaluate a list:

  1. evaluate every entry in the list. (take care of parenthesis)
  2. simplify the expression by applying each operator in order. (simplify chain)
  3. Return the only item left in the list.

Let’s write an eval function which does this, building upon simplify chain:

(defun eval-expr (expr)
  (if (listp expr)
        (car (reduce #'simplify-chain 
                     '(+ *)
                     :initial-value (mapcar #'eval-expr expr)))
        expr))

To summarize simplify-chain takes mathematical expressions and returns expressions that are simpler, while eval-expr always returns a number. Note that this works for any number of operators and adjusting precedence is as simple as changing the order of the '(+ *) list.

12/19/20

A fun mechanic in AOC is that the problem is divided into two parts, with the second only revealed after completion of the first. This challenges you to adapt an initial solution to new requirements, encouraging flexibility of design. In the case of problem 19, a seemingly minor change, completely transforms the requirements. It may seem you have to rethink everything from scratch, but there is actually a tiny change to part 1 which solves the problem

My personal ranking wasn’t great, but you can see I still managed to catch up a bit on part 2 (even taking an additional hour!) as others got stuck.

SPOILERS AHEAD

Part 1

Let’s outline a fairly standard solution to part 1. We will stick to psuedocode, but at then end I will show some of my Common Lisp code.

Parsing

We need to parse the rules and turn them into a structure we can work with. I chose to transform them into an expression tree. A rule such as:

3 4 | "a" "b"

Becomes:

(or (3 4) ("a" "b"))

All the rules are stored in an array, with their index corresponding to their rule number.

Lisp makes it easy to parse. I first replace the | with ^ because pipe has special semantics for the lisp reader. Whenever a rule contains a ^ I split it’s components into a left side and right side and wrap them in an or.

Evaluation

Now we need a recursive evaluation function to handle the rule. It takes a rule and a string as input. It returns FAIL if it cannot match, or the modified string location, if it did match. For example when the rule “a” is applied to the following string:

"abc"

It returns “bc” indicating successful matching.

The evaluation recursively handles the following cases:

• evaluate a char: check that the first character agrees
• evaluate a number: lookup the rule at that index and evaluate it.
• evaluate an or form (OR r1 r2): evaluate both options. If one succeeded, return that. Otherwise fail.
• evaluate a list (r1, ... ,rn): evaluate each rule in the list in order. Stop early if there is a failure. If the string ends before the list ends, the match failed.

Success is indicated by returning an empty string. This means we used everything up and it matched. Test how many entries pass and you’re done.

Part 2

What is going on?

Part 2 makes two substitutions to the rules list.

8: 42        ->    8: 42 | 42 8
11: 42 31    ->    11: 42 31 | 42 11 31

It’s not immediately clear why this breaks our recursive evaluation. In fact, if you run it, it will not crash and still give you an (incorrect) answer. So what is going on? (This question took most of my time!)

Consider the following rule:

"a" | "a" "b"

The string “ab” now matches BOTH possible choices. How do we know which one to pick? Logically, a disjunction (or) operation shouldn’t care. But whatever choice is made will actually have an effect on whether the remaining rules in the pattern match.

Here is an example of what can go wrong. Suppose our string is “abc”.

0: 1 2
1: "a" | "a" "b"
2: "c"

Starting with rule 0, we try rule 1. "abc" matches both "a" and "a" "b". So let’s pick the first rule. Now onto rule “c” which does not match “b c”. The match fails.

If we had chosen “a” “b” then we move to rule 2 “c” matches the remainder “c”. We have a successful match.

The Hard Solution

Every time we an encounter an OR and both cases match, we no longer know which one to pick. The global success of a match can no longer be determined locally.

We aren’t going to get false positives, but we may get false negatives if we choose the wrong path to take.

A proper solution must diverge at every or, creating branching possible evaluations. This can certainly be implemented, but is going to look very different than part 1, and be more complicated.

An Easy Solution

What we really want to know is whether there is SOME set of choices for each OR that will cause the pattern to match. Since we don’t know what they are, we can just try a whole bunch, and see if we can find a set that does. The easiest way to do this is by introducing probability. Randomly pick a branch at each or. We can then run the matcher a bunch of times and see if it finds a set of choices that match.

Let’s work through the details. We want to replace the code for “evaluate an or form (OR R1 R2)”. The original descrpition was:

• If one succeeded, return that. Otherwise fail.

Let’s change it to a randomized choice when we don’t have a better option:

• evaluate both options. If either one failed, return the other. Otherwise both succeeded, so randomly pick which one to return.

Now for each string run the pattern matcher a few hundred times. Each time it will (hopefully) try different choices for each OR. With enough tries, it should get lucky and find the best path if there is one.

Remember, changing branch directions never introduces false positives, only false negatives. So if it matches even one time, then we know it works. Experiment with different numbers of times until you get a match rate that stabilizes. If you don’t run it enough times, then you will get changing answers each time you run it. It works!

Want to learn more? Checkout chapter 4.3 of SICP on non-deterministic computing.

My Code

Here are the key snippets, some of it is a little ugly.

Eval (With Random)

(defun eval-rule (str rule all-rules)
  (cond ((stringp rule) (if (char= (car str) (char rule 0))
                            (cdr str)
                            'FAIL))

        ((numberp rule) (eval-rule str (aref all-rules rule) all-rules))
        ((and (listp rule) (eq 'or (car rule))) 
         (let ((first (eval-rule str (nth 1 rule) all-rules))
               (second (eval-rule str (nth 2 rule) all-rules)))
           (if (eq 'FAIL first)
               second
               (if (eq 'FAIL second)
                   first
                   (if (= (random 2) 1) 
                       first
                       second
                       )))))
        ((listp rule)
         (prog ((s str)
                (l rule)
                (result nil))

               step
               (setf result (eval-rule s (car l) all-rules))

               (if (eq result 'FAIL)
                   (return result))

               (setf s result)
               (setf l (cdr l))

               (if (consp l)
                   (if (consp s)
                        (go step)
                        (return 'FAIL))
                   (return s))
              ))
        (t (error "unknown rule "))))

Solve Part 2

(defun solve-2 (all-rules inputs)
  (remove-if-not 
    (lambda (string)
      ; use probablity!
      (some #'identity (loop for i from 1 to 500 collect
                             (not (eval-rule (coerce string 'list) (aref all-rules 0) all-rules)))))
      inputs))

11/15/20

The first time you encounter a theorem concluding with “an arrow that makes the diagram commute” can be quite confusing. But with a little thought you can typically find that the idea expressed the theorem is obvious. The provided arrow is simply the one thing that could possibly fit. It may be tedious to construct, but understanding the theorem clarifies why it must always be there, revealing patterns hidden in the construction.

Let’s see how this works by studying the universal property of quotients, which was the first example of a commutative diagram I encountered. If you are familiar with topology, this property applies to quotient maps. But we will focus on quotients induced by equivalence relation on sets and ignored additional structure.

An equivalence relation ~ is a binary relation satisfying the following properties:

• reflexive: x ~ x
• symmetric: x ~ y <=> y ~ z
• transitive: x ~ y and y ~ z implies x ~ z.

Examples include equality of real numbers, whether numbers are both even or odd (parity), matrix similarity, isomorphism, etc.

Exercise: Prove matrix similarity is an equivalence relation.

Given an equivalence relation ~ and an element x we can form it’s equivalence class [x] which is the set of things equivalent to it. The set of a equivalence classes form a new set X/~ with an analogous structure to the original, but with portions “grouped up” or “collapsed”

Exercise: Prove distinct equivalence classes are disjoint.

Universal Property of Quotients

Let q(x) = [x] be a map from an element to it’s equivalence class. Given a map f from X to Y which is constant on equivalence classes (q(x) = q(y) => f(x) = f(y)), we obtain a unique map from X/~ to Y making the diagram commute.

What is going on here? f isn’t just any map, it’s constant on equivalent elements of X. So a lot of the information in X isn’t really needed to compute the image of f. Since equal elements get sent to the same place, we could imagine picking just one element from each class and seeing where it goes.

In other words, we could define a function on each equivalence class. This map would have the same image as f and this is precisely what the universal property tells us is possible

Let’s apply this theorem to a particular example and attempt to fill in the diagram. Suppose our domain is the solid disc.

To construct an equivalence relation on the disk, think of properties that make points in the disc similar to one another. One such property is their distance from the center. Define x ~ y whenever ||x|| ~ ||y||. What does the quotient space X/~ look like under this relation? Each class is a ring at a particular radius L, so denote it [L]. The radii of course vary continuously so we get set of classes isomorphic to a closed interval [0, 1].

Exercise: Find an equivalence relation on D² without 0 (punctured disc) whose classes form the circle.

Now define a function f(x) = x^2 + y^2 so our codomain is the real numbers R. The graph is a multivariable calculus style paraboloid living in R^3 above the unit disc:

Can we apply the universal property? Almost, we need to confirm f is constant on equivalence classes. Suppose (x, y) ~ (a, b). Then sqrt(x^2 + y^2) = sqrt(a^2 + b^2). Squaring both sides we get that f(x, y) = f(a, b).

Ah! We can clearly see f only depends on radius. No matter what angle you are at f does the same thing. So the disc is more information than we need, we can define a similar function on the space of equivalence classes X/~ = [0, 1]. Can you figure out what it is?

…

The missing function is of course h([L]) = L^2. It’s graph is a parabola in 2D which carves out the same range as f in the real numbers.

To check commutativity take a point (x, y) and apply f(x, y) = x^2 + y^2. Now send it to it’s equivalence class q(x, y) = [L] where L = sqrt(x^2 + y^2). Now h(L) = x^2 + y^2 = f(x, y).

Easy right? You probably could see right away that f was just the square of the distance. In other examples constructing such a function h might be less obvious, but the universal property tells us it is always there.

Next time you an encounter a commutative diagram proof, try a few examples to figure out what basic idea it is telling you.

Exercise: Prove that h is unique. In other words, no other function could make the diagram commute.

Further Reading

Commutative diagrams are the central focus of category theory which attempts to understand such properties at a higher level of abstraction than set theory. From a category theory perspective the quotient set X/~ is the co-equalizer or co-limit of the diagram projecting and equivalent pair to it’s parts.

The book Topoi by Robert Goldblatt offers a fairly comprehensive look at category theory, starting at a basic understanding of sets and number theory.

]

Quotient spaces are studied in depth in Topology by Munkres.

]

The later chapter on Algebraic Topology have More elaborate constructions. These gave me a lot of practice with commutative diagrams, such as the chapter on Seifert-Van Kampen’s theorem.

10/8/20

Are Elections Actually Random?

A popular narrative explaining the outcome of the 2016 election is Hilary Clinton was just unlucky. On election night, the odds were in her favor, but when the dice were cast she unfortunately rolled snake eyes. In a weird way, if the election were just run another day, she would win.

The underlying assumption is that elections are sort of random. But, how similar is winning an election to playing roulette? Let’s briefly relate what actually happens on voting day. People who want to vote, visit a voting location. They select a name either by pushing a button or writing it in. Later, the votes are tallied and grouped by region and a winner is announced. Where is the randomness? At each step of the process it’s simple cause and effect.

An individual might have an unexpected accident, preventing them from voting, but like most other days, the vast majority of the population is able to carry out their intentions. It’s even difficult to detect rapidly changing variables which might appear random. The cultural and economic context have been slowly evolving over years. The media environment has been built up over months leading up to it. Individual’s political views move slowly, as they are predominantly a product of deeply rooted influences, including family upbringing, income, ethnicity, religion, and education.

If the unlucky dice roll doesn’t hold up to a little scrutiny, then why is this view so popular? Perhaps it is because its message avoids uncomfortable conclusions. If it was just bad lack, then it was nobody’s fault. We still have the popular ides. It’s not even clear that our strategy must change to win next time. The alternatives are much harder to look at.

Hold on a minute. Perhaps this is all a misunderstanding. People don’t actually believe elections function like a lottery. When journalists use percentages they are just referring to the confidence intervals of polls. Pollsters want to survey public opinion. But, they can’t talk to everyone in the country, so they talk to a sample, and then use probability to guess how likely it is that this sample represents the whole population. It’s basic statistics (as described in this Pew Research article).

This may correctly describe where the numbers come from. Perhaps in a back office somewhere, there actually is a Ph.D in Stats coolly scrutinizing surveying models. But this is clearly not how these figures are interpreted by journalists and otherwise intelligent or scientific people. If it was, they would be concerned with potential gaps between sample and population, instead of calculating their odds. Look at how randomness and chance is implied in the following popular media:

Election as a Pachinko/Plinko Machine

From: presidential-plinko.com, a site created by a Northwestern University professor.

Election as an Unlucky Dice Roll

From Paul Graham’s Twitter, a popular figure in tech and startups.

20% is still 1 in 5

A comment on the Silicon Valley forum “Hacker News”.

The unlikely combination

From Reuters.

If so many smart people are doing inappropriate probabilistic thinking in this area, are there others where we might be making similar mistakes?

A Cultural Belief in Randomness

Stats is at the forefront of business, marketing, medicine, sports, finance, academia, and science. Many of it’s concrete applications are no doubt useful and appropriate. But, our society’s interest in it goes well beyond it’s technical or economic usefulness. Its popularity is partially an extension of our love for science and logic. Just look at all the numbers, equations, and graphs! But, that also isn’t enough to explain its significance. Consider the following ideas prominent in popular culture:

• Don’t plan too far into the future, or commit to any one thing in your career. Nobody knows what the world will look like the future (see “Zero to One” chapter 6 for more on these ideas).
• Scientific discovery is mostly good luck. The best we can do is setup Fleming or Darwinian style projects to tip chance in our favor.
• Stats and data are the way smart people make decisions and overcomes biases. Don’t blinded by personal experience and intuition. Haven’t you seen Money Ball?
• Nobody knows which business ventures will be successful, just try a bunch of them. Invest in startup incubators or index funds.
• The universe is just atoms crashing into each other in the void, combined with strange quantum uncertainty. Who knows what will happen. It might even form a new universe.
• Nobody knows how to solve hard computer science problems like computer vision, or understanding natural language. If we correlate a bunch of data with a machine learning system, it will probably do figure out what’s going on.

If we apply the same step-by-step analysis as we did with the election, these ideas immediately begin to seem less plausible, or at least in need of some qualification and refinement. Perhaps the cultural prevalence of randomness nudges us to reach for that option more often than alternatives. Contributing to this bias is the fact that randomness can provide us with comfortable answers to hard questions, or mask the fact that the underlying mechanisms are not well understood.

When is Probability Useful?

Let’s step back a bit. Probability itself is a sound mathematical model. Just because an underlying process is not random, does not mean that probability cannot offer an accurate description or be a useful tool. Sometimes statistics is the best way to talk about complicated deterministic systems.

For example, statistics can be used in physics to describe the motion of bodies, which can otherwise be accurately described by deterministic cause and effect. A gas might be modeled as a cloud of individual particles, each of which is well understood, but there may just be too many of them to keep track of. Because, the task is overwhelming, and we care most about the global behavior, we settle for partial information (see Wikipedia).

Consider even a dice roll. With Newtonian mechanics we can understand gravity, the force of the throw, the friction in the air, the impact on the table, and with all this make an accurate prediction of the outcome. But no human can capture all those parameters in a split second and apply them. Without a controlled environment for study, probability is a good model.

Computers despite being artificially deterministic, benefit greatly from probability. Random numbers generated on a computer are usually just functions which are difficult to predict. Despite this, they can compute and solve problems using probability (see Monte Carlo methods).

The difference then between helpful and misleading attributions of randomness, is of course how closely the model actually resembles reality. Statistical models in physics don’t just start by assuming processes are random. They construct deterministic relationships between the elements; the energy, the area, the force, and then insert a tiny amount of probability to account for the arts that are too complex. The random processes are tightly constrained by the physical laws. Furthermore, they can usually quantify the information lost by assuming randomness.

A Universal Way of Understanding

It may seem like the only conclusion we can draw is, “lazily constructed models give poor results”. But, this analysis also reveals how backwards common statistical practices is. Statistics is the go-to tool when we have no ideas what’s going on. Fields like politics, financial markets, and human behavior come to mind. Instead of starting with well understood systems and then carefully relaxing assumptions and summarizing details, Data is automatically correlated, and everything is basically assumed to follow random distributions. Randomness replaces cause and effect, instead of working in concert with it.

The sophistication of the math involved, disguises all this. But, is it any surprise that we observe “black swan” events when we assume everything uniformly follows a bell curve?

In The Republic X, Plato observes a related attitude in painting and poetry:

The imitator, I said, is a long way off the truth, and can do all things because he lightly touches on a small part of them, and that part an image. For example: A painter will paint a cobbler, carpenter, or any other artist, though he knows nothing of their arts; and, if he is a good artist, he may deceive children or simple persons, when he shows them his picture of a carpenter from a distance, and they will fancy that they are looking at a real carpenter.

And whenever any one informs us that he has found a man knows all the arts, and all things else that anybody knows […] I think that we can only imagine to be a simple creature who is likely to have been deceived by some wizard or actor whom he met, and whom he thought all-knowing, because he himself was unable to analyze the nature of knowledge and ignorance and imitation.

And so, when we hear persons saying that the tragedians, and Homer, who is at their head, know all the arts and all things human, virtue as well as vice, and divine things too, for that the good poet cannot compose well unless he knows his subject, and that he who has not this knowledge can never be a poet, we ought to consider whether here also there may not be a similar illusion. Perhaps they may have come across imitators and been deceived by them

Statistics students and professionals hold the same remarkable belief about their own field today. They don’t need to study any subject or “applications”, besides statistics itself. Their statistical knowledge is immediately transferable to understanding any problem thrown their way. It’s a universal framework for understanding. This makes it the perfect career in the culture of randomness, and ironically minimize the need to make predictions about the future.

But can you understand business without running one? Can you understand voting patterns without understanding what is happening in people’s lives? Can you understand psychology without thinking, observing, and talking to people? In these areas, as in politics, we can conclude that probability and statistics are being used as poor substitutes for explanatory theory.

Update 09/25/2021: Heavily edited the second half to clarify intended message. The election discussion is largely the same.

09/26/20

As a C# programmer, you are probably familiar with Select, Where, and Aggregate, the LINQ equivalents of the fundamental functional programming operations map, filter, and reduce. GroupBy is an equally useful function, which you may be less familiar with, but should definitely learn about. It solves the following problem: Given a list of items, partition them into groups so that equivalent elements are together. For example, given a list of Students in the school, we may want to group them by teacher to build a class roster.

Functional programming aims to specify “what” needs to be done, rather than how to do it. As a programmer, this helps shift your focus and concern to a higher level of abstraction, leading to cleaner and more direct code. GroupBy is a beautiful example of these principles. It solves a general problem, in a reusable way, which would otherwise take a bit of thought to write correctly by hand (as we shall see). Furthmore, it combines harmoniously with the other LINQ operations.

How to use GroupBy

With the brief description above in mind, let’s examine the method signature:

IEnumerable<TResult> GroupBy<TSource, TKey, TElement, TResult> (
    IEnumerable<TSource> source,
    Func<TSource,TKey> keySelector,
    Func<TSource,TElement> elementSelector,
    Func<TKey,IEnumerable<TElement>,TResult> resultSelector,
    IEqualityComparer<TKey> comparer
);

Bleh! That looks pretty meaningless and intimidating. Ignore the generic types for now and I will explain the 4 functions that must be provided:

• TKey keySelector(TSource item) given an item, return a key which will determine which group to place this item. Typically this is a field on item.
• TElement elementSelector(TSource item) given an item, return the value that will actually be stored in the group, Often this returns a field on the item or the item itself.
• TResult resultSelector(TKey key, IEnumerable<TElement> contents) after elements are grouped together, this function will be called on each group to produce a final result. The key is the identifier for this group and the contents is an enumerable object containing the results. Most often you will just return contents, but you may want to store the result in a new class.
• int comparer(..) provide a function so that keys can be compared. This can be left off for key types like integer and string which are already comparable.

With an understanding of the functions, the meaning of the 4 generic types is now clear:

• TSource the type of the input items.
• TKey the type of the key that will identify which group the item belongs to.
• TElement each item has the opportunity to be transformed before being put into a group. This type is the output of that transformation.
• TResult The result type from obtaining a result from a group. If you were to return each group as an array, this type would be [TElement].

Example 1: Classroom roster by student id

Let’s try the example mentioned above. Given a list of Student objects, we want to group them by teacher to create class roster. However, in this example, we don’t need the whole student object, we only want their student ids.

List<Student> students = ...;

students.GroupBy(
   s => s.teacher,                 // keySelector
   s => s.id,                      // elementSelector
   (teacher, ids) => ids.ToArray()  // resultSelector
).ToArray();

// Result: [ [id1, id2, ..], [id1, id2] ] 

Example 2: A histogram of purchases

Let’s imagine we are building a personal finance application. One useful tool would be a histogram showing purchases withn typical price ranges (for example $0-$10, $10-$20, etc), For each range, we would want to know the total number of purchaes, and their total value. We will use a tuple to store these calculations for each resulting bucket.

struct Purchase {
   double amount;
   ...
}

List<Purchase> purchases = ...;

purchases.GroupBy(
   p => (int)Math.Floor(p.amount / 10.0),    // keySelector
   p => p.amount,                            // elementSelector
   (bucket, amounts) => {                    // resultSelector
      return ValueTuple.Create(
         bucket,
         amounts.Count(),
         amounts.Sum()
      );
   }
).ToArray();

// Result:
// [ (bucket, number of purchases, total purchase amount), ... ]

Note: Assuming the range of buckets was known up front, one could simply allocate an array with the proper number of buckets and place each item directly. But, GroupBy also works if the data is sparse and forgoes the awkward need to translate between bucket values and indices.

Implementing GroupBy with sort and merge

That may be all you want to know about GroupBy now. For those who are curious, let’s talk about how it works and performs. At first, grouping elements appears inherently complex. An initial idea might be to loop through each item, and then loop through all the other items to find matches. This works, but is an O(n^2) algorithm so will quickly become unwieldy.

The next natural implementation is to create a dictionary similar to Dictionary<TKey, List<TElement>>. Iterate over each item in the list once, and insert it into the list corresponding to it’s proper group. This solution has pretty good O complexity, but comes with some practical performance problem. The main concern is with how memory is used. Depending on the dictionary implementation, it has a memory allocation which it must resize and copy as it grows. Furthermore, each individual group array does the same thing! This is expensive, and not friendly to the cache.

We can do bit better in elegance and practical performance by “sorting and merging”. First, sort elements by their key (O(n log n)). After sorting, all the equivalent elements should be next to each other. We can then apply a reduce-like operation to merge adjacent neighbors into a group O(n). This is a general technique which can solve a lot of algorithm problems (See Stepanov’s bigrams for one other application).

Let’s give it a try:

public static IEnumerable<TResult> GroupBy<TSource, TKey, TElement, TResult> (
    IEnumerable<TSource> source,
    Func<TSource,TKey> keySelector,
    Func<TSource,TElement> elementSelector,
    Func<TKey,IEnumerable<TElement>,TResult> resultSelector
) where TKey: IComparable
{
    var sorted = source.Select(item => ValueTuple.Create(keySelector(item), elementSelector(item))).ToList();
    if (sorted.Count == 0) yield break;
    sorted.Sort((a, b) => a.Item1.CompareTo(b.Item1));

    var startIndex = 0;
    var endIndex = 1;
    var groupKey = sorted[0].Item1;

    while (endIndex < sorted.Count)
    {
        var key = sorted[endIndex].Item1;
        if (!groupKey.Equals(key))
        {
            var group = Enumerable.Range(startIndex, endIndex - startIndex).Select(i => sorted[i].Item2);
            yield return resultSelector(groupKey, group);
            startIndex = endIndex;
            groupKey = key;
        }
        ++endIndex;
    }

    var finalGroup = Enumerable.Range(startIndex, endIndex - startIndex).Select(i => sorted[i].Item2);
    yield return resultSelector(groupKey, finalGroup);
}

Note: this is almost certainly NOT how LINQ is exactly implemented (TKey is only required to have equality testing, not IComparable), but their approach is similar.

10/07/20

Lisp advocates are famous for having elaborate reasons for why Lisp is their favorite language. You might have heard that it’s the most powerful language, thanks to features like macros, first class functions, and homioconicity. Certainly, Lisp has no shortage of beautiful and influential ideas. But, almost every modern language has copied it’s most productive features. Are there still compelling reasons to use it in production?

For me there are. But, my interest is much more boring and practical. Fancy language abstractions don’t appeal to me, so the whole meta-programming isn’t as important. What I do care about is a reliable and stable ecosystem which allows me to write quality code that can last for a long time. Consider the following features:

• Standards: Common Lisp and Scheme are both fully standardized language with specifications. Consequently, they are well understood, cross-platform, and have diverse implementations, including several with free licenses.
• Documentation: SICP is “the book” for Scheme and computer science. Common Lisp has several great books worth reading even if you don’t care about Lisp (see PAIP). The CLHS is as reliable and thorough as any language.
• Stability: Lisp has been around since the 60s and the ANSI standard about 30 years. It is extremely stable. Code be written once, and run again years later, without modification.
• Efficiency: Lisp implementations are reasonably fast. The true believers claim Common Lisp is as fast as C, it’s probably much closer to Java. But, for a high-level, dynamic, language, it’s pretty fast.
• Design: Lisp is well designed and founded on solid computer science principles. It has a focused selection of features and an elegant evaluation model which make it easy to write and compose functionality.

You might think this list of features isn’t remarkable or unique to Lisp. That’s certainly true. C did them first, and established them as the recipe for releasing a successful language. Languages since have followed various steps, but the surprising thing is how few of them actually ended up with all the ingredients. Of course each item in this list is somewhat of a spectrum; some languages do more than others, but very few languages follow it to a degree that can be asserted with confidence. Usually, you can say a language somewhat satisfies it, followed by a list of ugly qualifications.

Take Python for example. It’s well designed, has great books and documentation, is fairly stable. It has an elegant design inspired by Lisp. But, is it standardized? Not really. They have a spec, but it only has one defacto implementation and changes all the time. Whatever the CPython project chooses to do defines the language. Alternative implementations exist, but they all have compatibility compromises. Python is also not very efficient. You can use a JIT implementation which makes it tolerable, but that has its own quirks, and you can’t use many libraries with it.

Lisp and C aren’t the only languages with all these benefits, but there aren’t as many as you think. Since Lisp has these great features and a design I enjoy using and studying it’s become my go-to language.

Update: 4/30/21: I recommend the article “Tips for stable and portable software” which discusses principles of stability at the language level, as well as the entire OS stack. It offers specific advice for reliable unix software.

Update: 09/23/21: Edited to clarify intended message.

09/26/20

How do you ensure that programmers are using time effectively? What can you do to help them be productive? Managers think a lot about these questions. But too often they get the answers wrong despite being very earnest and smart. This is because they have an unexamined mental model of how programming work is actually done. With an inaccurate model, managers optimize the wrong things and make bad decisions which would otherwise be good decisions in other work environments.

To understand the idea, let’s look at a few examples of how productivity varies between jobs. Simple manual labor, like digging holes, is easy to understand. Productivity comes down to moving as much dirt per minutes as possible. Progress is very measurable, and it easy to think about how to improve it. As a manager, you might ensure there is a rotation of workers digging at all time, and that everyone has the best tools possible. However that looks very different than the working model for a professional baseball scout. A scout can take a few years of evaluating players just to find one great pro. There aren’t simple metrics you can use to measure their progress. It’s hard to tell exactly what productive work should look like. Exposure to a lot of players can help, but ultimately it comes down to a few big choices.

What kind of work is programming most like? In my experience (especially in agencies), the typical model of a programmer is an expensive factory machine. This is the kind of machine that occupies a lot of space and attention in the factory. It’s operation is fragile, so it needs to be constantly checked on and maintained. But most importantly, every minute that it is not working on jobs is expensive money wasted. The primary metric for ensuring it stays productive is utilization time, even if that means spending time on throw away jobs.

Managers understand this work model very well, perhaps they read “The Goal” in school. They look for metrics to identify throughput and bottlenecks and do lots of shuffling of programmers around to ensure maximum utilization. In it’s most exaggerated form this led managers to obsess over metrics “lines of code per week”. Although we have moved beyond that. The underlying thinking about what’s going on has changed much. We just think about programmers as feature producers. The task the programmer needs to finish programmer needs to sit at the computer and type in the project.

Here is a common example of this thinking: Bob is busy finishing task 1 and then will start project 2. Alice has a bit of downtime before she starts project 3. Let’s have Alice get started on project 2, then we can have her hand off the incomplete work to Bob. In a few weeks when he is ready, Alice can move on to project 3 and leave Bob with a head start.

This seems like smart move. We made sure that Alice and Bob’s valuable time was well used, right?

A better Model

To see why this decision may actually waste time, we need a better understand what programming is like. The model I will describe is not perfect, it’s a great simplification, but its a rough estimation that’s far more accurate than the factory machine one. That better model is the following: each project consists of making 5-15 difficult software decisions. All of these decisions are of course somewhat dependent on one another. Each of those decisions that is made well, will lead to shipping the project, Each decision made wrong, will cause delays and waste time, now or down the road. That’s all. That’s the description of the job programmers need to do, and how they occupy their time. If programmers knew what to write beforehand, they could type it in no time! The time they spend fiddling at the keyboard is actually gather information through experiment and observation, in order to make decisions.

As a manager, the way to improve productivity is to simply think about what would help you make hard decisions. Here are some simple ideas that become immediately obvious, given this clearer model:

• Give programmers as much possible information about the project as early as possible. It’s difficult to make decisions with incomplete information. When you get ideas early they can sit in the back of your mind and get some sleep time. Organize information so its easy to process.

• Protect programmer time from interruptions. Design large blocks of uninterrupted time. Have quiet working environments. It’s hard to make good decisions without focus. Other writers have written extensively about the cost of “context switching” when you have to forget about what you were thinking and start thinking about something new.

• Ensure programmers have a clear understanding of project priorities and goals. Clearly communicate expectations and requirements. Making good decisions requires making trade-offs. Unclear priorities will lead to programmers making bad decisions. Changing priorities will invalidate previously good decisions.

• It’s OK if a programmer is not at their keyboard writing code. Taking a walk or a rest is often productive time. Distracting activities like watching a lot YouTube are not.

• Don’t treat programmers interchangeably. Give them ownership of specific project pieces so they can make long term decisions and build on previous ones. Would you hire a separate architect to design each room of a house?

• Ensure your incentive structure rewards good decision making. Do you reward band-aid approaches? Are programmers just trying to get it “over the fence” so someone else has to deal with it? Are programmers are the receiving side of the fence?

• Going through a crunch? Don’t add people to a late project. Throw more money and effort at building an focused environment to making decisions. Can you pay someone to errands at home so the programmer doesn’t have to think about them? Can you bring their family to the office for a lunch? (1)

This is just a short list of suggestions and you can probably can come up with even better ideas. What’s important is that as we think deeper about this model common productivity mistakes become apparent. (2)

Let’s analyze the Bob and Alice example again using this model. Did we help Alice and Bob make decisions or hurt them? Recall, Alice starts working on task 2 for a few weeks, to get it started for Bob. Then Bob picks it up, and Alice moves elsewhere. In our rough model we can assume that when Alice finishes, she has made 2 of the 15 decisions, and has some vague ideas about 5 others. The manager arranges a meeting for Alice to inform Bob of her work and she is swept off to task 3. Now Bob has to start thinking about task 2 from scratch, but not only that, he has to figure out what Alice was up to, whether the decisions she made were right, and what direction she was heading to address the other 13 decisions. This added information causes Bob to take more time than if he had started on his own. In practice, its even worse because Alice and Bob will need to continue to take time from each other to coordinate about what Alice had in mind. (3)

Smarter Alternatives

You might say, “that makes sense, but what am I supposed to do? Let Alice sit around for a week, wasting that time with an urgent project?”. Let’s use the model to work through better options. In this scenario Alice is scheduled to eventually own project 3. A better option would be to just let task 2 sit until Bob is ready, and get her stated on her project, even though it may be less urgent. Alice will be able to spend more time thinking about her project, and leave Bob with a fresh slate.

“But this is really urgent, I don’t even want to think about project 3 yet”. The next alternative would be to identify decisions which Alice can make which have the least impact on the other work Bob needs to do. One would be having Alice better define the requirements and organize resources for this project. This will only help Bob make his decisions. Another idea is to have Alice work on “black boxes” that Bob doesn’t need to know how they work. Maybe Alice has some special knowledge and can write a hard function he expects to use. What you absolutely don’t want to do is interrupt Bob and have him start thinking about what decisions Alice could be making, while he is making his decisions. That doesn’t make any sense!

Notes

1. I don’t think programmers deserve any of these special treatments over other jobs. I am not suggesting that programmers should have special privileges or are somehow more important than other kinds of work. It’s all about the productivity model for the kind of work they do.

2. In video game production, art tends to be easy to outsource while programming is notoriously difficult and often unsuccessful. Perhaps this is because 3D art production can be better fit into a factory production productivity model.

3. One reason “just write it from scratch” is so appealing for programmers is that they get to make their own decisions, instead of figuring out why others made (often bad) decisions. Who wants to live with someone elses choices? For large existing codebases this represents a lot of invested knowledge, so its usually bad, but for a new task, Why would you rob a programmer of this opportunity by playing musical chairs?

1/23/2020

The software industry is synonmous with rapid innovation. New discoveries require changing to better ways of doing things. For programmers to stay relevant they need to constantly keep up on the latest technologies, otherwise their skills will go out of date. At least, that’s what I’ve been told, but the more I study computer science from 50 years ago, the more I find “new” ideas are actually old. Perhaps the changes we see are only permutations of underlying ideas.

In many obvious ways software does change. Languages, libraries, and tools get replaced overtime. Nobody is writing desktop applications in assembly anymore. Many developers who wrote Windows applications moved on to mobile or web. But rarely are these changes due to a technological advancement or dramatic in change in how things are done, just new forms of popular products, or improved hardware that gives us some wiggle room.

A good engineer can adapt to these changes, in the same way they can switch to a company which uses different tools and processes. Learning Python after C# should be easy, because you understand programs, not the syntax. Moving to a language with significant design differences like Haskell requires a broader understanding of functions and computation. Now imagine there was similar knowledge that helped you approach every computer problem and perhaps all of nature. The knowledge I am describing is math and science! For software it consists of computer science subjects like OS theory, algorithms, software design, logic, and calculus. Because many programmers lack this, they struggle with change, and feel pressure to keep up with trends. ¹

Don’t believe me? Take a look at the hottest “new” areas of tech AI/machine learning, blockchain, and big data/data science. What barriers make it difficult for programmers to get into them and be successful? For machine learning the answer is a bit of linear algebra and multivariable calculus, evidenced by the many blog post promising to get readers up to speed. This math is (should be) covered by every computer science degree and has remained the same for at least 50 years, down to the presentations and illustrations used to teach it. Neural networks themselves have been around for a long time, although less mainstream.

For blockchain the essentials are an understanding of cryptography, and peer-to-peer networking. That doesn’t even mean fancy understandings of elliptic curves or number theory, just a solid grasp of hashes, signatures, and asymetric key encrpytion. Distributed systems is also a mature field of computer science.

Data science might be the most accessible. A solid understanding of introductory probablity and statistics and databases combined with a few tools such as linear regression, and polynomial interpolation might be enough for 90% of applications. But it’s going to be really hard, if you can’t understand a wikipedia page about polynomials.

I don’t mean to suggest that these skills make an expert. General theory is a longshot from research or novel contributions to the field. Nor is it sufficient to be a good programmer; learning linear algebra does not immediately make you good at writing machine learning programs (and scientists write some terrible code!). Rather this is the “hard stuff” that prevents programmers from getting into these fields. Once you know it, the other details are approachable. ²

If you analyze other software advances from the past, from database theory to graphics, you will find similar applications of rather unextraordinary math and science. Many programmers ask, how I can I predict what skills will be important in the future? What do I need to learn to have a successful career? Few can predict what specific trends will take off, but I bet whatever is important in the future is going to require understanding those broad areas of computer science. Keep practicing and specializing in the area you work in, but if you regularly refresh and broaden your base of fundamentals you will be prepared to learn anything new that starts to look interesting. [^3]

Understanding fundamentals gives programmers another significant advantage. They know what problems have been solved before. It’s unlikely you will remember every detail, but can say “I’ve heard of this before” and know where to learn more. Consider how many new tools are bad solutions to problems solved by simple bash scripts, of which the author was ignorant of. This is just a tiny represenative of how much duplication and complexity programmers are adding because they don’t know whats already there.

It may sound as though I am advocating a kind of tech hipsterism; everything interesting has already been done so we might as well stop looking for new things. Rather I am arguing that we will be able make more advancements, if we better understand the big ideas behind what has come before.

Properly understood this flips the progress narrative of technology on it’s head. We don’t have to chase headlines and blog posts about the latest frameworks and build tools. Nor do we have to guess which technologies might suddenly become useful, like day trading stocks. That’s for evangelists and IT consultants. The foundations for our next ideas has have already been built by generations of smart people. It’s all written down waiting to be dusted off and rediscovered. Rather than building and encyclopediac knowledge of novelties and press releases, we continuosuly revisit the same core subjects, over and over, in pursuit of mastery.

of those who study STEM seem to lose it after going through the motions.

popular book SICP. In just a few hundred pages it covers several major areas of CS and more than most programmers understand in their whole career. It’s able to do that because it’s not written for a general audience. It relies on the reader having a solid undergraduate understanding of another area of math or science as an MIT student would have.

[^3] Experts in a very particular system (for example OpenCL or V8 internals) can be extremely valuable. But, most of them only get that level of depth with a solid understanding of fundamental computer science. Expertise is also a subject for another day.

01/21/2020

What is the difference between MultiplyPoint and MultiplyPoint3x4 in Unity3D?

What steps of a matrix multiplication could Unity possibly be skipping to make an optimization? The optimization is possible because a scaling and rotation needs only a 3x3 matrix and another vector for the translation, so part of the full 4x4 matrix is unused and can be ignored.

Here is the source code taken from the decompilation here:

public Vector3 MultiplyPoint(Vector3 v)
{
    Vector3 vector3;
    vector3.x =  ( m00 *  v.x +  m01 *  v.y +  m02 *  v.z) + m03;
    vector3.y =  ( m10 *  v.x +  m11 *  v.y +  m12 *  v.z) + m13;
    vector3.z =  ( m20 *  v.x +  m21 *  v.y +  m22 *  v.z) + m23;
    float num = 1f / ( ( m30 *  v.x +  m31 *  v.y +  m32 *  v.z) + m33); // this is 1/1=1 when m30, m31, m32 = 0 and m33 = 1
    vector3.x *= num; // so then multiplying by 1 is pointless..
    vector3.y *= num;
    vector3.z *= num;
    return vector3;
}

This is a full 4x4 * 4x1 matrix multiplication, and then scaling the x y z of the result by the w of the result.

 | m_0_0 m_0_1 m_0_2 m_0_3 |     | x |
 | m_1_0 m_1_1 m_1_2 m_1_3 |  *  | y |
 | m_2_0 m_2_1 m_2_2 m_2_3 |     | z |
 | m_3_0 m_3_1 m_3_2 m_3_3 |     | 1 |

This w necessary for projective transformations, but not for rotation and scaling. This is why they have the faster 3x4 version. It assumes the w component of each column is 0 and skips that bottom row of multiplication and scaling, because the resulting w is always 1.

public Vector3 MultiplyPoint3x4(Vector3 v)
{
    Vector3 vector3;
    vector3.x =  ( m00 *  v.x +  m01 *  v.y +  m02 *  v.z) + m03;
    vector3.y =  ( m10 *  v.x +  m11 *  v.y +  m12 *  v.z) + m13;
    vector3.z =  ( m20 *  v.x +  m21 *  v.y +  m22 *  v.z) + m23;
    return vector3;
}

This function is equivalent to multiplying matrices which look like:

 | m_0_0 m_0_1 m_0_2 m_0_3 |     | x |
 | m_1_0 m_1_1 m_1_2 m_1_3 |  *  | y |
 | m_2_0 m_2_1 m_2_2 m_2_3 |     | z |
 | 0     0     0     1     |     | 1 |

6/8/19

Programmers love to discuss programming languages. We not only debate their technical merits and aesthetic qualities, but they become integrated into our personal identities, along with the values and traits that we associate with them. Some even defend a form of Linguistic Determinism that thinking is confined to what the language makes typable.

Since we spend so much time writing code, a keen interest in language design is justified. However, the character of these discussions suggests that we think of them as much more, and have perhaps forgotten their primary role. Programming languages are implementation tools for instructing machines, not thinking tools for expressing ideas. They are strict formal systems riddled with design compromises and practical limitations. At the end of the day, we hope they make controlling computers bearable for humans. In contrast, thoughts are best expressed through a medium which is free and flexible.

Thinking in Math

The natural language which has been effectively used for thinking about computation, for thousands of years, is mathematics. Most people don’t think of math as free or flexible. They think of scary symbols and memorizing steps to regurgitate on tests. Others hear math and think category theory, lambda calculus, or other methods of formalizing computation itself, but these are hardly necessary for programming itself.

I hope readers of this article have had a better experience regarding what math is about, such as a graph theory, algorithms, or linear algebra course; the kind that involves logic and theorems, and is written in prose with a mix of symbols (most symbols weren’t even invented until the 16th century). This kind of math is about creating logical models to understand real world problems, through careful definitions and deductions. If you don’t have a clear idea of what this looks like I recommend Trudeau, Stepanov, or Manber.

Math allows you to reason about logical structures, free from other constraints. This is also what programming requires: creating logical systems to solve problems. Take a look at the basic pattern for programming:

1. Identify a problem
2. Design algorithms and data structures to solve it
3. Implement and test them

In practice, work is not so well organized as there is interplay between steps. You may write code to inform the design. Even so, the basic pattern is followed over and over.

Notice that steps 1 and 2 are the ones that take most of our time, ability, and effort. At the same time, these steps don’t lend themselves to programming languages. That doesn’t stop programmers from attempting to solve them in their editor, but they end up with code that is muddled, slow, or that solves the wrong problem. It’s not that programming languages aren’t good enough yet. It’s that no formal language could be good at it. Our brains just don’t think that way. When problems get hard, we draw diagrams and discuss them with collaborators.

Ideally, steps 1 and 2 are solved first, and only then will a programming language be used to solve step 3. This has an added benefit of transforming the implementation process. With a mathematical solution in hand, you can then focus on choosing the best representation and implementation, and writing better code, knowing what the end goal will be.

Implementation Concerns

Why are programming languages burdensome thinking tools? One reason is that writing code is inseparably connected with implementation concerns. A computer is a device that must manage all kinds of tasks and while being bound by physical and economic constraints. Think about all the considerations for writing a simple function:

• What inputs should I provide?
• What should they be named?
• What types should they be? (Even dynamically typed languages must consider types, it’s just implicit.)
• Should I pass them by value or by reference?
• What file should I put the function in?
• Should the result be reused, or is it fast enough to recalculate it every time?

The list can go on. The point is that these considerations have nothing to do with what the function does. They distract from the problem the function is trying to solve.

Many languages aim to hide details such as these, which is helpful, especially for mundane tasks. However, they cannot transcend their role as an implementation tool. SQL is one of the most successful examples of this, but it is ultimately concerned with implementation concerns such as tables, rows, indices, and types. Because of this, programmers still design complicated queries in informal terms, like what they want to “get,” before writing a bunch of JOINs.

Inflexible Abstractions

Another limitation of programming languages is that they are poor abstraction tools. Typically, when we discuss abstraction in engineering, we mean hiding implementation details. A complex operation or process is packaged into a “black box” with its contents hidden and well-defined inputs and outputs exposed. Accompanying the box is fictional story that explains what it does, in a greatly simplified way.

Black boxes are essential for engineering large systems since the details are too overwhelming to hold in your head. They also have many well-known limitations. A black box leaks because its brief description cannot completely determine its behavior. The opaque interfaces introduce inefficiencies, like duplication and fragmented design.

Most importantly for problem-solving, black boxes are rigid. They must explicitly reveal some dials and knobs, and hide others, committing to a particular view about what it is essential to expose to the user, and what is noise. In doing so, they present a fixed level of abstraction which may be too high-level or too low-level for the problem, As an example, a high-level web server may provide a terrific interface for serving JSON, but be useless if one wants an interface for serving incomplete data streams, such as output from a program. In theory, you can always look inside the box, but in code, the abstraction level at any one time is fixed.

In contrast, the word abstraction in math is nothing like hiding information. Here, abstraction means extracting the essential features or characteristics of something, in relation to a particular context. Unlike black boxes, no information is hidden. They don’t leak in the same way. You are encouraged to adjust to the right level of abstraction and quickly jump between perspectives. You might ask:

• Is this problem best represented as a table? Or, a function?
• Can I look at the whole system as a function?
• Can I treat this collection of things as a single unit?
• Should I look at the whole system or a single part?
• What assumptions should I make? Should I make them stronger or weaker?

Just look at the many ways of looking at a function:

Thinking in math allows one to use whichever brings the most clarity at any moment.

It turns out most abstract concepts can be understood from many perspectives, just like functions. Studying math provides one with a versatile toolbox of perspectives for studying all kinds of problems. You might first describe a problem with a formula, and then switch to understanding it geometrically, then recognize some group theory (abstract algebra) is at play, and all of this combines to give insight and understanding.

To summarize, programming languages are great engineering tools for assembling black boxes; they provide functions, classes, and modules, all of which help wrap up code into nice interfaces. However, when trying to solve problems and design solutions, what you actually want is the math kind of abstraction. If you try to think at the keyboard, the black boxes available to you will warp your view.

Problem Representation

Just as programming languages are rigid in their ability to abstract, they also are rigid in how they represent data. The very act of implementing an algorithm or data structure is picking just one of the many possible ways to represent something; along with all the trade-offs that come with it. It is always easier to make trade-offs when one has use cases in mind and understand the problem well.

For example, graphs (sets of vertices and edges) appear in many programming problems such as internet networks, pathfinding, and social networks. Despite their simple definition, choosing how to represent them is hard and varies greatly depending on use case:

• The one which most closely matches the definition:
  vertices: vector<NodeData> edges: vector<pair<Int, Int>> (The vertices can be removed if you only care about connectivity.)

• If you want to traverse a node’s neighbors quickly, then you probably want a node structure:
  Node { id: Int, neighbors: vector<Node*> }

• You could use an adjacency matrix. Where each row stores the neighbors of a particular node: connectivity: vector<vector<int>> and the nodes themselves are implicit.

• Pathfinding algorithms often work on graphs implicitly from a board of cells:
  walls: vector<vector<bool>>.

• In a peer-to-peer network, each computer is a vertex and each socket is an edge. The entire graph isn’t even accessible from one machine!

Math allows you to reason about the graph itself, solve the problem, and then choose an appropriate representation. If you think in a programming language, you cannot delay this decision as your first line of code commits to a particular representation.

Note that the graph representations are too diverse to wrapped up in a polymorphic interface. (Consider again the graph representing a computer network, like the entire internet.) So creating a completely reusable library is impractical. It can only work on a few types, or force all graphs into an inappropriate representation. That doesn’t mean libraries or interfaces aren’t useful. Similar representations are needed again and again (like std::vector), but you cannot write a library which encapsulates the concept of “graph” once and for all. A simple generic or interface with a few types in mind is appropriate.

As a corollary, programming languages should focus primarily on being useful implementation tools, rather than theoretical tools. A good example of modern language feature which does this is async/await. It’s not hiding away complex details or introducing new conceptual theory. It takes a common practical problem and makes it easier to write.

Thinking in math also makes the “C style” of programming more appealing. When you understand a problem well, you don’t have to build up layers of framework and abstraction in anticipation of “what if”. You can write a program tailor made to the problem, with carefully chosen trade-offs.

Example Project

So what does thinking in math look like? For this section you may have to read a bit more slowly and carefully. Recently, I worked on an API at work for pricing cryptocurrency for merchants. It takes into account recent price changes and recommends that merchants charge a higher price during volatile times.

Although we did some homework on the theory, we wanted to empirically test it to see how it performed during various market conditions. To do so, I designed a bot to simulate a merchant doing business with our API, to see how it performs.

BTC/USD (1 day)

Preliminaries

Definition: The exchange rate r(t) is the market rate of fiat/crypto.

Definition: The merchant rate r'(t) is the modified exchange rate which the merchant is advised to charge customers.

Definition: When a customer buys an item, we call that event a purchase. A purchase consists of the price in fiat and a time. p = (f, t).

Theorem: The amount of crypto for a purchase is found by applying the modified exchange rate t(p) = p(1) / r'(p(2)).

Proof: p(1) / r'(p(2)) = fiat / (fiat/crypto) = fiat * crypto/fiat = crypto

Definition: When the merchant sells their crypto holdings, we call that event a sale. A sale consists of an amount in crypto and a timestamp. s = (c, t).

Theorem: The amount of fiat the merchant obtained from a sale is found by applying the exchange rate to the sale g(s) = s(1) * r(s(2)).

Proof: s(1) * r(s(2)) = crypto * (fiat/crypto) = fiat

Definition: The balance of a set of purchases and sales is the difference between all purchase crypto amounts and all sale crypto amounts. b(P, S) = sum from i to N of t(p_i) - sum from j to M of s_j(1)

Note that b(P, S) >= 0 must always hold.

Definition: The earnings of a set of purchases and sales is the difference between sale fiat amounts and purchase fiat amounts. e(P, S) = sum from j to M of g(s_j(1)) - sum from i to N of p_i(1) >= 0.

Objective

Definition: We say that the merchant rate is favorable iff the earnings are non-negative for most sets of typical purchases and sales. r'(t) is favorable iff e(P, S) >= 0.

In a favorable case, the merchant didn’t lose any fiat by accepting crypto.

most and typical will not be rigorously defined.

As part of typical, we can assume that merchants will sell their crypto in a timely manner. So assume s_i(2) - s_j(2) < W for i,j in {1.. M} for some bound W. Purchase amounts should be randomly distributed within a reasonable range that commerce is done. Perhaps $10-100.

The goal of the bot is to verify that r'(t) is favorable.

Note that this definition is only one measure of quality. Perhaps protecting against the worst case is more important than being favorable. In that case, we would be concerned about the ability to construct a set of purchases with very negative earnings.

Algorithm

Repeat many times:

1. Randomly choose a time range [t0, t1].

2. Generate a set of purchases at random times within [t0, t1]. The price should fall within a range [p0, p1] of typical prices.

3. Generate a set of sales at evenly spaced times (perhaps with slight random noise) within [t0, t1]. Each sale should be for the full balance at that time.

4. Calculate the earnings for these sets.
5. Record the earnings.

After:

1. Report how many earnings were negative and non-negative. Show a percentage for each.
2. Identify the minimum and maximum earnings and report them.

Conclusion

As you read this example, I think your tendency may be to think that its statements are obvious. Certainly, none of these steps are hard. However, it was surprising to me how many of my assumptions were corrected and how difficult it was to choose an objective definition of a favorable outcome. This process helped me become aware of assumptions I would not have even considered if I had started by simply writing code. Perhaps the greatest benefit was that after writing it, I was able to quickly review it with a co-worker and make corrections which were easy on paper, but would have been difficult to change in code.

I hope that thinking in the language of math will bring similar benefits to your projects! Note that this example is only one style of utilizing mathematical thinking.

05/12/2019

A good vector math library is essential for graphics and simulation programming. However, implementing one that is flexible, efficient, and easy to use is difficult. Due to so many choices, experienced programmers tend to write their own to accommodate their preference.

In this article I will survey a few of the most popular techniques and offer some design advice. I will specifically focus on math theory and C implementations.

Math

Before diving into the code. It’s helpful to review some of the math to understand what we are aiming for. One thing to watch for is operations that can be defined in terms of each other. Rarely do I see libraries take advantage of this.

Vector Operations

On vectors in R^N

• addition v + w
• subtraction v - w. Defined by addition: a - b = a + (-b)
• multiplication v * w
• scaling a * v
• normalization. Defined by length and scaling: 1/|v| * v

From R^N -> R

• dot product <v, w>
• length |v|. Defined by dot product sqrt(<v,v>)
• angle. Defined by dot product and length acos(<a,b>/|a||b|)

Only on specific dimension, such as R^2 or R^3

• cross product a X b
• angle (in the plane)

Matrix Operations

On all matrices M(n x m)

• addition A + B
• subtraction A - B Defined by addition A - B = A + (-B)
• scaling bA
• multiplication AB

On square matrices M(n x n)

• determinant det(A)
• inverse A^-1

Between vectors and matrices

• multiplication Av

Most programs use 2, 3, and 4 element vectors, and only a few operations are specific to a given dimension. So a lot of code can be condensed by writing algorithms on N dimensional vectors.

Matrix operations are also very general. But a few should be kept to a specific dimension (usually 3x3 or 4x4). You do not want to implement a general inverse or determinant function.

1. Simple Structs

typedef struct
{
    float x, y, z;
} vec3;

vec3 vec3_add(vec3 a, b)
{
    vec3 r;
    r.x = a.x + b.x;
    r.y = a.y + b.y;
    r.z = a.z + b.z;
    return r;  
}

This works well for smaller programs. The best part is that expressions look nice (a + 2.0*(b-d)):

vec3_add(a, vec3_scale(2.0, vec_sub(b, c)));

But, we have to copy this definition for every dimension. We also have to avoid any algorithms that use index or iteration. Matrix vector multiplication gets ugly.

If you only have a few functions that need indexing and you can index into a pointer to the first member:

vec3 a;
float* v = &a.x;
v[0];

Examples

• vec math

• stb vec

2. Arrays

For N dimensional vectors we might try to write functions which operate on arrays of floats. This is nice because it does not introduce another data structure, so other functions and vector libraries play nice with each other.

Unfortunately, C does not allow you to return an array from the stack. You can only return a pointer which must point to some valid region. So either we do something horrible like malloc in each operation, or pass in arrays for the return value. Passing in arrays works, but it destroys the ability to comfortably write simple expressions such as a + 2.0*(b-d):

void vecn_add(int n, float* a, float* b, float* ret);

// intermediate results everywhere
float temp[3];
vecn_sub(3, b, d, temp);

float temp2[3];
vecn_scale(3, 2.0, temp, temp2);

float final[3];
vecn_add(3, a, temp2, final);

Plain arrays may be appropriate for matrices since they are not typically involved in complex expressions. Matrices and large vectors, which would be inefficient to copy around would also be a good use case.

Depending on the application you may not want to sacrifice performance by introducing loops and branching into every operation. As long as the dimensions are input as a literals or macros, the small loops should be unrolled at compile time.

Examples:

• Accelerate
• linmath

3. Struct + Union

A workaround to return an array from a function is to include it in a struct. The tradeoff is that the size must be fixed and element access is a bit uglier as it requires at least an extra letter.

typedef struct
{
    float e[3];
} vec3;

vec3 v;
v.e[0] = 1;

The access syntax can be cleaned up with a union but, anonymous structs/unions are a GCC extension and are non-standard.

typedef union
{
    float v[3];
    struct
    {
        float x;
        float y;
        float z; 
    }; 
} vec3;

This gives you safe iterative access and nice named members, but it is hard to combine with generic functions. Either you use functions which operate on the internal arrays and deal with the intermediate results. Or, define fixed dimension functions which wrap the generic ones:

vec3 vec3_add(vec3 a, vec3 b);
{
    vec3 temp;
    vecn_add(3, a.v, b.v, temp.v);
    return temp;
}

I don’t love this option. If I need to write wrapper functions I might as well go back to method 1 and copy implementations around.

Examples:

• Math 3D (See Matrices)

4. Macros

Some clever macros can help you get the best of both worlds, and parameterize the scalar types. This can be combined with an array or union data structure. But, writing multi-line macros isn’t very fun.

#define DEFINE_VEC(T, N, SUF) \
\
void vec##N####SUF##_add(const T *a, const T *b,  T *r) \
{ \
    for (int i = 0; i < N; ++i) \
        r[i] = a[i] + b[i]; \
} \

Then define the types you need:

DEFINE_VEC(float, 2, f);
DEFINE_VEC(float, 3, f);
DEFINE_VEC(float, 4, f);

Usage:

vec3f_add(a, b);

Functions which only apply to a specific dimension can be defined outside of the macro:

void vec3f_cross(const float* a, const float* b, float* r)
{
    // ...
}

Examples:

• linmath

Closing Thoughts

In typical C fashion, I believe it is misguided to try to write the one true vector library to serve all purposes. These libraries are bloated and must choose tradeoffs which don’t fit your use case. Instead use the examples above to write to tailor make vector functions as needed.

For further reading, see On Vector Math Libraries. It focuses on C++ and has a few other handy tips. You can also read a discussion which led to these notes.

04/13/2019

One of my favorite courses in college was philosophy of language. Along with interesting philosphy it introduced me to the foundations of math project which has since become one of my favorite subjects to learn about.

Some of my favorite books of all time come out of this interest. I wanted to organize a few of these into a list for others who are interested in the topic. Note that many good books were left out in favor of the very best.

You will notice a theme in my commentary. The books I like most are those that don’t shy away from hard technical knowledge, but also explore the philosophical ideas behind them.

The list is arranged in a progressive sequence that will help prepare you for the next one.

Logicomix

By: Apostolos Doxiadis

Logicomix is actually a comic book! It tells an engaging historical narrative about the search for the foundations of math and the birth of analytic philosophy, in the early 20th century.

It introduces you to all the major characters such as Gödel, Russel, Frege, and Wittgenstein and motivates the kinds of problems they were trying to solve. The book also explores how these ideas connect to modern computer science.

It is an absolute joy to read and will give you a taste of whether this is an interesting subject for you.

Gödel, Escher, Bach

By: Douglas Hofstadter

You probably have seen this work recommended elsewhere. Gödel, Escher, Bach really deserves all the praise that it gets.

Hofstadter covers an enormous range of topics including formal systems, Godel’s proof, theory of computation, programming, molecular biology, and artificial intelligence. Every topic is presented beautifully and with a lot of philosophical discussion. In many ways it is an introduction to the big ideas in modern science. It is written for a general audience and assumes no mathematical background.

Gödel’s Proof.

By: James Newman & Ernest Nagel

Gödel, Escher, Bach does a good job of introducing the incompleteness theorem and discussing its ramifications, but if you are like me you probably still won’t completely understand it after a first reading.

This concise book offers another perspective and a clear explanation of the mathematics of the proof, its general strategy, and the historical context surrounding the incompleteness problem.

Descartes Dream

By: Phillip Davis & Reuben Hersch

In the 17th century Descartes had a dream in which he saw a future world driven by mathematical calculations and logical systems. The theme of this book is how this dream has become a reality.

The book explores how mathematics and computer science work together, surveys several interesting fields, and examines ethical issues in the technological world. This book is not technical, and should be appropriate for anyone interested in science or technology.

If you like this book, the authors wrote another called The Mathematical Experience which is focused more on pure mathematics.

Computation: Finite & Infinite

By: Marvin Minsky

Marvin Minsky is an incredibly clear and deep writer. In this work he provides a mathematical framework for thinking about mechanical machines and develops the theory of computation.

This book teaches you all you need to know about Turing machines, finite state, and neural networks. My project: McCulloch & Pitts Neural Net Simulator is based on this book.

To read this book, you should understand some logic and basic set theory such as that taught in an introductory proofs course.

Unfortunately, it is out of print and may be difficult to obtain (for a reasonable amount of money). I read it from my university’s library. If that is not an option, I recommend you find it online. If anyone knows of a place where I can reasonably purchase this book, let me know.

Structure and Interpretation of Computer Programs

By: Gerald Sussman & Hal Abelson

Read Online

This classic text is designed to teach programming to MIT students who have some technical background in another areas of math and science. It is a hard read, but it assumes no programming knowledge and teaches Scheme (a Lisp dialect) and its full inter-workings from the ground up.

If you want to be a professional programmer, this may be the only book you need to study. What other book teaches you to write a symbolic differentiator, interpreter, circuit simulator, and compiler?

Most of the material is mixed in the excercises so don’t skip them!

But, this is not just a programming book. It belongs in this list because it teaches the fundamental concepts of computation. See the section data as programs for an example.

Logical Foundations of Mathematics and Computational Complexity

By: Pavel Pudlak

This book is a massive and dense survey of topics including formal systems, set theory, abstract algebra, computability theory, analysis of algorithms, and quantum computing. The first chapter covered almost everything I had learned about algebra and meta-mathematics in my entire undergraduate degree!

Pudlak does a fantastic job of balancing technical information with philosophical discussion. I can’t recommend this book enough.

Reading this book definitely requires some mathematical maturity. The author does his best to explains every concept in the book but it would be hard for me to read about a “group” for the first time and really understand what he means.

If you read through the other books, and have technical knowledge you should be well prepared.

02/22/2019

Updated: 04/27/2020

A friend and I had a discussion about the basic skills that are often lacking in experienced programmers. How can a programmer work for ten or twenty years and never learn to write good code? Too often they need close supervision to ensure they go down the right path, and can never be trusted to take technical leadership on larger tasks. It seems they are just good enough to get by in their job, but they never become effective.

We thought about our experiences and came up with three fundamental skills that we find are most often missing. Note that these are not skills which take a considerable amount of talent or unique insight. Nor are they “trends” or “frameworks” to help you get a new job. They are basic fundamentals which are prerequisites to being a successful programmer.

Understand how the language works

Programmers cannot write good code unless they understand what they are typing. At the most basic level, this means they need to understand the rules of their programming language well. It is obvious when a programmer doesn’t because they solve problems in indirect ways and litter the code with unnecessary statements that they are clueless as to what they actually do. Their mental model of the program does not match with the actual behavior of the code.

You may have seen code which misunderstands how expressions work: (1)

if isDelivered and isNotified:
    isDone = True
else:
    isDone = false;

Instead of:

isDone = isDelivered and isNotified

In JavaScript, this is often indicated by new Promise inside a .then(). In C++, it is attaching virtual to every method and destructor and creating every object with new.

Debugging is also extremely difficult if you don’t understand the language. You may add a line of code because it fixes a bug for reasons you don’t understand. Bugs are mysteries that seem to appear organically, like dust on the shelves. The code has a mind of its own.

Understand the code you write. Know what every line does and why you put it there.

Once you understand the language well, its important to know about implementation; what goes on inside the computer or library? Do you know how the code gets to assembly? Do you know how a closure captures variables? Do you know how the garbage collector works? Do you know how a map/dictionary works? Do you know how an HTTP request is made?

Modern software and computers are too complex to know everything (2). The idea is not not to be too clever, but to avoid doing silly things. Silly mistakes result in sluggish code that is wasteful of system resources, or just does unexpected things. Removing an item from the front of a C++ vector requires copying the entire vector which is worth thinking about in large cases. Writing map<string, map<string, ...>> in C++ (a dictionary of dictionaries) creates a a self-balancing tree in which every node is a self-balancing tree, a data structure nobody would intentionally design. (3)

A muddy understanding of how things work is typical of beginners, but it is all too often a problem with experienced programmers if they are not curious and do not take time to learn how things work beyond their immediate job’s needs. Learn just a bit more about the stuff you use most works.

Anticipate problems

To write reliable code, you must be able to anticipate problems, not just patch individual use cases. I am shocked by the number of times I see code that puts the program in a broken state when a very likely error happens.

I recently reviewed some code that made an HTTP request to notify a server of a state change in which the programmer assumed the HTTP request would always succeed. If it failed, (and we know how often HTTP requests fail), a database record was put into an invalid state. The questions they should have asked when writing this code are: What happens if this fails? Is there another opportunity to send the notification? When is the correct time to record the state change? Careful programmers think through the possible states and transitions of their program.

Using sleep(), cron jobs, or setTimeout is almost always wrong because it typically means you are waiting for a task to finish and don’t know how long it will take. What if it takes longer than you expect? What if the scheduler gives resources to another program? Will that break your program? It may take a little bit of effort to rig a proper event, but it is always worth it.

Another common mistake I see is generating a random file or identifier and hoping that collisions will never happen (4). It is reasonable for an unlikely event to cause an error, but it is not ok if that puts your program in an unusable state. For example, if a successful login generates a session token and it collides with another token, you could reject the login and have the user try again. It is a freak accident that slightly inconveniences the user. On the other hand, what if you generate storage files with random names and you have a collision? You just lost someone’s data! “This probably won’t happen” is not a strategy for writing reliable code.

Unit testing can’t solve this problem either. It can help you stop and think about some inputs to write a test, but more than likely, the cases that you write tests are the ones you anticipated when you wrote the code! Unit testing cannot transform fragile code into reliable code.

Fragile code is often caused by a lack of experience regarding things that can go wrong, but it can also be the result of a long career of maintaining existing codebases. When working on a large existing system, you typically fix individual bugs and aren’t rewarded by your bosses for improving the system as a whole. You learn that programming is a never-ending patch. Increasing the sleep() time may fix the bug today, but never solve the underlying issue.

Organize and design systems

Even when armed with the other two skills, it’s hard to be effective unless you can organize code into a system that makes sense. I believe OOP and relational database get a lot of flack because programmers tend to be bad at design, not because they are broken paradigms. You simply can’t create rigid classes, schemas, and hierarchies without thinking them through. Design itself is too broad a topic to explore in this article (read Fred Brooks), so I want to focus on a few specific attributes that well-designed software tends to have.

You may have heard a rule like “don’t make functions or classes too long”. However, the real problem is writing code that mixes unrelated ideas. Poorly designed software lacks conceptual integrity. Its concepts and division of responsibilities are not well defined. It usually looks like a giant Rube Goldberg machine that haphazardly sets state and triggers events.

Accordingly, good software is built from well-defined concepts with clear responsibilities. Mathematicians and philosophers spend a lot of time discussing definitions because a good definition allows them to capture and understand some truth about the world. Programmers should think similarly and spend a comparable amount of effort grappling with ideas before writing code.

Good programmers ask questions like:

• “What is this function’s purpose?”
• “What does this data structure represent?”
• “Does this function actually represent two separate tasks?”
• “What is the responsibility of this portion of code? What shouldn’t it ‘know about’?”
• “What is necessary to be in the public interface?”

Luckily the field is ripe with strategies to help you design code. Design patterns and SOLID can give you guidelines for designing classes. Functional programming encourages writing pure functions (input -> output and no side effects) and maintaining as little state as possible. Model view controller aims to separate UI and storage concerns from program logic. On the other hand, React components form conceptual units by combining the HTML, CSS, and JS into a single component. Unix rejects categories and says everything is a file. All of these seemingly contradictory ideas are valid. The important thing is that the concepts make sense and map closely to the problem you are solving.

Software that is well-designed is also software that is easy to change. Of course, it’s too much to ask it to satisfy requirements that contradict its original intent. But, it should accommodate changes that are natural evolutions. A common mistake I see is solving a problem for a few cases, instead of N cases. (If you have a variable called button3.) Another is treating everything as a special case using switch statements instead of using polymorphism. (5)

I think the best way to learn about design is to write and study a lot of programs. Programmers who work only on old programs never learn to write new ones. The studying part is key too. Programmers who only work on small temporary projects (like an agency) may get by without ever improving how to design programs. Good design comes gradually with experience, but only if you think about it and try to improve.

There are no tricks or rules that you can follow to guarantee you will write good software. As Alex Stepanov said, “think, and then the code will be good.”

1. There may be some cases where the previous style is preferred. This example is only an illustration.

2. I was shocked when I wrote some multithreaded code and first faced bugs due to Cache coherence and instruction reordering!

3. A reader corrected me about the performance of modern map implementations. Map of map is just as good as alternatives, like map of pair. I think it is a good illustration of code being suprising, but perhaps I am doing a bit too much early optimization :) See his benchmark.

4. There are ways to do this sort of thing correctly using cryptographic hashes.

5. Once again, this is just an example. Switching on type may be perfectly appropriate.

These are notes sent to a friend, not a formal article. I thought they might be useful for others as well.

05/01/2017

If you’re looking for an easy way to do 3D its probably SceneKit or Unity, but I assume you want to learn how things work.

Everything you know and love in OpenGL is gone. glTranslate, glLookAt, glRotate, glVertex, glLight, etc. The reasoning is that these all describe a specific way of doing things, and OpenGL wanted to be as general as possible so that you can make whatever you want.

For example glLight has a built in algorithm for lighting. What if you want to make your own? You can’t.

The solution is to make everything programmable. Instead of having fixed GPU procedures, the GPU becomes flexible so you can write code which runs on the GPU that replaces the fixed procedures. These programs are called shaders.

Shaders

There are two kinds of shaders you need to worry about.

Vertex Shader: takes input data, outputs vertex information (position, color, etc)

The outputs from this vertex shader are interpolated across the triangle (or line) you are drawing, and the results are inputed into the fragment shader.

Fragment Shader: takes input data, and interpolated vertex information, and outputs a color.

Shaders are written in a C like language and compiled into native GPU code. I would say the first thing you need to do is practice writing some shaders, so you can get an idea of what data you need to give them.

The modern OpenGL API has been totally stripped down. It basically consists of creating shaders, providing them with data, and executing them. That is it. There is nothing else in OpenGL.

Data

So here is the kind of data you need to prepare for OpenGL. This is done in your main code.

vertex positions, vertex colors, vertex normals.

You use vertex buffers to upload this data to the GPU. Basically you can make a vertex struct:

struct vertex { Vec3 position, normal, color }

and then describe its structure, and then upload an array of them. You could make one vertex buffer for your sphere and then reuse it for each ball.

Textures

You upload images using glTexImage. This APIs has stayed mostly the same as old OpenGL.

Matrices

Since OpenGL has no translate, rotate, etc. You need to create your own transformation matrices for every object in the scene.

This has a few components:

• model matrix - this is the translation/rotation for each individual model.
• view matrix - this describes the camera in the scene (gluLookAt)
• projection matrix - this is the “3D lense” which projects 3D coordinates onto the 2D screen.

Basically you have to calculate all of these yourself, pass them to your vertex shader, and then use them to translate the vertex positions.

Usually the projection and matrices are constant for one draw of the scene, you create them at the beginning and pass them with everything. Then for each model you need to construct a new model matrix, describing its current state.

Recommended Resources

• Learn OpenGL
• Learning Modern 3D Graphics Programming

11/10/2016

At work, I recently started on a new medical application for the Microsoft HoloLens. I don’t follow popular technology news, and rarely get excited about new gadgets, but I feel that the HoloLens is a robust platform, with real substance beyond the usual VR gimmicks. In this article I will talk a little bit about my experience with the hardware and development environment, as well as outline some its shortcomings. As the current release is a “Developer Edition” I hope some of the items will be addressed in the consumer release.

Where the HoloLens succeeds most, is in offering an entirely distinct experience from VR headsets such as the Vive and Occulus. Everyone I have spoken to, who has tried both, immediately agrees the two are like apples and oranges. There is no question that the HTC Vive offers the most immersive graphical experience, but I get the feeling I could actually use the HoloLens for work, education, and other tasks beyond entertainment. The crisp display, is clear and readable, and since it doesn’t overwhelm your vision it doesn’t feel nauseating. The likelihood, of me setting up a full Vive room setup, to look at a piece of anatomy or the solar system is small, but quickly putting on the HoloLens to view a models from a true 3D perspective is helpful.

Microsoft recommends using Unity to get started with HoloLens development, while also providing native C++ and Direct 3D. Initially. using Unity didn’t excite me, but the it ends up working fairly well. Unity has a specific version of its engine adapted for HoloLens. This includes APIs and components for accessing the room scanning, world anchors, gestures, and voice controls. Once a build is exported from Unity, you can run, debug, and write code all in Visual Studio, until you need to modify the scene. The HoloLens can connect directly through USB or the WiFi for debugging. Microsoft has solid documentation and tutorials to help you figure everything out.

The voice controls are excellent, and easy to develop for. While in my quiet office, I have never had it mistake me for simple commands such as “back” and “select”. I’ll even admit to working with food stuffed in my mouth, and it still recognized my commands about half the time. Speaking commands to yourself is still as embarrassing as it sounds (less so than shouting at your phone), but the technology is solid, and I believe there is potential here. Development for voice controls is a pleasant process as well. In Unity you can simply register phrases you would like to listen for, and receive callbacks when they are spoken.

The most noticeable limitation is the small field of view. Even though the glasses surround almost your whole vision, the displayable portion takes up only a small rectangular box at the front of your vision. This means that any objects larger than a basketball get chopped off while standing only a few feet away. This ugliness is most noticeable when a large model fills the whole viewing rectangle, leaving everything outside blank. It looks as if you are looking at the world through an open cereal box. Unfortunately, I can’t show you what this looks like in a screenshot.

The gesture controls aren’t terrible, but still feel clunky. It takes a little bit of time to master, and then it’s sort of like having a mouse that fails to click every tenth time. They are made worse when other people are nearby to distract the sensor.

Since the entire system is contained within the headset, its GPU is more comparable to a mobile phone than a desktop machine. While not weak, the hardware feels a bit under powered. The frame rate cuts to about 30 frames, with roughly 50k triangles across 25 draw calls. I don’t expect a ton here, but a bit faster would be nice. In my experience an iPad 2 has better GPU performance, but probably does a lot less background work. Given that I only tested in Unity, it is possible that using native C++ and Direct X would be a dramatic improvement. If you are using Unity be sure to use the optimized shaders found in Microsoft’s Holokit, and be conscious of batches.

The shortcomings above can hopefully be addressed in future iterations of the hardware. In its current form, it is still a fundamentally innovative platform and I am excited to continue developing for it.

10/30/16

In 2012 I came across the paper OverCoat: An Implicit Canvas for 3D Painting which describes a technique for creating 3D models using traditional painting methods. At the time I didn’t have sufficient mathematical knowledge to understand all of the technical details described, but I picked up the general idea and wrote my own rough implementation. This article explains how my program was constructed and some of what I learned in the process.

The basic idea is to paint 3d strokes onto a rough 3D model which acts as a canvas. The model is simply rotated to a desired orientation, and drawn upon, as if one was painting ontop of a flat a picture. The difference is that the strokes are projected into 3D space giving them their own 3D form.

The model is only a placeholder which is eventually removed, leaving only the painting made up of strokes. To position the nodes which define a stroke, my program simply performed a ray-triangle test between the mouse ray and the mesh, and offset the intersection point by the face normal. The paper describes a sophisticated optimization method which evenly distributes stroke nodes, even across harshes changes in depth.

The image above shows several strokes projected onto the surface of the model. Along with the node positions, information such as a color and brush size are stored for each stroke.

I used the following data structures:

typedef struct
{
    Vec4_t color;
    int brush;
    float pressure;
} StrokeInfo_t;

typedef struct
{
    Vec3_t pos;
    float radius;
} StrokeNode3D_t;

typedef struct
{
    int nodeCount;
    StrokeNode3D_t nodes[MAX_STROKE_NODES];
    Vec4_t color;
    // used for depth sorting
    Vec3_t center;
    StrokeInfo_t info;
} Stroke3D_t;

typedef struct
{
    int strokeCount;
    Stroke3D_t strokes[MAX_STROKES];
    int dirty;
} Canvas3D_t;

To render the strokes, each node position is transformed from 3D onto the 2D screen, using the camera matrix. The brush image is then rendered onto the framebuffer at each of the 2D node positions on the screen. Even though these screenshots show only a few colors and a single brush, the program supports multiple brush types and RGB colors.

As the drawing becomes more dense, the strokes begin to resemble their own shape. Rendering strokes is a very efficient process because it only involves stamping the brush image in 2D, which is easily parallelized on the GPU. For the strokes to be drawn in the correct order they must be sorted by depth. This is by far the most computational expensive part of the process. My program solved this problem by only sorting strokes by their average position, which seemed to be an acceptable alternative to sorting each node. The order in which the nodes of a stroke are rendered is still important to ensure proper layering. An offline rendering process should sort each node properly.

Although my program cannot produce the beautiful paintings found in the original paper, it can still draw some fun pictures. I think the technique itself is very interesting and has great potential. It would be interesting to make a small game, or animation centered around the technique since the artwork is so easy and natural to produce. Because it requires no textures, other than the small brushes, its memory usage is also very small, while its vertex requirements are similar to triangle meshes. The next natural step would be to bind a painting to a skeleton for animation.

10/15/2015

I am not much of a visual artist, but I still enjoy 3D modeling and animation. This page contains pictures of some of my older models I have worked on.

Moon Rover

Low-poly Cargo Ship

Space Invader Sculpt

While working at Infuse Medical, I taught a course on 3D programming with OpenGL. As part of my course we created a small space invaders clone. Another developer created the low-poly model for the alien, which I use to create the high resolution sculpt shown below. Occlusion maps were generated from the sculpt for the game.

Low-poly Radio

Pixel Art

Another Ship

10/09/2015

I spent quite a bit of time experimenting with game technology when I was first learning how to program. I played with tons of ideas, many of which turned out to be no good, but I learned a ton about graphics and development in the process.

An extremely fast minecraft clone. Written in C and OpenGL.

I wrote my own lightmapper and UV unwrapper in Objective-C.

A small demo for a sci-fi adventure game engine. Written in C and OpenGL.

An adventure game in which players explore the ocean. One feature I like allows the player to take a picture and store it in an album. The game recognizes which fish are in the picture so it can reward them for collecting them all.

An action platform game where players avoid obstacles by jumping and running up walls.

A puzzle game I created for a school competition. Written in C++ and OpenGL.

A platform shooter game.

A prototype of a strategy game for iOS.