Snapshot Testing for the Slow Parts of your Tests

This week I discovered a technique for speeding up tests using snapshot testing[1] while avoiding the downsides of snapshot testing.

This technique works best if:

{
  "type": "unordered-list",
  "id": "222d4f75-48c3-4468-b44b-4e842a357857",
  "items": [
    {
      "id": "222d4f75-48c3-4468-b44b-4e842a357857",
      "content": [
        {
          "type": "text",
          "content": "you have a slow section of your test that you want to speed up"
        }
      ]
    },
    {
      "id": "508adaa3-5361-4992-8506-a2925a3f08c1",
      "content": [
        {
          "type": "text",
          "content": "that section's output entirely depends on its input (its idempotent)"
        }
      ]
    }
  ]
}

Let's say we have a test that looks like this:

{
  "id": "462923c0-9598-4aba-831b-9440ff58859b",
  "type": "code",
  "content": [
    {
      "type": "text",
      "content": "func Test(t *testing.T) {\n  mainCode := main.Generate(t)\n  writeFile(t, \"main.go\", mainCode)\n  actual := program.Run(t)\n  assert.Equal(t, `expected`, actual)\n}"
    }
  ],
  "language": "javascript"
}

Maybe program.Run is a slow operation. In my case, I was compiling and running generated Go code, but it could also be a database or network operation.

The technique I'm about to describe works best if

{
  "type": "unordered-list",
  "id": "100a1c74-849c-4ad7-baad-9b0ef0fd0f78",
  "items": [
    {
      "id": "100a1c74-849c-4ad7-baad-9b0ef0fd0f78",
      "content": [
        {
          "type": "text",
          "content": "you have a slow part of your test that you want to speed up"
        }
      ]
    },
    {
      "id": "21b91806-e1b4-4f26-bf3f-36325f34edd2",
      "content": [
        {
          "type": "text",
          "content": "that part's output depends on it's input (it's idempotent)"
        }
      ]
    }
  ]
}

Here's what you can do.

{
  "id": "e2019c4e-120a-46a9-92a9-40fb201ed417",
  "type": "code",
  "content": [
    {
      "type": "text",
      "content": "var fast = flag.Bool(\"fast\", false, \"run tests without running Go processes if possible\")\n\nfunc Test(t *testing.T) {\n  mainCode := main.Generate(t)\n  if *fast \u0026\u0026 snapshot.Exists(t.Name()) {\n    expected := snapshot.Read(t.Name())\n\t  diff.TestString(t, expected, mainCode)\n\t\treturn\n  }\n  actual := program.Run(t, mainCode)\n  assert.Equal(t, test.expected, actual)\n  snapshot.Read(t.Name(), mainCode)\n}"
    }
  ],
  "language": "javascript"
}

I've always felt uneasy about Snapshot Tests (aka Golden Tests).

{
  "id": "251ec3e4-5575-4cf8-8237-d5b877e5b8cd",
  "type": "code",
  "content": [
    {
      "type": "text",
      "content": "it('test', () =\u003e {\n  const actual = runCode()\n  expect(actual).toMatchSnapshot();\n});"
    }
  ],
  "language": "javascript"
}

On one hand, they take away of a lot of the toil of updating tests. Raise your hand if you've also spent an entire weekend just updating tests?

On the other hand, you just don't understand your tests and it's too easy to update them without thinking. The grunt work seems necessary to do it correctly.

Well this week, I discovered an interesting use case of snapshot testing that doesn't have

First we need a bit of context: I'm building a code generator for doing dependency injection. Code generation is the fast part, the slow part is compiling the code and running the code. Each tests takes about 600ms. I have about 100 tests so that's already a minute, just for that test suite and I'm just getting started. I found starting to make some weird tradeoffs to group tests and reduce the number of tests being run.

Snapshot testing offers a solution to this: whenever a test passes, take a snapshot. Then next time compare with that snapshot, if actual is different than the snapshot, fail.

I find the Snapshot testing you find in Jest way too easy to mess up. It's too easy to run -update and suddenly all your tests pass again.

For speeding up tests, snapshot testing is awesome.

For those of you in the Go

(aka Golden Tests)

I stumbled on a more useful for Snapshot testing.

When most people think of snapshot testing, they think of a test like this