
[accompanying code]: https://github.com/hallettj/hacker-news-ts
[tdd]: http://www.olioapps.com/blog/type-driven-development-with-typescript/

_This is a follow-up to [Type-Driven Development with TypeScript][tdd]._

The shape of data defines a program.
There are important benefits to writing out types for your data.

Let's consider a Hacker News client, which consumes stories and other
items from the [Hacker News API][]. This is a TypeScript type that
describes the format for stories:

[hacker news api]: https://github.com/HackerNews/API

```typescript
type Story = {
  type: "story"
  by: string // username
  dead?: boolean
  deleted?: boolean
  descendants: number
  id: number
  kids?: number[] // numeric IDs of comments
  score: number
  text?: string // HTML content if the story is a text post
  time: number // seconds since Unix epoch
  title: string
  url?: string // URL of linked article if the story is not a text post
}
```

In Javascript and other dynamically-typed languages, it is common to write
a program without any explicit description of a data structure like `Story`.
The shape of the data is implied in the code that manipulates the data.
But that means anyone reading the code has to mentally reconstruct that
shape from context, or refer to documentation outside of the program itself.

## Catching mistakes

TypeScript provides the option of documenting data structures in the form
of types.
An obvious advantage is that the type checker can identify mistakes like typos
in property names when accessing data.
The Hacker News API uses the property name `descendants`;
for some reason every time I try to type `descendants` I end up typing
`descendents` by mistake.
If I did not have a type checker to point out that `Story` does not have
a property named `descendents` I could end up wasting a lot of time debugging!

## List all changes

But this just scratches the surface. Types for data structures help keep
programmers oriented. When a data structure is suddenly required to change,
all you need to do is to update that particular type and the type checker
will list all of the changes that need to be made to work with this new type.

## Reducing cognitive load

When you come back to a program after you have been away from it long enough to
forget how everything works, having descriptions of data structures right
there in the code makes it much easier to understand what is going on.
The same is true if more than one person is working on the project.
Every detail that can be captured in types is one less detail that programmers
have to carry in their heads.
Reduced cognitive load leaves programmers with more energy for writing
important business logic.

## Bridging the gap with validators

But what if you make a mistake when you write the type?
I mentioned that I had problems mixing up `descendants` and `descendents`.
I actually made the same mistake the first time I wrote the `Story` type.
The type checker cannot help me if I give it bad information from the start!
Unfortunately, a static type checker cannot check types against data from an
external API.
But what you can do is to write a validator that will check at runtime that
incoming data has the shape that you expect.
Then you can extract a static type from the validator that is guaranteed to
match any values that pass validation.

There is a nifty library called [io-ts][] that works like magic.
Instead of the `Story` type above
We can define a validator using io-ts combinators:

[io-ts]: https://github.com/gcanti/io-ts/blob/master/README.md

```typescript
import * as t from "io-ts"

// The `V` in `StoryV` is short for `Validator`
const StoryV = t.type({
  type: t.literal("story"), // value of property called `type` is the exact string `"story"`
  by: t.string, // username
  dead: optional(t.boolean),
  deleted: optional(t.boolean),
  descendants: t.number, // number of comments
  id: t.number,
  kids: optional(t.array(t.number)), // IDs of comments on an item
  score: t.number,
  text: optional(t.string), // HTML content if story is a text post
  time: t.number, // seconds since Unix epoch
  title: t.string,
  url: optional(t.string), // URL of linked article if the story is not text post
})

// The `optional()` combinator is defined later in the article
```

This looks similar to the `Story` type from the beginning of the post.
`StoryV` expresses the properties of objects coming from the Hacker News API
with a type for each property.
(The `t.type()` combinator produces a validator that expects an object with the
given property names and types.)
But this time the "types" for each property are actually values supplied by
io-ts: `t.number`, `t.string`, `t.boolean`, etc.
Values can be referenced at runtime, types cannot.
With `StoryV` we can validate any arbitrary Javascript value by calling
`StoryV.decode(whateverValue)`.
If the given value is not an object with the expected properties then `decode`
will return an error value.

## From validator to type

What makes io-ts uniquely valuable is that it simultaneously defines
a runtime validator and a static type.

If `StoreV.decode()` returns a success result, then TypeScript knows that
the resulting value has a `descendants` property and does not have a `descendents` property.

If a value passes validation, then it is guaranteed to match that static type,
and we can use it to check the correctness of the rest of the program.
If a value does not pass, then you will get a failure with a clear point in the
program where it should be handled.

For example:

```typescript
import fetch from "node-fetch"

async function fetchTitle(storyId: number): Promise<string> {
  const res = await fetch(
    `https://hacker-news.firebaseio.com/v0/item/${storyId}.json`,
  )
  const data = await res.json()

  // If the data that is fetched does not match the `StoryV` validator then this
  // line will result in a rejected promise.
  const story = await decodeToPromise(StoryV, data)

  // This line does not type-check because TypeScript can infer from the
  // definition of `StoryV` that `story` does not have a property called
  // `descendents`.
  const ds = story.descendents

  // TypeScript infers that `story` does have a `title` property with a value of
  // type `string`, so this line passes type-checking.
  return story.title
}

// `decodeToPromise` is defined later in the article
```

Validating incoming data at runtime allows the program to fail fast if there is
a mismatch between the data and the program's expectations.
In development, that makes it easy to catch bugs early:
any mismatch is identified immediately at the point where you call
`decodeToPromise`.
You don't need fixtures or unit tests to check data ingress.
Yes,
the validation step could lead to failures in production that you would not
have seen otherwise if data comes in some unexpected shape under some condition -
but the alternative is for the program to limp along with unknown data leading to
possibly-undefined behavior.
Failing fast is better!

To minimize unnecessary validation errors it is a good idea to make your
validators permissive in what they accept.
For example,
err on the side of marking properties as optional if there is any possibility
that those properties will be absent in some cases.
And you can exclude properties from the validator that you are not going to use
in your program.

## Referencing types produced using io-ts

`StoryV` replaces the hand-written `Story` type -
so we no longer have a way to refer to the type of story objects.
But we can get that type back!
Io-ts provides a type operator called `t.TypeOf` that extracts a static type
from a validator.
We can define a new `Story` type like this:

```typescript
type Story = t.TypeOf<typeof StoryV>
```

Every TypeScript value has a type.
You can reference and manipulate the value at runtime.
Likewise, you can reference and manipulate the type at type check time.
The expression `typeof StoryV` uses TypeScript's built-in `typeof` operator to
get the typecheck-time representation of `StoryV`
which conveniently holds a complete description of the shape of story objects.
That description is wrapped in a validator type;
`t.TypeOf` pulls the shape description out into an independent type.

You can use the computed `Story` type in annotations in the rest of your
program:

```typescript
function formatStory(story: Story): string {
  return `"${story.title}" submitted by ${story.by}`
}
```

## When data comes in different shapes

The Hacker News API publishes more than just stories.
The `/v0/item/` endpoint alone also provides comments, job postings, polls, and
poll options,
which all have different shapes.
We want to be able to fetch an item from that endpoint
and use a runtime check on the `type` property in the returned object to
determine what type of item it is.
And we want the type checker to verify the correctness of the whole process.

Let's use io-ts to create some more item definitions.
These will be similar to the definition of `StoryV`.
Here are abbreviated definitions
(see the [accompanying code][] for complete definitions):

```typescript
const CommentV = t.type(
  {
    type: t.literal("comment"),
    parent: t.number,
    text: t.string, // HTML content
    /* ... */
  },
  "Comment",
) // The second argument is a label that makes validation messages nicer.
type Comment = t.TypeOf<typeof CommentV>

const JobV = t.type(
  {
    type: t.literal("job"),
    text: optional(t.string), // HTML content if job is a text post
    url: optional(t.string), // URL of linked page if the job is not text post
    /* ... */
  },
  "Job",
)
type Job = t.TypeOf<typeof JobV>

const PollV = t.type(
  {
    type: t.literal("poll"),
    descendants: t.number, // number of comments
    parts: t.array(t.number),
    /* ... */
  },
  "Poll",
)
type Poll = t.TypeOf<typeof PollV>

const PollOptV = t.type(
  {
    type: t.literal("pollopt"),
    poll: t.number, // ID of poll that includes this option
    score: t.number,
    text: t.string, // HTML content
    /* ... */
  },
  "PollOpt",
)
type PollOpt = t.TypeOf<typeof PollOptV>
```

The Hacker News item API could return a story or any of these types,
which means that the type of values from the item API is a union of all five
types.
More specifically the type is a _tagged union_:
the `type` property in API responses is a tag that we can use to distinguish
between types within the union.
A tagged union validator looks like this:

```typescript
const ItemV = t.taggedUnion(
  "type", // the name of the tag property
  [CommentV, JobV, PollV, PollOptV, StoryV],
  "Item", // a label to make validation messages nicer
)
type Item = t.TypeOf<typeof ItemV>
```

This is why it was important to use the `t.literal()` combinator instead of
`t.string` for the type of the `type` property in each item validator:
using `t.literal()` with a literal string makes the exact string value
available to the type checker.
With that information, TypeScript can use type guards to narrow the type of an
`item` to a specific item type based on the value of `item.type`.
For example:

```typescript
function formatItem(item: Item): string {
  switch (item.type) {
    case "story":
      // Stories have titles, so this is ok.
      return `"${item.title}" submitted by ${item.by}`
    case "job":
      return `job posting: ${item.title}`
    case "poll":
      // Only polls have a `parts` property - this would not pass type checking
      // without the type guard.
      const numOpts = item.parts.length
      return `poll: "${item.title}" - choose one of ${numOpts} options`
    case "pollopt":
      // In some item types `text` can be undefined, but not in poll options.
      return `poll option: ${item.text}`
    case "comment":
      const excerpt =
        item.text.length > 60 ? item.text.slice(0, 60) + "..." : item.text
      return `${item.by} commented: ${excerpt}`

    // Usually TypeScript will report an error if you do not include
    // a `default` case in a `switch`. But in this function TypeScript infers
    // that all possible item types have been handled.
  }
}
```

By the way, io-ts also supports intersections, untagged unions, and other fun
combinators.
Oh, and io-ts supports Flow too - not just TypeScript!

## Next steps

This was just a quick introduction to what io-ts is capable of,
and techniques for applying type-checking to external data.
The concepts here are not limited to consuming API data:
I recommend similar use of io-ts validators when working with data loaded from
a database, serialized messages between micro-services, user input, or any other
case where data can come in from outside the program.

The best way to cement your understanding of a pattern is to experiment with it.
I encourage you to check out the [accompanying code][]
and try adding some features.
One idea is to display ID numbers with story titles
and add an option so that if the user passes an ID as a command-line argument
when running the script it displays a link and some comments on the
corresponding story.

## Appendix A: definition for `optional`

In a hand-written definition for an object type you can use a question mark to
indicate that a property might be absent:

```typescript
type Story = {
  text?: string
  url?: string
}
```

There is no easy way to do that with io-ts because the argument to `t.type()`
is an actual object, and object properties are either present or not present.
There is another combinator, `t.partial()`, that describes an object where all
properties optional.
The idiomatic way to represent an object where some properties are optional is
to use an intersection of `t.type()` for required properties, and `t.partial()`
for optional properties:

```typescript
const StoryV = t.intersection(
  [
    t.type({
      // required properties
      type: t.literal("story"),
      descendants: t.number, // number of comments
    }),
    t.partial({
      // optional properties
      text: t.string, // HTML content if story is a text post
      url: t.string, // URL of linked article if the story is not text post
    }),
  ],
  "Story",
)
```

I used a different pattern in this article.
I didn't want to introduce too many concepts all at once;
so I didn't introduce intersections and nested definitions right away.

My `optional()` combinator is a union of the given type with `undefined`.
Technically this implies that we expect the given property to be present in
every case,
but that the value might be `undefined`.
In practice, that distinction often does not matter,
and io-ts will validate an object that is missing a required property if the
type of that property is allowed to be `undefined`.
But note that io-ts might make object validation more strict in the future!

This is the definition of `optional`:

```typescript
function optional<RT extends t.Any>(
  type: RT,
  name: string = `${type.name} | undefined`,
): t.UnionType<
  [RT, t.UndefinedType],
  t.TypeOf<RT> | undefined,
  t.OutputOf<RT> | undefined,
  t.InputOf<RT> | undefined
> {
  return t.union<[RT, t.UndefinedType]>([type, t.undefined], name)
}
```

That is adapted from the `maybe` combinator given in the io-ts README.
It is pretty dense for readers who do not have much experience with advanced
TypeScript use cases.
This is the sort of function that should be put into a library,
and I might do that in the future.

## Appendix B: definition for `decodeToPromise`

The built-in io-ts method `StoryV.decode()` returns an [`Either`][either] value,
which is a type from the package [fp-ts][] that can hold either an error or
a successful result.
It is similar to a promise except that it represents an immediate result,
not an asynchronous one.
The examples in this article use promises;
so I wrote a function, `decodeToPromise` to put validation results into the
more familiar `Promise` type.
Here is the definition:

```typescript
import { reporter } from "io-ts-reporters"

// Apply a validator and get the result in a `Promise`
function decodeToPromise<T, O, I>(
  validator: t.Type<T, O, I>,
  input: I,
): Promise<T> {
  const result = validator.decode(input)
  return result.fold(
    (errors) => {
      const messages = reporter(result)
      return Promise.reject(new Error(messages.join("\n")))
    },
    (value) => Promise.resolve(value),
  )
}
```

`fold()` is a method on the `Either` type.
It is used to collapse a possibility of success and a possibility of error into
one definite value.
TypeScript checks that the error-case callback and the value-case callback have
compatible return types.
One callback or the other will run depending on whether the `result` is an error or
a success value.

`decodeToPromise` also invokes an io-ts reporter to translate a set of
validation errors into a readable message.

[fp-ts]: https://github.com/gcanti/fp-ts
[either]: https://github.com/gcanti/fp-ts/blob/master/src/Either.ts


Checking Types Against the Real World in TypeScript


In 2016, I had the pleasure of testing the [Microsoft HoloLens](https://www.microsoft.com/en-us/hololens) courtesy of [Oregon Story Board](http://www.oregonstoryboard.org/). To begin, I pinched my fingers together and placed a ballerina on the desk in front of me. With a swipe of my hand, the little ballerina disappeared and the office transformed into Rome. I was fully immersed in a tour of the Colosseum before the floor began to lower beneath me and I was transported hundreds of feet below the ancient city to tour the ruins below. In the next session, the real-world office I was familiar with returned but the walls began to crack and the room was infiltrated by alien robots that I shot with my laser gun in an action-packed mixed reality game. Needless to say, I’ve been a major “mixed reality” fangirl ever since.

To kickoff a short series on exploring industrial applications for immersive reality technology (IRT), I wanted to outline some of the reasons why we, at Olio Apps, are excited about IRT and why you probably should be too. First, let’s start with some definitions. Because the technology is relatively new and rapidly evolving, a lot of the terms explaining this technology are also in flux. To clarify, I’ll be using these terms to refer to IRT.

**Virtual Reality** (VR) - Perhaps the oldest and most popular term to explain IRT. This refers to an opaque headset with a screen. The audio and video input from the screen replace your natural surroundings.

**Mixed Reality** (MR) - Like the hololens I described earlier, this is a wearable headset that overlays data, like a holograph, onto your view of the natural world.

**Augmented Reality** (AR) - Like Pokemon Go or Snapchat filters, this refers to a non-wearable screen that captures a snippet of the natural world and overlays data or distorts it.

**Immersive Reality Technology** - A blanket term that I (and other people) use that encompasses virtual reality, mixed reality, 360 video, and sometimes augmented reality.

In the past view years, immersive reality has fully emerged out the the realm of being “future tech” and is rapidly becoming more and more commercially viable and accessible. Many industries currently take advantage of this technology, but there is still a lot of room for growth. Immersive reality has some exciting advantages over traditional media as a way to see, experience, learn, and share. Here are a few of the reasons we’re geeking out over IRT.

## Immersive reality makes shared experiences personal

First, the literally in-your-face nature of immersive reality creates an environment that, while perhaps shared by thousands of others, is uniquely yours. Though the quality of the experience can vary from device to device, almost every VR or MR experience provides significantly more material than viewing a static image or watching traditional video. The experience allows you to focus on the details you want to and create your own experience and your own emotional response. This feature makes makes VR excellent for learning and skill sharing.

## Cost effective

Despite some (sometimes significant) upfront cost of creating an immersive reality experience, it can actually be highly cost effective. Unlike traditional video footage, immersive reality is primarily CGI (computer-generated imagery), i.e code, which makes creating IRT experiences more iterable, adjustable, and reusable over time even as things change.

The [astronauts at NASA](https://www.jpl.nasa.gov/news/news.php?feature=4451) are already understanding the value of technology like the hololens for training, simulated missions, and virtual assistance. Thanks to IRT, the environments of Mars and the international space station can be recreated at home for a fraction of the cost. Then, those virtual environments can be altered for an infinite number of simulations with no threat of deprecation.

## Safety

In addition to being cost effective, immersive reality technology allows users to experience high-risk situations relatively safely. Consider the astronauts from the previous example, or firefighters, who routinely train for dangerous scenarios. Using IRT allows these high-risk training simulations to be both realistic and completely harmless. Similarly, psychologists are starting to use IRT to help their patients [face their phobias](http://www.bbc.com/news/av/technology-35204681/virtual-reality-helps-people-overcome-their-phobias). The virtual environment created by the immersive headset allows the user to confront their fears in a safe and controlled space.

## Creates low distraction environment

The next exciting benefit of this technology is its ability to completely cut out distractions. The headsets create an environment that demands the users full attention and keeps them engaged on the task at hand. This feature makes immersive reality an excellent tool for focusing, learning, and even meditating.

Consider the dilemma of modern education. Class sizes are too large, sometimes 300 students in a lecture hall or a thousand students in an online classroom. This results in a number of issues, primarily that there are too many distractions and it becomes hard for students to stay focused. Now imagine a VR classroom with thousands of students, where each individual student receives a personal experience, free of distraction. Immersive reality technology could hold an interesting solution to improving both the quality and quantity of education.

## Immersive reality tech is awesome

Finally, I think Immersive Reality is exciting because it’s new and fun. The novelty of IRT encourages a lot of creativity and gives us a new way to learn, play, interact, and share. The limitations on what you can experience and create start to fall away when you have access to this technology and there is so much potential yet to be tapped. Much like the revolution that came along with the introduction of mobile technology, which allowed us to access information from almost anywhere, immersive reality technology allows us access to different environments and experiences from almost anywhere.

The gaming industry was quick to catch on to immersive reality, but IRT is emerging in industries across the board. At Olio Apps, we’re looking forward to seeing what comes next as immersive reality continues to develop. Stay tuned to our blog for future posts about immersive reality technology and how to get started with developing virtual reality and mixed reality software.


Why we're excited about Immersive Reality


I work on a lot of Javascript projects. The fashion in Javascript is to use build tools like [Gulp][] or [Webpack][]
that are written and configured in Javascript. I want to talk about the merits of [Make][] (specifically GNU Make).

Make is a general-purpose build tool that has been improved upon and refined continuously since its introduction over forty years ago. Make is great at expressing build steps concisely and is not specific to Javascript projects. It is very good at incremental builds, which can save a lot of time when you rebuild after changing one or two files in a large project.

Make has been around long enough to have solved problems that newer build tools are only now discovering for themselves.

[gulp]: https://gulpjs.com/
[webpack]: https://webpack.js.org/
[make]: https://www.gnu.org/software/make/

Despite the title of this post, Make is still widely used. But I think that it is underrepresented in Javascript development. You are more likely to see a `Makefile` in a C or C++ project, for example.

My guess is that a large portion of the Javascript community did not come from a background of Unix programming,
and never had a good opportunity to learn what Make is capable of.

I want to provide a quick primer here; I will go over the contents of the `Makefile` that I use with my own Javascript
projects.

You can find the complete file [here][my makefile].

[my makefile]: https://gist.github.com/hallettj/29b8e7815b264c88a0a0ee9dcddb6210

## When to stick with Webpack

The job that Webpack does is quite specialized. If you are writing a frontend
app and you need code bundling you should absolutely use Webpack (or a similar
tool like [Parcel][]).

On the other hand if your needs are more general Make is a good go-to tool.
I use Make when I am writing a client- or server-side library, or a Node app.
Those are cases where I do not benefit from the specialized features in Webpack.

[parcel]: https://parceljs.org/

## Why does Javascript need a build step?

Let's quickly address the question of why someone would want a build step in
a project that does bundle code.

I want to be able to write [Stage 4 ECMAScript][] while targeting browsers or recent stable versions of Node. I also like to include Flow type annotations in my code, and I want to distribute type definitions with my code; but I want the code that I distribute to be plain Javascript. So I use Make to transpile code using Babel.

[stage 4 ecmascript]: http://2ality.com/2015/11/tc39-process.html

## Introducing the Makefile

Make looks for a file called `Makefile` in the current directory.
A `Makefile` is a list of tasks that generally look like this:

```Makefile
target_file: prerequisite_file1 prerequisite_file2
	shell command to build target_file (must be indented with tabs, not spaces)
	another shell command (these commands are called the "recipe")
```

Unless you specify otherwise, Make assumes that the target
(`target_file` in this example) and prerequisites
(`prerequisite_file1` and `prerequisite_file2`) are files or directories.
You can ask Make to build a target from the command line like this:

```bash
$ make target_file
```

If the `target_file` does not exist,
or if `prerequisite_file1` or `prerequisite_file2` have been
modified since `target_file` was last built,
Make will run the given shell commands.
But first Make will check to see if there are recipes in the `Makefile` for
`prerequisite_file1` and `prerequisite_file2` and build or rebuild those if
necessary.

## A practical example of a Makefile rule

A minimal project might have a file called `src/index.js`.
We want a rule that tells Make to transpile that file and write the result to
`lib/index.js`.
But Make looks at things the other way around:
Make expects to be told the desired result,
and it uses rules to work out how to produce that result.
So we write a `Makefile` with a rule where the target is `lib/index.js` and
`src/index.js` is a prerequisite:

```Makefile
lib/index.js: src/index.js
	mkdir -p $(dir $@)
	babel $< --out-file $@ --source-maps
```

The recipe uses `babel` to produce `lib/index.js` using `src/index.js` as input.
The shell commands in a `Makefile` recipe are almost exactly what you would type
in bash -
but note that Make substitutes variables and expressions prefixed with `$`
before commands are executed.
You can escape a `$` in a recipe command by doubling it (e.g. `cd $$HOME`).
In the recipe above there are two special variables:
`$<` is a shorthand for the list of prerequisites (`src/index.js` in this case)
and `$@` is the target (`lib/index.js`).
We will see why those variables are indispensable in a moment.

The `mkdir -p` line creates the `lib/` directory in case it does not already
exist.
The function [`dir`][file name functions] extracts the directory portion from
a file path.
So `$(dir $@)` is read as "the path to the directory that contains the file
referenced by `$@`".

[file name functions]: https://www.gnu.org/software/make/manual/html_node/File-Name-Functions.html

## Generalized rules

When we add more files to the project it would be tedious to write a `Makefile`
target for each Javascript file.
A target and it's prerequisites can include wildcards to create a pattern:

```Makefile
lib/%: src/%
	mkdir -p $(dir $@)
	babel $< --out-file $@ --source-maps
```

This tells Make that any file path that begins with `lib/` can be built using
the given steps,
and that the target depends on a matching path under `src/`.
Whatever string Make substitutes in the position of the `%` in the target,
it substitutes the same string for `%` on the prerequisite's side.
Now it becomes clear why the variables `$<` and `$@` are necessary:
we won't know what the values of those variables will be until the rule is
invoked.

## Why invoke Babel separately for each source file?

Babel can transpile all files in a directory tree with one invocation.
But the rule above will run `babel` separately for every file under `src/`.
There is some startup time overhead every time `babel` runs;
so invoking `babel` many times is slower when building from a fresh checkout.
But thanks to Make's talent for incremental builds separate invocations make
incremental builds much faster.
When we ask Make to transpile all files under `src/` it will skip files that
already have up-to-date results under `lib/`.
I run incremental builds far more often than full builds
so I appreciate the speedup!

Edit: several commenters on Hacker News (falcolas, Jtsummers, jlg23, nzoschke) point out that Make can run tasks in parallel. Because Make rules explicitly list dependencies for each target Make knows which tasks can be run in parallel safely. Using the command `make --jobs=4` will run up to four instances of Babel at once, which can offset some of the performance loss of running a separate instance of Babel for each source file.

## Locating Babel

I have the above rule in my `Makefile` with one small change:

```Makefile
babel := node_modules/.bin/babel

lib/%: src/%
	mkdir -p $(dir $@)
	$(babel) $< --out-file $@ --source-maps
```

The `babel` executable is provided by the babel-cli NPM package.
I prefer to install babel-cli as a project dev dependency,
which causes `babel` executable to be installed at the path
`node_modules/.bin/babel`.
That way anyone who wants to build my project does not have to take a special
step to install babel-cli globally.
But then `babel` will not be in the executable `$PATH` on most machines.
To avoid typing out the path to the executable I assign the location of `babel`
to a variable in the `Makefile` (`babel := node_modules/.bin/babel`),
and use Make's variable substitution to splice that path into recipe commands.

(Pro tip: you can add `node_modules/.bin` to your shell `$PATH` like this:
`PATH="node_modules/.bin:$PATH"`.
That makes it easy to run executables installed by dependencies of the project
in your current directory.
Executables installed with the project will take precedence over executables
installed globally.
NPM automatically makes this `$PATH` adjustment when you run NPM scripts.
I type out the path to `babel` in my `Makefile` because I do not want to assume
that other people have made the same `$PATH` modification, and I do not always
run `make` from an NPM script.)

## Transpiling the whole project

With the above rule in place you can transpile a Javascript source file with
this command:

```bash
$ make lib/index.js  # outputs lib/index.js and lib/index.js.map
```

Make finds the matching target in your `Makefile` (`lib/%`),
expands the wildcard,
finds the source matching file by expanding `src/%`,
and runs `babel`.
But you probably do not want to run `make` manually for every source file.
What you want is to be able to just type `make` and have it transpile
all source files.
Remember that Make needs to be told the results that you want.
To do that,
first compute a list of all source files and assign it to a variable:

```Makefile
src_files := $(shell find src/ -name '*.js')
```

The expression on the right side of that assignment uses Make's built-in
[`shell`][shell function] function to run an external shell command.
In this case we use the `find` command to recursively list all files under
`src/` that have the extension `.js`.
You could use another command like [`fd`][] -
but `find` is more likely to be installed on your colleagues' workstations and on
your CI server.

[shell function]: https://www.gnu.org/software/make/manual/html_node/Shell-Function.html
[fd]: https://github.com/sharkdp/fd

That gives us a list of files that we _have_.
But we need to tell Make which files we _want_.
For every file under `src/` we want a transpiled file with a matching path under
`lib/`.
We can compute that list by applying Make's [`patsubst`][text functions]
function to the path of every source file:

```Makefile
transpiled_files := $(patsubst src/%,lib/%,$(src_files))
```

[text functions]: https://www.gnu.org/software/make/manual/html_node/Text-Functions.html

The substitution expression uses `%` as a wildcard in the same way as the rule
that we wrote earlier.

Now we can define a target that lists the files that we want as prerequisites.
When we request that target, Make will automatically build a transpiled result
for every source file:

```Makefile
all: $(transpiled_files)
```

The target name `all` is special:
When you run `make` with no target specified it will evaluate the `all` target
by default.
This is a case where the target is not a file or directory - `all` is just
a label.
You should declare non-file targets in your Makefile like this so that Make does
not waste time or confuse itself trying to find matching files in your project:

```Makefile
.PHONY: all clean
```

Oh yeah,
you probably want a way to remove build artifacts so that you can build cleanly.
With this target you can run `make clean` to do that:

```Makefile
clean:
	rm -rf lib
```

## Automatically install node modules when `package.json` changes

Make is powerful enough to accomplish pretty much any task that you can imagine.
Do you ever pull updates to a project,
and find out after some debugging that you forgot to run `yarn install` to
update your dependencies?
You can catch that with Make!
When you run `yarn install` the result is that the `node_modules` directory is
created or updated.
You can add a rule for the `node_modules` target to represent that fact to Make.
The state of `node_modules` depends on the content of `package.json` and
`yarn.lock`, so those files should be listed as prerequisites:

```Makefile
node_modules: package.json yarn.lock
	yarn install  # could be replaced with `npm install` if you prefer
```

This change to the `all` target adds `node_modules` as a prerequisite:

```Makefile
all: node_modules $(transpiled_files)
```

Now Make will run `yarn install` if and only if `package.json` or `yarn.lock`
has changed since the last build.
I put `node_modules` before `$(transpiled_files)` just in case new dependencies
include items such as updates to Babel modules that might affect the way that
project files should be built.

## Watch files and rebuild on changes

Every build tool should have a watch-files-for-changes option for rapid
development.
You can get that effect by pairing Make with a general purpose file-watching
tool:

```bash
$ yarn global add watch
$ watch make src/
```

Just make sure that you do not watch `lib/` or you will get into an infinite
build loop.

## Using Make to distribute Flow type definitions

I mentioned that I often use Flow to type-check my projects.
I want to distribute plain Javascript,
but I also want anyone consuming my library who also uses Flow to benefit from
my type annotations.
Flow supports that by looking for a files with the `.js.flow` extension.
For example when you import a module called `User` the Javascript runtime looks
for a file called `User.js`;
Flow will additionally look for a file called `User.js.flow` in the same
directory,
which should be the original source file with type annotations.
My `Makefile` copies every file under `src/` to the
corresponding path under `lib/` and adds a `.flow` extension according to this
rule:

```Makefile
lib/%.js.flow: src/%.js
	mkdir -p $(dir $@)
	cp $< $@
```

To make sure that Flow runs this step for all source files I compute the list of
`.flow` files that I expect the same way that we computed the list of transpiled
files that we expect:

```Makefile
flow_files := $(patsubst %.js,%.js.flow,$(transpiled_files))
```

And I include `flow_files` in the prerequisites of the `all` task:

```Makefile
all: node_modules $(flow_files) $(transpiled_files)
```

## Going further

Make has many capabilities that I have not touched on here.
For example Make supports macros that can compute rules on-the-fly for
especially complex use cases.
And a `Makefile` can delegate to targets in other `Makefile`s, which is useful
when distributing Make libraries,
or for multi-tiered projects where a build process involves combining artifacts
from building multiple subprojects.
There is much information to be found in the [Gnu Make Manual][].

[gnu make manual]: https://www.gnu.org/software/make/manual/


The Lost Art of the Makefile


I am always interested in making my coding process faster and more robust.
I want to be confident that my code will work as expected.
And I want to spend as little time as possible debugging, testing, and hunting
down pieces of code that I forgot to update when I make changes to a project.
That is why I am a fan of correct-by-construction methods.
Those are methods of constructing programs where the program will not build
/ compile unless it behaves the way that it is supposed to.

![letterpress - Photo by Marcus dePaula on Unsplash](/assets/blog/2018-02-12-type-driven-development-with-typescript/marcus-depaula-43304.jpg)

## Correct-by-construction

Test-driven development is one form of correct-by-construction method.
The philosophy of test-driven development is that your tests are the
specification for how your program should behave.
If you look at your test suite as a mandatory part of your build process,
if your tests do not pass the program does not build because it is not correct.
Of course the limitation is that the correctness of your program is only as
certain as the completeness of your tests.
Nevertheless studies have found that
[test-driven development can result in 40%-80% fewer bugs in production][tdd].

[tdd]: http://evidencebasedse.com/?q=node/78

### Writing your program around types

Type-driven development is the practice of writing your program around types,
and of choosing types that make it easy for the type checker to catch logic
errors.
In type-driven development your data types and type signatures are the program
specification.
Types also serve as a form of documentation that is guaranteed to be up-to-date.
Type-driven development is another correct-by-construction method.
As with test-driven development,
type-driven development can improve your confidence in your code,
and can save you time when making changes to a large codebase.

## Planning your types

But it takes more than writing type signatures and running the type checker to
gain those benefits.
The [same studies][tdd] that show the effectiveness of test-driven development
found that writing tests after the fact is not as effective as the "test first,
then implement" flow of test-driven development.
That could be because writing tests first encourages programmers to write code
that can be tested effectively.
The same idea applies to type-driven development:
you will get the most benefit from your type checker when you expose as much
information as possible in type signatures.
That means planning your types, and writing code to match.

## Types vs Tests

Types and tests are not an either/or proposition.
There are plenty of classes of errors that type-checking is bad at catching.
You should catch those with tests!
On the other hand there are classes of errors that are tedious to write tests
for but that type-checking can catch easily.
My opinion is that types are quick to write,
quick to read,
and do not require fixtures, mock data, test databases, setup, or teardown;
so I lean on types when I can.

## A simple example - sanitized user input

Let's look at a simple example in TypeScript.
From time to time I end up working on a project where keeping track of whether
some user input has already been sanitized becomes kinda tricky.
Problems come up when HTML sanitization is run where it shouldn't be.
That can lead to double-escaping issues, where content like "Types & Tests"
ends up rendered as "Types &amp;amp; Tests"
(thankfully some sanitizers are smart enough to avoid this);
or you can run into situations where functionality is lost because trusted HTML
content is stripped.
On the other hand if content is never sanitized you get a code injection
vulnerability.

### Distinguishing trusted vs untrusted content

From a type-driven perspective the problem is that the program specification
(i.e. type signatures)
makes no distinction between user-submitted content (type `string`),
and trusted HTML content (type `string`).
If we use two different types we can make the program specification
more precise,
providing essential information to the type checker and to programmers reading
the code.

```ts
// HTML.ts

import sanitizeHtml from "sanitize-html"

export type HTML = {
  trustedHtml: string
}

export function sanitize(input: string): HTML {
  return {
    trustedHtml: sanitizeHtml(input),
  }
}

export function trustedHtml(input: string): HTML {
  return {
    trustedHtml: input,
  }
}

export function fromHtml(input: HTML): string {
  return input.trustedHtml
}
```

Untrusted input still has type `string`.
The new type `HTML` represents content that has been sanitized or has been
explicitly declared to be trusted.
We don't want callers to have to know about the internal representation of the
`HTML` type.
(We might want to change that representation in the future.)
So the `HTML.ts` module defines constructors for producing an `HTML` value from
either untrusted or trusted input,
and an extractor to get the underlying string from an `HTML` value.
A function that ultimately outputs HTML might look like this:

```ts
import { HTML, fromHtml } from "./HTML"
import { ServerResponse } from "http"

function renderContent(
  response: ServerResponse,
  html: HTML,
  cb: (err?: Error) => void,
) {
  // Simplified for clarity
  response.write("<body>" + fromHtml(html) + "</body>", "utf8", cb)
}
```

## Verifying content is sanitized exactly once

The result is that the type checker is able to do the hard work of checking
that untrusted input is sanitized exactly once through the entire logic flow of
the program.
Content that has never been sanitized or declared trusted will not have the
correct type when it is passed to `renderContent`,
and you will get a type error.
If content has already been sanitized once then you will get a type error if
`sanitize` is called a second time because the input will have type `HTML`,
not `string`.

```ts
const untrusted = "hello <script>alert('world')</script>"
renderContent(response, untrusted, () => {})

// error TS2345: Argument of type '"hello <script>alert('world')</script>"' is not assignable to parameter of type 'HTML'.

const trusted = sanitize(untrusted)
sanitize(trusted)

// error TS2345: Argument of type 'HTML' is not assignable to parameter of type 'string'.
```

You could make the same change in plain Javascript without static type checking,
and get similar errors at runtime -
which would be almost as helpful.
But without type checking you would need thorough test coverage to catch mismatches.
With type checking if we had an existing `renderContent` function that took
a `string` input,
and changed it to take `HTML` input the type checker would immediately
highlight all of the call sites that need to be updated.

Leaning on the type checker pays off when the program accumulates multiple
layers of content processing:

```ts
import { HTML, fromHtml, sanitize, trustedHtml } from "./HTML"

function logInjectionAttempts(content: string): string {
  if (content.match(/<\s*script|onclick\s*=/i)) {
    console.warn("injection attempt", content)
  }
  return content
}

function appendLikeButton(content: HTML): HTML {
  return trustedHtml(fromHtml(content) + '<button class="like">+1</button>')
}

function appendWordCount(content: HTML): HTML {
  const str = fromHtml(content)
  const count = str.split(/[\s]/).length
  return trustedHtml(str + `<aside>word count: ${count}</aside>`)
}

const responseContent = appendWordCount(
  appendLikeButton(sanitize(logInjectionAttempts(userInput))),
)
renderContent(response, responseContent, cb)
```

When a program gets big it can be easy to accidentally put content processing
steps in the wrong order,
or to lose track of what steps have been performed at any given point in
a program.
Type-driven development helps to set you straight.

### Opaque type aliases

The implementation of the `HTML` type in the examples here requires an object
wrapper that is not strictly necessary.
It would be OK for untrusted input and HTML to both be strings at runtime if we
could assign them distinct types at type-checking time.
We could use a trick described by Charles Pick at codemix to define an
[opaque type alias][],
which we could use to declare types that TypeScript treats as incompatible,
but that actually represent the same runtime type.
That would be a "zero-cost abstraction":
an abstraction that has no runtime overhead.
I did not use an opaque type alias for this post because that would get into
TypeScript features that are a bit more advanced than I wanted to cover in this
post.
But maybe we will revisit that idea another time.

[opaque type alias]: https://codemix.com/opaque-types-in-javascript/

## Type-driven development and functional programming

Type-driven development goes hand-in-hand with functional programming (FP).
That is because FP discourages side-effects and mutation.
A type signature can tell you all about the inputs and outputs of a function or
method;
but most languages do not have options for declaring side-effects in type
signatures.
Writing code in a functional style is how you make important information
available to your type checker.
For example the processing steps above could have been written in a more
imperative fashion:

```ts
type Result = {
  status: number
  content: string
}

function logInjectionAttempts(result: Result) {
  if (result.content.match(/<\s*script|onclick\s*=/i)) {
    console.warn("injection attempt", result.content)
    result.status = 401
  }
}

function appendLikeButton(result: Result) {
  result.content += '<button class="like">+1</button>'
}

function appendWordCount(result: Result) {
  const count = result.content.split(/[\s]/).length
  result.content += `<aside>word count: ${count}</aside>`
}

const result = { status: 200, content: userInput }
logInjectionAttempts(result)
appendLikeButton(result)
appendWordCount(result)
response.write("<body>" + result.content + "</body>", "utf8", cb)
```

## Imperative mutations are weakly typed

This version uses a command pattern,
where each function updates a data structure and does not return a value.
Without return values, there is nowhere in the type signatures to indicate
where content changes from an untrusted `string` value to a trusted `HTML`
value.
The input types in a type signature specify requirements that must be satisfied
before a piece of the program can run.
The return type describes changes to program state.
A signature with a `void` return type does not give any information about state
changes.
`void` functions are sometimes useful and necessary;
but if they are used excessively then the program ends up underspecified.

This was an introduction to the idea of type-driven development with some
simple examples to give you a peek what types can do for you.
Watch this space for posts that will dive deeper into thinking in types.

<span style="font-size: 0.7em">Photo by [Marcus dePaula](https://unsplash.com/photos/tk7OAxsXNL0?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) on [Unsplash](https://unsplash.com/search/photos/letterpress?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText)</span>


Type-Driven Development with TypeScript


Code documentation is the collection of easy to understand images and written descriptions that explain what a codebase does and how it can be used.

It can be simple explanatory comments above functions and blocks, or a full-fledged developer handbook, complete with prescriptive style dos-and-don'ts, overviews of each part of the application, and approaches to the most common types of coding tasks.

It's often best to find a happy medium between these two extremes when you make the conscious decision to comment your code and outline the gnarlier areas of a codebase in plain language.

## Why document your code?

Why go to the trouble writing _about_ your code, instead of just _writing_ code? Wouldn't that just be more productive anyway?

![condescending wonka meme](https://user-images.githubusercontent.com/5914687/34965265-a6b6f904-fa07-11e7-92ff-22729040dd0a.png)

### Mental Save Points

Documentation is like a mental save point for those times when you finally get what's going on at the end of the day and don't want to lose momentum. Well documented code will ensure that when you need to dive back in tomorrow morning (or several months from now), you won't have to take as much time getting up to speed.

### Documented code gets reused

The difference between a used library and an unused library often comes down to its documentation.

Who wants to use an undocumented open-source library or contribute to an undocumented project? Almost no one.

In most cases, you'd rather use a less powerful library that has docs. Why? Because the documentation conveys other points of information about a codebase. The developers believe that the project is worth the time and energy to write about it in plain language so anyone who might be interested can quickly get started and learn how it works, and why key decisions were made.

Finding a new, interesting, and well-documented project in your favorite language can be very exciting and fun. Let's look at the extreme alternative: being required to use an undocumented code library and potentially contribute to a seemingly byzantine codebase... it sounds pretty painful, right?

So why haven't you documented your own code?

## Documenting code is part of writing good code

A cornerstone of good code is maintainability, achieved through understandable, legible documentation.

There are multiple ways of documenting code:

- choosing good names for variables and functions
- leaving brief comments within the code to help give context to future readers
- adding illustrative images such as sequence and entity relationship diagrams
- providing API docs, describing each class, method, argument and return value
- using a statically typed language, such as TypeScript (types as documentation)

## Compare two examples

Let's say you're getting some data from a backend API and "ingressing" it - converting it into a more useful shape for a front-end UI. Here's the first of two possible solutions.

```javascript
// wdgt.js

export const converter = (w) => {
    wdgtHdr = {
        name: w.nm,
        id: w.id,
        ts: w.lastTS
    }

    wdgtDtl = {
        dtlId: w.dtlId,
        dtlName: w.dtlNm,
        openTS: w.oTS,
        closeTS: w.cTS
    }

    wdgtRev = w.revs.map((r) => {
        const { note, prsn1Id, prsn1Ln, prsn1Fn, amt, cTS } = r
        const rr = {
            assocId: prsn1Id,
            assoc: `${prsn1Fn} ${prsn1Ln}`,
            amount: amt,
            note: note,
            revTS: cTS,
        }
        return rr
    }

    return {
        wdgtHdr, wdgtDtl, wdgtRev
    }
}
```

The above code has the potential to be **painful**. You can see by the name that this is some sort of converter that takes `w` as its param.

It looks like it returns a header, maybe details, and something called revs. Where did revs come from, anyway? Of course, there's no details about the incoming variable names `note, prsn1Id, prsn1Ln, prsn1Fn, amt, cTS`... or is there? Is `rr` short for `returned rev`? Who can say.

Let's look at the second example.

```typescript
// widget_utils.ts

// widgetConverter :: widgetRespDTO -> widget
export const widgetConverter = (wResp: widgetRespDTO): Widget => {
    const widgetHeader: WidgetHeader = {
        name: wResp.nm,
        id: wResp.Id,
        ts: wResp.lastTS), // timestamp
    }

    const widgetDetail: WidgetDetail = {
        id: wResp.widgetId,
        name: wResp.dtlNm,
        openTS: wResp.oTS, // timestamp
        closeTS: wResp.cTS // timestamp
    }

    // [WidgetRevisionsRespDTO] is nested inside of WidgetRespDTO

    // [WidgetRevisionsRespDTO].map -> [WidgetRevision]
    const widgetRevisions: ReadonlyArray<WidgetRevision> =
        wResp.revs.map((wRevs: WidgetRevisionsRespDTO): WidgetRevision => {
            // how that other team names their variables...!
            const { cmt, prsn1Id, prsn1Ln, prsn1Fn, amt, cTS } = wRevs

            const comment: string = cmt
            const associateId: string = prsn1Id
            const associateName: string = `${prsn1Fn} ${prsn1Ln}`
            const amount: number = amt
            const revisionTS: number = cTS // unix timestamp, and "closeTS" fyi

            return {
                associateId,
                associateName,
                amount,
                comment,
                revisionTS,
            }
    }

    return {
        widgetHeader,
        widgetDetail,
        widgetRevisions,
    }
}
```

Wow, how different! This function is in a `util` file, so the main file is less cluttered already. It's also written in TypeScript instead of plain JavaScript, so we have the benefit of type defintions to help guide us.

We're converting `widgetRespDTO`s into `widget`s. Though shortened, we have full knowledge of what `wResp` is. We're creating a `widgetHeader`, and a `widgetDetail`, that's easy to see. We also understand what `oTS` and `cTS` are. We can tell `WidgetRevisionsRespDTO`s are nested inside of `WidgetRespDTO`.

Someone very kindly renamed the incoming variables so _everyone_ who sees this code knows what they are. Finally, we see a `Widget` must be composed of the returned `{widgetHeader, widgetDetail, widgetRevisions}` object, since it must match the return type `Widget` specified at the top.

Which code would you feel better about using?

## Who benefits?

Well documented code benefits many people:

### 1. Junior developers

Because it's in plain language and easy to understand, documentation helps junior developers and new team members feel confident jumping into a codebase. This prevents frustration and early abandonment of a task because it gets too complicated too quickly.

It's also helpful for junior developers to write their own documentation so they can share their code with others and so they can get a better understanding of their code.

### 2. Senior developers

By taking the time to document now, senior devs spend less time explaining their code to others in the future.

They can also write more complex code. Plain language documentation lets others use it like a black box, without having to understand the inner workings.

### 3. Teams & open source projects

Personnel changes can bring a tremendous slowdown to a project. Up-to-date and well documented code can serve as insurance against such slowdowns and allow the remaining team members to take a step back, review the code from a high level and decide on the best course of action moving forward, and if needed, on-board new employees.

### 4. You

When documenting code, you must summarize it. In order to summarize properly, you must be able to understand it, holding the different parts in your head at the same time. Writing or contributing to the documentation is a shortcut to really understanding a codebase, making huge contributions, and feeling like a great part of a great team.

## Good documentation can DRY up our coding experience

As a programmer you've probably heard of the _Don't Repeat Yourself_ principle as a key to writing clean and concise code. Instead of repeating the same code block several times, write a function that can be written once and employed all over your application.

In the same way, you and your team shouldn't be running into similar problems and then reinventing the wheel each time you attempt to solve them. For example, have you ever heard one of your coworkers or found yourself saying:

_"Agh, I solved something very similar to this but I never remember how I do it when it comes up... I should probably write it down somewhere, huh?"_

You then find yourself scanning the same Stack Overflow threads again and again.

Save yourself and your coworkers this grief by following the spirit of the DRY principle and writing a document that you can employ every time the issue comes up! This will help you and others on your team save time, and be more effective in writing simpler, more manageable, and more efficient code.

How has code documentation, or the lack thereof, affected you?


The Value of Code Documentation


Pull requests are an essential collaborative tool for modern software development. I wanted to capture the kinds of questions a developer should ask themselves before asking another developer for a peer review.

These are general guidelines, and are not exhaustive of the types of due diligence every software engineer undertake prior to seeking a peer review, with the aim to:

- Reduce the amount of recurrent defects
- Reduce the impact to the other developer’s time and energy spent defects in your code
- Increase the overall autonomy and productivity of individual developers

Developers should ask these types of questions prior to submitting work for review.

## Prior to coding

- Can you articulate what problem you are trying to solve?
- Can you articulate your proposed solution before coding it?
- If the stakeholder is present/available, have you validated your solution with them?

## Learning from previous reviews

Have you had a code review in the area of the code affected by your previous PR? If so, have you double checked your current PR for the existence of similar defects?

Some people may find it useful to maintain a (private) journal or log of previous PRs.

## Code hygiene

- Does the code compile?
- Does the code pass all linting and sanity checks locally? How about the the CI server?
- If you are working on a UI project - does your code conform to the UI coding and naming conventions agreed upon by your organization?
- Have you removed any unused or dead code (such as scaffolding or work product code)? Anything related to debugging?
- Are there any merge conflicts you need to resolve as a result of your PR?

## Code organization and correctness

- If an algorithm is material to your code changes, have you ensured that its as efficient and correct as possible?
- Can a hardened third-party component be used in place of anything you’ve hand coded for your solution?
- Assuming the code compiles, runs and spits out the correct behavior -- have you gone back and optimized the solution with the benefit of hindsight?
- If you are working on a multifactor project (react native + react web for example) - - are you properly separating component logic from the visual components for maximum reuse?
- If you are adding logic as part of your PR -- are you adding it in the right level? Is this logic appropriate, for example, in the component versus the reducers, versus the sagas?

## Documentation

- Are you working on a ticket that is undocumented? Eg. if we are using trello tickets to track sprint activities -- have you added at ticket capturing your effort?
- Are there any external or internal facing documents that need to updated should your PR be accepted?
- Have you made comments in the code you are touching that will be necessary to explain what you did?
- If the change you are effecting originates from a trello ticket, JIRA, or some other bug tracking system -- have you documented there how your changes can be tested?
- In your PR, have you pasted links to the to the originating trello or JIRA item, so the reviewer has context for testing and review?
- If your PR affects the user interface, have you added screenshots of the affected UIs?

## Testing

- Can you describe what functional change your PR is trying to effect? For example, what behaviors are changing as a result of the code you are submitting?
- Assuming your code is runnable, does it actually fulfill the behavioral requirements set forth in the comps, functional requirements spec, trello tickets, etc.?
- Have you manually tested the behavior you are affecting in the PR?
- If there is an automated testing framework in place, have you built tests that cover the code touched by your PR?
- If you are working on a UI project, have you:
  - Considered the cross browser implications? Have you tried testing it in the target browsers?
  - Considered the operating system implications? Have you tried testing it in the target operating systems and the versions thereof?
  - Considered the device implications? Have you tried testing it in the target devices and form factors?

## Resource Bundles

The types of applications that Olio creates will often use resource bundles to describe the text or colors presented to the user.

- If text variabilization is supported (via internationalization or otherwise) -- have you made sure that any text you’ve added or modified isn’t a hard coded string?
- If color variabilization is supported -- have you made sure that any colors you’ve added or modified isn’t a hard coded color or hex string?

## Exiting the current review

- Based on the PR feedback, have you identified new or existing areas of knowledge or ability to work on?
- Have you thanked your peer reviewer, and offered to reciprocate? :)

Do you have any suggestions for check list items or questions to add to our list?


Pull Request Checklist


While writing about [optimizing data for performance](/blog/optimizing-api-data/), I rewrote my transformation function a few different ways and benchmarked the performance of each. Wow! The results were surprising.

## Some background

Immutability and [persistent data structures](https://en.wikipedia.org/wiki/Persistent_data_structure) are all the rage lately. They can reduce errors by preventing modifications to existing data, so functions can rely on consistency.

Languages like Clojure have [built-in immutable data types](https://clojure.org/about/functional_programming#_immutable_data_structures) that use [structural sharing](http://hypirion.com/musings/understanding-persistent-vector-pt-1).

JavaScript has no built in immutable data structures, but libraries like [Immutable.js](https://facebook.github.io/immutable-js/) and [Mori](https://github.com/swannodette/mori) provide this. When you modify an array with Immutable.js, it returns a new array that references the remaining objects from the previous array.

But, Immutable.js requires some additional ceremony, like initializers, getters and setters:

```javascript
const { Map } = require("immutable")
const map1 = Map({ a: 1, b: 2, c: 3 })
const map2 = map1.set("b", 50)
map1.get("b") // 2
map2.get("b") // 50
```

## Using ES6 Spread Operator for Immutability

With the advent of ES6, you can avoid mutations using the spread operator, which looks like an ellipsis `...`

With the spread operator, you can copy the elements of an existing object or array to a new one, thus you have a new object with your desired modification, without changing the original:

```javascript
const newArray = [...oldArray, newItem]
const newObject = { ...oldObject, newItem: "foo" }
```

The best part about the spread operator is that it works with standard JavaScript objects and arrays, no new libraries, getters or setters needed. But, it makes a complete copy of every element from the previous array. Sometimes, that's exactly what you want, so it's very useful.

## TypeScript, tslint-immutable

Why do we care? In our most recent frontend projects, we've been using [TypeScript](https://www.typescriptlang.org) to great effect. Increased type safety, fewer run time bugs, and with [tslint-immutable](https://github.com/jonaskello/tslint-immutable), we found a way to _enforce immutability_ at compile time instead of at runtime and ditched the use of Immutable.js, opting for TypeScript `readonly` properties and `ReadonlyArray`'s instead.

The only drawback is, sometimes you need to loop through one array and add items to another, repeatedly in a loop. If we enforce immutability on arrays, we can't use `Array.push()` to mutate an existing array. To work around that, we can create a new array and use the spread operator to copy the previous items, on every loop iteration. Wait a second...

## Hmm, that might be slow

Question was, how slow was it?

So, I made [a few variations](https://github.com/olioapps/benchmarking-js/blob/master/index.js#L13-L85) of a little function that transforms arrays of objects into a map of objects and an array of keys, to see how fast they performed.

The task was to transform this structure:

```javascript
const authorArray = [
  {
    id: "1",
    name: "Scott",
  },
  {
    id: "2",
    name: "Aron",
  },
]
```

into this one:

```javascript
const authors = {
  // a map of items, keyed by id, for fast lookups
  items: {
    1: {
      id: "1",
      name: "Scott",
    },
    2: {
      id: "2",
      name: "Aron",
    },
  },

  // an ordered array of keys, for maintaining original sort order
  keys: ["1", "2"],
}
```

Turns out you can implement this in widely varying degrees of efficiency. To cut to the chase...

This was the most efficient implementation:

```javascript
function parseUsers3(apiUsers) {
  let acc = { items: {}, keys: [] }
  for (let i = 0; i < apiUsers.length; i++) {
    acc.items[apiUsers[i].id] = apiUsers[i]
    acc.keys.push(apiUsers[i].id)
  }
  return acc
}
```

Note that besides the `let` keyword, we're using bog standard old school imperative JavaScript, with a for loop and mutable objects.

And here was the least efficient:

```javascript
function parseUsers2(apiUsers) {
  return apiUsers.reduce(
    (acc, user) => ({
      items: {
        ...acc.items,
        [user.id]: user,
      },
      keys: [...acc.keys, user.id],
    }),
    {
      items: {},
      keys: [],
    },
  )
}
```

Here we are using a reduce function, but that's not the slow part. Note the use of the spread operator to avoid mutations to the existing accumulator object, `acc`. Each iteration creates a brand new accumulator object to hand back to the reducer, with fresh copies of all items and keys from the previous iterations. That, it turns out, is not a cheap operation.

## Benchmarking

In my initial attempt at measuring code performance, I rolled my own janky benchmarking code that measured how long it took to call a function N number of times.

Then it occurred _"perhaps this is not the best benchmarking technique ever conceived. Maybe there's already a library out there by people who thought about how to do it right."_

I found that in [Benchmark.js](https://benchmarkjs.com/), _"a library that supports high-resolution timers & returns statistically significant results"_.

Benchmark.js spits out some nice results like this:

```javascript
parseUsers1(apiUsers) x 5.93 ops/sec ±7.56% (20 runs sampled)
parseUsers2(apiUsers) x 5.15 ops/sec ±1.83% (17 runs sampled)
parseUsers3(apiUsers) x 29,350 ops/sec ±1.37% (83 runs sampled)
parseUsers4(apiUsers) x 19,607 ops/sec ±3.26% (80 runs sampled)
parseUsers5(apiUsers) x 11,763 ops/sec ±8.97% (68 runs sampled)
parseUsers6(apiUsers) x 20,284 ops/sec ±1.66% (81 runs sampled)

Fastest is parseUsers3(apiUsers)
```

Note that the fastest implementation could run my benchmark test about 30,000 times a second, while the slowest would run about 5 times a second. That's an insane difference! I guess we're ditching the spread operator here and going with good old `Array.push()`.

## Writing Benchmark Tests

Writing a benchmark test with Benchmark.js is very simple. Create a suite and add a test case for each implementation that you want to compare:

```javascript
var Benchmark = require("benchmark")
var suite = new Benchmark.Suite()

// add tests
suite
  .add("parseUsers1(apiUsers)", () => {
    parseUsers1(apiUsers)
  })
  .add("parseUsers2(apiUsers)", () => {
    parseUsers2(apiUsers)
  })
  .add("parseUsers3(apiUsers)", () => {
    parseUsers3(apiUsers)
  })
  .add("parseUsers4(apiUsers)", () => {
    parseUsers4(apiUsers)
  })
  .add("parseUsers5(apiUsers)", () => {
    parseUsers5(apiUsers)
  })
  .add("parseUsers6(apiUsers)", () => {
    parseUsers6(apiUsers)
  })

  // add listeners
  .on("cycle", (event) => {
    console.log(String(event.target))
  })
  .on("complete", () => {
    console.log("Fastest is " + this.filter("fastest").map("name"))
  })
  // run async
  .run({ async: true })
```

Here's [the repository on Github](https://github.com/olioapps/benchmarking-js) containing all 6 implementations, and the tests which you can run yourself.

## Conclusions

There are many ways of accomplishing a task in code. Premature optimization should generally be avoided, but if you suspect something might be slow, and it's going to occur frequently, measure performance! Solid benchmarking libraries exist in most languages these days that make it easy.

I certainly learned something from this experiment: use the right tool for the job. Don't blindly use immutability or the spread operator for every situation. When accumulating items into arrays or objects, it's silly, and kills performance.


Benchmarking and Optimizing Slow JavaScript


Sometimes APIs return data in a form that's not ideal for the client. We can deal with this by applying data transformation functions to the incoming data, to reshape it into exactly what we need.

A common use case is to render a list of items, and for each item, show info about some related item. For example, a list of comments might contain the author id, but visually we need to also show the author name.

```javascript
// GET /comments
;[
  {
    id: 1,
    text: "Great post!",
    author_id: 1,
  },
  {
    id: 2,
    text: "I disagree!",
    author_id: 2,
  },
]
```

If you have the luxury of GraphQL, perhaps you can query for the list of comments, and the author name of each in a single request.

If you have a REST api which gives you separate endpoints for comments and authors, both must be fetched to get the complete picture.

```javascript
// GET /authors
;[
  {
    id: "1",
    name: "Scott",
  },
  {
    id: "2",
    name: "Aron",
  },
]
```

In this case, when displaying comments, we'll need to look up the author for a given author_id.

## Optimizing Arrays for Fast lookups

We could just store data in arrays. When we need to find an item, loop through the array to find the item with the matching unique identifier.

As the arrays grow larger, this becomes extremely inefficient, especially when nested within another loop:

```javascript
// render list of comments

// loop through comment array
// for each comment ...
// get the comment's author by looping through author array
// gets slower as the number of authors grows
```

To make this more efficient, we could transform the arrays into maps, with ids for keys and items for values.

```javascript
const authors = {
  1: {
    id: "1",
    name: "Scott",
  },
  2: {
    id: "2",
    name: "Aron",
  },
}
```

This makes look ups fast, but if we discard the array, we might lose important sort order information, so we can also store an array of keys, to keep that information in tact:

```javascript
const authorKeys = ["1", "2"]
```

That way we can get the best of both worlds - a sorted array of items, and fast lookup of individual items by id. It's sort of like having a relational database table with a primary key index, except all in memory!

### Map of Items, Array of Keys

A data structure we've found useful is a compound object that stores both a map of the items, keyed by id, and an array of the id keys:

```javascript
const authors = {
  items: {
    1: {
      id: "1",
      name: "Scott",
    },
    2: {
      id: "2",
      name: "Aron",
    },
  },
  keys: ["1", "2"],
}
```

### Transformation Function

Using JavaScript's handy reduce function, we can generate an efficient structure for lookups from the API response array of users with a single iteration through the array:

```javascript
function parseItems(apiItems) {
  return apiItems.reduce(
    (acc, item) => {
      acc.items[item.id] = item
      acc.keys.push(item.id)
      return acc
    },
    {
      items: {},
      keys: [],
    },
  )
}
```

Now, when looping through comments, we can avoid iterating through the author array again and again for lookups!


Optimizing API Data for Faster Frontend Performance


We frequently work on front-ends along side other teams building APIs. Sometimes those APIs are not yet available. But deadlines and sprint demos are approaching. Are we to sit around twiddling our thumbs? Nope!

The approach we take is to get the API spec if available, or if not, imagine how we would want this API to look and behave.

Then we create mock API functions _within_ our front-end application that do not make any requests, but behave exactly as if calling the real API, returning fake data.

An alternative is to use a hosted fake API server like [json-server](https://github.com/typicode/json-server). In some cases, that might be a better choice - such as if the front-end is deployed to a staging site, and you want to be able to edit the fake responses without redeploying.

During the development phase, we prefer to use mocking functions instead, so we don't increase our dependencies or have to rely on an external service for testing.

This allows us to decouple our development process from that of other teams we are relying on, and quickly put together working demos that illustrate how the application will perform once the API is online.

## Example API - a list of tips

Suppose our backend API will return a set of "tips". A tip is a short piece of content, to show to a user on start up. Think _Clippy_ or, a "Did you know? ..." pop-up. The front end application will fetch these tips from the API and show them to the user.

We have mockups, but the API is still under development. Fortunately, the shape and contract of the API is known. If we build the UI to be data driven now, we won't have to go back and do partial rework later. When the API comes online, we can flip a switch and it should "just work".

## API response shape

This API will exist on an authenticated HTTPS endpoint. Let's say its `GET https://special.api/tips`. According to the spec, the response JSON shape will be:

```json
{
  "tips": [
    {
      "id": 1,
      "type": "info",
      "title": "Set a daily or weekly reminder to track your time",
      "subtitle": "Tracking",
      "description": "Over time, we can provide insights and suggestions for improvement",
      "link": "https://url/to/set-reminder",
      "label": "Add Reminder"
    }
    // ... more tips
  ]
}
```

## Static Typing

At Olio Apps we've been enjoying [TypeScript](https://www.typescriptlang.org/) lately, which allows us to create static types for things like this json structure:

```typescript
// the tip itself
interface ApiTip {
  readonly id: number
  readonly type: string
  readonly title: string
  readonly subtitle: string
  readonly description: string
  readonly link: string
  readonly label: string
}

// the api response envelope, an object that has an array of tips
interface ApiTips {
  readonly tips: ReadonlyArray<ApiTip>
}
```

## Immutable Data

Note the use of `readonly` and `ReadonlyArray<T>` which enforce immutability. We use [tslint-immutable](https://github.com/jonaskello/tslint-immutable) to ensure all data types are immutable and checked at compile time, instead of at run time with something like [Immutable.js](https://github.com/facebook/immutable-js/). This yields higher performance and more straightforward code.

## API Client

The following function makes a request to fetch tips:

```typescript
export function getTips(): Promise<HandledResp> {
  if (useMock) {
    return fakeGetTips().then((res: Resp) => handleResp(res))
  }

  return request
    .get(buildUrl("/tips"))
    .set(authHeader())
    .type("application/json")
    .accept("application/json")
    .then((res: Resp) => handleResp(res))
    .catch((res: Resp) => handleResp(res))
  }
}
```

## The Mock API Function

Note the `useMock` check. If this configuration variable is `true`, we'll call `fakeGetTips()` instead of making a request.

Here's what `fakeGetTips()` looks like:

```typescript
export function fakeGetTips(): Promise<Resp> {
  const payload: ApiTips = {
    tips: [
      {
        id: 1,
        type: "info",
        title: "Set a daily or weekly reminder to track your time",
        subtitle: "Tracking",
        description:
          "Over time, we can provide insights and suggestions for improvement",
        link: "https://url/to/set-reminder",
        label: "Add Reminder",
      },
      // more fake tips
    ],
  }

  return Promise.resolve({
    text: JSON.stringify(payload, null, 4),
    status: 200,
    statusText: "",
  })
}
```

Just like a standard request, `fakeGetTips()` return a promise that resolves to a standard response object:

```typescript
interface Resp {
  readonly text: string
  readonly status: number
  readonly statusText: string
}
```

## Standard Response Handling

In both versions, when the response value comes back in a promise, we handle it with `.then((res: Resp) => handleResp(res))`.

The `handleResp` function converts the response object into a normalized structure, where any text from the response is parsed as json and returned as `items`, along with the `status` and `statusText`.

```typescript
const handleResp = (resp: Resp): HandledResp => {
  // if text present, parse as json, otherwise empty array
  const items: {} = resp.text ? JSON.parse(resp.text) : []
  // return normalized response object
  return {
    items,
    status: resp.status,
    statusText: resp.statusText,
  }
}

interface HandledResp {
  readonly items: {} // can be any javascript type
  readonly status: number
  readonly statusText: string
}
```

All downstream functions can now rely on `items` to be the response data, converted back into a javascript object. Once the `tips` API comes online, we can simply change `useMock` to false, and it should work as long as the API data structure hasn't changed.


Reinforced Learning