When building large database-backed Elixir applications using Ecto, it is inevitable for your database schema to evolve over time. As your database schema changes, it's possible for there to be discrepancies between the tables in the database and your local schema definitions. Tracking inconsistencies between the schema defined in your application's modules and what's actually in the database can be tedious.

Thankfully, there's a way we can look for inconsistencies programmatically.

Finding all schema modules

A normal Ecto schema will look something like the following:

defmodule User do
  use Ecto.Schema

  schema "users" do
    field :name, :string
    field :age, :integer, default: 0
    field :password, :string, redact: true
    has_many :posts, Post
  end
end

schema is actually a macro, that, when expanded, will declare a number of functions. We can use the presence of the __schema__ function to filter our application's modules to grab only those that have a schema definition.

To get all modules defined in your application we can do:

{:ok, modules} = :application.get_key(:your_app, :modules)

Now we want to filter out modules without the __schema__ function:

schema_modules =
  modules 
  |> Enum.filter(&({:__schema__, 1} in &1.__info__(:functions))) 

This will give us all modules where the __schema__ field is present. This will include modules that define embedded schemas however. Since embedded schemas don't directly map to a database table we'll want to exclude those modules. Taking a look at the implementation of the schema and embedded macros reveals that only schema definitions will add an additional __meta__ field to the module struct.

  |> Enum.filter(&(:__meta__ in Map.keys(&1.__schema__(:loaded))))

Comparing the schemas with the database

We can now iterate over all schema modules. Let's define a function for this:

defmodule SchemaChecker do
  def schema_modules do
    {:ok, modules} = :application.get_key(:your_app, :modules)
    
    modules 
    |> Enum.filter(&({:__schema__, 1} in &1.__info__(:functions))) 
    |> Enum.filter(&(:__meta__ in Map.keys(&1.__schema__(:loaded))))
  end
  
  def check_module(module) do
    [...]
  end
end

To obtain the list of fields on each schema we can continue using the __schema__ function.

fields = module.__schema__(:fields)

This is almost perfect, but fields in our schema can have different names than the database column. For this we need to get the :field_source, so we change the above to:

fields = 
  module.__schema__(:fields)
  |> Enum.map(&module.__schema__(:field_source, &1))
  |> Enum.map(&Atom.to_string/1)

You'll note that we also convert the atoms to strings. This is to more easily compare the lists later on.

We can now run a simple dummy query against our database with a LIMIT 0 that will return an empty result set but list out all available columns. By computing the difference between our list of fields and the database columns we can determine which columns are no longer referenced by any schema:

table_name = module.__schema__(:source)

{:ok, %{columns: columns}} = DB.Repo.query("SELECT * FROM #{table_name} LIMIT 0")

columns -- fields

You may have to play around with the result set a bit in case you are not using PostgreSQL.

Summary

This post is the result of several database cleanup efforts in my dayjob. I was positively surprised how easy it was to leverage Ecto's powerful API and well-written documentation to get this task done.

You can find the final version at: https://gist.github.com/halfdan/6853a8cc8994eca4c9311ecad04a0eb0

Mixmaster, Broadside, Windcharger, Ransack, Ramjet, Wheeljack, Skyfire, Slingshot and Wideload. Those were project names I had to deal with in my previous role. Some of those projects were backend services, some libraries and some were for a frontend.

"Why is there a RansackRunner in Windcharger that actually runs Ramjet?
- Some engineer

Unlike the author of "Names should be cute, not descriptive" I believe the opposite to be true.

The main arguments from the linked article can be summarized as:

• Projects change and names may no longer fully reflect the responsibility of the project
• Descriptive names are hard to say and remember and are not "fun"

Names serve a major purpose: They help everyone involved in a project understand what is being talked about. In a company with many projects in can quickly get confusing when you have tens of "cute" project names thrown around on a daily basis. Almost every month I would have to spend time in meetings explaining the difference between two similar sounding "cute" project names and reminding everyone what those projects actually did. Additionally, while the engineers would often refer to the projects by their cute name, the PMs would not bother remembering the names and instead talk about "the backend" and "the frontend" which caused significant friction and confusion on both sides.

The moment we switched to more descriptive names this changed. By following the Domain Driven Design's "Ubiquitous Language" between our different functions we were able to find names that were easy to remember, specific enough for our business context but generic enough not to limit the projects' growth.

Not only was communication easier with the descriptive project names it also helped onboarding new engineers and PMs who now didn't have to spend the first few weeks on the job producing their version of an up to date glossary to map codenames to "what does this thing do".  

Projects would still evolve and the names would eventually longer fully describe the responsibility of the underlying service or library. When this happened there were three options:

• alter the name to once again fully capture the responsibility of our project
• refactor the project and split it up
• keep the name as it was still "good enough"

Renaming projects is often harder than finding the name in the first place because project names percolate into many places such as environment variables, class names and functions. For this reason we never actually renamed a project, we simply phased it our over time and replaced it with a project with a more descriptive name.

It's rare and speaks of bad decision making when a project suddenly does a 180 and somehow no longer matches any of the original intent embodied by its name. When you hide behind cute project names it's easy to let something like this slip, but when you have a descriptive name it allows you to catch problems like this early on.

Sometimes though the name is simply still accurate enough and an argument could be made that feature X is close enough in context for it end up in the specific project. This is the case when a feature is merely extending the original project and doesn't introduce something completely new.

There may be a middle ground to all of this. If you are a company with a massive monolithic application then the name will probably not matter too much. It will be fine to give it a "cutesy" name. The moment you expand to multiple codebases or even microservices I strongly suggest sprinkling some descriptiveness into the name. It's entirely valid to combine cute and descriptive and produce a name like "godzilla-auth-service".

The world is pretty ok. Let's be kind to each other and use descriptive names and not make work harder than it needs to be.

I'm currently taking the Deep Learning nano degree offered by Udacity to deepen my knowledge of modern Machine Learning and Deep Learning.

The course walks through the basics of ML and introduces PyTorch and will require knowledge of it in later stages.

This post will be short and sweet and simply walk you through the steps you need to take to make use of PyTorch using a standard Python installation and have a GPU enabled. This is all possible with PyTorch nightly which introduces a new MPS backend:

The new MPS backend extends the PyTorch ecosystem and provides existing scripts capabilities to setup and run operations on GPU.

This was previously announced on the PyTorch Blog and is a good read in and by itself. The expected improvement over regular CPU training/evaluation looks spectacular, but

Installing Python

I use asdf to manage the languages I work with. It can be installed with brew install asdf. This adds a small shell script that you can use to install plugins and individual versions. We'll start by adding the Python plugin:

asdf plugin add python

We can then list all available versions using:

asdf list all python

To install a version and make it the global default we simply run:

asdf install python 3.10.7
asdf global python 3.10.7

Refer to the asdf documentation to learn more about all its features and supported plugins.

Note that at the time of this writing PyTorch is not compatible with Python 3.11 - we'll use the latest 3.10.x release instead.

Setting up our work environment

For the first project I want to set up a virtual environment with all the core dependencies and a Jupyter Notebook installation for easy exploration. We'll start by creating a virtual environment and activating it:

python -m venv .venv
. .venv/bin/activate

From here we can use the configuration tool on the PyTorch website to get the installation command for the nightly version.

pip3 install --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cpu

With this we should have PyTorch installed and ready to go. We can validate our installation in the Python REPL:

$ python
Python 3.10.7 (main, Nov 17 2022, 15:26:39) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.backends.mps.is_available()
True

At this point I was able to follow the PyTorch tutorial and leverage my GPU. In places where the tutorial references a CUDA device, you can simply use the mps device.

I'm excited to have a powerful GPU readily available on my machine without the need to build a separate rig with CUDA cores. Given that I'm new to this field I'm not sure how far the Apple GPUs will take me, but for now they can support me in my learning journey.

Git is an immensely powerful tool with a vast set of features. Worktrees are one feature that was unknown to me until recently, and boy do I wish I had discovered it sooner.

When working in a team, especially when working on larger codebases, it will be common to switch branches frequently. I usually git stash any uncommitted changes or simply git commit -m "wip" them. This can quickly get out of control when I switch multiple times and then become complicated when juggling numerous different stashes.

Git worktrees are a fantastic solution to improve this workflow. It's not a new feature by any standard. It has been around for years.

What is git worktree?

"Manage multiple working trees attached to the same repository."

When you git switch <branch> to another branch, your working directory will be changed in place. The changes from the other branch are no longer directly accessible. This is precisely why you have to stash or commit any changes before switching. With a normal git clone you get exactly one working directory.

With git worktree you can have multiple directories for the same repository. Each worktree will point to a different commit/branch. This opens up the ability to quickly switch between branches without interrupting your current work.

I personally started using worktrees to check out code from other people's PRs for review or development or to simply switch to a different set of work when I get stuck on something.

Usage

Git worktree requires that you create a bare clone of your repository. This will look a bit weird if you haven't seen it before. Instead of a .git directory alongside your code, you will have a directory with all the contents that normally go into .git.

git clone --bare https://github.com/golang/go.git

With this on your disc, you can now create a worktree by typing:

git worktree add master

This will create a new folder inside your bare clone with the name master that will contain all the code from the master branch. From there, you can interact with your code and git just like you usually would.

At this point, we've just reached feature parity. The usefulness of this feature becomes apparent the moment you create a second worktree, i.e., to work on a separate ticket:

git worktree add feat-123

You now have two separate versions of your code side by side that you can work on without having to stash or commit code when switching between branches.

We can just as easily remove worktrees we no longer need with git worktree remove feat-123

Editor Support

Despite their usefulness, I haven't found great support for worktrees in code editors. There's an open issue for VSCode requesting support for worktrees, but it seems it's lacking community support to get something shipped.

There exists a plugin for neovim however, that was written by popular twitch.tv streamer ThePrimeagen. This plugin allows rapid switching between and managing worktrees from within neovim. He even explains the concept and his plugin in a video on Youtube:

Recap

Git worktree is a hidden gem that is unknown to many. They can drastically help simplify development workflows by allowing users to switch between branches without committing their work quickly.

I'm almost mad I hadn't discovered this feature of git sooner. The number of times I have lost some of my work in a messy git stash could have easily been zero had I used worktrees from the get-go.

Earlier this year, and for the first time in my professional life, I experienced pain caused by repetitive stress injury (RSI). An incorrect desk setup or keyboard can have many detrimental effects on your ability to sustain long hours in front of a computer. With RSI not wanting to go away, I was on the market for a new, more ergonomic keyboard that put less stress on my forearms.

As I was traveling shortly after experiencing RSI-related issues, I had time to research and explore different keyboard options. For the last few years, I've been typing away on a pretty standard flat Apple magic wireless keyboard which I've enjoyed. The keyboard is super flat and portable, easy to stow away in a bag and its battery lasts forever. Combined with the magic trackpad, it's been a good travel companion.

Initially, I bought a Ducky One 2 Mini RGB. Not at all an ergonomic upgrade, but I thought it would be enough of a change to relieve me of the pain. After just a few days of typing on it, I could already tell this wasn't working for me. I also struggled to get used to not having arrow keys available and ended up not liking the high keys as much as I thought.

QWERTZ/QWERTY

The first computer we owned in 1998 came with a standard German keyboard layout. Once I got into programming, I very quickly converted to the English QWERTY layout as the keys we use as programmers are much more accessible with this layout. That was probably around 2001 or 2002 when I got my own computer in my room.

I never correctly learned touch typing; I was an auto-didact. I mainly use my index, middle, and ring fingers for typing. My thumbs both rest on the space bar, and I'll use them for keyboard shortcuts that include the Option or Command key. With shortcuts that use the Function or Control key, I'll switch to my left little finger, which I also use to press the Shift key. Further up the keyboard, I will use my ring finger to hit Tab or backspace. My right little finger will only hit enter. It's worth mentioning that my right Shift key goes completely unused.

With this weird, self-taught typing style, I manage to hit speeds of 110 WPM with an average somewhere in the low-00s with an accuracy around 98-99%. This seems to place me well above average professional typists.

All that out of the way, I still feel like my typing isn't particularly efficient. My fingers jump all over the keyboard, and I have to make some awkward moves to nail specific keyboard shortcuts.

Dvorak & Touch Typing

This finally brings me to the Kinesis Advantage 2 QD. I've first come across this keyboard in a review by Martin Fowler. After more research and a deep look into my pockets, I decided to give this keyboard a shot. After all, not being able to type because of wrist pain would eventually cost me more than this keyboard.

I was always quite intrigued by weird-looking ergonomic keyboards. Sometimes you'd find them split into two halves, sometimes they're propped up, and then there's the Advantage 2 with its strange hollow body.

Similarly, I always wanted to learn Dvorak. Years ago, I tried to learn it for a few weeks, but it simply never stuck with me. Since I would have to learn to touch type on the Advantage 2, I might as well learn Dvorak with it.

Luckily, Kinesis offers the QD model, which, and you can see this if you look closely enough, comes with the keys labeled for QWERTY and Dvorak. While I never really look down while typing, I figured this would be handy for the first few weeks when learning Dvorak.

Progress has admittedly been slow, and I had to switch back and forth between my new and my old keyboard to stay productive at work. I have been using keybr.com to practice typing Dvorak as it's too difficult for me to remain productive while adjusting to the new keyboard and keyboard layout. Below you can see my current progress after eight hours of practice (plus some additional time I just spent typing away on Slack with the new keyboard).

There's visible progress, and I have just added the last remaining letters to the practice set. I'm nowhere near my average typing speed, but I'm also just using the keyboard for about an hour every day so far.

My pain is gone, however, so I'm more than happy to have made the switch. The keyboard takes a few days to get used to. There's a lot of muscle memory to rewire when you've used a regular keyboard for two decades, so I often found myself hitting backspace when I wanted to hit space.

It's fascinating to watch the brain make new connections while I'm not at the keyboard. I'd get frustrated after an hour of practice making more and more mistakes, but coming back the next day and I can immediately type faster, make fewer errors while using new keys.

The keyboard forces you to touch type and the experience so far is that my fingers flow a lot smoother over the keyboard. Without touch typing, it sometimes feels as if my fingers were performing some kind of breakdance.

(Real) Programmer Dvorak

Dvorak is optimized for English text. While my main communication language is English, as a programmer I frequently need special characters. This exact need has brought on several variations of Dvorak called "Programmer Dvorak" and "Real Programmer's Dvorak". Both aim to optimize the position of special characters.

I briefly tried to set up my Advantage 2 to emulate Real Programmer's Dvorak, but I haven't found it difficult to properly remap all keys on the Advantage 2 to make it work. For now, I've decided to stick to regular Dvorak and explore additional remaps once I'm more used to the layout and keyboard.

Resources

There are plenty of resources to use for learning to type with a new keyboard layout. The two sites that I can wholeheartedly recommend, are both free and provide generated word sequences (or quotes).

• https://monkeytype.com/
• https://www.keybr.com/

keybr.com stands out as it will start you with a small set of letters and lets you only add more letters once you've reached a certain WPM threshold. Other than that, the only thing required is time and practice.

Our underwater journey continues. It can get dark underwater, and today is no exception. The only thing we can see apparently is a giant squid that has attached itself to the outside of our submarine.

It's only logical that we decide to play Bingo with the squid. Hey, maybe we should check the oxygen generator and co2 levels again; we seem to be making strange decisions.

Part 1

The rules of Bingo are simple: A unique number is drawn from a pool. If your board contains this number, you mark it off. The first player with five numbers marked in a horizontal or vertical direction wins the game.

Since the squid is excellent at multi-tasking, it will play many boards simultaneously.

7,4,9,5,11,17,23,2,0,14,21,24,10,16,13,6,15,25,12,22,18,20,8,19,3,26,1

22 13 17 11  0
 8  2 23  4 24
21  9 14 16  7
 6 10  3 18  5
 1 12 20 15 19
 
 3 15  0  2 22
 9 18 13 17  5
19  8  7 25 23
20 11 10 24  4
14 21 16 12  6

I'll spare you the boring part of parsing the input numbers and bingo boards; that's quickly done with a few strings.Fields.

For the first part, we only need to find the board that wins first. We compute its score by taking the sum of all unmarked numbers and multiplying that with the number that was last drawn.

I chose to represent the board as a struct, containing the list of numbers, a list of booleans of the same length, and a boolean flag to indicate whether the board has won already.

type Board struct {
	numbers  []int
	matched  []bool
	hasBingo bool
}

After constructing all the bingo boards, we can loop over the list of drawn numbers:

Loop:
for i := range numbers {
    for b := range boards {
        if boards[b].hasBingo {
            continue
        }
        boards[b].markDigit(numbers[i])

        if boards[b].isBingo() {
            score = boards[b].sumUnmarked() * numbers[i]
            break Loop
        }
    }
}

For each new number, we call markDigit (implementation omitted for brevity) on each board. The method checks whether the number is on the board and marks it as matched. Once a board has a bingo, we compute its score and break out of the loop.

func (b *Board) isBingo() bool {
	// Check rows
	isBingo := true
	for i := 0; i < len(b.matched); i += 5 {
		isBingo = true
		for j := 0; j < 5; j++ {
			isBingo = isBingo && b.matched[i+j]
		}
		if isBingo {
			return true
		}
	}

	// Check columns
	for i := 0; i < 5; i++ {
		isBingo = true
		for j := 0; j < 5; j++ {
			isBingo = isBingo && b.matched[i+j*5]
		}
		if isBingo {
			return true
		}
	}
	return false
}

sumUnmarked iterates over the list of all numbers to find those that are unmarked.  

func (b *Board) sumUnmarked() (sum int) {
	for i := range b.numbers {
		if !b.matched[i] {
			sum += b.numbers[i]
		}
	}
	return
}

Let the games begin!

Part 2

In part two, we worry that it might be wiser to let the squid win, who knows what it'll do to our precious submarine if it lost. We can easily modify our loop from part one and keep track of the board that wins last.

for i := range numbers {
    for b := range boards {
        if boards[b].hasBingo {
            continue
        }
        boards[b].markDigit(numbers[i])

        if boards[b].isBingo() {
            if firstScore == 0 {
                firstScore = boards[b].sumUnmarked() * numbers[i]
            }
            lastScore = boards[b].sumUnmarked() * numbers[i]
            boards[b].hasBingo = true
        }
    }
}

I don't think I've ever solved the second part of any Advent of Code puzzle quite this quickly.

Wrap up

There's little to say about today's puzzle. It was an easy puzzle and serves as a good exercise to practice some basic programming.  

A possible optimization for isBingo would be using a binary representation  marked since we could then leverage bitmasks. Given that we're dealing with a low number of boards and performance wasn't an issue, I'll leave that as an exercise for later.

Deep under the water, we find ourselves hearing creaking noises. Our submarine handily gives us a diagnostics report. Sadly, as is oft too common in AoC, the report format is complete garbage, to put it mildly.

Our puzzle input comes in the form of binary numbers. The task is to compute a gamma rate and an epsilon rate based on a simple algorithm. The example report given to us looks like this:

00100
11110
10110
10111
10101
01111
00111
11100
10000
11001
00010
01010

Instead of taking these binary numbers as-is, we need to construct new ones to get the gamma and epsilon rates. We do this by looking at the binary numbers bit by bit. The gamma rate results from taking the most common bit amongst all binary numbers for each position.

A short example can illustrate this much better than words, so let's take the first three binary numbers from above:

00100
11110
10110
=====
10110

By looking at the first bit of each number, we can see that there are two 1s and only one 0, so the first digit of the gamma rate is 1. We continue this for each digit until we get a gamma rate of 10110 or 22 in decimal.

The epsilon rate is obtained the same way, but we're taking the least common bit this time. A careful observer will notice that this can also be obtained by simply negating each bit of the gamma rate.

Part 1

Naïvely, we can skip converting the input to their actual binary representation and work on them as ASCII characters. We can read the file into a []string. For each position in the bitstring, we use a counter that increases for every 1 and decreases for every 0.

var cntr []int = make([]int, len(lines[0]))
for _, line := range lines {
	for idx, ch := range line {
		switch ch {
			case '1':
				cntr[idx] += 1
			case '0':
				cntr[idx] -= 1
		}
	}
}

In the next step, we can use this to construct the gamma and epsilon values using binary shifts:

var gamma, epsilon uint64
for idx, val := range cntr {
	if val > 0 {
		gamma = gamma | 1 << (len(cntr) - 1 - idx)
	} else {
		epsilon = epsilon | 1 << (len(cntr) - 1 - idx)
	}
}

We can then compute the puzzle answer as the product of gamma and epsilon.

Part 2

Part two is a bit of a tricky variation on part one. Instead of finding the most common bit value at each position, we use the most common bit as a cumulative filter for the whole list of numbers.

Initially, I solved this by building a binary tree where the edges represented a zero or a one. By traversing the tree and walking in the direction of the subtree with the most child nodes, it was easy to find the solution.

A binary isn't very efficient for this type of puzzle - after all, we're just dealing with binary numbers, and if there's something computers are fast at, it's binary.

A much faster solution comes from my friend @javorszky, who solved this part recursively. His algorithm makes use of bitmasks. To explain this, let's take the first four numbers:

00100
11110
10110
10111

For each position, we construct a bitmask. For the first position, this is simply 1<<5 or 10000. Any number bigger or equal to the bitmask has a one in that position; any number smaller has a zero. We continue this until we've filtered all the sublists down into a single element.

func filterNumbers(list []uint, pos int, larger bool) uint {
	if len(list) == 1 {
		return list[0]
	}
	mask := uint(1 << pos)
	ones := []uint{}
	zeros := []uint{}

	for _, item := range list {
		if item&mask >= mask {
			ones = append(ones, item)
		} else {
			zeros = append(zeros, item)
		}
	}

	if (len(ones) >= len(zeros) && larger) || (len(ones) < len(zeros) && !larger) {
		return filterNumbers(ones, pos-1, larger)
	}
	return filterNumbers(zeros, pos-1, larger)
}

Wrapping up

This was an interesting challenge that offered many solutions. We're not yet in the realm of puzzles that need heavily optimized algorithms, so it's great to play around a little. Implementing a binary tree from scratch isn't something I get to do very often. I'm definitely looking forward to more challenging puzzles going forward, but this was another well-crafted puzzle.

Still with me? Great! Let's head on to solving day 2.

The story tells us, that we're still in the submarine and are trying to learn how to navigate it. The submarine can take a bunch of simple commands in the form of direction unit.

forward 5
down 5
forward 8
up 3
down 8
forward 2

Someone has already programmed a course for us and it's our task to figure out where we're going. We're to compute the final horizontal position and our final depth. An important note here is that down increases the depth while up decreases the depth of the submarine.

Part 1

Since I'm trying to learn some concepts of Go I decided to represent the commands as a struct:

type command struct {
    direction string
    steps int
}

This is easy enough. We have a direction and a number of steps in the defined direction.

We can now write go on to read our puzzle input.

func getInputs() []command {
	bytes, _ := ioutil.ReadFile(filename)

	lines := strings.Split(string(bytes), "\n")
	cmds := []command{}

	for _, line := range lines {
		if len(line) == 0 {
			continue
		}
		instr := strings.Split(line, " ")
		val, _ := strconv.Atoi(instr[1])
		cmds = append(cmds, command{instr[0], val})
	}
	return cmds
}

Once again, there's not too much magic going on here. We read the file, split it into lines, and then further split the lines into the two components representing our direction and a number. A more experienced Go programmer would probably use bufio.ReadLine to read the file line-by-line, but for now, I'm content that this will do the job just fine.

Computing our horizontal position and depth is now as easy as looping over all commands and tallying up the steps based on the direction:

var pos, depth int

for _, cmd := range cmds {
	switch cmd.direction {
	case "forward":
		pos += cmd.steps
	case "down":
		depth += cmd.steps
	case "up":
		depth -= cmd.steps
	}
}

Part 2

The second part of the story suggests that the final position we calculated doesn't make any sense, and so an alternative algorithm is proposed. The down and up commands don't directly affect our position but represent changes to the submarine's aim.

Assume for a moment our instructions were:

forward 5
down 1
forward 2

In Part 1 this would have resulted in a horizontal position of 7 and a depth of 1. In Part 2 the interpretation changes to:

• Move 5 steps forward
• Adjust aim to 1 (aim down!)
• Move 2 more steps forward and 2 steps down in depth (since our aim is at 1 and we're stepping two times)

In other words, every movement forward by X steps will change our depth by X * aim.

We need to modify our code ever so slightly to keep track of our sub's aim:

var aim int
pos = 0
depth = 0

for _, cmd := range cmds {
    switch cmd.direction {
    case "forward":
        pos += cmd.steps
        depth += aim * cmd.steps
    case "down":
        aim += cmd.steps
    case "up":
        aim -= cmd.steps
    }
}

That's it. We're done with part 2.

Summary

Day 2 built really well on the previous day. It's good enough to practice I/O and processing text input. The task should be easy to solve even if the programming language you're using is relatively new to you.

Over the last few years, Advent of Code has gained a lot of popularity. Many developers use it as a creative way to challenge themselves or practice their skills in new programming languages. Nearly 200k people completed the first day this year.

Myself, I'm using this year to learn and practice Go. While I was one of the early adopters of Go in 2009 and used it for about a year during my postgraduate studies, I have not touched it since, since my career took me in a different direction. With the rising popularity of Kubernetes and microservices, Go seems like an excellent addition to my toolset.

I'll attempt to cover all of this year's challenges and explain my thought process going into them. Since the later exercises usually take a lot of time to solve, I expect this series to run well into 2022.

Part 1

As per usual, the first day starts relatively easily. We're in a submarine with a sonar sweeping the ocean floor and recording the depth at certain intervals. The first task is to count the number of times a depth measurement increases.

We first need to read our puzzle input which is given as a sequence of positive integers with each number on a new line.

func loadMeasurements() []int {
    bytes, err := ioutil.ReadFile(filename)
    if err != nil {
        panic(err)
    }

    lines := strings.Split(string(bytes), "\n")

    nums := []int{}
    for _, val := range lines[:len(lines)-1] {
        num, _ := strconv.Atoi(val)
        nums = append(nums, num)
    }

    return nums
}

This snippet gives us an int-slice with all numbers from the file. It's now trivial to loop over this slice and count the number of increments:

var count, last int

measurements := loadMeasurements()
last = measurements[0]
for _, num := range measurements {
	if num > last {
		count++
	}
	last = num
}
fmt.Printf("Number of increments: %d\n", count)

This solves part 1.

Part 2

We've now concluded that looking at every single measurement isn't useful, so we decide to instead look at the last three measurements. The goal is still the same, count the number of times the sum of three measurements is bigger than the three a measurement ago.

To visualize this a little bit, imagine the numbers are: 1 2 3 4 5 6. In this example the sum of the first three measurements 1 2 3 is 6. In the next step, we're looking at 2 3 4 whose sum is 9. This continues until we've run out of numbers.

The code for this is fairly simple to write. Start with the first three measurements and then step through the entire slice one by one:

count = 0
prev := math.MaxInt
for i := 0; i < len(measurements)-2; i++ {
	sum := measurements[i] + measurements[i+1] + measurements[i+2]
	if sum > prev {
		count++
	}
	prev = sum
}

It's important to recognize that we want to ignore the first measurement, and we can do this by initializing prev to math.MaxInt since it ensures that our first measurement can't be bigger than prev.

Summary

That's it for day 1. No crazy algorithms yet, but enough to prepare you for reading the problem descriptions and doing basic IO.

Diagrams.net (formerly draw.io) is a fantastic website and tool that allows you to create rich diagrams. The service is entirely free and diagrams can be saved to your Google Drive, Dropbox, or downloaded to your computer. Additionally, diagrams.net allows you to export your diagrams to various formats such as SVG, JPG, and PNG.

Recently it was pointed out to me that you can actually load an exported PNG diagram back into the tool and edit it again. This got me thinking - how are they doing this? Surely they aren't using image recognition techniques to identify objects in the image?

You may wonder: What does all of this have to do with the title of this post? Let's talk about the PNG format a little.

PNG File Format

PNG stands for Portable Network Graphics. The format has existed in some form since the mid-nineties. Like all binary file formats, it follows a specification. From the specification, we can learn a lot about how the image and its metadata is represented on disk.

File Header

A PNG file starts with an 8-byte signature. This signature tells a decoder that all bytes that follow are to be interpreted based on the PNG spec. In hexadecimal representation the header is: 89 50 4e 47 0d 0a 1a 0a

Chunks

The remainder of the PNG format follows a very simple structure. Data is represented in chunks. Each chunk starts with 4 bytes describing the length of chunk data. Then follow 4 bytes for the chunk type. This again is followed by length bytes of chunk data and finally 4 more bytes for a CRC (cyclic-redundancy check). The CRC can be computed over the chunk type and chunk data.

The file specification mentions that while the length is represented using 4 bytes or 32bits the maximum length of chunk data is actually 2^31.

The chunk types are more interesting as there are plenty of them. I won't go into much detail here and instead only cover the relevant bits for this post. I encourage you to go read the specification for yourself to understand the nifty encoding techniques used here.

Since the chunk type is represented by 4 bytes, they can (mostly) be represented using 4 ASCII characters. Chunk types are split into critical and ancillary chunks - a decoder must understand all critical chunks but can safely ignore the ancillary chunks.

The critical chunks are as follows:

• IHDR must be the first chunk in the file. It contains in specific order the image width, height, bit depth, color type, compression method, filter method and interlace method.
• PLTE contains information about the color palette used
• IDAT contains the actual image. There can be multiple IDAT chunks which is what allows PNG to be a streamable format in which the first smaller IDAT chunk allows a pre-render of the full image before all data is received.
• IEND marks the end of the file

A selection of ancillary chunks:

• tIME stores the time the image was last changed
• tEXt stores key-value metadata. The text is encoded in ISO 8859-1. The key must be between 1 and 79 characters long and is terminated by a null character. The remainder of chunk data is the value.

Exporting a diagram from diagrams.net

Before we get started with writing some Julia code, let's first export a PNG file from diagrams.net.

This is fairly straightforward, just head over to diagrams.net, click together a diagram and hit File > Export and choose PNG. Make sure to keep the "Include a copy of my diagram" checkbox checked.

File IO in Julia

With everything prepared we can start looking into I/O. We're not going to do anything advanced here so we'll just look at the basics.

Interacting with files, regardless of the language, always follows the same pattern:

• Open a file for reading/writing
• Read/write
• Close the file descriptor

Julia is no exception to this. We can use Base.open to open a file. This will give us an IOStream instance which in turn wraps the OS file descriptor. We can either do it in a block, in which case the file will be closed automatically at the end of the block, or we call open/close separately.

open("myfile.txt", "r") do io
   # ...
end;

Furthermore, there are multiple ways to read data from a file.

We'll need read and readbytes!. They both take an IOStream (the result of the open call) as the first argument. read takes a primitive type as a second argument telling it to read a single value of that type from the IO and return it. I.e. read(io, UInt32) will read the 4 bytes it takes to represent a UInt32.

readbytes! requires a vector-like object to be passed as its second argument. It will read as many bytes as the vector can hold as long as there's data to read.

Reading in the PNG file

Let's put what we've just learned together. Here's the plan:

• Open the PNG file
• Check for the file header (remember those 8 bytes mentioned above?)
• Read in PNG chunks by first consuming the length, the type, the data based on the length field and finally the CRC.

We can represent PNG chunks using a struct with named fields for each of the elements. The easiest way to represent a sequence of bytes is using a Vector{UInt8}. Here's the struct I came up with:

struct PNGChunk
    length::UInt32
    type::Vector{UInt8}
    data::Vector{UInt8}
    crc::Vector{UInt8}
end

It's also useful to declare a constant for holding the PNG header:

const PNG_HEADER = [0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]

Let's now open the PNG file and read in the first 8 bytes for the header:

io = open("Diagram.png", "r")
header = Vector{UInt8}(undef, 8)
readbytes!(io, header)

readbytes! takes an IOStream handle and a variable that it will try to fill. You can pass it an additional integer to indicate the number of bytes to read but it defaults to the length of the second argument which we've declared as a vector of UInt8s with 8 elements.

By simply comparing header with PNG_HEADER we can determine whether we're dealing with a valid PNG file:

if header ≠ PNG_HEADER
    throw(ArgumentError("File is not a PNG"))
end

Assuming our file is valid we can now attempt to read in all the chunks in the file. It's easiest to do this iteratively with a loop and consume the file until we hit EOF. Luckily Julia provides an eof function that takes an IOStream and returns whether or not we've reached the end of the file.

while !eof(io)
    length = hton(read(io, UInt32))

    type = Vector{UInt8}(undef, 4)
    readbytes!(io, type)

    data = Vector{UInt8}(undef, length)
    readbytes!(io, data)

    crc = Vector{UInt8}(undef, 4)
    readbytes!(io, crc)

    push!(chunks, PNGChunk(length, type, data, crc))
end

I'm calling hton to get the length represented properly. This is because my system (Intel-based MacBook Pro) is a little-endian system (meaning the least significant byte comes first) but PNG represents all data in big-endian requiring us to reorder bytes.

The loop will continue to consume bytes for the chunk type, data, and the CRC and construct a PNGChunk that will then be pushed into a vector.

Note: The above code will work for a valid PNG file. There's no error checking at all so if one of the fields is corrupted or the file ends prematurely this will throw an error and fail.

Displaying chunks

Now that we're done reading the file we should take a look at its contents. For this, we can add a bunch of helper functions.

We essentially want to run something like:

for chunk in chunks
    print(chunk)
end

but executing this will result in a lot of gibberish being displayed. To tell Julia how to display a PNGChunk we need to implement Base.show for our type. Base.show takes an IO object and an instance of a type. You can compare this with __repr__ in Python. An implementation that will display the length and the type of a chunk might look as follows:

function Base.show(io::IO, c::PNGChunk)
    println(io, length(c), "\t", type(c))
end

Where in other languages you declare methods on classes, in Julia you simply declare a function that operates on a type. To make the implementation of Base.show work we need to define length and type:

length(c::PNGChunk) = c.length
type(c::PNGChunk) = String(Char.(c.type))

While we could simply access chunk.length directly it's common practice to consider struct fields "private" and write functions to access them. This way you get a layer of abstraction and can easily change the layout of structs without breaking code all over the place.

To deconstruct what's going on in the second line let's start by looking at c.type. We declared the type to be a Vector{UInt8} and we consumed 4 bytes while reading the PNG file. The first thing we want to do is convert each item in the vector to its ASCII character representation. Julia provides the Char data type to represent 32-bit characters. Simply calling Char(c.type) would result in Julia attempting to consume all 4 bytes (32 bit) and won't give us the desired result.

Instead, we can iterate over the items in the vector and convert each item to a Char. This could be written using a list comprehension like [Char(ch) for ch in c.type] which is rather lengthy but standard if you're coming from Python. Julia conveniently offers the dot-operator (also called broadcast) which can be applied to any function. By writing Char.(c.type) we're essentially expressing "apply each element in c.type to the Char function".

Finally, we wanted to obtain the string representation of those characters and by passing a Vector{Char} to the String function we can cast it into a string.

More tenured Julia developers would probably write all of the above simply as collect(Char, c.type) |> join, but we're going to ignore this for now.

Ok, back to displaying the chunk. With Base.show and our two functions out of the way we can loop over the chunks and see what's inside our file:

13      IHDR
970     tEXt
3379    IDAT
0       IEND

So that's cool - we've got three chunks with data. IHDR contains height, width, color depth and some other metadata about the file and IDAT contains the actual image. This leaves tEXt which could contain anything.

Extracting information from IHDR

Let's see if we can make sense of the data in the IHDR chunk. First we're going to modify our Base.show implementation to also display the data field when we recognize the chunk type.

function Base.show(io::IO, c::PNGChunk)
    println(io, length(c), "\t", type(c) ,"\t", datastr(c))
end

The specification tells us that there are 13 bytes reserved for the IHDR data field and how many bytes are reserved for different properties.

The IHDR chunk must appear FIRST. It contains:

   Width:              4 bytes
   Height:             4 bytes
   Bit depth:          1 byte
   Color type:         1 byte
   Compression method: 1 byte
   Filter method:      1 byte
   Interlace method:   1 byte

The multi-byte fields will require endian conversion. Since we have already read in all data we need to reinterpret the data from our Vector{UInt8}. That's exactly the name of a Julia function that helps with reinterpreting data into another type:

hton(reinterpret(UInt32, c.data[1:4])[1])

This will take the first four bytes of chunk data and reinterpret them into a UInt32. The wrapping hton will make sure to convert from host byte order to big endian. We can repeat this for the height field and then read all the individual bytes.

function datastr(c::PNGChunk)
    if type(c) == "IHDR"
        height = hton(reinterpret(UInt32, c.data[1:4])[1])
        width = hton(reinterpret(UInt32, c.data[5:8])[1])
        depth, ct, cm, fm, im = c.data[9:13]
        return "h=$height, w=$width, d=$depth, color type=$ct, compression method=$cm, filter method=$fm, interlace method=$im"
    end
    ""
end

For my diagram I get the following output:

h=121, w=201, d=8, color type=6, compression method=0, filter method=0, interlace method=0

Obtaining the original diagram from tEXt

Finally, let's peek inside the tEXt chunk. We can first extend our datastr(c::PNGChunk) function to also have a branch to catch the tEXt type and simply print the contents of the data field:

 mxfile %3Cmxfile%20host%3D%22app.diagrams.net%22%20modified%3D%222021-05-24T09%3A22%3A42.489Z%22%20agent%3D%225.0%20
(Macintosh%3B%20Intel%20Mac%20OS%20X%2010_15_7)....

That's a bunch of gibberish. Consulting the specification tells us that the data field for tEXt consists of a key and value pair separated by a null-byte. That should be easy to parse:

key, value = split(String(Char.(c.data)), '\0')

But that's only half the equation. It looks like the value part may be URL encoded and so we need to find a way to decode it. I couldn't find this functionality in the standard library and so I ended up installing URLParser.jl which implements unescape.

(@v1.6) pkg> add URLParser

Putting everything together we can complete our datastr function by adding tEXt handling:

    elseif type(c) == "tEXt"
        key, value = split(String(Char.(c.data)), '\0')
        value = unescape(value)
        return "$key, $value"
    end

And so the final output is:

13      IHDR    h=121, w=201, d=8, color type=6, compression method=0, filter method=0, interlace method=0
970     tEXt    mxfile, <mxfile host="app.diagrams.net" modified="2021-05-24T09:22:42.489Z" agent="5.0 (Maci
ntosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" et
ag="MSmUq0enpJxQ3pDGyP_L" version="14.3.0"><diagram id="py8BCTe_me7SkJGnhe6H" name="Page-1">zZRNb4MwDEB/DcdJ
EDbaHruOdYdNm9TDdo2IC5kCRsF89dcvjFCKWKvtUGmXijw7dfxwcPxN2mw1z5MXFKAc5orG8R8cxpZLZn470FqwcHsQayl65I1gJw9g4ZBW
SgHFJJEQFcl8CiPMMohowrjWWE/T9qimVXMewwzsIq7m9F0KSiz1gtUYeAIZJ7b0ki36QMqHZNtJkXCB9QnyQ8ffaETqn9JmA6pzN3jp9z2e
iR4PpiGj32wItquqWN+tgjC+3fPXj0Ks4cbv/6XiqrQN28NSOxgAYYTYJWpKMMaMq3Ck9xrLTEBXxjWrMecZMTfQM/ATiFr7dnlJaFBCqbLR
vmZX6GxvFhVY6gguNDTMCNcx0IU8dnwDZnIBUyDdmn0aFCdZTc/B7QzFx7xRs3mwpv9g3ZtZX6f8ILN4Jn9U23mqE0mwy/m3gdrct580VqAJ
mssi543bDcy102qvqzdc1/pk+IeJTk7mPnCv5IrNXL0ppOLfmfK965kyy/E78R07+dj64Rc=</diagram></mxfile>
3379    IDAT
0       IEND

The secret to how diagrams.net embeds the diagram is solved. It's urlencoded XML embedded into a tEXt chunk inside the PNG file (now that's a fun sentence to say!).

The full code can be found at https://github.com/halfdan/geekmonkey/tree/main/julia/lwm-03

Summary

In this article, we've covered a lot of different concepts in Julia. If you struggled to keep up - don't worry I'll go over all the concepts mentioned here in more detail in future posts. My approach to learning is often guided by the projects I want to do and so I often jump in at the deep end. As a result, this article introduced concepts rather rapidly without spending too much time on the mechanics.

It's always fascinating when you think about how many things we take for granted in tech without thinking about the underlying mechanics. I was definitely surprised by how easy it was to extract some metadata from a binary format like PNG. I've used PNG files for decades without ever thinking about their inner structure. Clearly, we've only scratched the surface and haven't looked at the IDAT chunk containing all the image information, but we'll get there with time.

Before we really dive into Julia I wanted to go over the tools and learning resources I have and will be using going forward. These resources fit my learning journey and may not directly apply to you so I encourage you to spend some time to see what's out there.

Learning Resources

I have compiled a list of learning resources that I intend to explore over the next 6 months as I progress in the Julia language. Any learning journey is prone to fail if you don't work towards an end goal. As stated in my previous post I want to use Julia to get into data science and machine learning and so it should come as no surprise that my selection of learning resources is biased toward these topics.

It was difficult to settle on a good book since the majority of books listed on the Julia website seem to be targeted towards programming beginners and seem to be lacking the depth I am looking for. Some other books have terrible ratings and so I excluded them. I ended up including only one book in the list below.

Basic

The following resources should help get a basic understanding of programming in Julia and provide a good set of problems and challenges to practice learned skills.

• The Julia track on exercism.io where you can get your coding solutions mentored by volunteers. This provides plenty of exercises to apply some of the basics and get high quality feedback.
• The Julia Language - A concise tutorial gitbook, as the name suggests, gives a concise overview of several important topics such as I/O, data structures, meta programming, package development and concurrency.
• Julia Academy provides a number of interesting courses.
• Finally, the official Julia documentation will always be my first go-to resource should I get stuck in a problem. As a shorthand for browsing the documentation I'll resort to using Julia's built-in help mode in the REPL.

Intermediate/Advanced

Eventually I'd like to dive a lot deeper into Julia performance and how to squeeze the best performance out of my code.

• The MIT course Introduction to Computational Thinking is an incredibly well received course that uses Julia and Pluto.jl. The course teaches image analysis, machine learning, climate modelling and dynamics on networks.
• The only book that stood out to me is Julia High Performance - Second Edition by Avik Sengupta, Alan Edelman. Just judging from its cover it seems to go into a lot more depth than most books about Julia out there.

Tools

When it comes to tools, there really isn't that much needed for Julia other than a decent computer that runs any of the three main OSes. Most of my development these days happens on Mac OS, but I'm also exploring developing on a more powerful Windows machine with WSL2.

For now, I'm going to be using:

• Visual Studio Code with the Julia for VSCode extension. It's definitely not a requirement to use this particular editor or even the extension but it's an editor I'm already very comfortable with.
• Pluto.jl for interactive notebooks and quick data exploration

That's really all that's needed to get started. I'm sure my development workflow and tools will evolve over time but for now

If you are interested in seeing a different setup using neovim and tmux I suggest you check out Jacob Zelko's excellent post on his writing and coding workflow.

Project-driven Learning

My method of learning a new (programming) language has evolved over time. Immediately applying what you learn is essential for retention. As I read through my resources and explore Julia packages I will develop small project ideas. These projects aren't necessarily meant to evolve into production-grade software, but rather produce something interesting or help me explore an area of technology.

Not all of my time will be spent on larger projects, however. There's value in exploring small coding challenges from exercism.io, projecteuler and others when you're short on time or can't make any progress in your project.

As I dive deeper into Julia and get more comfortable navigating the package ecosystem I expect the focus of my projects to also shift and my projects to grow in size.

I intend to document most of these projects and invite you to follow along as I learn this exciting language and discover its secrets.

Welcome to Learn With Me: Julia. A series where you can follow me along my journey of learning Julia, Data Science and Machine Learning. This series is heavily inspired by Learn With Me: Elixir, a series by Kevin Peter / The Inquisitive Developer and the format of this post will follow his introductory post for Elixir.

I realised that while there are plenty of resources about Julia already out there, it would be interesting to document my journey in picking up the language and some fundamental data science and machine learning with it.

The Julia community, to a large degree, consists of academics. The level of discourse on the Julia Slack / Zulip is often too advanced for me to understand. Researchers from all kinds of fields, space engineering, bio engineering, mathematics all come together to practice Julia. The 2020 Community Survey nicely shows this:

About You

I'm going to be writing this series for someone who has some programming knowledge and like me wants to learn about Julia. Some familiarity with computer science and algorithms will be useful and you should also have an interest in mathematics as I like to lean into the mathematics of machine learning when I'm ready to explore it with Julia.

For the rest of this section I'll quote Kevin Peter, since it also applies here:

So while this series is not meant for beginning programmers, you don't have to be a master programmer to follow along either. I very much doubt I will be delving into any advanced theoretical concepts or heavy mathematics. I'm aiming for practical stuff that a typical experienced software developer will be able to read and understand. I aim to be easily readable and informative.

If you need a resource on how to get started with Julia and a quick overview of why I chose this language you can read my Getting started with Julia post.

About Me

I left academia six years ago and have since been working with Ruby, JavaScript and Python in a professional setting almost exclusively. In my job I build web application backends and ETL pipelines and work closely with data scientists.

My personal interest in Machine Learning is what's driving me to Julia. Its expressivity over other languages like Python intrigues me and makes me think that it's only going to grow going forward.

Recently I've challenged myself to practice Julia 45 minutes every (week)day as part of a #100daysofcode challenge. I'm 25 days into this challenge and have explored the popular libraries such as Pluto, Plots, Revise and Javis.

You can find more information about who I am on the About page

I use Windows for gaming. It's been a long time since I've last done any serious development work on my Windows machine and yet I still spent a good chunk of money on building out a beefy machine for my efforts to learn machine learning. It has taken me a few months to finally sit down and get this machine ready for anything other than gaming.

With the recent announcement of GUI support for WSLg I got really excited to try out WSL and see how good the GPU support actually is, but that's not the main reason. I've been shying away from developing on Windows because I'm used to a *NIX environment. WSL gives you that, but up until recently you wouldn't have been able to interact with any GPU - and this all changed with this announcement!

You can watch the video below to see what's coming for WSL2.

The first half of this article will show you how to get everything set up and in the second half we'll set up CUDA.jl in Julia. Since the latter part is about CUDA I'll assume that you have a compatible nVidia GPU.

Installation

Here's a summary of what we need to go through to prepare our environment:

• Update Windows 10 to latest release on the dev channel
• Install nVidia CUDA drivers
• Install Ubuntu 20.04 in WSL2
• Install Linux CUDA packages
• 🎉

Windows 10 Insider Preview

At the time of writing all of the features are only available through the Windows Insider Program. The Windows Insider Program allows you to receive new Windows features before the hit the main update line. The program is split into three channels: Dev, Beta and Release Preview.

To receive the update with WSLg and GPU support we will need to switch to the dev channel.

Note: The dev channel comes with some rough edges and potential for system instability. Be mindful of this when you switch and make sure you have backups!

After installing all downloaded updates you should end up with OS Build 21364 or higher. You can check your OS Build by running winver in PowerShell/cmd.

With this all set we can hop on to install the latest WSL2 compatible CUDA drivers.

CUDA drivers

NVIDIA are providing special CUDA drivers for Windows 10 WSL. The link below will take you to the download page. It's required to sign up for the NVIDIA Developer Program, which is free.

GPU in Windows Subsystem for Linux (WSL)

CUDA on Windows Subsystem for Linux (WSL) - Public Preview Microsoft Windows is a ubiquitous platform for enterprise, business, and personal computing systems. However, industry AI tools, models, frameworks, and libraries are predominantly available on Linux OS. Now all users of AI - whether they ar…

NVIDIA Developer

Follow the setup wizard (I chose the express installation which keeps existing settings in place).

Note: There's also a documentation page provided by NVIDIA around setting up your GPU for WSL. I found the docs to be outdated and not working on my machine.

Installing Ubuntu 20.04 LTS with WSL2

Before we can proceed with installing Ubuntu I advise to update the WSL kernel by running:

wsl --update

In case you're like me and don't enjoy the default terminal Windows comes with I suggest you install Windows Terminal from the Microsoft Store. This terminal comes is a lot more pleasant to use than either cmd or the PowerShell terminal ever were.

It's also a good idea to set WSL to default to version 2:

wsl --set-default-version 2

Finally let's install Ubuntu with:

wsl --install --distribution Ubuntu-20.04

In case you were wondering what other distributions are available you can simply run: wsl --list --online.

Ubuntu 20.04 LTS

With Ubuntu installed we have a couple of final steps. First we will add an upstream repo to apt for getting the latest CUDA builds directly from NVIDIA:

sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"

We also need to add NVIDIA's GPG key for the apt repo:

sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub 

And finally to make sure that we prefer the packages provided by NVIDIA over packages in mainline Ubuntu we need to pin the apt repo:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600

With this out of the way, we're ready to install the CUDA drivers inside our WSL Ubuntu installation:

 sudo apt update && sudo apt install -y cuda-drivers

In case setting up WSL2 with GPU support was all you wanted to do - we're done!

Julia + WSL2 + Cuda.jl

Ok, so as promised in the headline we're now going to install Julia inside our Ubuntu 20.04 and set up Cuda.jl.

Install Julia

At the time of writing Julia 1.6.1 is the latest version available (make sure to check for updates on Julia's download page).

First let's fetch the latest Julia tarball:

wget https://julialang-s3.julialang.org/bin/linux/x64/1.6/julia-1.6.1-linux-x86_64.tar.gz

Extract the .tar.gz:

tar -xvzf julia-1.6.1-linux-x86_64.tar.gz

Move the extracted folder to /opt:

sudo mv -r julia-1.6.1 /opt/

Finally, create a symbolic link to julia inside the /usr/local/bin folder:

sudo ln -s /opt/julia-1.6.1/bin/julia /usr/local/bin/julia

You may pick a different target directory for your installation of Julia or use a version manager like asdf-vm.

Install Cuda.jl

At this point simply run julia in your terminal and you should be dropped into the Julia REPL. I assume you've worked with Julia before and know how to operate it's package manager.

Hit ] to enter pkg mode and install CUDA with:

activate --temp
add CUDA

From here hit backspace and import CUDA. CUDA.jl provides a useful function called functional which will confirm that we've done everything right (well, that's the hope at least, right?).

using CUDA
CUDA.functional()

You can additionally run CUDA.versioninfo() to get a more detailed breakdown of the supported features on your GPU.

At this point you should have a working installation with WSL2, Ubuntu 20.04, Julia and CUDA.jl. In case you're new to CUDA.jl I suggest you follow the excellent introduction to GPU programming by JuliaGPU or jump in at the deep end with FluxML's GPU support.

If you like articles like this one, please consider subscribing to my free newsletter where at least once a week I send out my latest work covering Julia, Python, Machine Learning and other tech.

You can also follow me on Twitter.

Some time in 2020 I came to the realisation that my 2013 built gaming computer for getting a bit outdated. With becoming a father and working a full time job I didn't have as much time as I used to to play games so for the past few years I barely had any use for a proper gaming PC and ended up only casually playing old games like Skyrim or Starcraft 2 which my computer was more than capable of handling.

With the pandemic came more time at home and at the same time a renewed interest in learning some machine learning and utilizing GPUs - it was time to build a new computer!

I set out to build a computer that is powerful enough to run new and upcoming games like Cyberpunk 2077 but also serve as a tool for machine learning experiments. It doesn't need beat all the benchmarks but it should provide me with enough computing power to last me several years.

My new machine

• CPU: AMD Ryzen 7 3700X 8x 3.60GHz So.AM4 BOX 269.00€
• RAM: 32GB (2x 16384MB) G.Skill Aegis DDR4-3200 Dual Kit 96.90€
• Motherboard: ASRock X470 Master SLI AMD X470 So.AM4 Dual Channel DDR4 ATX 121.62€
• SSD: 256GB Transcend 110S M.2 2280 PCIe 3.0 x4 3D-NAND TLC 256 36.40€
• SSD: 1000GB PNY XLR8 CS3030 M.2 2280 PCIe 3.0 x4 NVMe 1.3 3D-NAND TLC 134.70€
• Power: be quiet! Power Zone 750W 103.61€
• Case: be quiet! Pure Base 500 69.03€

This makes a total of 1884.16 Euros.

Case

The case was selected purely based on cost, but I ended up with a good looking case that even has a transparent side panel. The be quiet! Pure Base 500 is a pretty standard ATX case with 2 USB ports, headphone and microphone jack on the frontpanel. It's optimized for a quiet environment and hence perfect for my needs.

Power Supply

As a power supply I bought the be quiet! Power Zone 750W. It's a midrange power supply and probably a bit overkill for my build, but I wanted to have the option to upgrade to a dual graphics card setup down the road.

As with the case the be quiet! power supply comes with a really quiet fan that is barely noticeable even under massive load. The cable management with this power supply is super easy.

CPU

Strong, stable and ready to support all my needs. I realised I knew too little about the current CPU market and after deliberating with a friend I went with an AMD that just seemed right. 8 cores, 3.6Ghz which go up to 4.4Ghz on an AM4 socket.

Motherboard

I picked my motherboard after picking my CPU which needed a AM4 socket. The ASRock X470 Master SLI was the cheapest board available with support for AMD Matisse generation CPUs.

The only downside to this motherboard is that it was delivered with an old BIOS that didn't yet support Matisse. I had to buy a cheap AMD Athlon 200GE 2x 3.20GHz to perform a BIOS upgrade before finally installing my CPU.

RAM

RAM is one of those things where more is always better. Experiencing the slow down when applications start swapping memory to disc is excruciating and a small upfront investment can save me from it.

My motherboard supports up to 64 GB of RAM - for now I went with 32 GB to see how long it would last me.

The 2x16GB G.Skill Aegis DDR4-3200 have been more than sufficient and my motherboard automatically detected the memory frequency and adjusted its settings. I'm considering upgrading to 64 GB purely because .. "I can" not because there's any imminent need for me to upgrade.

Storage

My storage setup consists of 2 NVMe PCIe M.2  SSDs. That's quite the mouthful, but essentially instead of using old-school HDDs with spinning discs or SATA SSDs these SSDs simply sit on your mainboard allowing for much faster read/write than with those traditional options.

The SSDs are went with are:

• 256GB Transcend 110S M.2 2280 PCIe 3.0 x4 3D-NAND TLC (TS256GMTE110S)
• 1000GB PNY XLR8 CS3030 M.2 2280 PCIe 3.0 x4 NVMe 1.3 3D-NAND TLC (M280CS3030-1TB-RB)

I'm using the 256GB model for my operating system and the larger 1TB model for most of my file storage. If you looked closely enough there's also an old-school SATA SSD

Graphics Card

Writing this in 2021 where graphics cards are harder to find than gold I consider myself lucky to have gotten my 11GB GeForce RTX 2080 Ti last year.

While the 2080 Ti is now superseded by nVidia's new 30X0 generation I think this card will last me for years to come. It's powerful enough to play games like Cyberpunk 2077 on max settings and it sure as hell can run a lot of machine learning with its 4352 CUDA cores.

Benchmark

Benchmarks are silly. That said I ran the PassMark performance test to see how my computer would rank against "the world".

Personally I'm more than happy with the result. I wasn't trying to build the ultra ultimate extreme power machine, but instead a computer that would satisfy my needs and after using this computer for a good 6 months I can say that I succeeded.