ln - The Natural Log Function

One of the most essential things in software is a good interface for logging data to places. Logging is a surprisingly hard problem and there are many approaches to doing it. This time, we're going to talk about my favorite logging library in Go that uses my favorite function I've ever written in Go.

Today we're talking about ln, the natural log function. ln works with key value pairs and logs them to somewhere. By default it logs things to standard out. Here is how you use it:

package main

import (
  "context"

  "within.website/ln"
)

func main() {
  ctx := context.Background()
  ln.Log(ctx, ln.Fmt("hello %s", "world"), ln.F{"demo": "usage"})
}

ln works with key value pairs called F. This type allows you to log just about anything you want, including custom data types with an Fer. This will let you annotate your data types so that you can automatically extract the important information into your logs while automatically filtering out passwords or other secret data. Here's an example:

type User struct {
  ID       int
  Username string
  Password []byte
}

func (u User) F() ln.F {
	return ln.F{
		"user_id":       u.ID,
		"user_name": u.Username,
	}
}

Then if you create that user somehow, you can log the ID and username without logging the password on accident:

var theDude User = abides()

ln.Log(ctx, ln.Info("created new user"), theDude)

This will create a log line that looks something like this:

level=info msg="created new user" user_name="The Dude" user_id=1337

The way this is all glued together is that F itself is an Fer, meaning that the Log/Error functions take a variadic set of Fers. This is where my favorite Go function comes into play, it is the implementation of the Fer interface for F. Here is that function verbatim:

// F makes F an Fer
func (f F) F() F {
	return f
}

I love how this function looks like some kind of abstract art. This function holds this library together.

If you end up using ln for your projects in the future, please let me know what your experience is like. I would love to make this library the best it can possibly be. It is not a nanosecond scale zero allocation library (I think those kind of things are a bit of a waste of time, because most of the time your logging library is NOT going to be your bottleneck), but it is designed to have very usable defaults and solve the problem good enough that you shouldn't need to care. There are a few useful tools in the ex package nested in ln. The biggest thing is the HTTP middleware, which has saved me a lot of effort when writing web services in Go.

kalama pali pi kulupu Kala

I've wanted to write a novel for a while, and I think I've finally got a solid idea for it. I want to write about the good guys winning against an oppressive system. I've been letting the ideas and thoughts marinate in my heart for a long time; these short stories are how I am exploring the world and other related concepts. I want to use language as a tool in this world. So here is my take on a creation myth for the main species of this world, the Kala (the title of this post roughly translates to "creation story of the Kala").

This is day 2 of my 100 days to offload.

---

In the beginning, the gods roamed the skies. Pali, Sona and Soweli talked and talked about their plans.

tenpo wan la sewi li lon e sewi. sewi Pali en sewi Sona en sewi Soweli li toki.

Soweli went down to the world Pali had created. Animals of all kinds followed them as Soweli moved about the earth.

sewi Soweli li tawa e sike. soweli li kama e sike.

Sona followed and went towards the whales. Sona took a liking to how graceful they were in the water, and decided to have them be the arbiters of knowledge. Sona also reshaped them to look like the gods did. The Kala people resulted.

sewi Sona li tawa e soweli sike. sewi Sona li tawa e kala suli. sewi Sona li lukin li pona e kala suli. sewi Sona li pana e sona e kon tawa kala suli. sewi Sona li pali e jan kama kala suli. kulupu Kala li lon.

Pali had created the entire world, so Pali fell into a deep slumber in the ocean.

tenpo pini la sewi Pali li pali e sike. sewi Pali li lape lon telo suli.

Soweli had created all of the animals on the whole world, so Soweli fell asleep in Soweli mountain.

tenpo pini la sewi Soweli li pali e soweli ale. sewi Soweli li lape e nena Soweli.

Sona lifted themselves into the skies to watch the Kala from above. Sona keeps an eye on us to make sure we are using their gift responsibly.

sewi Sona li tawa e sewi. sewi Sona li lukin e kulupu Kala. kulupu Kala li jo sona li jo toki. kulupu Kala li pona e sewi Sona.

The Itch

I write a lot. I code a lot. This leads to people asking me questions like "how do you have the energy to do that?" or "why do you keep doing that day in and day out?". I was reading this post that I found linked in the Forbidden Orange Site's comments and it really resonated with me.

At the core, I have this deep burning sensation to try things out to see what they are like. It's like this itch deep in me that I can only scratch with writing, coding or sometimes even just answering people's questions in chatrooms. This itch is a catalyst to my productivity. It powers my daily work and makes me able to do what I do in order to make things better for everyone.

However, sometimes the itch isn't there. Sometimes it makes me want to focus on something else. Trying to do something else without the itch empowering me can feel like swimming upstream with heavy chains wrapped around me. My greatest boon is simultaneously my greatest vice.

I don't really know how to handle the days where it's not working. I try to save up my sick and vacation days so that I can avoid burning myself out on the bad days. Things like this are why I am a huge fan of unlimited vacation policies. Unlimited vacation does mean that I get paid out less money when I leave a job; however it means that I have the freedom to have bad days and let the good days tank me through the bad days so that I come out above average.

Trying to explain this to people can feel stressful. Especially to a manager. I've had some bad experiences with that in the past. Phrase this wrong, and some people will hear "I don't want to do this work ever" instead of "I can't do this work today". This especially sucks when deadlines roll in and that vital itch goes away, leaving me at half capacity at the worst possible time.

This itch leads me to set increasing standards on myself too. It's had some negative sides in that it makes me feel like I need to make everything better than the last thing. Each post better than the previous ones. Each project implementation better than the last. Onwards and onwards into a spiral that sets the bar so high I stress myself out trying to approach it.

I haven't kept to my informal goal to have at least one post per week on this blog because of that absurdly high standard I set for myself. I'm going to try and change this. I'm going to start participating in 100 days to offload. Expect some shorter and more focused posts for the immediate future. I am going to be working on the Rust series, however each part of it will be in isolation from here on out instead of the longer multifaceted posts.

This is day 1 of my 100 days to offload.

Also be sure to check out my post on Palisade, a version bumping tool for GitHub repositories.

How Mara Works

Recently I introduced Mara to this blog and I didn't explain much of the theory and implementation behind them in order to proceed with the rest of the post. There was actually a significant amount of engineering that went into implementing Mara and I'd like to go into detail about this as well as explain how I implemented them into this blog.

Mara's Background

Mara is an anthropomorphic shark. They are nonbinary and go by they/she pronouns. Mara enjoys hacking, swimming and is a Chaotic Good Rogue in the tabletop games I've played her in. Mara was originally made to help test my upcoming tabletop game The Source, and I have used them in a few solitaire tabletop sessions (click here to read the results of one of these).

The Theory

My blogposts have a habit of getting long, wordy and sometimes pretty damn dry. I notice that there are usually a few common threads in how this becomes the case, so I want to do these three things to help keep things engaging.

1. I go into detail. A lot of detail. This can make paragraphs long and wordy because there is legitimately a lot to cover. fasterthanlime's Cool Bear's Hot Tip is a good way to help Amos focus on the core and let another character bring up the finer details that may go off the core of the message.
2. I have been looking into how to integrate concepts from The Socratic method into my posts. The Socratic method focuses on dialogue/questions and answers between interlocutors as a way to explore a topic that can be dry or vague.
3. Soatok's blog was an inspiration to this. Soatok dives into deep technical topics that can feel like a slog, and inserts some stickers between paragraphs to help keep things upbeat and lively.

I wanted to make a unique way to help break up walls of text using the concepts of Cool Bear's Hot Tip and the Socratic method with some furry art sprinkled in and I eventually arrived at Mara.

How Mara is Implemented

I write my blogposts in Markdown, specifically a dialect that has some niceties from GitHub flavored markdown as parsed by comrak. Mara's interjections are actually specially formed links, such as this:

[Hi! I am saying something!](conversation://Mara/hacker)

Notice how the destination URL doesn't actually exist. It's actually intercepted in my markdown parsing function and then a HTML template is used to create the divs that make up the image and conversation bits. I have intentionally left this open so I can add more characters in the future. I may end up making some stickers for myself so I can reply to Mara a-la this blogpost by fasterthanlime (search for "What's with the @@GLIBC_2.2.5 suffixes?"). The syntax of the URL is as follows:

conversation://<character>/<mood>[?reply]

This will then fetch the images off of my CDN hosted by CloudFlare. However if you are using Tor to view my site, this may result in not being able to see the images. I am working on ways to solve this. Please bear with me, this stuff is hard.

You may have noticed that Mara sometimes has links inside her dialogue. Understandably, this is something that vanilla markdown does not support. However, I enabled putting raw HTML in my markdown which lets this work anyways! Consider this:

In the markdown source, that actually looks like this:

[My art was drawn by <a href="https://selic.re">Selicre</a>!](conversation://Mara/hacker)

This is honestly one of my favorite parts of how this is implemented, though others I have shown this to say it's kind of terrifying.

The <picture> Element and Image Formats

Something you might notice about the HTML template is that I use the <picture> element like this:

<picture>
    <source srcset="https://cdn.christine.website/file/christine-static/stickers/@character.to_lowercase()/@(mood).avif" type="image/avif">
    <source srcset="https://cdn.christine.website/file/christine-static/stickers/@character.to_lowercase()/@(mood).webp" type="image/webp">
    <img src="https://cdn.christine.website/file/christine-static/stickers/@character.to_lowercase()/@(mood).png" alt="@character is @mood">
</picture>

The <picture> element allows me to specify multiple versions of the stickers and have your browser pick the image format that it supports. It is also fully backwards compatible with browsers that do not support <picture> and in those cases you will see the fallback image in .png format. I went into a lot of detail about this in a twitter thread, but in short here are how each of the formats looks next to its filesize information:

The avif version does have the ugliest quality when blown up, however consider how small these stickers will appear on the webpages:

At these sizes most people will not notice any lingering artifacts unless they look closely. However at about 5-6 kilobytes per image I think the smaller filesize greatly wins out. This helps keep page loads fast, which is something I want to optimize for as it makes people think my website loads quickly.

I go into a lot more detail on the twitter thread, but the commands I use to get the webp and avif versions of the stickers are as follows:

#!/bin/sh

cwebp \
      $1.png \
      -o $1.webp
avifenc \
      $1.png \
      -o $1.avif \
      -s 0 \
      -d 8 \
      --min 48 \
      --max 48 \
      --minalpha 48 \
      --maxalpha 48

I plan to automate this further in the future, but for the scale I am at this works fine. These stickers are then uploaded to my cloud storage bucket and CloudFlare provides a CDN for them so they can load very quickly.

---

Anyways, this is how Mara is implemented and some of the challenges that went into developing them as a feature (while leaving the door open for other characters in the future). Mara is here to stay and I have gotten a lot of positive feedback about her.

As a side note, for those of you that are not amused that I am choosing to have Mara (and consequentially furry art in general) as a site feature, I can only hope that you can learn to respect that as an independent blogger I am free to implement my blog (and the content that I am choosing to provide FOR FREE even though I've gotten requests to make it paid content) as I see fit. Further complaints will only increase the amount of furry art in future posts.

Be well all.

Rust Crates that do What the Go Standard library Does

One of Go's greatest strengths is how batteries-included the standard library is. You can do most of what you need to do with only the standard library. On the other hand, Rust's standard library is severely lacking by comparison. However, the community has capitalized on this and been working on a bunch of batteries that you can include in your rust projects. I'm going to cover a bunch of them in this post in a few sections.

Logging

Go has logging out of the box with package log. Package log is a very uncontroversial logger. It does what it says it does and with little fuss. However it does not include a lot of niceties like logging levels and context-aware values.

In Rust, we have the log crate which is a very simple interface. It uses the error!, warn!, info!, debug! and trace! macros which correlate to the highest and lowest levels. If you want to use log in a Rust crate, you can add it to your Cargo.toml file like this:

[dependencies]
log = "0.4"

Then you can use it in your Rust code like this:

use log::{error, warn, info, debug, trace};

fn main() {
  trace!("starting main");
  debug!("debug message");
  info!("this is some information");
  warn!("oh no something bad is about to happen");
  error!("oh no it's an error");
}

This is because the log crate doesn't directly log anything anywhere, it is a facade that other packages build off of. pretty_env_logger is a commonly used crate with the log facade. Let's add it to the program and work from there:

[dependencies]
log = "0.4"
pretty_env_logger = "0.4"

Then let's enable it in our code:

use log::{error, warn, info, debug, trace};

fn main() {
  pretty_env_logger::init();

  trace!("starting main");
  debug!("debug message");
  info!("this is some information");
  warn!("oh no something bad is about to happen");
  error!("oh no it's an error");
}

And now let's run it with RUST_LOG=trace:

$ env RUST_LOG=trace cargo run --example logger_test
    Finished dev [unoptimized + debuginfo] target(s) in 0.07s
     Running `/home/cadey/code/christine.website/target/debug/logger_test`
 TRACE logger_test > starting main
 DEBUG logger_test > debug message
 INFO  logger_test > this is some information
 WARN  logger_test > oh no something bad is about to happen
 ERROR logger_test > oh no it's an error

There are many other consumers of the log crate and implementing a consumer is easy should you want to do more than pretty_env_logger can do on its own. However, I have found that pretty_env_logger does just enough on its own. See its documentation for more information.

Flags

Go's standard library has the flag package out of the box. This package is incredibly basic, but is surprisingly capable in terms of what you can actually do with it. A common thing to do is use flags for configuration or other options, such as here:

package main

import "flag"

var (
	program      = flag.String("p", "", "h program to compile/run")
	outFname     = flag.String("o", "", "if specified, write the webassembly binary created by -p here")
	watFname     = flag.String("o-wat", "", "if specified, write the uncompiled webassembly created by -p here")
	port         = flag.String("port", "", "HTTP port to listen on")
	writeTao     = flag.Bool("koan", false, "if true, print the h koan and then exit")
	writeVersion = flag.Bool("v", false, "if true, print the version of h and then exit")
)

This will make a few package-global variables that will contain the values of the command-line arguments.

In Rust, a commonly used command line parsing package is structopt. It works in a bit of a different way than Go's flag package does though. structopt focuses on loading options into a structure rather than into globally mutable variables.

Here's a quick example copied from pa'i:

#[derive(Debug, StructOpt)]
#[structopt(
    name = "pa'i",
    about = "A WebAssembly runtime in Rust meeting the Olin ABI."
)]
struct Opt {
    /// Backend
    #[structopt(short, long, default_value = "cranelift")]
    backend: String,


    /// Print syscalls on exit
    #[structopt(short, long)]
    function_log: bool,


    /// Do not cache compiled code?
    #[structopt(short, long)]
    no_cache: bool,


    /// Binary to run
    #[structopt()]
    fname: String,


    /// Main function
    #[structopt(short, long, default_value = "_start")]
    entrypoint: String,


    /// Arguments of the wasm child
    #[structopt()]
    args: Vec<String>,
}

This has the Rust compiler generate the needed argument parsing code for you, so you can just use the values as normal:

fn main() {
  let opt = Opt::from_args();
  debug!("args: {:?}", opt.args);
}

You can even handle subcommands with this, such as in palisade. This package should handle just about everything you'd do with the flag package, but will also work for cases where flag falls apart.

Errors

Go's standard library has the error interface which lets you create a type that describes why functions fail to do what they intend. Rust has the Error trait which lets you also create a type that describes why functions fail to do what they intend.

In my last post I described eyre and the Result type. However, this time we're going to dive into thiserror for making our own error type. Let's add thiserror to our crate:

[dependencies]
thiserror = "1"

And then let's re-implement our DivideByZero error from the last post:

use std::fmt;
use thiserror::Error;

#[derive(Debug, Error)]
struct DivideByZero;

impl fmt::Display for DivideByZero {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "cannot divide by zero")
    }
}

The compiler made our error instance for us! It can even do that for more complicated error types like this one that wraps a lot of other error cases and error types in maj:

#[derive(thiserror::Error, Debug)]
pub enum Error {
    #[error("TLS error: {0:?}")]
    TLS(#[from] TLSError),

    #[error("URL error: {0:?}")]
    URL(#[from] url::ParseError),

    #[error("Invalid DNS name: {0:?}")]
    InvalidDNSName(#[from] webpki::InvalidDNSNameError),

    #[error("IO error: {0:?}")]
    IO(#[from] std::io::Error),

    #[error("Response parsing error: {0:?}")]
    ResponseParse(#[from] crate::ResponseError),

    #[error("Invalid URL scheme {0:?}")]
    InvalidScheme(String),
}

Serialization / Deserialization

Go has JSON encoding/decoding in its standard library via package encoding/json. This allows you to define types that can be read from and write to JSON easily. Let's take this simple JSON object representing a comment from some imaginary API as an example:

{
  "id": 31337,
  "author": {
    "id": 420,
    "name": "Cadey"
  },
  "body": "hahaha its is an laughter image",
  "in_reply_to": 31335
}

In Go you could write this as:

type Author struct {
  ID   int    `json:"id"`
  Name string `json:"name"`
}

type Comment struct {
  ID        int    `json:"id"`
  Author    Author `json:"author"`
  Body      string `json:"body"`
  InReplyTo int    `json:"in_reply_to"`
}

Rust does not have this capability out of the box, however there is a fantastic framework available known as serde which works across JSON and every other serialization method that you can think of. Let's add serde and its JSON support to our crate:

[dependencies]
serde = { version = "1", features = ["derive"] }
serde_json = "1"

So, to use serde for our comment type, we would write Rust that looks like this:

use serde::{Deserialize, Serialize};

#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct Author {
  pub id: i32,
  pub name: String,
}

#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct Comment {
  pub id: i32,
  pub author: Author,
  pub body: String,
  pub in_reply_to: i32,
}

And then we can load that from JSON using code like this:

fn main() {
  let data = r#"
  {
    "id": 31337,
    "author": {
      "id": 420,
      "name": "Cadey"
    },
    "body": "hahaha its is an laughter image",
    "in_reply_to": 31335
  }
  "#;
  
  let c: Comment = serde_json::from_str(data).expect("json to parse");
  println!("comment: {:#?}", c);
}

And you can use it like this:

$ cargo run --example json
   Compiling xesite v2.0.1 (/home/cadey/code/christine.website)
    Finished dev [unoptimized + debuginfo] target(s) in 0.43s
     Running `target/debug/examples/json`
comment: Comment {
    id: 31337,
    author: Author {
        id: 420,
        name: "Cadey",
    },
    body: "hahaha its is an laughter image",
    in_reply_to: 31335,
}

HTTP

Many APIs expose their data over HTTP. Go has the net/http package that acts as a production-grade (Google uses this in production) HTTP client and server. This allows you to get going with new projects very easily. The Rust standard library doesn't have this out of the box, but there are some very convenient crates that can fill in the blanks.

Client

For an HTTP client, we can use reqwest. It can also seamlessly integrate with serde to allow you to parse JSON from HTTP without any issues. Let's add reqwest to our crate as well as tokio to act as an asynchronous runtime:

[dependencies]
reqwest = { version = "0.10", features = ["json"] }
tokio = { version = "0.2", features = ["full"] }

And then let's integrate with that imaginary comment api at https://xena.greedo.xeserv.us/files/comment.json:

use eyre::Result;
use serde::{Deserialize, Serialize};

#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct Author {
    pub id: i32,
    pub name: String,
}

#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct Comment {
    pub id: i32,
    pub author: Author,
    pub body: String,
    pub in_reply_to: i32,
}

#[tokio::main]
async fn main() -> Result<()> {
  let c: Comment = reqwest::get("https://xena.greedo.xeserv.us/files/comment.json")
      .await?
      .json()
      .await?;
  println!("comment: {:#?}", c);
  
  Ok(())
}

And then let's run this:

$ cargo run --example http
   Compiling xesite v2.0.1 (/home/cadey/code/christine.website)
    Finished dev [unoptimized + debuginfo] target(s) in 2.20s
     Running `target/debug/examples/http`
comment: Comment {
    id: 31337,
    author: Author {
        id: 420,
        name: "Cadey",
    },
    body: "hahaha its is an laughter image",
    in_reply_to: 31335,
}

We can change the code to something like this:

let c: Comment = reqwest::get("https://xena.greedo.xeserv.us/files/comment2.json")
    .await?
    .error_for_status()?
    .json()
    .await?;

And then when we run it we get an error back:

$ cargo run --example http_fail
   Compiling xesite v2.0.1 (/home/cadey/code/christine.website)
    Finished dev [unoptimized + debuginfo] target(s) in 1.84s
     Running `/home/cadey/code/christine.website/target/debug/examples/http_fail`
Error: HTTP status client error (404 Not Found) for url (https://xena.greedo.xeserv.us/files/comment2.json)

This combined with the other features in reqwest give you an very capable HTTP client that does even more than Go's HTTP client does out of the box.

Server

As for HTTP servers though, let's take a look at warp. warp is a HTTP server framework that builds on top of Rust's type system. You can add warp to your dependencies like this:

[dependencies]
warp = "0.2"

Let's take a look at its "Hello, World" example:

use warp::Filter;

#[tokio::main]
async fn main() {
    // GET /hello/warp => 200 OK with body "Hello, warp!"
    let hello = warp::path!("hello" / String)
        .map(|name| format!("Hello, {}!", name));

    warp::serve(hello)
        .run(([127, 0, 0, 1], 3030))
        .await;
}

We can then build up multiple routes with its or pattern:

let hello = warp::path!("hello" / String)
    .map(|name| format!("Hello, {}!", name));
let health = warp::path!(".within" / "health")
    .map(|| "OK");
let routes = hello.or(health);

And even inject other datatypes into your handlers with filters such as in the printer facts API server:

let fact = {
    let facts = pfacts::make();
    warp::any().map(move || facts.clone())
};

let fact_handler = warp::get()
    .and(warp::path("fact"))
    .and(fact.clone())
    .and_then(give_fact);

warp is an extremely capable HTTP server and can work across everything you need for production-grade web apps.

Templating

Go's standard library also includes HTML and plain text templating with its packages html/template and text/template. There are many solutions for templating HTML in Rust, but the one I like the most is ructe. ructe uses Cargo's build.rs feature to generate Rust code for its templates at compile time. This allows your HTML templates to be compiled into the resulting application binary, allowing them to render at ludicrous speeds. To use it, you need to add it to your build-dependencies section of your Cargo.toml:

[build-dependencies]
ructe = { version = "0.12", features = ["warp02"] }

You will also need to add the mime crate to your dependencies because the generated template code will require it at runtime.

[dependencies]
mime = "0.3.0"

Once you've done this, create a new folder named templates in your current working directory. Create a file called hello.rs.html and put the following in it:

@(title: String, message: String)

<html>
  <head>
    <title>@title</title>
  </head>
  <body>
    <h1>@title</h1>
    <p>@message</p>
  </body>
</html>

Now add the following to the bottom of your main.rs file:

include!(concat!(env!("OUT_DIR"), "/templates.rs"));

And then use the template like this:

use warp::{http::Response, Filter, Rejection, Reply};

async fn hello_html(message: String) -> Result<impl Reply, Rejection> {
    Response::builder()
        .html(|o| templates::index_html(o, "Hello".to_string(), message).unwrap().clone()))
}

And hook it up in your main function:

let hello_html_rt = warp::path!("hello" / "html" / String)
    .and_then(hello_html);
    
let routes = hello_html_rt.or(health).or(hello);

For a more comprehensive example, check out the printerfacts server. It also shows how to handle 404 responses and other things like that.

---

Wow, this covered a lot. I've included most of the example code in the examples folder of this site's GitHub repo. I hope it will help you on your journey in Rust. This is documentation that I wish I had when I was learning Rust.

TL;DR Rust

Recently I've been starting to use Rust more and more for larger and larger projects. As things have come up, I realized that I am missing a good reference for common things in Rust as compared to Go. This post contains a quick high-level overview of patterns in Rust and how they compare to patterns in Go. This will focus on code samples. This is no replacement for the Rust book, but should help you get spun up on the various patterns used in Rust code.

Also I'm happy to introduce Mara to the blog!

Let's start somewhere simple: functions.

Making Functions

Functions are defined using fn instead of func:

func foo() {}

fn foo() {}

Arguments

Arguments can be passed by separating the name from the type with a colon:

func foo(bar int) {}

fn foo(bar: i32) {}

Returns

Values can be returned by adding -> Type to the function declaration:

func foo() int {
  return 2
}

fn foo() -> i32 {
  return 2;
}

In Rust values can also be returned on the last statement without the return keyword or a terminating semicolon:

fn foo() -> i32 {
  2
}

fn foo() -> i32 {
    if some_cond {
        2
    }
    
    4
}

Let's find out! The compiler spits back an error:

error[E0308]: mismatched types
 --> src/lib.rs:3:9
  |
2 | /     if some_cond {
3 | |         2
  | |         ^ expected `()`, found integer
4 | |     }
  | |     -- help: consider using a semicolon here
  | |_____|
  |       expected this to be `()`

This happens because most basic statements in Rust can return values. The best way to fix this would be to move the 4 return into an else block:

fn foo() -> i32 {
    if some_cond {
        2
    } else {
        4
    }
}

Otherwise, the compiler will think you are trying to use that if as a statement, such as like this:

let val = if some_cond { 2 } else { 4 };

Functions that can fail

The Result type represents things that can fail with specific errors. The eyre Result type represents things that can fail with any error. For readability, this post will use the eyre Result type.

import "errors"

func divide(x, y int) (int, err) {
  if y == 0 {
    return 0, errors.New("cannot divide by zero")
  }
  
  return x / y, nil
}

use eyre::{eyre, Result};

fn divide(x: i32, y: i32) -> Result<i32> {
  match y {
    0 => Err(eyre!("cannot divide by zero")),
    _ => Ok(x / y),
  }
}

Let's try that, however we will need to make our own error type because the eyre! macro creates its own transient error type on the fly.

First we need to make our own simple error type for a DivideByZero error:

use std::error::Error;
use std::fmt;

#[derive(Debug)]
struct DivideByZero;

impl fmt::Display for DivideByZero {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "cannot divide by zero")
    }
}

impl Error for DivideByZero {}

So now let's use it:

fn divide(x: i32, y: i32) -> Result<i32, DivideByZero> {
  match y {
    0 => Err(DivideByZero{}),
    _ => Ok(x / y),
  }
}

However there is still one thing left: the function returns a DivideByZero error, not any error like the error interface in Go. In order to represent that we need to return something that implements the Error trait:

fn divide(x: i32, y: i32) -> Result<i32, impl Error> {
    // ...
}

And for the simple case, this will work. However as things get more complicated this simple facade will not work due to reality and its complexities. This is why I am shipping as much as I can out to other packages like eyre or anyhow. Check out this code in the Rust Playground to mess with this code interactively.

The ? Operator

In Rust, the ? operator checks for an error in a function call and if there is one, it automatically returns the error and gives you the result of the function if there was no error. This only works in functions that return either an Option or a Result.

func doThing() (int, error) {
  result, err := divide(3, 4)
  if err != nil {
    return 0, err
  }
  
  return result, nil
}

use eyre::Result;

fn do_thing() -> Result<i32> {
  let result = divide(3, 4)?;
  Ok(result)
}

If the second argument of divide is changed to 0, then do_thing will return an error.

It works with eyre because eyre has its own error wrapper type called Report, which can represent anything that implements the Error trait.

Macros

Rust macros are function calls with ! after their name:

println!("hello, world");

Variables

Variables are created using let:

var foo int
var foo = 3
foo := 3

let foo: i32;
let foo = 3;

Mutability

In Rust, every variable is immutable (unchangeable) by default. If we try to change those variables above we get a compiler error:

fn main() {
    let foo: i32;
    let foo = 3;
    foo = 4;
}

This makes the compiler return this error:

error[E0384]: cannot assign twice to immutable variable `foo`
 --> src/main.rs:4:5
  |
3 |     let foo = 3;
  |         ---
  |         |
  |         first assignment to `foo`
  |         help: make this binding mutable: `mut foo`
4 |     foo = 4;
  |     ^^^^^^^ cannot assign twice to immutable variable

As the compiler suggests, you can create a mutable variable by adding the mut keyword after the let keyword. There is no analog to this in Go.

let mut foo: i32 = 0;
foo = 4;

Lifetimes

Rust does garbage collection at compile time. It also passes ownership of memory to functions as soon as possible. Lifetimes are how Rust calculates how "long" a given bit of data should exist in the program. Rust will then tell the compiled code to destroy the data from memory as soon as possible.

For example, this code will fail to compile because quo was moved into the second divide call:

let quo = divide(4, 8)?;
let other_quo = divide(quo, 5)?;

// Fails compile because ownership of quo was given to divide to create other_quo
let yet_another_quo = divide(quo, 4)?;

To work around this you can pass a reference to the divide function:

let other_quo = divide(&quo, 5);
let yet_another_quo = divide(&quo, 4)?;

Or even create a clone of it:

let other_quo = divide(quo.clone(), 5);
let yet_another_quo = divide(quo, 4)?;

Passing Mutability

Sometimes functions need mutable variables. To pass a mutable reference, add &mut before the name of the variable:

let something = do_something_to_quo(&mut quo)?;

Project Setup

Imports

External dependencies are declared using the Cargo.toml file:

# Cargo.toml

[dependencies]
eyre = "0.6"

This depends on the crate eyre at version 0.6.x.

Dependencies can also have optional features:

# Cargo.toml

[dependencies]
reqwest = { version = "0.10", features = ["json"] }

This depends on the crate reqwest at version 0.10.x with the json feature enabled (in this case it enables reqwest being able to automagically convert things to/from json using Serde).

External dependencies can be used with the use statement:

// go

import "github.com/foo/bar"

use foo; //      -> foo now has the members of crate foo behind the :: operator
use foo::Bar; // -> Bar is now exposed as a type in this file

use eyre::{eyre, Result}; // exposes the eyre! and Result members of eyre

Async/Await

Async functions may be interrupted to let other things execute as needed. This program uses tokio to handle async tasks. To run an async task and wait for its result, do this:

let printer_fact = reqwest::get("https://printerfacts.cetacean.club/fact")
  .await?
  .text()
  .await?;
println!("your printer fact is: {}", printer_fact);

This will populate response with an amusing fact about everyone's favorite household pet, the printer.

To make an async function, add the async keyword before the fn keyword:

async fn get_text(url: String) -> Result<String> {
  reqwest::get(&url)
    .await?
    .text()
    .await?
}

This can then be called like this:

let printer_fact = get_text("https://printerfacts.cetacean.club/fact").await?;

Public/Private Types and Functions

Rust has three privacy levels for functions:

• Only visible to the current file (no keyword, lowercase in Go)
• Visible to anything in the current crate (pub(crate), internal packages in go)
• Visible to everyone (pub, upper case in Go)

Structures

Rust structures are created using the struct keyword:

type Client struct {
  Token string
}

pub struct Client {
  pub token: String,
}

If the pub keyword is not specified before a member name, it will not be usable outside the Rust source code file it is defined in:

type Client struct {
  token string
}

pub(crate) struct Client {
  token: String,
}

Encoding structs to JSON

serde is used to convert structures to json. The Rust compiler's derive feature is used to automatically implement the conversion logic.

type Response struct {
  Name        string  `json:"name"`
  Description *string `json:"description,omitempty"`
}

use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize, Debug)]
pub(crate) struct Response {
  pub name: String,
  pub description: Option<String>,
}

Strings

Rust has a few string types that do different things. You can read more about this here, but at a high level most projects only uses a few of them:

• &str, a slice reference to a String owned by someone else
• String, an owned UTF-8 string
• PathBuf, a filepath string (encoded in whatever encoding the OS running this code uses for filesystems)

The strings are different types for safety reasons. See the linked blogpost for more detail about this.

Enumerations / Tagged Unions

Enumerations, also known as tagged unions, are a way to specify a superposition of one of a few different kinds of values in one type. A neat way to show them off (along with some other fancy features like the derivation system) is with the structopt crate. There is no easy analog for this in Go.

#[derive(StructOpt, Debug)]
#[structopt(about = "A simple release management tool")]
pub(crate) enum Cmd {
    /// Creates a new release for a git repo
    Cut {
        #[structopt(flatten)]
        common: Common,
        /// Changelog location
        #[structopt(long, short, default_value="./CHANGELOG.md")]
        changelog: PathBuf,
    },

    /// Runs releases as triggered by GitHub Actions
    GitHubAction {
        #[structopt(flatten)]
        gha: GitHubAction,
    },
}

Enum variants can be matched using the match keyword:

match cmd {
    Cmd::Cut { common, changelog } => {
        cmd::cut::run(common, changelog).await
    }
    Cmd::GitHubAction { gha } => {
        cmd::github_action::run(gha).await
    }
}

All variants of an enum must be matched in order for the code to compile.

Testing

Test functions need to be marked with the #[test] annotation, then they will be run alongside cargo test:

mod tests { // not required but it is good practice
  #[test]
  fn math_works() {
    assert_eq!(2 + 2, 4);
  }
  
  #[tokio::test] // needs tokio as a dependency
  async fn http_works() {
    let _ = get_html("https://within.website").await.unwrap();
  }
}

Avoid the use of unwrap() outside of tests. In the wrong cases, using unwrap() in production code can cause the server to crash and can incur data loss.

---

This is by no means comprehensive, see the rust book or Learn X in Y Minutes Where X = Rust for more information. This code is written to be as boring and obvious as possible. If things don't make sense, please reach out and don't be afraid to ask questions.

My Org Mode Flow

At almost every job I've worked at, at least one of my coworkers has noticed that I use Emacs as my main text editor. People have pointed me at IntelliJ, VS Code, Atom and more, but I keep sticking to Emacs because it has one huge ace up its sleeve that other editors simply cannot match. Emacs has a package that helps me organize my workflow, focus my note-taking and even keep a timeclock for how long I spend working on tasks. This package is called Org mode, and this is my flow for using it.

Org mode is a TODO list manager, document authoring platform and more for GNU Emacs. It uses specially formatted plain text that can be managed using version control systems. I have used it daily for about five years for keeping track of what I need to do for work. Please note that my usage of it barely scratches the surface of what Org mode can do, because this is all I have needed.

~/org

My org flow starts with a single folder: ~/org. The main file I use is todo.org and it looks something like this:

#+TITLE: TODO

* Doing
** TODO WAT-42069 Unfrobnicate the rilkef for flopnax-ropjar push...
* In Review
** TODO WAT-42042 New Relic Dashboards...
* Reviews
** DONE HAX-1337 Security architecture of wasmcloud
* Interrupt
* Generic todo
* Overhead
** 09/08/2020
*** DONE workday start...
*** DONE standup...

Each level of stars creates a new heading level, and these headings can be treated like a tree. You can use the tab key to open and close the heading levels and hide those parts of the tree if they are not relevant. Let's open up the standup subtree with tab:

*** DONE standup
    CLOSED: [2020-09-08 Tue 10:12]
    :LOGBOOK:
    CLOCK: [2020-09-08 Tue 10:00]--[2020-09-08 Tue 10:12] =>  0:12
    :END:

Org mode automatically entered in nearly all of the information in this subtree for me. I clocked in (alt-x org-clock-in with that TODO item highighted) when the standup started and I clocked out by marking the task as done (alt-x org-todo with that TODO item highlighted). If I am working on a task that takes longer than one session, I can clock out of it (alt-x org-clock-out) and then the time I spent (about 20 minutes) will be recorded in the file for me. Then I can manually enter the time spent into tools like Jira.

When I am ready to move a task from In Progress to In Review, I close the subtree with tab and then highlight the collapsed subtree, cut it and paste it under the In Review header. This will keep the time tracking information associated with that header entry.

I will tend to let tasks build up over the week and then on Monday morning I will move all of the done tasks to done.org, which is where I store things that are done. As I move things over, I double check with Jira to make sure the time tracking has been accurately updated. This can take a while, but doing this has caught cases where I have misreported time and then had the opportunity to correct it.

Clocktables

Org mode is also able to generate tables based on information in org files. One of the most useful ones is the clock table. You can use these clock tables to make reports about how much time was spent in each task. I use these to help me know what I have done in the day so I can report about it in the next day's standup meeting. To add a clock table, add an empty block for it and press control-c c on the BEGIN line. Here's an example:

#+BEGIN: clocktable :block today
#+END:

This will show you all of the things you have recorded for that day. This may end up being a bit much if you nest things deep enough. My preferred clock table is a daily view only showing the second level and lower for the current file:

#+BEGIN: clocktable :maxlevel 2 :block today :scope file
#+CAPTION: Clock summary at [2020-09-08 Tue 15:47], for Tuesday, September 08, 2020.
| Headline                    |   Time |      |
|-----------------------------|--------|------|
| *Total time*                | *6:14* |      |
|-----------------------------|--------|------|
| In Progress                 |   2:09 |      |
| \_  WAT-42069 Unfrobnica... |        | 2:09 |
| Overhead                    |   4:05 |      |
| \_  09/08/2020              |        | 4:05 |
#+END:

This allows me to see that I've been working today for about 6.25 hours for the day, so I can use that information when deciding what to do next.

Other Things You Can Do

In the past I used to use org mode for a lot of things. In one of my older files I have a comprehensive list of all of the times I smoked weed down to the amount smoked and what I felt about it at the time. In another I have a script that I used for applying ansible files across a cluster. The sky really is the limit.

However, I have really decided to keep things simple for the most part. I leave org mode for work stuff and mostly use iCloud services for personal stuff. There are mobile apps for using org-mode on the go, but they haven't aged well at all and I have been focusing my time into actually doing things instead of configuring WEBDAV servers or the like.

This is how I keep track of things at work.

The Within Go Repo Layout

Go repository layout is a very different thing compared to other languages. There's a lot of conflicting opinions and little firm guidance to help steer people along a path to more maintainable code. This is a collection of guidelines that help to facilitate understandable and idiomatic Go.

At a high level the following principles should be followed:

• If the code is designed to be consumed by other random people using that repository, it is made available for others to import
• If the code is NOT designed to be consumed by other random people using that repository, it is NOT made available for others to import
• Code should be as close to where it's used as possible
• Documentation helps understand why, not how
• More people can reuse your code than you think

Folder Structure

At a minimum, the following folders should be present in the repository:

• cmd/ -> houses executable commands
• docs/ -> houses human readable documentation
• internal/ -> houses code not intended to be used by others
• scripts/ -> houses any scripts needed for meta-operations

Any additional code can be placed anywhere in the repo as long as it makes sense. More on this later in the document.

Additional Code

If there is code that should be available for other people outside of this project to use, it is better to make it a publicly available (not internal) package. If the code is also used across multiple parts of your program or is only intended for outside use, it should be in the repository root. If not, it should be as close to where it is used as makes sense. Consider this directory layout:

repo-root
├── cmd
│   ├── paperwork
│   │   ├── create
│   │   │   └── create.go
│   │   └── main.go
│   ├── hospital
│   │   ├── internal
│   │   │   └── operate.go
│   │   └── main.go
│   └── integrator
│       ├── integrate.go
│       └── main.go
├── internal
│   └── log_manipulate.go
└── web
    ├── error.go
    └── instrument.go

This would expose packages repo-root/web and repo-root/cmd/paperwork/create to be consumed by outside users. This would allow reuse of the error handling in package web, but it would not allow reuse of whatever manipulation is done to logging in package repo-root/internal.

repo-root/cmd/

This folder has subfolders with go files in them. Each of these subfolders is one command binary. The entrypoint of each command should be main.go so that it is easy to identify in a directory listing. This follows how the go standard library does this.

For example:

repo-root
└── cmd
    ├── paperwork
    │   └── main.go
    ├── hospital
    │   └── main.go
    └── integrator
        └── main.go

This would be for three commands named paperwork, hospital, and integrate respectively.

As your commands get more complicated, it's tempting to create packages in repo-root/internal/ to implement them. This is probably a bad idea. It's better to create the packages in the same folder as the command, or optionally in its internal package. Consider if paperwork has a command named create, hospital has a command named operate and integrator has a command named integrate:

repo-root
└── cmd
    ├── paperwork
    │   ├── create
    │   │   └── create.go
    │   └── main.go
    ├── hospital
    │   ├── internal
    │   │   └── operate.go
    │   └── main.go
    └── integrator
        ├── integrate.go
        └── main.go

Each of these commands has the logic separated into different packages.

paperwork has the create command as a subpackage, meaning that other parts of the application can consume that code if they need to.

hospital has the operate command inside its internal package, meaning only cmd/foo/ and anything that has the same import path prefix can use that code. This makes it easier to isolate the code so that other parts of the repo cannot use it.

integrator has the integrate command as a separate go file in the main package of the command. This makes the integrate command code only usable within the command because main packages cannot be imported by other packages.

Each of these methods makes sense in some contexts and not in others. Real-world usage will probably see a mix of these depending on what makes sense.

repo-root/docs/

This folder has human-readable documentation files. These files are intended to help humans understand how to use the program or reasons why the program was put together the way it was. This documentation should be in the language most common to the team of people developing the software.

The structure inside this folder is going to be very organic, so it is not entirely defined here.

repo-root/internal/

The internal folder should house code that others shouldn't consume. This can be for many reasons. Generally if you cannot see a use for this code outside the context of the program you are developing, but it needs to be used across multiple packages in different areas of the repo, it should default to going here.

If the code is safe for public consumption, it should go elsewhere.

repo-root/scripts/

The scripts folder should contain each script that is needed for various operations. This could be for running fully automated tests in a docker container or packaging the program for distribution. These files should be documented as makes sense.

Test Code

Code should be tested in the same folder that it's written in. See the upstream testing documentation for more information.

Integration tests or other things should be done in an internal subpackage called "integration" or similar.f

Questions and Answers

Why not use pkg/ for packages you intend others to use?

The name pkg is already well-known in the Go ecosystem. It is the folder that compiled packages (not command binaries) go. Using it creates the potential for confusion between code that others are encouraged to use and the meaning that the Go compiler toolchain has.

If a package prefix for publicly available code is really needed, choose a name not already known to the Go compiler toolchain such as "public".

How does this differ from https://github.com/golang-standards/project-layout?

This differs in a few key ways:

• Discourages the use of pkg, because it's obvious if something is publicly available or not if it can be imported outside of the package
• Leaves the development team a lot more agency to decide how to name things

The core philosophy of this layout is that the developers should be able to decide how to put files into the repository.

But I really think I need pkg!

Set up another git repo for those libraries then. If they are so important that other people need to use them, they should probably be in a libraries repo or individual git repos.

Besides, nothing is stopping you from actually using pkg if you want to. Some more experienced go programmers will protest though.

Examples of This in Action

Here are a few examples of views of this layout in action:

• https://github.com/golang/go/tree/master/src
• https://github.com/golang/tools
• https://github.com/PonyvilleFM/aura
• https://github.com/Xe/ln
• https://github.com/goproxyio/goproxy
• https://github.com/heroku/x

Colemak Layout - First Week

A week ago I posted the last post in this series where I announced I was going all colemak all the time. I have not been measuring words per minute (to avoid psyching myself out), but so far my typing speed has gone from intolerably slow to manageably slow. I have been only dipping back into qwerty for two main things:

1. Passwords, specifically the ones I have in muscle memory
2. Coding at work that needs to be done fast

Other than that, everything else has been in colemak. I have written DnD-style game notes, hacked at my own "Linux distro", started a few QMK keymaps and more all via colemak.

Here are some of the lessons I've learned:

Let Your Coworkers Know You Are Going to Be Slow

This kind of thing is a long tirm investment. In the short term, your productivity is going to crash through the floor. This will feel frustrating. It took me an entire workday to implement and test a HTTP handler/client for it in Go. You will be making weird typos. Let your coworkers know so they don't jump to the wrong conclusions too quickly.

Also, this goes without saying, but don't do this kind of change during crunch time. That's a bit of a dick move.

Print Out the Layout

I have the layout printed and taped to my monitor and iPad stand. This helps a lot. Instead of looking at the keyboard, I look at the layout image and let my fingers drift into position.

I also have a blank keyboard at my desk, this helps because I can't look at the keycaps and become confused (however this has backfired with typing numbers, lol). This keyboard has cherry MX blues though, which means it can be loud when I get to typing up a storm.

Have Friends Ask You What Layout You Are Using

Something that works for me is to have friends ask me what keyboard layout I am using, so I can be mindful of the change. I have a few people asking me that on the regular, so I can be accountable to them and myself.

macOS and iPadOS have Colemak Out of the Box

The settings app lets you configure colemak input without having to jailbreak or install a custom keyboard layout. Take advantage of this.

Someone has also created a colemak windows package for windows that includes an IA-64 (Itanium) binary. It was last updated in 2004, and still works without hassle on windows 10. It was the irst time I've ever seen an IA-64 windows binary in the wild!

Relearn How To Type Your Passwords

I type passwords from muscle memory. I have had to rediscover what they actually are so I can relearn how to type them.

---

The colemak experiment continues. I also have a ZSA Moonlander and the kit for a GergoPlex coming in the mail. Both of these run QMK, which allows me to fully program them with a rich macro engine. Here are a few of the macros I plan to use:

// Programming
SUBS(ifErr,     "if err != nil {\n\t\n}", KC_E, KC_I)
SUBS(goTest,    "go test ./...\n",        KC_G, KC_T)
SUBS(cargoTest, "cargo test\n",           KC_C, KC_T)

This will autotype a few common things when I press the keys "ei", "gt", or "ct" at the same time. I plan to add a few more as things turn up so I can more quickly type common idioms or commands to save me time. The if err != nil combination started as a joke, but I bet it will end up being incredibly valuable.

Be well, take care of your hands.

Colemak Layout - Beginning

I write a lot. On average I write a few kilobytes of text per day. This has been adding up and is taking a huge toll on my hands, especially considering the Covid situation. Something needs to change. I've been working on learning a new keyboard layout: Colemak.

This post will be shorter than most of my posts because I'm writing it with Colemak enabled on my iPad. Writing this is painfully slow at the moment. My sentences are short and choppy because those are easier to type.

I also have a ZSA Moonlander on the way, it should be here in October or November. I will also be sure to write about that once I get it in the mail.

So far, I have about 30 words per minute on the homerow, but once I go off the homerow the speed tanks to less than about five.

However, I am making progress!

Be well all, don't stress your hands out.

The Fear Of Missing Out

Humans have evolved over thousands of years with communities that are small, tight-knit and where it is easy to feel like you know everyone in them. The Internet changes this completely. With the Internet, it's easy to send messages, write articles and even publish books that untold thousands of people can read and interact with. This has lead to an instinctive fear in humanity I'm going to call the Fear of Missing Out [1].

[1]: The Fear of Missing Out

The Internet in its current form capitalizes and makes billions off of this. Infinite scrolling and live updating pages that make it feel like there's always something new to read. Uncountable hours of engineering and psychological testing spent making sure people click and scroll and click and consume all day until that little hit of dopamine becomes its own addiction. We have taken a system for displaying documents and accidentally turned it into a hulking abomination that consumes the souls of all who get trapped in it, crystallizing them in an endless cycle of checking notifications, looking for new posts on your newsfeed, scrolling down to find just that something you think you're looking for.

When I was in high school, I bagged groceries for a store. I also had the opportunity to help customers out to their cars and was able to talk with them. Obviously, I was minimum wage and had a whole bunch of other things to do; however there were a few times that I could really get to talk with regular customers and feel like I got to know them. What comes to mind however is a story where that is not the case. One day I was helping this older woman to her car, and she eventually said something like "All of these people just keep going, going, going nonstop. It drives me mad. How can't they see where they are is good enough already?" I thought for a moment and I wasn't able to come up with a decent reply.

The infinite scrollbars and newsfeeds of the web just keep going, going, going, going, going, going, going and going until the user gives up to do something elses. There's no consideration of how the content is discovered, and why the content is discovered, it's just an endless feed of noise. One subtle change in your worldview after another, just from the headlines alone. Not to mention the endless torrent of advertising.

However, I think there may be a way out, a kind of detox from the infinite scrolling, newsfeeds, notifications and the like for the internet, and I think a good step towards that is the Gemini [2] protocol.

[2]: Gemini Protocol

Gemini is a protocol that is somewhere between HTTP and Gopher. A user sends a request to a Gemini server and the user gets a response back. This response could be anything, but a little header tells the client what kind of data it is. There's also a little markup format that's a very lightweight take on markdown [3], but overall the entire goal of the project is to be minimal and just serve documents.

[3]: Gemtext markup

I've noticed something as I browse through the known constellation of Gemini capsules though. I keep refreshing the CAPCOM feed of posts. I keep refreshing the mailing list archives. I keep refreshing my email client, looking for new content and feel frustrated when it doesn't show up like I expect it to. I'm addicted to the newsfeeds. I'm caught in the trap that autoplay put me in. I'm a victim to infinite scrolling and that constant little hit of dopamine that modern social media has put on us all. Realizing this feels like I am realizing an addiction to a drug (but I'd argue that it somewhat is a drug, by design, what better way to get people to be exposed to ads than to make the service that serves the ads addictive!).

I'm not sure how to best combat this. It feels kind of scary. I'm starting to attempt to detox though. I'm writing a lot more on my Gemini capsule [4] [5]. I'm starting to really consider the Fear of Missing Out when I design and implement things in the future. So many things update instantly on the modern internet, it may be a good idea to attempt to make something that updates weekly or even monthly.

[4]: My Gemini capsule
[5]: [experimental] My Gemini capsule over HTTP

I'm still going to attempt a few ideas that I have regarding long term archival of the Gemini constellation, but I'm definitely going to make sure that I take the time to actually consider the consequences of my actions and what kind of world it creates. I want to create the kind of world that enables people to better themselves.

Let's work together to detox from the harmful effects of what we all have created. I'm considering opening up a Gemini server that other people can have accounts on and write about things that interest them.

If you want to get started with Gemini, I suggest taking a look at the main site through the Gemini to HTTP proxy [6]. There are some clients listed in the pages there, including a very good iOS client that is currently in TestFlight. Please do keep in mind that Gemini is very much a back-button navigation kind of experience. The web has made people expect navigation links to be everywhere, which can make it a weird/jarring experience at first, but you get used to it. You can see evidence of this in my site with all the "Go back" links on each page. I'll remove those at some point, but for now I'm going to keep them.

[6]: Project Gemini

Don't be afraid of missing out. It's inevitable. Things happen. It's okay for them to happen without you having to see them. They will still be there when you look again.

Book Release: Musings from Within

I am happy to announce that I have successfully created an eBook compilation of the best of the posts on this blog plus a bunch of writing I have never before made public, and the result is now available for purchase on itch.io and the Kindle Store (TODO(Xe): add kindle link here when it gets approved) for USD$5. This book is the product of 5 years of effort writing, getting better at writing, failing at writing and everything inbetween.

I have collected the following essays, poems, recipes and stories:

• Against Label Permanence
• A Letter to Those Who Bullied Me
• All There is is Now
• Alone
• Barrier
• Bricks
• Chaos Magick Debugging
• Chicken Stir Fry
• Creator’s Mission
• Death
• Died to Save Me
• Don't Look Into the Light
• Every Koan Ever
• Final Chapter
• Gratitude
• h
• How HTTP Requests Work
• Humanity
• I Love
• Instant Pot Quinoa Taco Bowls
• Instant Pot Spaghetti
• I Put Words on this Webpage so You Have to Listen to Me Now
• I Remember
• It Is Free
• Listen to Your Rubber Duck
• MrBeast is Postmodern Gold
• My Experience Cursing Out God
• Narrative of Sickness
• One Day
• Plurality-Driven Development
• Practical Kasmakfa
• Questions
• Second Go Around
• Self
• Sorting Time
• Tarot for Hackers
• The Gears and The Gods
• The Origin of h
• The Service is Already Down
• The Story of Hol
• The Sumerian Creation Myth
• Toast Sandwich Recipe
• Untitled Cyberpunk Furry Story
• We Exist
• What It’s Like to Be Me
• When Then Zen
• When Then Zen: Anapana
• When Then Zen: Wonderland Immersion
• You Are Fine

Most of these are available on this site, but a good portion of them are not available anywhere else. There's poetry about shamanism, stories about reincarnation, koans and more.

I am also uploading eBook files to my Patreon page, anyone who supports me for $1 or more has immediate access to the DRM-free ePub, MOBIPocket and PDF files of this book.

If you are facing financial difficulties, want to read my book and just simply cannot afford it, please contact me and I will send you my book free of charge.

Feedback and reviews of this book are more than welcome. If you decide to tweet or toot about it, please use the hashtag #musingsfromwithin so I can collect them into future updates to the description of the store pages, as well as assemble them below.

Enjoy the book! My hope is that you get as much from it as I've gotten from writing these things for the last 5 or so years. Here's to five more. I'll likely create another anthology/collection of them at that point.

RSS/Atom Feeds Fixed and Announcing my Flight Journal

I have released version 2.0.1 of this site's code. With it I have fixed the RSS and Atom feed generation. For now I have had to sacrifice the post content being in the feed, but I will bring it back as soon as possible.

Victory badges:

Thanks to W3Schools for having a minimal example of an RSS feed and this Flickr image for expanding it so I can have the post dates be included too.

Flight Journal

I have created a Gemini protocol server at gemini://cetacean.club. Gemini is an exploration of the space between Gopher and HTTP. Right now my site doesn't have much on it, but I have added its feed to my feeds page.

Please note that the content on this Gemini site is going to be of a much more personal nature compared to the more professional kind of content I put on this blog. Please keep this in mind before casting judgement or making any kind of conclusions about me.

If you don't have a Gemini client installed, you can view the site content here. I plan to make a HTTP frontend to this site once I get Maj up and functional.

Maj

I have created a Gemini client and server framework for Rust programs called Maj. Right now it includes the following features:

• Synchronous client
• Asynchronous server framework
• Gemini response parser
• text/gemini parser

Additionally, I have a few projects in progress for the Maj ecosystem:

• majc - an interactive curses client for Gemini
• majd - An advanced reverse proxy and Lua handler daemon for people running Gemini servers
• majsite - A simple example of the maj server framework in action

I will write more about this in the future when I have more than just this little preview of what is to come implemented. However, here's a screenshot of majc rendering my flight journal:

Site Update: Rewrite in Rust

Hello there! You are reading this post thanks to a lot of effort, research and consultation that has resulted in a complete from-scratch rewrite of this website in Rust. The original implementation in Go is available here should anyone want to reference that for any reason.

If you find any issues with the RSS feed, Atom feed or JSONFeed, please let me know as soon as possible so I can fix them.

This website stands on the shoulder of giants. Here are just a few of those and how they add up into this whole package.

comrak

All of my posts are written in markdown. comrak is a markdown parser written by a friend of mine that is as fast and as correct as possible. comrak does the job of turning all of that markdown (over 150 files at the time of writing this post) into the HTML that you are reading right now. It also supports a lot of common markdown extensions, which I use heavily in my posts.

warp

warp is the web framework I use for Rust. It gives users a set of filters that add up into entire web applications. For an example, see this example from its readme:

use warp::Filter;

#[tokio::main]
async fn main() {
    // GET /hello/warp => 200 OK with body "Hello, warp!"
    let hello = warp::path!("hello" / String)
        .map(|name| format!("Hello, {}!", name));

    warp::serve(hello)
        .run(([127, 0, 0, 1], 3030))
        .await;
}

This can then be built up into something like this:

let site = index
    .or(contact.or(feeds).or(resume.or(signalboost)).or(patrons))
    .or(blog_index.or(series.or(series_view).or(post_view)))
    .or(gallery_index.or(gallery_post_view))
    .or(talk_index.or(talk_post_view))
    .or(jsonfeed.or(atom).or(rss.or(sitemap)))
    .or(files.or(css).or(favicon).or(sw.or(robots)))
    .or(healthcheck.or(metrics_endpoint).or(go_vanity_jsonfeed))
    // ...

which is the actual routing setup for this website!

ructe

In the previous version of this site, I used Go's html/template. Rust does not have an equivalent of html/template in its standard library. After some research, I settled on ructe for the HTML templates. ructe works by preprocessing templates using a little domain-specific language that compiles down to Rust source code. This makes the templates become optimized with the rest of the program and enables my website to render most pages in less than 100 microseconds. Here is an example template (the one for /patrons):

@use patreon::Users;
@use super::{header_html, footer_html};

@(users: Users)

@:header_html(Some("Patrons"), None)

<h1>Patrons</h1>

<p>These awesome people donate to me on <a href="https://patreon.com/cadey">Patreon</a>.
If you would like to show up in this list, please donate to me on Patreon. This
is refreshed every time the site is deployed.</p>

<p>
    <ul>
        @for user in users {
            <li>@user.attributes.full_name</li>
        }
    </ul>
</p>

@:footer_html()

The templates compile down to Rust, which lets me include other parts of the program into the templates. Here I use that to take a list of users from the incredibly hacky Patreon API client I wrote for this website and iterate over it, making a list of every patron by name.

Build Process

As a nice side effect of this rewrite, my website is now completely built using Nix. This allows the website to be built reproducibly, as well as a full development environment setup for free for anyone that checks out the repo and runs nix-shell. Check out naersk for the secret sauce that enables my docker image build. See this blogpost for more information about this build process (though my site uses GitHub Actions instead of Drone).

jsonfeed Go package

I used to have a JSONFeed package publicly visible at the go import path christine.website/jsonfeed. As far as I know I'm the only person who ended up using it; but in case there are any private repos that I don't know about depending on it, I have made the jsonfeed package available at its old location as well as its source code here. You may have to update your go.mod file to import christine.website/jsonfeed instead of christine.website. If something ends up going wrong as a result of this, please file a GitHub issue here and I can attempt to assist further.

go_vanity crate

I have written a small go vanity import crate and exposed it in my Git repo. If you want to use it, add it to your Cargo.toml like this:

[dependencies]
go_vanity = { git = "https://github.com/Xe/site", branch = "master" }

You can then use it from any warp application by calling go_vanity::github or go_vanity::gitea like this:

let go_vanity_jsonfeed = warp::path("jsonfeed")
    .and(warp::any().map(move || "christine.website/jsonfeed"))
    .and(warp::any().map(move || "https://tulpa.dev/Xe/jsonfeed"))
    .and_then(go_vanity::gitea);

I plan to add full documentation to this crate soon as well as release it properly on crates.io.

patreon crate

I have also written a small Patreon API client and made it available in my Git repo. If you want to use it, add it to your Cargo.toml like this:

[dependencies]
patreon = { git = "https://github.com/Xe/site", branch = "master" }

This client is incredibly limited and only supports the minimum parts of the Patreon API that are required for my website to function. Patreon has also apparently started to phase out support for its API anyways, so I don't know how long this will be useful.

But this is there should you need it!

Dhall Kubernetes Manifest

I also took the time to port the kubernetes manifest to Dhall. This allows me to have a type-safe kubernetes manifest that will correctly have all of the secrets injected for me from the environment of the deploy script.

---

These are the biggest giants that my website now sits on. The code for this rewrite is still a bit messy. I'm working on making it better, but my goal is to have this website's code shine as an example of how to best write this kind of website in Rust. Check out the code here.

Continuous Deployment to Kubernetes with Gitea and Drone

Recently I put a complete rewrite of the printerfacts server into service based on warp. I have it set up to automatically be deployed to my Kubernetes cluster on every commit to its source repo. I'm going to explain how this works and how I set it up.

Nix

One of the first elements in this is Nix. I use Nix to build reproducible docker images of the printerfacts server, as well as managing my own developer tooling locally. I also pull in the following packages from GitHub:

• naersk - an automagic builder for Rust crates that is friendly to the nix store
• gruvbox-css - the CSS file that the printerfacts service uses
• nixpkgs - contains definitions for the base packages of the system

These are tracked using niv, which allows me to store these dependencies in the global nix store for free. This lets them be reused and deduplicated as they need to be.

Next, I made a build script for the printerfacts service that builds on top of these in printerfacts.nix:

{ sources ? import ./nix/sources.nix, pkgs ? import <nixpkgs> { } }:
let
  srcNoTarget = dir:
    builtins.filterSource
    (path: type: type != "directory" || builtins.baseNameOf path != "target")
    dir;
  src = srcNoTarget ./.;

  naersk = pkgs.callPackage sources.naersk { };
  gruvbox-css = pkgs.callPackage sources.gruvbox-css { };
  
  pfacts = naersk.buildPackage {
    inherit src;
    remapPathPrefix = true;
  };
in pkgs.stdenv.mkDerivation {
  inherit (pfacts) name;
  inherit src;
  phases = "installPhase";

  installPhase = ''
    mkdir -p $out/static

    cp -rf $src/templates $out/templates
    cp -rf ${pfacts}/bin $out/bin
    cp -rf ${gruvbox-css}/gruvbox.css $out/static/gruvbox.css
  '';
}

And finally a simple docker image builder in default.nix:

{ system ? builtins.currentSystem }:

let
  sources = import ./nix/sources.nix;
  pkgs = import <nixpkgs> { };
  printerfacts = pkgs.callPackage ./printerfacts.nix { };

  name = "xena/printerfacts";
  tag = "latest";

in pkgs.dockerTools.buildLayeredImage {
  inherit name tag;
  contents = [ printerfacts ];

  config = {
    Cmd = [ "${printerfacts}/bin/printerfacts" ];
    Env = [ "RUST_LOG=info" ];
    WorkingDir = "/";
  };
}

This creates a docker image with only the printerfacts service in it and any dependencies that are absolutely required for the service to function. Each dependency is also split into its own docker layer so that it is much more efficient on docker caches, which translates into faster start times on existing servers. Here are the layers needed for the printerfacts service to function:

• libunistring - Unicode-safe string manipulation library
• libidn2 - An internationalized domain name decoder
• glibc - A core library for C programs to interface with the Linux kernel
• The printerfacts binary/templates

That's it. It packs all of this into an image that is 13 megabytes when compressed.

Drone

Now that we have a way to make a docker image, let's look how I use drone.io to build and push this image to the Docker Hub.

I have a drone manifest that looks like this:

kind: pipeline
name: docker
steps:
  - name: build docker image
    image: "monacoremo/nix:2020-04-05-05f09348-circleci"
    environment:
      USER: root
    commands:
      - cachix use xe
      - nix-build
      - cp $(readlink result) /result/docker.tgz
    volumes:
      - name: image
        path: /result

  - name: push docker image
    image: docker:dind
    volumes:
      - name: image
        path: /result
      - name: dockersock
        path: /var/run/docker.sock
    commands:
      - docker load -i /result/docker.tgz
      - docker tag xena/printerfacts:latest xena/printerfacts:$DRONE_COMMIT_SHA
      - echo $DOCKER_PASSWORD | docker login -u $DOCKER_USERNAME --password-stdin
      - docker push xena/printerfacts:$DRONE_COMMIT_SHA
    environment:
      DOCKER_USERNAME: xena
      DOCKER_PASSWORD:
        from_secret: DOCKER_PASSWORD

  - name: kubenetes release
    image: "monacoremo/nix:2020-04-05-05f09348-circleci"
    environment:
      USER: root
      DIGITALOCEAN_ACCESS_TOKEN:
        from_secret: DIGITALOCEAN_ACCESS_TOKEN
    commands:
      - nix-env -i -f ./nix/dhall.nix
      - ./scripts/release.sh

volumes:
  - name: image
    temp: {}
  - name: dockersock
    host:
      path: /var/run/docker.sock

This is a lot, so let's break it up into the individual parts.

Configuration

Drone steps normally don't have access to a docker daemon, privileged mode or host-mounted paths. I configured the cadey/printerfacts job with the following settings:

• I enabled Trusted mode so that the build could use the host docker daemon to build docker images
• I added the DIGITALOCEAN_ACCESS_TOKEN and DOCKER_PASSWORD secrets containing a Digital Ocean API token and a Docker hub password

I then set up the volumes block to create a few things:

volumes:
  - name: image
    temp: {}
  - name: dockersock
    host:
      path: /var/run/docker.sock

• A temporary folder to store the docker image after Nix builds it
• The docker daemon socket from the host

Now we can get to the building the docker image.

Docker Image Build

I use this docker image to build with Nix on my Drone setup. As of the time of writing this post, the most recent tag of this image is monacoremo/nix:2020-04-05-05f09348-circleci. This image has a core setup of Nix and a few userspace tools so that it works in CI tooling. In this step, I do a few things:

name: build docker image
image: "monacoremo/nix:2020-04-05-05f09348-circleci"
environment:
  USER: root
commands:
  - cachix use xe
  - nix-build
  - cp $(readlink result) /result/docker.tgz
volumes:
  - name: image
    path: /result

I first activate my cachix cache so that any pre-built parts of this setup can be fetched from the cache instead of rebuilt from source or fetched from crates.io. This makes the builds slightly faster in my limited testing.

Then I build the docker image with nix-build (nix-build defaults to default.nix when a filename is not specified, which is where the docker build is defined in this case) and copy the resulting tarball to that shared temporary folder I mentioned earlier. This lets me build the docker image without needing a docker daemon or any other special permissions on the host.

Pushing

The next step pushes this newly created docker image to the Docker Hub:

name: push docker image
image: docker:dind
volumes:
  - name: image
    path: /result
  - name: dockersock
    path: /var/run/docker.sock
commands:
  - docker load -i /result/docker.tgz
  - docker tag xena/printerfacts:latest xena/printerfacts:$DRONE_COMMIT_SHA
  - echo $DOCKER_PASSWORD | docker login -u $DOCKER_USERNAME --password-stdin
  - docker push xena/printerfacts:$DRONE_COMMIT_SHA
environment:
  DOCKER_USERNAME: xena
  DOCKER_PASSWORD:
    from_secret: DOCKER_PASSWORD

First it loads the docker image from that shared folder into the docker daemon as xena/printerfacts:latest. This image is then tagged with the relevant git commit using the magic $DRONE_COMMIT_SHA variable that Drone defines for you.

In order to push docker images, you need to log into the Docker Hub. I log in using this method in order to avoid the chance that the docker password will be leaked to the build logs.

echo $DOCKER_PASSWORD | docker login -u $DOCKER_USERNAME --password-stdin

Then the image is pushed to the Docker hub and we can get onto the deployment step.

Deploying to Kubernetes

The deploy step does two small things. First, it installs dhall-yaml for generating the Kubernetes manifest (see here) and then runs scripts/release.sh:

#!/usr/bin/env nix-shell
#! nix-shell -p doctl -p kubectl -i bash

doctl kubernetes cluster kubeconfig save kubermemes
dhall-to-yaml-ng < ./printerfacts.dhall | kubectl apply -n apps -f -
kubectl rollout status -n apps deployment/printerfacts

This uses the nix-shell shebang support to automatically set up the following tools:

• doctl to log into kubernetes
• kubectl to actually deploy the site

Then it logs into kubernetes (my cluster is real-life unironically named kubermemes), applies the generated manifest (which looks something like this) and makes sure the deployment rolls out successfully.

This will have the kubernetes cluster automatically roll out new versions of the service and maintain at least two active replicas of the service. This will make sure that you users can always have access to high-quality printer facts, even if one or more of the kubernetes nodes go down.

---

And that is how I continuously deploy things on my Gitea server to Kubernetes using Drone, Dhall and Nix.

If you want to integrate the printer facts service into your application, use the /fact route on it:

$ curl https://printerfacts.cetacean.club/fact
A printer has a total of 24 whiskers, 4 rows of whiskers on each side. The upper
two rows can move independently of the bottom two rows.

There is currently no rate limit to this API. Please do not make me have to create one.

The Dwarven Cavern - A Beginner 6E Adventure

Recently itch.io had one of the largest game bundles in history and one of the things in it was this humble game named 6E. Some friends and I have started up a small group that meets on the weekends to spend a few hours with an adventure. I've been writing a few adventures for them, and I would like to start sharing their archetypes. These will all be included in a small zine that describes the systems we have built on top of 6E that I'm calling The Source. This PDF will be available publicly once it is closer to done (however if you really want a copy early on to dig at it, let me know and we can surely work something out).

Today, I would like to share the details that went into writing the most recent adventure: The Dwarven Cavern. This was derived from One Page Dungeons by Geoffrey Cullop, specifically this is a variant of Kobold Caverns on page 5. Please note that the experience, gold and hitpoints of enemies are balanced for the group I play with, and will probably need to be adjusted for other parties of adventurers. This should work for players at level 1-3. By the end they should gain enough experience to level up once. My group is also very stun-heavy, so that makes my job of attempting to balance things really interesting.

Like most great adventures, this starts at a humble tavern, The Flying Ombudsman.

The Flying Ombudsman

The players start off in The Flying Ombudsman, a tavern in the town of LAST_TOWN_YOUR_PLAYERS_WERE_IN. There are a few people sitting at the bar and drinking steamy mugs of grog. There is a salesperson sitting at one of the tables fidgeting with a golden scepter head and looks like she has plenty of items should people want them. There is someone rather sad sitting at another of the tables, looking like he has suffered a great loss.

For extra immersion, have the NPC's speak with a slightly Irish accent. For extra fun, throw in random words from scottish, english and australian accents. Keeps the players thinking.

When players ask the bartender for a mug o' grog, he will sell them one for 5 gold (limit 4 each player). If players ask for the history behind the name, explain to the players the information in The Story of Hol below.

The salesperson at the top sells the following:

Name	Effect	Price
3x potion of cure poison	Cures poison and grants 2hp	15g each
Golden Scepter head	Doesn't look like it does anything if it's not on top of a staff	50g

You may want to add a few more items to this list. I need to draw up a table for items that salespeople like this can have.

The salesperson can also get you a room at the inn for 30g. It is big enough to hold all of the party.

When players talk to the person at the bottom, they get a sad story about the hoe-pocalypse that threatens the end of the village the person comes from. Dwarves snuck up from underground and stole the hoe from him, making it difficult to feed his hamlet. He asks the adventurers to go to the cavern the dwarves live in and get them back. The players should eventually agree to do it, and the NPC will progressively offer more and more gold as a reward. A charisma check will get him to throw in some salted meat as a reward.

The Story of Hol

The bartender previously ran a failing tavern, and was running out of hope and money. One day, the bartender found a weird looking mug and found that it could be used to talk with the gods. He eventually found a god named Hol. Hol offered him the recipe for the strongest alcohol he could possibly make, for a price. The bartender agreed without further thought and gained the ability to make intensely strong alcohol, but lost his ability to negotiate permanently.

One day the local ombudsman got news of how strong the liquor was and decided to enforce some obscure liquor control law to get the strength reduced. The bartender refused to negotiate and it escalated into a situation where the bartender kicked the ombudsman so hard that he flew into the wall. People started calling the tavern "the place where the ombudsman flew" and that eventually evolved into the name The Flying Ombudsman.

Business has been booming ever since.

The Cave

The first part has 1 mechanical trap around the second corner, players will need to work their way through it with low light or to grab a working torch off the wall. The trap does 1d4 damage.

The tunnel opens into a guard station, the dwarf archers wait behind cover for players to get in range of their shortbows. If they get within 10 feet of the archers, they fall back to the room behind them to the left. The Archers have 6 hp and grant 100xp on defeat.

The room behind and to the left is a labratory that is very well-lit. Lots of weird liquid in flasks, scientific equipment, etc. There is a Dwarf Grenadier there and will attack you on sight. He throws alchemical grenades at range (1d6 damage, =6 -> random elemental effect), If the players get within melee range of the Grenadier then he will use a poisoned dagger. If the guards retreated then he will use them as meat shields. He has 25 hp. He drops 1d4 alchemical grenades and also drops his poisoned dagger. He grants 300xp on defeat.

The bigger chamber is the dwarven living area. Plenty of beds to hide behind, but there are 1d6 dwarves living in there. They attack on sight, but flee south if their attack goes poorly. The dwarves each grant 100xp on defeat. There is a chest in there that is actually a mimic. It drops an Amulet of Chest on defeat.

The room south is the home of Bubba the Bugbear, a hired goon of the dwarves. Bubba and the remaining dwarves make their last stand here. Bubba grants 500xp on defeat and has 30hp and 1d6+2 attack damage with his giant club.

The final room is the treasure room, which contains the stolen hoe, some other farming equipment and 250 gold. After this the dungeon is cleared and players gain experience from their journey, then go back to The Flying Ombudsman victorious. The quest NPC will award them with however many gold they agreed to and any items they also agreed to.

Enemies

Dwarf Archer

Decent guard, horrible foresight

• Hit Points 6

STR	DEX	CON	INT	WIS	CHA
0	+1	0	-2	-2	0

• Condition Immunities drunk, groggy
• Senses Night-vision
• Languages Dwarf
• Challenge 2 (100xp)

Keep-away. Will flee at the first sign of trouble.

Actions

Crossbow. Ranged Attack: 1d4 damage

Dwarf Grenadier

Mad scientist, crazier inventions

• Hit Points 25

STR	DEX	CON	INT	WIS	CHA
0	+1	-2	+2	0	+1

• Condition Immunities drunk, groggy
• Senses Night-vision
• Languages Dwarf
• Challenge 6 (300xp)

Insane. Will do things that normal enemies would not.

Actions

Alchemical Grenade. Ranged Attack: 1d6 damage plus elemental effect if 6 damage. See table for Alchemical Grenades.

Poisoned Dagger. Melee Attack: 1d4 damage. On hit player must roll for constitution. If the check fails they get disadvantage for their throws the next 1d3 turns.

Dwarf

Underground folk that love digging

Hit Points 6

STR	DEX	CON	INT	WIS	CHA
0	+1	0	-2	-2	0

• Condition Immunities drunk, groggy
• Senses Night-vision
• Languages Dwarf
• Challenge 2 (100xp)

Short and Stout. Easily able to hit enemies below the belt.

Actions

Dagger. Melee Attack: 1d4 damage

Mimic

Just an ordinary chest, don't question it

• Hit Points 14

STR	DEX	CON	INT	WIS	CHA
0	0	-2	0	0	-1

• Condition Immunities burn, poison
• Languages None
• Challenge 5 (250xp)

Unsuspicious. Come on, the chest wouldn't be alive, would it?

Actions

Omnomnom Melee Attack: 1d6 piercing damage, gets advantage if the player tries to open the chest.

Bubba the Bugbear

From Bubba with Love

• Hit Points 30

STR	DEX	CON	INT	WIS	CHA
+2	+2	+1	-2	0	-1

• Condition Immunities blinded
• Languages Dwarf
• Challenge 10 (500xp)

Unreasonable. Does not respond well to trickery.

Actions

Whomp. Melee Attack: 1d6+2 damage, piercing if the player fails a constitution check.

Items

These are the unique items specific to this quest.

Mug of Grog

A wooden/iron mug full of the barkeep's grog. Consuming it gives you the following stat boosts for 3 turns:

STR	DEX	CON	INT	WIS	CHA
+0	-1	+0	-1	-1	+3

This bonus does not stack. When you drink the grog, you keep the mug and can use it to bludgeon people for normal attack damage. It can also be used as a tool to check for traps.

Does not sell, purchasable for 5g.

Golden Scepter Head

A golden scepter head that looks like The Grand Nagus from Star Trek. It has no effect outside of when it's put on a staff. When it is on a staff it gives you 25% more gold when you collect gold from places. This has 5 charges and cannot be refreshed.

Sells for 50g, purchases for 50g.

Alchemical Grenades

Standard grenades that look like they are made out of wood, metal, insanity and magic. They do d6 damage normally, but when you crit with one it also causes one of the following status effects (roll a d4):

Alchemical Effects

Roll	Effect
1	Acid makes the entity take off their armor
2	The entity rolls a constitution check, if it fails they get poisoned and need to roll for constitution before every action, passing removes the poison, failing makes them not have any action.
3	The entity is burned for 1d4 damage every turn they fail a constitution check.
4	The entity is stunned for 1d3 turns.

Sells for 25g each, unpurchaseable.

Poisoned Dagger

A standard dagger that gives no distinct bonuses. However when you use it as a normal dagger, you need to roll for constitution in addition to rolling for strength. If you fail the constitution check (but pass the strength check), you get poisoned from the poison streaking out of the dagger.

The poison has 5 charges.

Merchants do not want to take the risk buying it and will have you pay them for its disposal. This cannot be purchased.

Bubba's Clubba

A rather large mace that does +2 damage. It is a giant thing designed for a 9-foot tall centaur. It takes up three slots in your inventory. It looks imposing and may require a lot of strength to use properly.

Sells for 25g.

Amulet of Chest

This cursed amulet lets the holder transform into a harmless looking treasure chest, but also grants them advantage if an enemy tries to open it. After 4 transformations, they need to roll a charisma check when they try to turn back. If that fails, they stay a chest for an hour (with arms/legs/a mouth/etc). After 4 more transformations, they turn into a mimic permanently. The transformation count stays persistent until someone becomes a mimic, then it resets to zero.

Merchants will pay 100 gold for it, but will only accept it after the player rolls for charisma.

Dungeon Map

Adventure Highlights

When I ran this yesterday, the following amazing things happened:

• The players used a crossbow from a previous adventure and a torch to hit the grenades held by the grenadier, vaporizing it from 5 grenades exploding at once.
• The Artificer shield-bashed one of the dwarves so hard that it ricocheted into the other dwarves like pool balls. That caused a lot of damage to all of the enemies.
• The Monk critted a Ki art and shredded an archer with their claws, making the other archer flee.
• The Thief straight up banished a dwarf to the shadow realm with a slingshot hit.

I still wonder how they are going to use that Cursed Amulet of Chest though.

---

Thank you @infinite_mao for making 6E. I can only hope that my buying your stuff separately and making this content for 6E can help give back to the community.

Feedback on the balance of this is very welcome. I openly welcome any and all feedback about how this quest could be rebalanced to be a bit less lopsided in favor of the players. I also wanted to err on the side of balancing towards the players to avoid an unwanted party death.

V Update - June 2020

Every so often I like to check in on the V Programming Language. It's been about six months since my last post, so I thought I'd take another look at it and see what progress has been done in six months.

Last time I checked, V 0.2 was slated for release in December 2019. It is currently June 2020, and the latest release (at time of writing) is 0.1.27.

Feature Updates

Interestingly, the V author seems to have walked back one of their original listed features of V and now has an abstract syntax tree for representing the grammar of the language. They still claim that functions are "pure" by default, but allow functions to perform print statements while still being "pure". Printing data to standard out is an impure side effect, but if you constrain the definition of "side effects" to only include mutability of memory, this could be fine. There seems to be an issue about this on the github tracker, but it was closed.

The next stable release 0.2 seems to be planned for June 2020 (according to the readme); and according to the todo list in the repo, memory management seems to be one of the things that will be finished. V is also apparently in alpha, but will also apparently jump from alpha directly to stable? Given the track record of constantly missed release windows, I am not very confident that V 0.2 will be released on time.

Tools like this need to be ready when they are ready. Trying to rush things is a very unproductive thing to do and can result in more net harm than good.

Build

Testing V is a bit more difficult for me now as its build process is incompatible with my Linux tower's NixOS install (I tend to try and package all the programs I use for testing this stuff so it is easier to reproduce my environment on other machines). The V scripts also do not work on my NixOS tower because it doesn't have a /usr/local/bin. The correct way to make a shell script cross-platform is to use the following header:

#!/usr/bin/env v

This makes the env program search for the V binary in your $PATH, and will function correctly on all platforms (this may not work on environments like Termux due to limitations of how Android works, but it will solve 99% of cases. I am unsure how to make a shell script that will function properly across Android and non-Android environments).

The Makefile in the V source tree seems to do network calls, specifically a git clone. Remember that this is on the front page of the website:

V can be bootstrapped in under a second by compiling its code translated to C with a simple

cc v.c

No libraries or dependencies needed.

Git is a dependency, which means perl is a dependency, which means a shell is a dependency, which means glibc is a dependency, which means that a lot of other things (including posix threads) are also dependencies. Pedantically, you could even go as far as saying that you could count the Linux kernel, the processor being used and the like as dependencies, but that's a bit out of scope for this.

I claim that the V compiler has dependencies because it requires other libraries or programs in order to function. For an example, see the output of ldd (a program that lists the dynamically linked dependencies of other programs) on the V compiler and a hello world program:

$ ldd ./v
        linux-vdso.so.1 (0x00007fff2d044000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2fb3e4c000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2fb3a5b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f2fb4345000)

$ ldd ./hello
        linux-vdso.so.1 (0x00007ffdfdff2000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fed25771000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fed25d88000)

If these binaries were really as dependency-free as the V website claims, the output of ldd would look something like this:

$ ldd $HOME/bin/dhall
        not a dynamic executable

The V compiler claims to have support for generating machine code directly, but in my testing I was unable to figure out how to set the compiler into this mode.

Memory Management

V doesn't use garbage collection or reference counting. The compiler cleans everything up during compilation. If your V program compiles, it's guaranteed that it's going to be leak free.

Accordingly, the documentation still claims that memory management is both a work in progress and has (or will have, it's not clear which is accurate from the documentation alone) perfect accuracy for cleaning up things at compile time. Every one of these posts I have run a benchmark against the V compiler, I like to call it the "how much ram do you leak compiling hello world" test. Last it leaked 4,600,383 bytes (or about 4.6 megabytes) and before that it leaked 3,861,785 bytes (or about 3.9 megabytes). This time:

$ valgrind ./v hello.v
==5413== Memcheck, a memory error detector
==5413== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==5413== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==5413== Command: ./v hello.v
==5413==
==5413==
==5413== HEAP SUMMARY:
==5413==     in use at exit: 7,232,779 bytes in 163,690 blocks
==5413==   total heap usage: 182,696 allocs, 19,006 frees, 11,309,504 bytes allocated
==5413==
==5413== LEAK SUMMARY:
==5413==    definitely lost: 2,673,351 bytes in 85,739 blocks
==5413==    indirectly lost: 4,265,809 bytes in 77,711 blocks
==5413==      possibly lost: 256,000 bytes in 1 blocks
==5413==    still reachable: 37,619 bytes in 239 blocks
==5413==         suppressed: 0 bytes in 0 blocks
==5413== Rerun with --leak-check=full to see details of leaked memory
==5413==
==5413== For counts of detected and suppressed errors, rerun with: -v
==5413== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

It seems that the memory managment really is a work in progress. This increase in leakage means that the compiler building itself now creates 7,232,779 bytes of leaked ram (which still is amusingly its install size in memory, when including git deltas, temporary files and a worktree copy of V).

Doom

The Doom translation project still has one file translated (and apparently it breaks sound effects but not music). I have been looking forward to the full release of this as it will show a lot about how readable the output of V's C to V translation feature is.

1.2 Million Lines of Code

Let's re-run the artificial as heck 1.2 million lines of code benchmark from the last post:

$ bash -c 'time ~/code/v/v main.v'

real    7m54.847s
user    7m32.860s
sys     0m14.212s

Compared to the last time this benchmark was run, this took 2 minutes less (last time it took about 10 minutes). This is actually a major improvement, and means that V's claims of speed are that much closer to reality at least on my test hardware.

Concurrency

A common problem that shows up when writing multi-threaded code are race conditions. Effectively, race conditions are when two bits of code try to do the same thing at the same time on the same block of memory. This leads to undefined behavior, which is bad because it can corrupt or crash programs.

As an example, consider this program raceanint.v:

fn main() {
  foo := [ 1 ]
  go add(mut foo)
  go add(mut foo)

  for {}
}

fn add(mut foo []int) {
  for {
    foo[0] = foo[0] + 1
  }
}

In theory, this should have two threads infinitely trying to increment foo[0], which will eventually result in foo[0] getting corrupted by two threads trying to do the same thing at the same time (given the tight loops invovled). This leads to undefined behavior, which can be catastrophic in production facing applications.

However, I can't get this to build:

==================
/home/cadey/.cache/v/raceanint.tmp.c: In function ‘add_thread_wrapper’:
/home/cadey/.cache/v/raceanint.tmp.c:1209:6: error: incompatible type for argument 1 of ‘add’
  add(arg->arg1);
      ^~~
/home/cadey/.cache/v/raceanint.tmp.c:1198:13: note: expected ‘array_int * {aka struct array *}’ but argument is of type ‘array_int {aka struct array}’
 static void add(array_int* foo);
             ^~~
/home/cadey/.cache/v/raceanint.tmp.c: In function ‘strconv__v_sprintf’:
/home/cadey/.cache/v/raceanint.tmp.c:3611:7: warning: variable ‘th_separator’ set but not used [-Wunused-but-set-variable]
  bool th_separator = false;
       ^~~~~~~~~~~~
/home/cadey/.cache/v/raceanint.tmp.c: In function ‘print_backtrace_skipping_top_frames_linux’:
...
==================
(Use `v -cg` to print the entire error message)

builder error:
==================
C error. This should never happen.

If you were not working with C interop, please raise an issue on GitHub:

https://github.com/vlang/v/issues/new/choose

Like I said before, I also cannot file new issues about this. So if you are willing to help me out, please open an issue about this.

---

EDIT(Xe): 2020 M06 23

I do not plan to make any future update posts about the V programming language in the future. The V community is something I would really rather not be associated with. This is an edited-down version of the post that was released last week (2020 M06 17).

As of the time of writing this note to the end of this post and as far as I am aware, I am banned from being able to contribute to the V language in any form. I am therefore forced to consider that the V project will respond to criticism of their language with bans. This subjective view of reality may not be accurate to what others see.

I would like to see this situation result in a net improvement for everyone involved. V is an interesting take on a stagnant field of computer science, but I cannot continue to comment on this language or give it any of the signal boost I have given it with this series of posts.

Thank you for reading. I will continue with my normal posts in the next few days.

Be well.

Made in Drawpile with an iPad Pro connected as a display to a MacBook. I can upload the .ora file on request.

Why I Use Suckless Tools

Software is complicated. Foundational building blocks of desktop environments tend to grow year over year until it's difficult to understand or maintain them. Suckless offers an alternative to this continuous cycle of bloat and meaningless redesign. Suckless tools aim to keep things simple, minimal, usable and hackable by default. Their window manager dwm is just a window manager. It doesn't handle things like transparency, compositing or volume control. Their terminal st is just a terminal. It doesn't handle fancy things like ancient terminal kinds that died out long ago. It just displays text. It doesn't handle things that tmux or similar could take care of, because tmux can do a better job at that than st ever could on its own.

Suckless tools are typically configured in C, the language they are written in. However as a side effect of suckless tools having their configuration baked into the executable at compile time, they start up instantly. If something goes wrong while using them, you can easily jump right into the code that implements them and nail down issues using basic debugger skills.

However, even though the window manager is meager, it still offers places for you to make it look beautiful. For examples of beautiful dwm setups, see this search of /r/unixporn on reddit.

I would like to walk through my dwm setup, how I have it configured all of the parts at play as well as an example of how I debug problems in my dwm config.

My dwm Config

As dwm is configured in C, there's also a community of people creating patches for dwm that add extra features like additional tiling methods, the ability to automatically start things with dwm, transparency for the statusbar and so much more. I use the following patches:

• alpha
• autostart
• bottomstack
• dwmc
• pertag
• systray
• uselessgap

This combination of patches allows me to make things feel comfortable and predictable enough that I can rely entirely on muscle memory for most of my window management. Nearly all of it is done with the keyboard too.

Here is my config file. It's logically broken into two big sections:

• Variables
• Keybinds

I'll go into more detail about these below.

Variables

The main variables in my config control the following:

• border width
• size of the gaps when tiling windows
• the snap width
• system tray errata
• the location of the bar
• the fonts
• colors
• transparency values for the bar
• workspace names (mine are based off of the unicode emoticon (ﾉ◕ヮ◕)ﾉ*:･ﾟ✧)
• app-specific hacks
• default settings for the tiling layouts
• if windows should be forced into place or not
• window layouts

All of these things control various errata. As a side effect of making them all compile time constants, these settings don't have to be loaded into the program because they're already a part of it. I use the Hack font on my desktop and with emacs.

Keybinds

The real magic of tiling window managers is that all of the window management commands are done with my keyboard. Alt is the key I have devoted to controlling the window manager. All of my window manager control chords use the alt key.

Here are the main commands and what they do:

Command	Effect
Alt-p	Spawn a program by name
Alt-Shift-Enter	Open a new terminal window
Alt-b	Hide the bar if it is shown, show the bar if it is hidden
Alt-j	Move focus down the stack of windows
Alt-k	Move focus up the stack of windows
Alt-i	Increase the number of windows in the primary area
Alt-d	Decrease the number of windows in the primary area
Alt-h	Make the primary area smaller by 5%
Alt-l	Make the primary area larger by 5%
Alt-Enter	Move the currently active window into the primary area
Alt-Tab	Switch to the most recently active workspace
Alt-Shift-C	Nicely ask a window to close
Alt-t	Select normal tiling mode for the current workspace
Alt-f	Select floating (non-tiling) mode for the current workspace
Alt-m	Select monocle (fullscreen active window) mode for the current workspace
Alt-u	Select bottom-stacked tiling mode for the current workspace
Alt-o	Select bottom-stacked horizontal tiling mode for the current workspace (useful on vertical monitors)
Alt-e	Open a new emacs window
Alt-Space	Switch to the most recently used tiling method
Alt-Shift-Space	Detach the currently active window from tiling
Alt-1 thru Alt-9	Switch to a given workspace
Alt-Shift-1 thru Alt-Shift-9	Move the active window to a given workspace
Alt-0	Show all windows on all workspaces
Alt-Shift-0	Show the active window on all workspaces
Alt-Comma and Alt-Period	Move focus to the other monitor
Alt-Shift-Comma and Alt-Shift-Period	Move the active window to the other monitor
Alt-Shift-q	Uncleanly exit dwm and kill the session

This is just enough commands that I can get things done, but not so many that I get overwhelmed and forget what keybind does what. I have most of this committed to muscle memory (and had to look at the config file to write out this table), and as a result nearly all of my window management is done with my keyboard.

The rest of my config handles things like Alt-Right-Click to resize windows arbitrarily, signals with dwmc and other overhead like that.

The Other Parts

The rest of my desktop environment is built up using a few other tools that build on top of dwm. You can see the NixOS modules I've made for it here and here:

• xrandr to set up my multiple monitors and rotation for them
• feh to set my wallpaper
• picom to handle compositing effects like transparency, blur and drop shadows for windows
• pasystray for controlling my system volume
• dunst for notifications
• xmodmap for rebinding the caps lock key to the escape key
• cabytcini to show the current time and weather in my dwm bar

Each of these tools has their own place in the stack and they all work together to give me a coherent and cohesive environment that I can use for Netflix, programming, playing Steam games and more.

cabytcini is a program I created for myself as part of my goal to get more familiar with Rust. As of the time of this post being written, it uses only 11 megabytes of ram and is configured using a config file located at ~/.config/cabytcini/gaftercu'a.toml. It scrapes data from the API server I use for my wall-mounted clock to show me the weather in Montreal. I've been meaning to write more about it, but it's currently only documented in Lojban.

Debugging dwm

Software is imperfect, even smaller programs like dwm can still have bugs in them. Here's the story of how I debugged and bisected a problem with my dwm config recently.

I had just gotten the second monitor set up and noticed that whenever I sent a window to it, the entire window manager seemed to get locked up. I tried sending the quit command to see if it would respond to that, and it failed. I opened up a virtual terminal with control-alt-F1 and logged in there, then I launched htop to see if the process was blocked.

It reported dwm was using 100% CPU. This was odd. I then decided to break out the debugger and see what was going on. I attached to the dwm process with gdb -p (pgrep dwm) and then ran bt full to see where it was stuck.

The backtrace revealed it was stuck in the drawbar() function. It was stuck in a loop that looked something like this:

for (c = m->clients; c; c = c->next) {
    occ |= c->tags;
    if (c->isurgent)
            urg |= c->tags;
}

dwm stores the list of clients per tag in a singly linked list, so the root cause could be related to a circular linked list somehow, right?

I decided to check this by printing c and c->next in GDB to see what was going on:

gdb> print c
0xfad34f
gdb> print c->next
0xfad34f

The linked list was circular. dwm was stuck iterating an infinite loop. I looked at the type of c and saw it was something like this:

struct Client {
	char name[256];
	float mina, maxa;
	float cfact;
	int x, y, w, h;
	int oldx, oldy, oldw, oldh;
	int basew, baseh, incw, inch, maxw, maxh, minw, minh;
	int bw, oldbw;
	unsigned int tags;
	int isfixed, isfloating, isurgent, neverfocus, oldstate, isfullscreen;
	Client *next;
	Client *snext;
	Monitor *mon;
	Window win;
};

So, next is a pointer to the next client (if it exists). Setting the pointer to NULL would probably break dwm out of the infinite loop. So I decided to test that by running:

gdb> set var c->next = 0x0

To set the next pointer to null. dwm immediately got unstuck and exited (apparently my quit command from earlier got buffered), causing the login screen to show up. I was able to conclude that something was wrong with my dwm setup.

I know this behavior worked on release versions of dwm, so I decided to load up KDE and then take a look at what was going on with Xephyr and git bisect.

I created two fake monitors with Xephyr:

$ Xephyr -br -ac -noreset -screen 800x600 -screen 800x600 +xinerama :1 &

And then started to git bisect my dwm fork:

$ cd ~/code/cadey/dwm
$ git bisect init
$ git bisect bad HEAD
$ git bisect good cb3f58ad06993f7ef3a7d8f61468012e2b786cab

I registered the bad commit (the current one) and the last known good commit (from when dwm 6.2 was released) and started to recreate the conditions of the hang.

I set the DISPLAY environment variable so that dwm would use the fake monitors:

$ export DISPLAY=:1

and then rebuilt/ran dwm:

$ make clean && rm config.h && make && ./dwm

Once I had dwm up and running, I created a terminal window and tried to send it to the other screen. If it worked, I marked the commit as good with git bisect good, and if it hung I marked the commit as bad with git bisect bad. 7 iterations later and I found out that the attachbelow patch was the culprit.

I reverted the patch on the master branch, rebuilt and re-ran dwm and tried to send the terminal window between the fake monitors. It worked every time. Then I committed the revert of attachbelow, pushed it to my NUR repo, and then rebuilt my tower's config once it passed CI.

Being a good internet citizen, I reported this to the suckless mailing list and then was able to get a reply back not only confirming the bug, but also with a patch for the patch to fix the behavior forever. I have yet to integrate this meta-patch into my dwm fork, but I'll probably get around to it someday.

This really demonstrates one of the core tenets of the suckless philosophy perfectly. I am not very familiar with how the dwm codebase works, but I am able to dig into its guts and diagnose/fix things because it is intentionally kept as simple as possible.

If you use Linux on a desktop/laptop, I highly suggest taking a look at suckless software and experimenting with it. It is super optimized for understandability and hacking, which is a huge breath of fresh air these days.

gitea-release Tool Announcement

I'm a big fan of automating things that can possibly be automated. One of the biggest pains that I've consistently had is creating/tagging releases of software. This has been a very manual process for me. I have to write up changelogs, bump versions and then replicate the changelog/versions in the web UI of whatever git forge the project in question is using. This works great at smaller scales, but can quickly become a huge pain in the butt when this needs to be done more often. Today I've written a small tool to help me automate this going forward, it is named gitea-release. This is one of my largest Rust projects to date and something I am incredibly happy with. I will be using it going forward for all of my repos on my gitea instance tulpa.dev.

gitea-release is a spiritual clone of the tool github-release, but optimized for my workflow. The biggest changes are that it works on gitea repos instead of github repos, is written in Rust instead of Go and it automatically scrapes release notes from CHANGELOG.md as well as reading the version of the software from VERSION.

CHANGELOG.md and VERSION files

The CHANGELOG.md file is based on the Keep a Changelog format, but modified slightly to make it easier for this tool. Here is an example changelog that this tool accepts:

# Changelog
All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## 0.1.0

### FIXED

- Refrobnicate the spurious rilkefs

## 0.0.1

First release, proof of concept.

When a release is created for version 0.1.0, this tool will make the description of the release about as follows:

### FIXED

- Refrobnicate the spurious rilkefs

This allows the changelog file to be the ultimate source of truth for release notes with this tool.

The VERSION file plays into this as well. The VERSION file MUST be a single line containing a semantic version string. This allows the VERSION file to be the ultimate source of truth for software version data with this tool.

Release Process

When this tool is run with the release subcommand, the following actions take place:

• The VERSION file is read and loaded as the desired tag for the repo
• The CHANGELOG.md file is read and the changes for the VERSION are cherry-picked out of the file
• The git repo is checked to see if that tag already exists
  • If the tag exists, the tool exits and does nothing
• If the tag does not exist, it is created (with the changelog fragment as the body of the tag) and pushed to the gitea server using the supplied gitea token
• A gitea release is created using the changelog fragment and the release name is generated from the VERSION string

Automation of the Automation

This tool works perfectly well locally, but this doesn't make it fully automated from the gitea repo. I use drone as a CI/CD tool for my gitea repos. Drone has a very convenient and simple to use plugin system that was easy to integrate with structopt.

I created a drone plugin at xena/gitea-release that can be configured as a pipeline step in your .drone.yml like this:

kind: pipeline
name: ci/release
steps:
  - name: whatever unit testing step
    # ...
  - name: auto-release
    image: xena/gitea-release:0.2.5
    settings:
      auth_username: cadey
      changelog_path: ./CHANGELOG.md
      gitea_server: https://tulpa.dev
      gitea_token:
        from_secret: GITEA_TOKEN
    when:
      event:
        - push
      branch:
        - master

This allows me to bump the VERSION and CHANGELOG.md, then push that commit to git and a new release will automatically be created. You can see an example of this in action with the drone build history of the gitea-release repo. You can also how the CHANGELOG.md file grows with the CHANGELOG of gitea-release.

Once the release is pushed to gitea, you can then use drone to trigger deployment commands. For example here is the deployment pipeline used to automatically update the docker image for the gitea-release tool:

kind: pipeline
name: docker
steps:
  - name: build docker image
    image: "monacoremo/nix:2020-04-05-05f09348-circleci"
    environment:
      USER: root
    commands:
      - cachix use xe
      - nix-build docker.nix
      - cp $(readlink result) /result/docker.tgz
    volumes:
      - name: image
        path: /result
    when:
      event:
        - tag

  - name: push docker image
    image: docker:dind
    volumes:
      - name: image
        path: /result
      - name: dockersock
        path: /var/run/docker.sock
    commands:
      - docker load -i /result/docker.tgz
      - echo $DOCKER_PASSWORD | docker login -u $DOCKER_USERNAME --password-stdin
      - docker push xena/gitea-release
    environment:
      DOCKER_USERNAME:
        from_secret: DOCKER_USERNAME
      DOCKER_PASSWORD:
        from_secret: DOCKER_PASSWORD
    when:
      event:
        - tag

volumes:
  - name: image
    temp: {}
  - name: dockersock
    host:
      path: /var/run/docker.sock

This pipeline will use Nix to build the docker image, load it into a Docker daemon and then log into the Docker Hub and push it. This can then be used to do whatever you want. It may also be a good idea to push a docker image for every commit and then re-label the tagged commits, but this wasn't implemented in this repo.

---

I hope this tool will be useful. I will accept feedback over any contact method. If you want to contribute directly to the project, please feel free to create issues or pull requests. If you don't want to create an account on my git server, get me the issue details or code diffs somehow and I will do everything I can to fix issues and integrate code. I just want to make this tool better however I can.

Be well.

ReConLangMo 8: Storytelling

In the last episode of ReConLangMo, we covered conversational discourse as well as formality and other grammatical moods. I also covered my goals for the gender system of L'ewa. Here I will cover the closest thing L'ewa has to culture, the storytelling and poetry norms. L'ewa is also a language designed for spellcraft and sigil magick, so those norms will be covered too. This is a response to this prompt.

Stories

Stories are told as statements that happened in the past. Stories are structured in the same way that you would structure them in English. There is a scenario, a call to action, a refusal of the call, then the story goes on in the standard way. Casual retelling of events is done without a narrative, and the events are just relayed using casual sentences.

The particle qu can be repeated at the beginning of a story to enable the "story time" flag. Story time sentences can be figurative. Each sentence in the story progressively builds up the narrative to explain the themes and lessons that are trying to be conveyed.

Stories are told using the narrative present tense. Speakers also relay secondhand information directly.

Poetry

One of the morphological side effect of L'ewa root words is that only the first four letters of each word are unique. As a side effect of this, you can make any word rhyme with any other word if you want it to. L'ewa can become l'ewi, l'ewo, le'we or l'ewu if the poetry demands it. Poetry can be done in any meter or rhythm depending on the mood of the speaker. Poetry can also be formatted using fixed-width text. Here are a few examples:

le l'ewa de kirta
xi firga to renma

The language of Creators
is beneficial to all people

a'o ro zimpu ti
e'o so vorto

I hope you understand this
How many words?

I don't think I have enough vocabulary to make any more yet.

Spellcraft

These poems are worked into sigils by interlocking the words together into a larger figure. Here is an example based on the first poem:

Ideally this would would include the letters spiraling around things, but my current tools are limited in what they can do. Sigils don't need to follow normal grammar rules. They can bend and break them as much as they want or need in order to flow nicer. If they need to, they can also make up words that don't normally exist in the dictionary. These words should be documented in the dictionary at some point, but there is no big rush.

Gender and Third Person Pronouns

Previously, I haven't gone into details about the third person pronouns in L'ewa. This was done very, very intentionally. One of the goals of L'ewa's handling of gender is to abolish the gender binary as much as possible. This means that any content word can end up being used as a pronoun. In order to avoid ambiguity, only part of the content word is used to form something matching the particle rules I was vaguely gesturing about in the post that had details about colors. To recap:

Compound words still need to be fleshed out, but generally all CVCCV words will have wordparts made out of the first, second and fifth letter, unless the vowel pair is illegal and all CCVCV words are the first, third and fifth letter unless this otherwise violates the morphology rules.

Let's say that your gender is the word for "is meat", or dextu. This would mean the third person pronoun form would be de'u (eu isn't a valid vowel pair so the glottal stop is used to break it).

de'u qu tulpa lo l'ewa
They (de'u) built a language.

mao qu madsa lo spalo
They (mao) ate an apple.

If you want to declare your gender, you can declare it with the word zedra:

lo spalo xi zedra
An apple is (my) gender.

This would then make their pronoun sao.

You can ask someone what their gender is with the gender question particle zei:

<Mai> xoi
<Cadey> xoi ro zei
<Mai> lo mlato xi zedra
<Mai> zei
<Cadey> lo 'orka xi zedra

From then Mai would be referred to using the pronoun mao and Cadey would be referred to using the pronoun 'ka.

If you need a generic third person pronoun, use ke'o.

---

This seems to be the end of the ReConLangMo series in /r/conlangs, but I will definitely continue to develop this on my own and post about it as I make larger accomplishments. This has been a fun series and I hope it gave people a high level overview of what is needed to make a speakable language from nothing.

Be well.

ReConLangMo 7: Discourse

Previously on ReConLangMo, we covered a lot of new words for the lexicon of L'ewa. This helps to flesh out a lot of what can be said, but conversations themselves can be entirely different from formal sentences. Conversations flow and ebb based on the needs/wants of the interlocutors. This post will start to cover a lot of the softer skills behind L'ewa as well as cover some other changes I'm making under the hood. This is a response to this prompt.

Information Structure

L'ewa doesn't have any particular structure for marking previously known information, as normal sentences should suffice in most cases. Consider this paragraph:

I saw you eat an apple. Was it tasty?

Since an apple was the last thing mentioned in the paragraph, the vague "it" pronoun in the second sentence can be interpreted as "the apple".

L'ewa doesn't have a way to mark the topic of a sentence, that should be obvious from context (additional clauses to describe things will help here). In most cases the subject should be equivalent to the topic of a sentence.

L'ewa doesn't directly offer ways to emphasize parts of sentences with phonemic stress like English does (eg: "I THOUGHT you ate an apple" vs "I thought you ATE an apple"), but emotion words can be used to help indicate feelings about things, which should suffice as far as emphasis goes.

Discourse Structure

Conversationally, a lot of things in L'ewa grammar get dropped unless it's ambiguous. The I/yous that get tacked on in English are completely unneeded. A completely valid conversation could look something like this:

<Mai> xoi
<Cadey> xoi
<Mai> xoi madsa?
<Cadey> lo spalo

And it would roughly equate to:

<Mai> Hi
<Cadey> Hi, you doing okay?
<Mai> Yes, have you eaten?
<Cadey> Yes, I ate an apple

People know when they can speak after a sufficient pause between utterances. Interrupting is not common but not a social faux-pas, and can be used to stop a false assumption from being said.

Utterances

An utterance in L'ewa is anything from a single content word all the way up to an entire paragraph of sentences. An emotion particle can be a complete utterance. A question particle can be a complete utterance, anything can be an utterance. A speaker may want to choose more succinct options when the other detail is already contextually known or simply not relevant to the listener.

L'ewa has a few discourse particles, here are a few of the more significant ones:

L'ewa	Function
xi	signals that the verb of the sentence is coming next
ko	ends a noun phrase
ka	marks something as the subject of the sentence
ke	marks something as the verb of the sentence
ku	marks something as the object of the sentence

Formality

The informal dialect of L'ewa drops everything it can. The formal dialect retains everything it can, to the point where it includes noun phrase endings, the verb signaler, ka/ke/ku and every single optional particle in the language. The formal dialect will end up sounding rather wordy compared to informal slangy speech. Consider the differences between informal and formal versions of "I eat an apple":

mi madsa lo spalo.

ka mi ko xi ke madsa ku lo spalo ko.

Nearly all of those particles are not required in informal speech (you could even get away with madsa lo spalo depending on context), but are required in formal speech to ensure there is as little contextual confusion as possible. Things like laws or legal rulings would be written out in the formal register.

Greetings and Farewell

"Hello" in L'ewa is said using xoi. It can also be used as a reply to hello similar to «ça va» in French. It is possible to have an entire conversation with just xoi:

<Mai> xoi
<Cadey> xoi
<Mai> xoi

The other implications of xoi are "how are you?" "I am good, you?", "I am good", etc. If more detail is needed beyond this, then it can be supplied instead of replying with xoi.

"Goodbye" is said using xei. Like xoi it can be used as a reply to another goodbye and can form a mini-conversation:

<Cadey> xei
<Mai> xei
<Cadey> xei

Emotion Words

Feelings in L'ewa are marked with a family of particles called "UI". These can also be modified with other particles. Here are the emotional markers:

L'ewa	English
a'a	attentive
a'e	alertness
ai	intent
a'i	effort
a'o	hope
au	desire
a'u	interest
e'a	permission
e'e	competence
ei	obligation
e'i	constraint
e'o	request
e'u	suggestion
ia	belief
i'a	acceptance
ie	agreement
i'e	approval
ii	fear
i'i	togetherness
io	respect
i'o	appreciation
iu	love
i'u	familiarity
o'a	pride
o'e	closeness
oi	complaint/pain
o'i	caution
o'o	patience
o'u	relaxation
ua	discovery
u'a	gain
ue	surprise
u'e	wonder
ui	happiness
u'i	amusement
uo	completion
u'o	courage
uu	pity
u'u	repentant

If an emotion is unknown in a conversation, you can ask with kei:

<Mai> xoi, so kei?
      hi,  what-verb what-feeling?

<Cadey> madsa ui
        eating :D

This system is wholesale stolen from Lojban.

Connectives

Connectives exist to link noun phrases and verbs together into larger noun phrases and verbs. They can also be used to link together sentences. There are four simple connectives: fa (OR), fe (AND), fi (connective question), fo (if-and-only-if) and fu (whether-or-not).

OR

ro au madsa lo spalo fa lo hafto?
Do you want to eat an apple or an egg?

AND

ro au madsa lo spalo fe lo hafto?
Do you want to eat an apple and an egg?

If and Only If

ro 'amwo mi fo mi madsa hafto?
Do you love me if I eat eggs?

Whether or Not

mi 'amwo ro. fu ro madsa hafto.
I love you, whether or not you eat eggs.

Connective Question

ro au madsa lo spalo fi lo hafto?
Do you want to eat apples and/or eggs?

Changes Being Made to L'ewa

Early on, I mentioned that family terms were gendered. This also ended up with me making some gendered terms for people. I have since refactored out all of the gendered terms in favor of more universal terms. Here is a table of some of the terms that have been replaced:

English	L'ewa term	L'ewa word
brother/sister	sibling	xinga
mother/father	parent	pa'ma
grandfather/grandmother	grandparent	gra'u
aunt/uncle	parent	pa'ma
cousin	sibling	xinga
man/woman	Creator	kirta
man/woman	human	renma

In some senses, gender exists. In other senses, gender does not. With L'ewa I want to explore what is possible with language. It would be interesting to create a language where gender can be discussed as it is, not as the categories that it has historically fit into. Consider colors. There are millions of colors, all sightly different but many follow general patterns. No one or two colors can be thought of as the "default" color, yet we can have long and meaningful conversations about what color is and what separates colors from eachother.

I aim to have the same kind of granularity in L'ewa. As a goal of the language, I should be able to point to any content word in the dictionary and be able to say "that's my gender" in the same way I can describe color or music with that tree. These will implicitly be metaphors (which does detract a bit from the logical stance L'ewa normally takes) because gender is almost always a metaphor in practice. L'ewa will not have binary gender.

Issue number two on the L'ewa repo will help track the creation and implementation of a truly non-binary "gender" system for L'ewa.

---

I've been chugging through the Swaedish list more and more to build up more of L'ewa's vocabulary in preparation for starting to translate sentences more complicated than simple "I eat an apple" or "Do you like eating plants?". One of the first things I want to translate is the classic tower of babel story.

Be well.

maybedoer: the Maybe Monoid for Go

I recently posted (a variant of) this image of some Go source code to Twitter and it spawned some interesting conversations about what it does, how it works and why it needs to exist in the first place:

This file is used to sequence functions that could fail together, allowing you to avoid doing an if err != nil check on every single fallible function call. There are two major usage patterns for it.

The first one is the imperative pattern, where you call it like this:

md := new(maybedoer.Impl)

var data []byte

md.Maybe(func(context.Context) error {
  var err error
 
  data, err = ioutil.ReadFile("/proc/cpuinfo")
 
  return err
})

// add a few more maybe calls?

if err := md.Error(); err != nil {
  ln.Error(ctx, err, ln.Fmt("cannot munge data in /proc/cpuinfo"))
}

The second one is the iterative pattern, where you call it like this:

func gitPush(repoPath, branch, to string) maybedoer.Doer {
  return func(ctx context.Context) error {
    // the repoPath, branch and to variables are available here
    return nil
  }
}

func repush(ctx context.Context) error {
  repoPath, err := ioutil.TempDir("", "")
  if err != nil {
    return fmt.Errorf("error making checkout: %v", err)
  }

  md := maybedoer.Impl{
    Doers: []maybedoer.Doer{
      gitConfig, // assume this is implemented
      gitClone(repoPath, os.Getenv("HEROKU_APP_GIT_REPO")), // and this too
      gitPush(repoPath, "master", os.Getenv("HEROKU_GIT_REMOTE")),
    },
  }
  
  err = md.Do(ctx)
  if err != nil {
    return fmt.Errorf("error repushing Heroku app: %v", err)
  }
  
  return nil
}

Both of these ways allow you to sequence fallible actions without having to write if err != nil after each of them, making this easily scale out to arbitrary numbers of steps. The design of this is inspired by a package used at a previous job where we used it to handle a lot of fiddly fallible actions that need to happen one after the other.

However, this version differs because of the Doers element of maybedoer.Impl. This allows you to specify an entire process of steps as long as those steps don't return any values. This is very similar to how Haskell's Data.Monoid.First type works, except in Go this is locked to the error type (due to the language not letting you describe things as precisely as you would need to get an analog to Data.Monoid.First). This is also similar to Rust's and_then combinator.

If we could return values from these functions, this would make maybedoer closer to being a monad in the Haskell sense. However we can't so we are locked to one specific instance of a monoid. I would love to use this for a pointer (or pointer-like) reference to any particular bit of data, but interface{} doesn't allow this because interface{} matches literally everything:

var foo = []interface{
  1,
  3.4,
  "hi there",
  context.Background(),
  errors.New("this works too!"),
}

This could mean that if we changed the type of a Doer to be:

type Doer func(context.Context) interface{}

Then it would be difficult to know how to handle returns from the function. Arguably we could write some mechanism to check if it is an error:

result := do(ctx)
if result != nil {
  switch result.(type) {
  case error:
    return result // result is of type error magically
  default:
    md.return = result
  }
}

But then it would be difficult to know how to pipe the result into the next function, unless we change Doer's type to be:

type Doer func(context.Context, interface{}) interface{}

Which would require code that looks like this:

func getNumber(ctx context.Context, _ interface{}) interface{} {
  return 2
}

func double(ctx context.Context, num interface{}) interface{} {
  switch num.(type) {
  case int:
    return 2+2
  default:
    return fmt.Errorf("wanted num to be an int, got: %T", num)
  }
  
  return nil
}

But this kind of repetition would be required for every function. I don't really know what the best way to solve this in a generic way would be, but I'm fairly sure that these fundamental limitations in Go prevent this package from being genericized to handle function outputs and inputs beyond what you can do with currying (and maybe clever pointer usage).

I would love to be proven wrong though. If anyone can take this source code under the MIT license and prove me wrong, I will stand corrected and update this blogpost with the solution.

This kind of thing is more easy to solve in Rust with its Result type; and arguably this entire problem solved in the Go package is irrelevant in Rust because this solution is in the standard library of Rust.

ReConLangMo 6: Lexicon

Previously in this series, we've covered a lot of details about how sentences work, tenses get marked and how words work in general; however this doesn't really make L'ewa a language. Most of the difficulty in making a language like this is the vocabulary. In this post I'll be describing how I am making the vocabulary for L'ewa and I'll include an entire table of the dictionary words. This answers this prompt.

Word Distinctions

L'ewa is intended to be a logical language. One of the side effects of L'ewa being a logical language is that each word should have as minimal and exact of a meaning/function as possible. English has lots of words that cover large semantic spaces (like go, set, run, take, get, turn, good, etc.) without much of a pattern to it. I don't want this in L'ewa.

Let's take the word "good" as an example. Off the top of my head, good can mean any of the following things:

• beneficial
• aesthetically pleasing
• favorful taste
• saintly (coincidentally this is the source of the idiom "God is good")
• healthy

I'm fairly sure there are more "senses" of the word good, but let's break these into their own words:

L'ewa	Definition
firgu	is beneficial/nice to
n'ixu	is aesthetically pleasing to
flawo	is tasty/has a pleasant flavor to
spiro	is saintly/holy/morally good to
qanro	is healthy/fit/well/in good health

Each of these words has a very distinct and fine-grained meaning, even though the range is a bit larger than it would be in English. These words also differ from a lot of the other words in the L'ewa dictionary so far because they can take an object. Most of the words so far are adjective-like because it doesn't make sense for there to be an object attached to the color blue.

By default, if a word that can take an object doesn't have one, it's assumed to be obvious from context. For example, consider the following set of sentences:

mi qa madsa lo spalo. ti flawo!

I am eating an apple. It's delicious!

I am working at creating more words using a Swaedish list.

Family Words

Family words are a huge part of a language because it encodes a lot about the culture behind that language. L'ewa isn't really intended to have much of a culture behind it, but the one place I want to take a cultural stance is here. The major kinship word is kirta, or "is an infinite slice of an even greater infinite". This is one of the few literal words in L'ewa that is defined using a metaphor, as there is really no good analog for this in English.

There are also words for other major family terms in English:

L'ewa	Definition
brota	is the/a brother of
sistu	is the/a sister of
mamta	is the/a mother of
patfu	is the/a father of
grafa	is the/a grandfather of
grama	is the/a grandmother of
wanto	is the/a aunt of
tunke	is the/a uncle of

Cousins are all called brother/sister. None of these words are inherently gendered and brota can refer to a female or nonbinary person. The words are separate because I feel it flows better, for now at least.

Idioms

L'ewa strives to have as few idioms as possible. If something is meant non-literally (or as a conceptual metaphor), the particle ke'a can be used:

ti firgu
This is beneificial

ti ke'a firgu
This is metaphorically/non-literally beneficial

---

I have been documenting L'ewa and all of its words/grammar in a git repo. The layout of this repo is as follows:

Folder	Purpose
book	The source files and build scripts for the L'ewa book (this book may end up being published)
nix	Nix crud, custom packages for the eBook render and development tools
script	Where experiments for the written form of L'ewa live
tools	Tools for hacking at L'ewa in Rust/Typescript (none published yet, this is where the dictionary server code will live)
words	Where the definitions of each word are defined in Dhall, this will be fed into the dictionary server code

I also have the entire process of building and testing everything (from the eBook to the unit tests of the tools) automated with Drone. You can see the past builds here. After I merge the information from the latest blogpost into this repo, I will put a rendered version of it here. This will allow you to browse through the chapters of the eBook while it is being written. Eventually this will be automatically deployed to my Kubernetes cluster and the book will be a subpath/subdomain of lewa.christine.website.

I have created a system of defining words that allows you to focus on each word at once, but then fit it back into the greater whole of the language. For example here is kirta.dhall:

-- kirta.dhall
let ContentWord = ../types/ContentWord.dhall

in  ContentWord::{
    , word = "kirta"
    , gloss = "Creator"
    , definition =
        "is an infinite slice of an even greater infinite/our Creator/a Creator"
    }

This is put in words/roots because it is a root (or uncombined) word. Then it is added to the dictionary.dhall:

-- dictionary.dhall
let ContentWord = ./types/ContentWord.dhall

let ParticleWord = ./types/ParticleWord.dhall

in  { rootWords =
      [ -- ...
      ./roots/kirta.dhall
      -- ...
      ]
    , particles [ -- ...
    ]

And then the build process will automatically generate the new dictionary from all of these definitions. Downside of this is that each new kind of word needs subtle adjustments to the build process of the dictionary and that removals/changes to lots of words requires a larger-scale refactor of the language, but I feel the tradeoff is worth the effort. I will undoubtedly end up creating a few tools to help with this.

I will keep working on additional vocabulary on my own, but here is the list of vocabulary that has been written up so far.

Be well.

How HTTP Requests Work

Reading this webpage is possible because of millions of hours of effort with tens of thousands of actors across thousands of companies. At some level it's a minor miracle that this all works at all. Here's a preview into the madness that goes into hitting enter on christine.website and this website being loaded.

Beginnings

The user types in https://christine.website into the address bar and hits enter on the keyboard. This sends a signal over USB to the computer and the kernel polls the USB controller for a new message. It's recognized as from the keyboard. The input is then sent to the browser through an input driver talking to a windowing server talking to the browser program.

The browser selects the memory region normally reserved for the address bar. The browser then parses this string as an RFC 3986 URI and scrapes out the protocol (https), hostname (christine.website) and path (/). The browser then uses this information to create an abstract HTTP request object with the Host header set to christine.website, HTTP method (GET), and path set to the path. This request object then passes through various layers of credential storage and middleware to add the appropriate cookies and other headers in order to tell my website what language it should localize the response to, what compression methods the browser understands, and what browser is being used to make the request.

Connections

The browser then checks if it has a connection to christine.website open already. If it does not, then it creates a new one. It creates a new connection by figuring out what the IP address of christine.website is using DNS. A DNS request is made over UDP on port 53 to the DNS server configured in the operating system (such as 8.8.8.8, 1.1.1.1 or 75.75.75.75). The UDP connection is created using operating system-dependent system calls and a DNS request is sent.

The packet that was created then is destined for the DNS server and added to the operating system's output queue. The operating system then looks in its routing table to see where the packet should go. If the packet matches a route, it is queued for output to the relevant network card. The network card layer then checks the ARP table to see what mac address the ethernet frame should be sent to. If the ARP table doesn't have a match, then an arp probe is broadcasted to every node on the local network. Then the driver waits for an arp response to be sent to it with the correct IP -> MAC address mapping. The driver then uses this information to send out the ethernet frame to the node that matches the IP address in the routing table. From there the packet is validated on the router it was sent to. It then unwraps the packet to the IP layer to figure out the destination network interface to use. If this router also does NAT termination, it creates an entry in the NAT table for future use for a site-configured amount of time (for UDP at least). It then passes the packet on to the correct node and this process is repeated until it gets to the remote DNS server.

The DNS server then unwraps the ethernet frame into an IP packet and then as a UDP packet and a DNS request. It checks its database for a match and if one is not found, it attempts to discover the correct name server to contact by using a NS record query to its upstreams or the authoritative name server for the WEBSITE namespace. This then creates another process of ethernet frames and UDP packets until it reaches the upstream DNS server which hopefully should reply with the correct address. Once the DNS server gets the information that is needed, it sends this back the results to the client as a wire-format DNS response.

UDP is unreliable by design, so this packet may or may not survive the entire round trip. It may take one or more retries for the DNS information to get to the remote server and back, but it usually works the first time. The response to this request is cached based on the time-to-live specified in the DNS response. The response also contains the IP address of christine.website.

Security

The protocol used in the URL determines which TCP port the browser connects to. If it is http, it uses port 80. If it is https, it uses port 443. The user specified HTTPS, so port 443 on whatever IP address DNS returned is dialed using the operating system's network stack system calls. The TCP three-way handshake is started with that target IP address and port. The client sends a SYN packet, the server replies with a SYN ACK packet and the client replies with an ACK packet. This indicates that the entire TCP session is active and data can be transferred and read through it.

However, this data is UNENCRYPTED by default. Transport Layer Security is used to encrypt this data so prying eyes can't look into it. TLS has its own handshake too. The session is established by sending a TLS ClientHello packet with the domain name (christine.website), the list of ciphers the client supports, any application layer protocols the client supports (like HTTP/2) and the list of TLS versions that the client supports. This information is sent over the wire to the remote server using that entire long and complicated process that I spelled out for how DNS works, except a TCP session requires the other side to acknowledge when data is successfully received. The server on the other end replies with a ClientHelloResponse that contains a HTTPS certificate and the list of protocols and ciphers the server supports. Then they do an encryption session setup rain dance that I don't completely understand and the resulting channel is encrypted with cipher (or encrypted) text written and read from the wire and a session layer translates that cipher text to clear text for the other parts of the browser stack.

The browser then uses the information in the ClientHelloResponse to decide how to proceed from here.

HTTP

If the browser notices the server supports HTTP/2 it sets up a HTTP/2 session (with a handshake that involves a few roundtrips like what I described for DNS) and creates a new stream for this request. The browser then formats the request as HTTP/2 wire format bytes (binary format) and writes it to the HTTP/2 stream, which writes it to the HTTP/2 framing layer, which writes it to the encryption layer, which writes it to the network socket and sends it over the internet.

If the browser notices the server DOES NOT support HTTP/2, it formats the request as HTTP/1.1 wire formatted bytes and writes it to the encryption layer, which writes it to the network socket and sends it over the internet using that complicated process I spelled out for DNS.

This then hits the remote load balancer which parses the client HTTP request and uses site-local configuration to select the best application server to handle the response. It then forwards the client's HTTP request to the correct server by creating a TCP session to that backend, writing the HTTP request and waiting for a response over that TCP session. Depending on site-local configuration there may be layers of encryption involved.

Application Server

Now, the request finally gets to the application server. This TCP session is accepted by the application server and the headers are read into memory. The path is read by the application server and the correct handler is chosen. The HTML for the front page of christine.website is rendered and written to the TCP session and travels to the load balancer, gets encrypted with TLS, the encrypted HTML gets sent back over the internet to your browser and then your browser decrypts it and starts to parse and display the website. The browser will run into places where it needs more resources (such as stylesheets or images), so it will make additional HTTP requests to the load balancer to grab those too.

---

The end result is that the user sees the website in all its glory. Given all these moving parts it's astounding that this works as reliably as it does. Each of the TCP, ARP and DNS requests also happen at each level of the stack. There are layers upon layers upon layers of interacting protocols and implementations.

This is why it is hard to reliably put a website on the internet. If there is a god, they are surely the one holding all these potentially unreliable systems together to make everything appear like it is working.

ReConLangMo 5: Sentence Structure

The last post in this series was more of a grammar dump with few concrete examples or much details about things (mostly because of a lack of vocabulary to make examples with). I'll fix this in the future, but for now let's continue on with sentence structure goodness. This is a response to this prompt.

Independent Clause Structure

Most of the time L'ewa sentences have only one clause. This can be anything from a single verb to a subject, verb and object. However, sometimes more information is needed. Consider this sentence:

The dog which is blue is large.

This kind of a relative clause would be denoted using hoi, which would make the sentence roughly the following in L'ewa:

le wufra hoi blanu xi brado.

The particle xi is needed here in order to make it explicit that the subject noun-phrase has ended.

Similarly, an incidental relative clause is done with with joi:

le  wufra  joi              blanu    ke brado
the dog,   which by the way is blue,    is big.

Questions

There are a few ways to ask questions in L'ewa. They correlate to the different kinds of things that the speaker could want to know.

ma

ma is the particle used to fill in a missing/unknown noun phrase. Consider these sentences:

ma   blanu?
what is blue?

ro  qa madsa   ma?
you are eating what?

no

no is the particle used to fill in a missing/unknown verb. Consider these sentences:

ro no?
How are you doing?

le wufra xi no?
The dog did what?

so

so is the particle used to ask questions about numbers, similar to the "how many" construct in English.

ro madsa so spalo?
You ate how many apples?

le so zasko xi qa'te glowa
How many plants grow quickly?

Color Words

L'ewa uses a RGB color system like computers. The basic colors are red, green and blue, with some other basic ones for convenience:

English	L'ewa
blue	blanu
red	delja
green	qalno
yellow	yeplo
teal	te'ra
pink	hetlo
black	xekri
white	pu'ro
50% gray	flego

Colors will be mixed by creating compound words between base colors. Compound words still need to be fleshed out, but generally all CVCCV words will have wordparts made out of the first, second and fifth letter, unless the vowel pair is illegal and all CCVCV words are the first, second and fifth letter unless this otherwise violates the morphology rules. Like I said though, this really needs to be fleshed out and this is only a preview for now.

For example a light green would be puoqa'o (pu'lo qalno, white-green).

---

I hit a snag while hacking at the tooling for making word creation and the like easier. I am still working on it, but most of my word creation is manual and requires me to keep a phonology information document up on my monitor while I sound things out. As part of writing this article I had to add the letters f and r to L'ewa for the word wufra.

I am documenting my work for this language here. This repo will build the grammar book PDF, website and eBook. This will also be the home of the word generation, similarity calculation, dictionary and (eventually) automatic translation tools. I am also documenting each of the words in the language in their own files that will feed into the grammar book generation. More on this when I have more of a coherent product!

ReConLangMo 4: Noun and Verb Morphology

Last time on ReConLangMo I covered word order and some of the finer points about how sentences work. This time we are covering how nouns and verbs get modified (some languages call this conjugation or declension). This is a response to this prompt.

Other Noun Things

At a high level, noun-phrases can be marked for direct ownership or number. The general pattern is like this:

<article> [pronoun] [negation] [number] <verb>

Pronouns

Here's some of the pronouns:

English	L'ewa
me, I	mi
My system and I	mi'a
you	ro
we (all-inclusive)	mi'o
your system and you	ro'a
This (near me)	ti
That (near you)	ta
That (far away)	tu

Numbers

Numbers are in base six. Here are a few numerals:

Decimal	Seximal	L'ewa
0	0	zo
1	1	ja
2	2	he
3	3	xu
4	4	ho
5	5	qi
6	10	jazo
36	100	gau

Here are few non-numerals-but-technically-still-numbers-I-guess:

English	L'ewa
all	to
some	ra'o
number-question	so

Negation

As L'ewa is more of a logical language, it has several forms of negation. Here are a few:

English	L'ewa
contradiction	na
total scalar negation	na'o
particle negation	nai

na can be placed before the sentence's verb too:

ti na spalo
This is something other than an apple

Verb Forms

Verbs have one form in L'ewa. Aspects like tense or the perfective aspect are marked with particles. Here's a table of the common ones:

English	L'ewa
past tense	qu
present tense	qa
future tense	qo
perfective aspect	qe

Modality

Modality is going to be expressed with emotion words. These words have not been assigned yet, but their grammar will be a lot looser than the normal L'ewa particle grammar. They will allow any two vowels in any combination that might otherwise make them not "legal" for particles.

• VV (ii)
• V'V (i'i)

Explicitly Ending Noun Phrases

In case it is otherwise confusing, ko can be used to end noun phrases grammatically.

---

I will probably be fleshing this out some more, but for now this is how all of this works.

ReConLangMo 3: Morphosyntactic Typology

In the last post of this series, we covered the sounds and word patterns of L'ewa. This time we are covering morphosyntactic typology, or how words and sentences are formed out of root words, details about sentences, word order and those kinds of patterns. I'll split each of these into their own headings so it's a bit easier to grok. This is a response to this prompt.

Word Order

L'ewa is normally a Subject-Verb-Object (SVO) language like English. However, the word order of a sentence can be changed if it is important to specify some part of the sentence in particular.

I haven't completely finalized the particles for this, but I'd like to use ka to denote the subject, ke to denote the verb and ku to denote the object. For example if the input sentence is something like:

/mi/ /mad.sa/ /lo/ /spa.lo/
mi   madsa    lo   spalo
 I   eat      an   apple

You could emphasize the eating with:

/kɛ/ /mad.sa/ /ka/ /mi/ /lo/ /spa.lo/
[ke] madsa    ka   mi   lo   spalo
V    eat      S    I    an   apple

(the ke is in square brackets here because it is technically not required, but it can make more sense to be explicit in some cases)

or the apple with:

/ku/ /lo/ /spalo/ /kɛ/ /mad.sa/ /mi
ku   lo   spalo   ke   madsa    mi
O    an   apple   V    eat      I

L'ewa doesn't really have adjectives or adverbs in the normal indo-european sense, but it does have a way to analytically combine meanings together. For example if qa'te is the word for is fast/quick/rapid in rate, then saying you are quickly eating (or wolfing food down) would be something like:

/qaʔ.tɛ/          /mad.sa/
qa'te             madsa
is fast [kind of] eat

These are assumed to be metaphorical by default. It's not always clear what someone would mean by a fast kind of language (would they be referencing Speedtalk?)

L'ewa doesn't always require a subject or object if it can be figured out from context. You can just say "rain" instead of "it's raining". By default, the first word in a sentence without an article is the verb. The ka/ke/ku series needs to be used if the word order deviates from Subject-Verb-Object (it functions a lot like the selma'o FA from Lojban).

Morphological Typology

L'ewa is a analytic language. Every single word has only one form and particles are used to modify the meaning or significance of words. There are only two word classes: content and particles.

Alignment

L'ewa is a nominative-accusative language. Other particles may be introduced in the future to help denote the relations that exist in other alignments, but I don't need them yet.

Word Classes

As said before, L'ewa only has two word classes, content (or verbs) and particles to modify the significance or relations between content. There is also a hard limit of two arguments per verb, which should help avoid the problems that Lojban has with its inconsistent usage of the x3, x4 and x5 places.

As the content words are all technically verbs, there is no real need for a copula. The ka/ke/ku series can also help to break out of other things that modify "noun-phrases" (when those things exist). There are also no nouns, adjectives or adverbs, because analytically combining words completely replaces the need for them.

Nouns and verbs do not inflect for numbers. If numbers are needed they can be provided, otherwise the default is to assume "one or more".

Conscript

I am still working on the finer details of the conscript for L'ewa, but here is a sneak preview of the letter forms I am playing with (this image below might not render properly in light mode):

My inspirations for this script were zbalermorna, Hangul, Hanzi, Katakana, Greek, international computer symbols, traditional Japanese art and the International Phonetic Alphabet.

This script is very decorative, and is primarily intended to be used in spellcraft and other artistic uses. It will probably show up in my art from time to time, and will definitely show up in any experimental video production that I work on in the future. I will go into more detail about this in the future, but here is my prototype. Please do let me know what you think about it.

---

As a side note, the words madsa, spalo and qa'te are now official L'ewa words, I guess. The entire vocabulary of the language can now be listed below:

Content Words

L'ewa word	IPA	English
l'ewa	/lʔ.ɛwa/	is a language
madsa	/mad.sa/	eats/is eating
qa'te	/qaʔ.tɛ/	is fast/quick/rapid in rate
zasko	/ʒa.sko/	is a plant/is vegetation
spalo	/spa.lo/	is an apple

Particles

L'ewa word	IPA	English
lo	/lo/	a, an, indefinite article
le	/lɛ/	the, definite article
ka	/ka/	subject marker
ke	/kɛ/	verb marker
ku	/ku/	object marker
mi	/mi/	the current speaker

Gamebridge: Fitting Square Pegs into Round Holes since 2020

Recently I did a stream called Twitch Plays Super Mario 64. During that stream I both demonstrated and hacked on a tool I'm calling gamebridge. Gamebridge is a tool that lets you allow games to interoperate with programs they really shouldn't be able to interoperate with.

Gamebridge works by aggressively hooking into a game's input logic (through a custom controller driver) and uses a pair of Unix fifos to communicate between it and the game it is controlling. Overall the flow of data between the two programs looks like this:

You can view the source code of this diagram in GraphViz dot format here.

The main magic that keeps this glued together is the use of blocking I/O. This means that the bridge input thread will be blocked at the kernel level for the vblank signal to be written, and the game will also be blocked at the kernel level for the bridge input thread to write the desired input. This effectively uses the Linux kernel to pass around a scheduling quantum like you would in the L4 microkernel. This design consideration also means that gamebridge has to perform as fast as possible as much as possible, because it realistically only has a few hundred microseconds at best to respond with the input data to avoid humans noticing any stutter. As such, gamebridge is written in Rust.

Implementation

When implementing gamebridge, I had a few goals in mind:

• Use blocking I/O to have the kernel help with this
• Use threads to their fullest potential
• Unix fifos are great, let's use them
• Understand linear interpolation better
• Create a surreal demo on Twitch
• Only have one binary to start, the game itself

As a first step of implementing this, I went through the source code of the Mario 64 PC port (but in theory this could also work for other emulators or even Nintendo 64 emulators with enough work) and began to look for anything that might be useful to understand how parts of the game work. I stumbled across src/pc/controller and then found two gems that really stood out. I found the interface for adding new input methods to the game and an example input method that read from tool-assisted speedrun recordings. The controller input interface itself is a thing of beauty, I've included a copy of it below:

// controller_api.h
#ifndef CONTROLLER_API
#define CONTROLLER_API

#include <ultra64.h>

struct ControllerAPI {
    void (*init)(void);
    void (*read)(OSContPad *pad);
};

#endif

All you need to implement your own input method is an init function and a read function. The init function is used to set things up and the read function is called every frame to get inputs. The tool-assisted speedrunning input method seemed to conform to the Mupen64 demo file spec as described on tasvideos.org, and I ended up using this to help test and verify ideas.

The thing that struck me was how simple the format was. Every frame of input uses its own four-byte sequence. The constants in the demo file spec also helped greatly as I figured out ways to bridge into the game from Rust. I ended up creating two bitflag structs to help with the button data, which ended up almost being a 1:1 copy of the Mupen64 demo file spec:

bitflags! {
    // 0x0100 Digital Pad Right
    // 0x0200 Digital Pad Left
    // 0x0400 Digital Pad Down
    // 0x0800 Digital Pad Up
    // 0x1000 Start
    // 0x2000 Z
    // 0x4000 B
    // 0x8000 A
    pub(crate) struct HiButtons: u8 {
        const NONE = 0x00;
        const DPAD_RIGHT = 0x01;
        const DPAD_LEFT = 0x02;
        const DPAD_DOWN = 0x04;
        const DPAD_UP = 0x08;
        const START = 0x10;
        const Z_BUTTON = 0x20;
        const B_BUTTON = 0x40;
        const A_BUTTON = 0x80;
    }
}

Input

This is where things get interesting. One of the more interesting side effects of getting inputs over chat for a game like Mario 64 is that you need to hold buttons or even the analog stick in order to do things like jumping into paintings or on ledges. When you get inputs over chat, you only have them for one frame. Therefore you need some kind of analog input (or an emulation of that) that decays over time. One approach you can use for this is linear interpolation (or lerp).

I implemented support for both button and analog stick lerping using a struct I call a Lerper (the file it is in is named au.rs because .au. is the lojban emotion-particle for "to desire", the name was inspired from it seeming to fake what the desired inputs were).

At its core, a Lerper stores a few basic things:

• the current scalar of where the analog input is resting
• the frame number when the analog input was set to the max (or above)
• the maximum number of frames that the lerp should run for
• the goal (or where the end of the linear interpolation is, for most cases in this codebase the goal is 0, or neutral)
• the maximum possible output to return on apply()
• the minimum possible output to return on apply()

Every frame, the lerpers for every single input to the game will get applied down closer to zero. Mario 64 uses two signed bytes to represent the controller input. The maximum/minimum clamps make sure that the lerped result stays in that range.

Twitch Integration

This is one of the first times I have ever used asynchronous Rust in conjunction with synchronous rust. I was shocked at how easy it was to just spin up another thread and have that thread take care of the Tokio runtime, leaving the main thread to focus on input. This is the block of code that handles running the asynchronous twitch bot in parallel to the main thread:

pub(crate) fn run(st: MTState) {
    use tokio::runtime::Runtime;
    Runtime::new()
        .expect("Failed to create Tokio runtime")
        .block_on(handle(st));
}

Then the rest of the Twitch integration is boilerplate until we get to the command parser. At its core, it just splits each chat line up into words and looks for keywords:

let chatline = msg.data.to_string();
let chatline = chatline.to_ascii_lowercase();
let mut data = st.write().unwrap();
const BUTTON_ADD_AMT: i64 = 64;

for cmd in chatline.to_string().split(" ").collect::<Vec<&str>>().iter() {
    match *cmd {
        "a" => data.a_button.add(BUTTON_ADD_AMT),
        "b" => data.b_button.add(BUTTON_ADD_AMT),
        "z" => data.z_button.add(BUTTON_ADD_AMT),
        "r" => data.r_button.add(BUTTON_ADD_AMT),
        "cup" => data.c_up.add(BUTTON_ADD_AMT),
        "cdown" => data.c_down.add(BUTTON_ADD_AMT),
        "cleft" => data.c_left.add(BUTTON_ADD_AMT),
        "cright" => data.c_right.add(BUTTON_ADD_AMT),
        "start" => data.start.add(BUTTON_ADD_AMT),
        "up" => data.sticky.add(127),
        "down" => data.sticky.add(-128),
        "left" => data.stickx.add(-128),
        "right" => data.stickx.add(127),
        "stop" => {data.stickx.update(0); data.sticky.update(0);},
        _ => {},
    }
}

This implements the following commands:

Command	Meaning
a	Press the A button
b	Press the B button
z	Press the Z button
r	Press the R button
cup	Press the C-up button
cdown	Press the C-down button
cleft	Press the C-left button
cright	Press the C-right button
start	Press the start button
up	Press up on the analog stick
down	Press down on the analog stick
left	Press left on the analog stick
stop	Reset the analog stick to center

Currently analog stick inputs will stick for about 270 frames and button inputs will stick for about 20 frames before drifting back to neutral. The start button is special, inputs to the start button will stick for 5 frames at most.

Debugging

Debugging two programs running together is surprisingly hard. I had to resort to the tried-and-true method of using gdb for the main game code and excessive amounts of printf debugging in Rust. The pretty_env_logger crate (which internally uses the env_logger crate, and its environment variable configures pretty_env_logger) helped a lot. One of the biggest problems I encountered in developing it was fixed by this patch, which I will paste inline:

diff --git a/gamebridge/src/main.rs b/gamebridge/src/main.rs
index 426cd3e..6bc3f59 100644
@@ -93,7 +93,7 @@ fn main() -> Result<()> {
                     },
                 };
 
-                sticky = match stickx {
+                sticky = match sticky {
                     0 => sticky,
                     127 => {
                         ymax_frame = data.frame;

Somehow I had been trying to adjust the y axis position of the stick by comparing the x axis position of the stick. Finding and fixing this bug is what made me write the Lerper type.

---

Altogether, this has been a very fun project. I've learned a lot about 3d game design, historical source code analysis and inter-process communication. I also learned a lot about asynchronous Rust and how it can work together with synchronous Rust. I also got to make a fairly surreal demo for Twitch. I hope this can be useful to others, even if it just serves as an example of how to integrate things into strange other things from unixy first principles.

You can find out slightly more about gamebridge on its GitHub page. Its repo also includes patches for the Mario 64 PC port source code, including one that disables the ability for Mario to lose lives. This could prove useful for Twitch plays attempts, the 5 life cap by default became rather limiting in testing.

Be well.

ReConLangMo 2: Phonology & Writing

Continuing from the last post, one of the next steps in this process is to outline the phonology and basic phonotactics of L'ewa. A language's phonology is the set of sounds that are allowed to be in words. The phonotactics of a language help people understand where the boundaries between syllables are. I will then describe my plans for the L'ewa orthography and how L'ewa is romanized. This is a response to the prompt made here.

Phonology

I am taking inspiration from Lojban, Esperanto, Mandarin Chinese and English to design the phonology of L'ewa. All of the phonology will be defined using the International Phonetic Alphabet. If you want to figure out how to pronounce these sounds, a lazy trick is to google them. Wikipedia will have a perfectly good example to use as a reference. There are two kinds of sounds in L'ewa, consonants and vowels.

Consonants

Consonant inventory: /d f g h j k l m n p q s t w ʃ ʒ ʔ ʙ̥/

Manner/Place	Bilabial	Alveolar	Palato-alveolar	Palatal	Velar	Labio-velar	Uvular	Glottal
Nasal	m	n
Stop	p	t d			k g		q	ʔ
Fricative	f	s	ʃ ʒ					h
Approximant				j		w
Trill	ʙ̥	r
Lateral approximant		l

The weirdest consonant is /ʙ̥/, which is a voiceless bilabial trill, or blowing air through your lips without making sound. This is intended to imitate a noise an orca would make.

Vowels

Vowel inventory: /a ɛ i o u/

Diphthongs: au, oi, ua, ue, uo, ai, ɛi

	Front	Back
High	i	u
High-mid		o
Low-mid	ɛ
Low	a

Phonotactics

I plan to have two main kinds of words in L'ewa. I plan to have content and particle words. The content words will refer to things, properties, or actions (such as tool, red, run) and the particle words will change how the grammar of a sentence works (such as the or prepositions).

The main kind of content word is a root word, and they will be in the following forms:

• CVCCV (/ʒa.sko/)
• CCVCV (/lʔ.ɛwa/)

Particles will mostly fall into the following forms:

• V (/a/)
• VV (/ai/)
• CV (/ba/)
• CVV (/bai/)
• CV'V (/baʔ.i)

Proper names should end with consonants, but there is no hard requirement.

L'ewa is a stressed language, with stress on the second-to-last (penultimate) syllable. For example, the word "[z]asko" would be pronounced "[Z]Asko".

Syllables end on stop consonants if one is present in a consonant cluster. Two stop consonants cannot follow eachother in a row.

Writing

I haven't completely fleshed this part out yet, but I want the writing system of L'ewa to be an abugida. This is a kind of written script that has the consonants make the larger shapes but the vowels are small diacritics over the consonants. If the word creation process is done right, you can actually omit the vowels entirely if they are not relevant.

I plan to have this script be written by hand with pencils/pen and typed into computers, just like English. This script will also be a left-to-right script like English.

Romanisation

L'ewa's romanization is intentionally simple. Most of the IPA letters keep their letters, but the ones that do not match to Latin letters are listed below:

Pronunciation	Spelling
/j/	y
/ɛ/	e
/ʃ/	x
/ʒ/	z
/ʔ/	'
/ʙ̥/	b

This is designed to make every letter typeable on a standard US keyboard, as well as mapping as many letters as possible on the home row of a QWERTY keyboard.

---

I am still working on the tooling for word creation and the like. I plan to use the Swaedish lists (this site is having certificate issues at the time of writing this post) to help guide the creation of a base vocabulary. I will go into more detail in the future.

Super Bootable 64

Super Mario 64 was the launch title of the Nintendo 64 in 1996. This game revolutionized an entire generation and everything following it by delivering fast, smooth and fun 3d platforming gameplay to gamers all over the world. This game is still played today by speedrunners, who do everything from beating it while collecting every star, the minimum amount of stars normally required, 0 stars and without pressing the A jump button.

This game was the launch title of the Nintendo 64. As such, the SDK used to develop it was pre-release and had an optimization bug that forced the game to be shipped without optimizations due to random crashiness issues (watch the linked video for more information on this than I can summarize here). Remember that the Nintendo 64 shipped games on write-once ROM cartridges, so any bug that could cause the game to crash randomly was fatal.

When compiling something without optimizations, the output binary is effectively a 1:1 copy of the input source code. This means that exceptionally clever people could theoretically go in, decompile your code and then create identical source code that could be used to create a byte-for-byte identical copy of your program's binary. But surely nobody would do that, that would be crazy, wouldn't it?

Someone did. The fruits of this effort are available here. This was mostly a proof of concept and is a masterpiece in its own right. However, because it was decompiled, this means that the engine itself could theoretically be ported to run on any other platform such as Windows, Linux, the Nintendo Switch or even a browser.

Someone did this and ended up posting it on 4chan. Thanks to a friend, I got my hands on the Linux-compatible source code of this port and made an archive of it on my git server. My fork of it has only minimal changes needed for it to build in NixOS.

nixos-generators is a tool that lets you create custom NixOS system definitions based on a NixOS module as input. So, let's create a bootable ISO of Super Mario 64 running on Linux!

Setup

You will need an amd64 Linux system. NixOS is preferable, but any Linux system should theoretically work. You will also need the following things:

• sm64.us.z64 (the release rom of Super Mario 64 in the US version 1.0) with an sha1 sum of 9bef1128717f958171a4afac3ed78ee2bb4e86ce
• nixos-generators installed (nix-env -f https://github.com/nix-community/nixos-generators/archive/master.tar.gz -i)

So, let's begin by creating a folder named boot2sm64:

$ mkdir ~/code/boot2sm64

Then let's create a file called configuration.nix and put some standard boilerplate into it:

# configuration.nix

{ pkgs, lib, ... }:

{
  networking.hostName = "its-a-me";
}

And then let's add dwm as the window manager. This setup will be a little bit more complicated because we are going to need to add a custom configuration as well as a patch to the source code for auto-starting Super Mario 64. Create a folder called dwm and run the following commands in it to download the config we need and the autostart patch:

$ mkdir dwm
$ cd dwm
$ wget -O autostart.patch https://dwm.suckless.org/patches/autostart/dwm-autostart-20161205-bb3bd6f.diff
$ wget -O config.h https://gist.githubusercontent.com/Xe/f5fae8b7a0d996610707189d2133041f/raw/7043ca2ab5f8cf9d986aaa79c5c505841945766c/dwm_config.h

And then add the following before the opening curly brace:

{ pkgs, lib, ... }:

let
  dwm = with pkgs;
    let name = "dwm-6.2";
    in stdenv.mkDerivation {
      inherit name;

      src = fetchurl {
        url = "https://dl.suckless.org/dwm/${name}.tar.gz";
        sha256 = "03hirnj8saxnsfqiszwl2ds7p0avg20izv9vdqyambks00p2x44p";
      };

      buildInputs = with pkgs; [ xorg.libX11 xorg.libXinerama xorg.libXft ];

      prePatch = ''sed -i "s@/usr/local@$out@" config.mk'';
      
      postPatch = ''
        cp ${./dwm/config.h} ./config.h
      '';

      patches = [ ./dwm/autostart.patch ];

      buildPhase = " make ";

      meta = {
        homepage = "https://suckless.org/";
        description = "Dynamic window manager for X";
        license = stdenv.lib.licenses.mit;
        maintainers = with stdenv.lib.maintainers; [ viric ];
        platforms = with stdenv.lib.platforms; all;
      };
    };
in {
  environment.systemPackages = with pkgs; [ hack-font st dwm ];

  networking.hostName = "its-a-me";
}

Now let's create the mario user:

{
  # ...
  users.users.mario = { isNormalUser = true; };
  
  system.activationScripts = {
    base-dirs = {
      text = ''
        mkdir -p /nix/var/nix/profiles/per-user/mario
      '';
      deps = [ ];
    };
  };
  
  services.xserver.windowManager.session = lib.singleton {
    name = "dwm";
    start = ''
      ${dwm}/bin/dwm &
      waitPID=$!
    '';
  };

  services.xserver.enable = true;
  services.xserver.displayManager.defaultSession = "none+dwm";
  services.xserver.displayManager.lightdm.enable = true;
  services.xserver.displayManager.lightdm.autoLogin.enable = true;
  services.xserver.displayManager.lightdm.autoLogin.user = "mario";
}

The autostart file is going to be located in /home/mario/.dwm/autostart.sh. We could try and place it manually on the filesystem with a NixOS module, or we could use home-manager to do this for us. Let's have home-manager do this for us. First, install home-manager:

$ nix-channel --add https://github.com/rycee/home-manager/archive/release-20.03.tar.gz home-manager
$ nix-channel --update

Then let's add home-manager to this config:

{
  # ...

  imports = [ <home-manager/nixos> ];

  home-manager.users.mario = { config, pkgs, ... }: {
    home.file = {
      ".dwm/autostart.sh" = {
        executable = true;
        text = ''
          #!/bin/sh
          export LIBGL_ALWAYS_SOFTWARE=1 # will be relevant later
        '';
      };
    };
  };
}

Now, for the creme de la creme of this project, let's build Super Mario 64. You will need to get the base rom into your system's Nix store somehow. A half decent way to do this is with quickserv:

$ nix-env -if https://tulpa.dev/Xe/quickserv/archive/master.tar.gz
$ cd /path/to/folder/with/baserom.us.z64
$ quickserv -dir . -port 9001 &
$ nix-prefetch-url http://127.0.0.1:9001/baserom.us.z64

This will pre-populate your Nix store with the rom and should return the following hash:

148xna5lq2s93zm0mi2pmb98qb5n9ad6sv9dky63y4y68drhgkhp

If this hash is wrong, then you need to find the correct rom. I cannot help you with this.

Now, let's create a simple derivation for the Super Mario 64 PC port. I have a tweaked version that is optimized for NixOS, which we will use for this. Add the following between the dwm package define and the in statement:

# ...
  sm64pc = with pkgs;
    let
      baserom = fetchurl {
        url = "http://127.0.0.1:9001/baserom.us.z64";
        sha256 = "148xna5lq2s93zm0mi2pmb98qb5n9ad6sv9dky63y4y68drhgkhp";
      };
    in stdenv.mkDerivation rec {
      pname = "sm64pc";
      version = "latest";

      buildInputs = [
        gnumake
        python3
        audiofile
        pkg-config
        SDL2
        libusb1
        glfw3
        libgcc
        xorg.libX11
        xorg.libXrandr
        libpulseaudio
        alsaLib
        glfw
        libGL
        unixtools.hexdump
      ];

      src = fetchgit {
        url = "https://tulpa.dev/saved/sm64pc";
        rev = "c69c75bf9beed9c7f7c8e9612e5e351855065120";
        sha256 = "148pk9iqpcgzwnxlcciqz0ngy6vsvxiv5lp17qg0bs7ph8ly3k4l";
      };

      buildPhase = ''
        chmod +x ./extract_assets.py
        cp ${baserom} ./baserom.us.z64
        make
      '';

      installPhase = ''
        mkdir -p $out/bin
        cp ./build/us_pc/sm64.us.f3dex2e $out/bin/sm64pc
      '';

      meta = with stdenv.lib; {
        description = "Super Mario 64 PC port, requires rom :)";
      };
    };
# ...

And then add sm64pc to the system packages:

{
  # ...
  environment.systemPackages = with pkgs; [ st hack-font dwm sm64pc ];
  # ...
}

As well as to the autostart script from before:

{
  # ...
  home-manager.users.mario = { config, pkgs, ... }: {
    home.file = {
      ".dwm/autostart.sh" = {
        executable = true;
        text = ''
          #!/bin/sh
          export LIBGL_ALWAYS_SOFTWARE=1
          ${sm64pc}/bin/sm64pc
        '';
      };
    };
  };

}

Finally let's enable some hardware support so it's easier to play this bootable game:

{
  # ...
  
  hardware.pulseaudio.enable = true;
  virtualisation.virtualbox.guest.enable = true;
  virtualisation.vmware.guest.enable = true;
}

Altogether you should have a configuration.nix that looks like this.

So let's build the ISO!

$ nixos-generate -f iso -c configuration.nix

Much output later, you will end up with a path that will look something like this:

/nix/store/fzk3psrd3m6x437m6xh9pc7bnv2v44ax-nixos.iso/iso/nixos.iso

This is your bootable image of Super Mario 64. Copy it to a good temporary folder (like your downloads folder):

cp /nix/store/fzk3psrd3m6x437m6xh9pc7bnv2v44ax-nixos.iso/iso/nixos.iso ~/Downloads/mario64.iso

Now you are free to do whatever you want with this, including booting it in a virtual machine.

This is why I use NixOS. It enables me to do absolutely crazy things like creating a bootable ISO of Super Mario 64 without having to understand how to create ISO files by hand or how bootloaders on Linux work in ISO files.

It Just Works.

ReConLangMo 1: Name, Context, History

I've been curious about how language works for a very long time. This curiosity has lead me down many fascinating rabbit holes, but for a long time I have either been cribbing off of other people's work or studying natural languages that don't have a cohesive plan or core to them. Constructed Languages (or conlangs as I will probably be calling them from here on out) are a simpler model of this. You might be familiar with Klingon from the Star Trek series, the various forms of Elvish as described by J. R. R. Tolkien or Dothraki from Game of Thrones. This series will show an example of how one of those kinds of languages are created.

Recently a challenge came up on /r/conlangs called ReConLangMo and I've decided to take a stab at this and flesh this out into a personal language.

This post will be the first in a series (with articles to be listed below) and is following the prompt made here.

L'ewa Overview

The language I am going to create will be called L'ewa (⁄l.ʔɛ.wa⁄, also romanized lewa for filesystems). This word is identical in English and in L'ewa. It means "is a language". The name came to me in a shower a while ago and I'm not entirely sure where it came from.

This language is being designed as a personal language to help me keep a diary (more on that later) and to act as a testbed for writing a computational knowledge engine, much like IBM's Watson. I do not expect anyone else to use this language. I may pull this language into fiction (if that ever gets off the ground) or into other projects as it makes sense.

Some of the high level things I want to try in this language are ways to make me think differently. I'm following the weak form of the Sapir-Whorf hypothesis by this logic. I want to see what would happen if I give myself a tool that I can use to help myself think in different ways. Other features I plan to include are:

• A seximal number system
• A predicate-argument system similar to Lojban
• Nounlessness (only having verbs for content words) like Salishan languages
• An a-priori (or made up) vocabulary
• Grammatical markers for the identity of the thinker of a sentence/phrase/word
• Make each grammatical feature and word logical, or working in one way only
• Typeable with standard QWERTY en-US keyboards
• A decorative script that I'll turn into a font

L'wea as A Diary Language

When I was younger, I used to keep a diary/journal file on my computers off and on. I was detailed about what I was feeling and what I was considering and going through. This all ended abruptly after my parents were snooping through my computer in middle school and discovered that I was questioning fundamental aspects of myself like my gender. I have never really felt comfortable keeping a diary file since then. I have made a few attempts at this (including by using a dedicated diary machine, air-gapped TempleOS machines and the like), but they all feel too vulnerable and open for anyone to read them.

This is my logic for using a language that I create for myself. If people really want to go through and take the time to learn the ins and outs of a tool I created for myself to archive my personal thoughts, they probably deserve to be able to read them. Otherwise, this would allow me to write my diary from pretty much anywhere, even in plain sight out in public. People can't shoulder-surf and read what they literally cannot understand.

---

I plan to continue going through this series as the prompts come out and will put my responses on my blog along with explanations, analysis and sample code (where relevant). I will probably also reformat these posts (and relevant dictionary files) to an eBook and later into a reference grammar book.

Like I said though, this project is for myself. I do not expect this language to change the world for anyone but me. Let's see where this rabbit hole goes.

My NixOS Desktop Flow

Before I built my current desktop, I had been using a 2013 Mac Pro for at least 7 years. This machine has seen me through living in a few cities (Bellevue, Mountain View and Montreal), but it was starting to show its age. Its 12 core Xeon is really no slouch (scoring about 5 minutes in my "compile the linux kernel" test), but with Intel security patches it was starting to get slower and slower as time went on.

So in March (just before the situation started) I ordered the parts for my new tower and built my current desktop machine. From the start, I wanted it to run Linux and have 64 GB of ram, mostly so I could write and test programs without having to worry about ram exhaustion.

When the parts were almost in, I had decided to really start digging into NixOS. Friends on IRC and Discord had been trying to get me to use it for years, and I was really impressed with a simple setup that I had in a virtual machine. So I decided to jump head-first down that rabbit hole, and I'm honestly really glad I did.

NixOS is built on a more functional approach to package management called Nix. Parts of the configuration can be easily broken off into modules that can be reused across machines in a deployment. If Ansible or other tools like it let you customize an existing Linux distribution to meet your needs, NixOS allows you to craft your own Linux distribution around your needs.

Unfortunately, the Nix and NixOS documentation is a bit more dense than most other Linux programs/distributions are, and it's a bit easy to get lost in it. I'm going to attempt to explain a lot of the guiding principles behind Nix and NixOS and how they fit into how I use NixOS on my desktop.

What is a Package?

Earlier, I mentioned that Nix is a functional package manager. This means that Nix views packages as a combination of inputs to get an output:

This is how most package managers work (even things like Windows installer files), but Nix goes a step further by disallowing package builds to access the internet. This allows Nix packages to be a lot more reproducible; meaning if you have the same inputs (source code, build script and patches) you should always get the same output byte-for-byte every time you build the same package at the same version.

A Simple Package

Let's consider a simple example, my gruvbox-inspired CSS file's default.nix file':

{ pkgs ? import <nixpkgs> { } }:

pkgs.stdenv.mkDerivation {
  pname = "gruvbox-css";
  version = "latest";
  src = ./.;
  phases = "installPhase";
  installPhase = ''
    mkdir -p $out
    cp -rf $src/gruvbox.css $out/gruvbox.css
  '';
}

This creates a package named gruvbox-css with the version latest. Let's break this down its default.nix line by line:

{ pkgs ? import <nixpkgs> { } }:

This creates a function that either takes in the pkgs object or tells Nix to import the standard package library nixpkgs as pkgs. nixpkgs includes a lot of utilities like a standard packaging environment, special builders for things like snaps and Docker images as well as one of the largest package sets out there.

pkgs.stdenv.mkDerivation {
  # ...
}

This runs the stdenv.mkDerivation function with some arguments in an object. The "standard environment" comes with tools like GCC, bash, coreutils, find, sed, grep, awk, tar, make, patch and all of the major compression tools. This means that our package builds can build C/C++ programs, copy files to the output, and extract downloaded source files by default. You can add other inputs to this environment if you need to, but for now it works as-is.

Let's specify the name and version of this package:

pname = "gruvbox-css";
version = "latest";

pname stands for "package name". It is combined with the version to create the resulting package name. In this case it would be gruvbox-css-latest.

Let's tell Nix how to build this package:

src = ./.;
phases = "installPhase";
installPhase = ''
  mkdir -p $out
  cp -rf $src/gruvbox.css $out/gruvbox.css
'';

The src attribute tells Nix where the source code of the package is stored. Sometimes this can be a URL to a compressed archive on the internet, sometimes it can be a git repo, but for now it's the current working directory ./..

This is a CSS file, it doesn't make sense to have to build these, so we skip the build phase and tell Nix to directly install the package to its output folder:

mkdir -p $out
cp -rf $src/gruvbox.css $out/gruvbox.css

This two-liner shell script creates the output directory (usually exposed as $out) and then copies gruvbox.css into it. When we run this through Nix withnix-build, we get output that looks something like this:

$ nix-build ./default.nix
these derivations will be built:
  /nix/store/c99n4ixraigf4jb0jfjxbkzicd79scpj-gruvbox-css.drv
building '/nix/store/c99n4ixraigf4jb0jfjxbkzicd79scpj-gruvbox-css.drv'...
installing
/nix/store/ng5qnhwyrk9zaidjv00arhx787r0412s-gruvbox-css

And /nix/store/ng5qnhwyrk9zaidjv00arhx787r0412s-gruvbox-css is the output package. Looking at its contents with ls, we see this:

$ ls /nix/store/ng5qnhwyrk9zaidjv00arhx787r0412s-gruvbox-css
gruvbox.css

A More Complicated Package

For a more complicated package, let's look at the build directions of the website you are reading right now:

{ pkgs ? import (import ./nix/sources.nix).nixpkgs }:
with pkgs;

assert lib.versionAtLeast go.version "1.13";

buildGoPackage rec {
  pname = "christinewebsite";
  version = "latest";
  
  goPackagePath = "christine.website";
  src = ./.;
  goDeps = ./nix/deps.nix;
  allowGoReference = false;

  preBuild = ''
    export CGO_ENABLED=0
    buildFlagsArray+=(-pkgdir "$TMPDIR")
  '';

  postInstall = ''
    cp -rf $src/blog $bin/blog
    cp -rf $src/css $bin/css
    cp -rf $src/gallery $bin/gallery
    cp -rf $src/signalboost.dhall $bin/signalboost.dhall
    cp -rf $src/static $bin/static
    cp -rf $src/talks $bin/talks
    cp -rf $src/templates $bin/templates
  '';
}

Breaking it down, we see some similarities to the gruvbox-css package from above, but there's a few more interesting lines I want to point out:

{ pkgs ? import (import ./nix/sources.nix).nixpkgs }:

My website uses a pinned or fixed version of nixpkgs. This allows my website's deployment to be stable even if nixpkgs changes something that could cause it to break.

with pkgs;

With expressions are one of the more interesting parts of Nix. Essentially, they let you say "everything in this object should be put into scope". So if you have an expression that does this:

let
  foo = {
    ponies = "awesome";
  };
in with foo; "ponies are ${ponies}!"

You get the result "ponies are awesome!". I use with pkgs here to use things directly from nixpkgs without having to say pkgs. in front of a lot of things.

assert lib.versionAtLeast go.version "1.13";

This line will make the build fail if Nix is using any Go version less than 1.13. I'm pretty sure my website's code could function on older versions of Go, but the runtime improvements are important to it, so let's fail loudly just in case.

buildGoPackage {
  # ...
}

buildGoPackage builds a Go package into a Nix package. It takes in the Go package path, list of dependencies and if the resulting package is allowed to depend on the Go compiler or not.

It will then compile the Go program (and all of its dependencies) into a binary and put that in the resulting package. This website is more than just the source code, it's also got assets like CSS files and the image earlier in the post. Those files are copied in the postInstall phase:

postInstall = ''
  cp -rf $src/blog $bin/blog
  cp -rf $src/css $bin/css
  cp -rf $src/gallery $bin/gallery
  cp -rf $src/signalboost.dhall $bin/signalboost.dhall
  cp -rf $src/static $bin/static
  cp -rf $src/talks $bin/talks
  cp -rf $src/templates $bin/templates
'';

This results in all of the files that my website needs to run existing in the right places.

Other Packages

For more kinds of packages that you can build, see the Languages and Frameworks chapter of the nixpkgs documentation.

If your favorite language isn't shown there, you can make your own build script and do it more manually. See here for more information on how to do that.

nix-env And Friends

Building your own packages is nice and all, but what about using packages defined in nixpkgs? Nix includes a few tools that help you find, install, upgrade and remove packages as well as nix-build to build new ones.

nix search

When looking for a package to install, use $ nix search name to see if it's already packaged. For example, let's look for graphviz, a popular diagramming software:

$ nix search graphviz

* nixos.graphviz (graphviz)
  Graph visualization tools

* nixos.graphviz-nox (graphviz)
  Graph visualization tools

* nixos.graphviz_2_32 (graphviz)
  Graph visualization tools

There are several results here! These are different because sometimes you may want some features of graphviz, but not all of them. For example, a server installation of graphviz wouldn't need X windows support.

The first line of the output is the attribute. This is the attribute that the package is imported to inside nixpkgs. This allows multiple packages in different contexts to exist in nixpkgs at the same time, for example with python 2 and python 3 versions of a library.

The second line is a description of the package from its metadata section.

The nix tool allows you to do a lot more than just this, but for now this is the most important thing.

nix-env -i

nix-env is a rather big tool that does a lot of things (similar to pacman in Arch Linux), so I'm going to break things down into separate sections.

Let's pick an instance graphviz from before and install it using nix-env:

$ nix-env -iA nixos.graphviz
installing 'graphviz-2.42.2'
these paths will be fetched (5.00 MiB download, 13.74 MiB unpacked):
  /nix/store/980jk7qbcfrlnx8jsmdx92q96wsai8mx-gts-0.7.6
  /nix/store/fij1p8f0yjpv35n342ii9pwfahj8rlbb-graphviz-2.42.2
  /nix/store/jy35xihlnb3az0vdksyg9rd2f38q2c01-libdevil-1.7.8
  /nix/store/s895dnwlprwpfp75pzq70qzfdn8mwfzc-lcms-1.19
copying path '/nix/store/980jk7qbcfrlnx8jsmdx92q96wsai8mx-gts-0.7.6' from 'https://cache.nixos.org'...
copying path '/nix/store/s895dnwlprwpfp75pzq70qzfdn8mwfzc-lcms-1.19' from 'https://cache.nixos.org'...
copying path '/nix/store/jy35xihlnb3az0vdksyg9rd2f38q2c01-libdevil-1.7.8' from 'https://cache.nixos.org'...
copying path '/nix/store/fij1p8f0yjpv35n342ii9pwfahj8rlbb-graphviz-2.42.2' from 'https://cache.nixos.org'...
building '/nix/store/r4fqdwpicqjpa97biis1jlxzb4ywi92b-user-environment.drv'...
created 664 symlinks in user environment

And now let's see where the dot tool from graphviz is installed to:

$ which dot
/home/cadey/.nix-profile/bin/dot

$ readlink /home/cadey/.nix-profile/bin/dot
/nix/store/fij1p8f0yjpv35n342ii9pwfahj8rlbb-graphviz-2.42.2/bin/dot

This lets you install tools into the system-level Nix store without affecting other user's environments, even if they depend on a different version of graphviz.

nix-env -e

nix-env -e lets you uninstall packages installed with nix-env -i. Let's uninstall graphviz:

$ nix-env -e graphviz

Now the dot tool will be gone from your shell:

$ which dot
which: no dot in (/run/wrappers/bin:/home/cadey/.nix-profile/bin:/etc/profiles/per-user/cadey/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin)

And it's like graphviz was never installed.

Notice that these package management commands are done at the user level because they are only affecting the currently logged-in user. This allows users to install their own editors or other tools without having to get admins involved.

Adding up to NixOS

NixOS builds on top of Nix and its command line tools to make an entire Linux distribution that can be perfectly crafted to your needs. NixOS machines are configured using a configuration.nix file that contains the following kinds of settings:

• packages installed to the system
• user accounts on the system
• allowed SSH public keys for users on the system
• services activated on the system
• configuration for services on the system
• magic unix flags like the number of allowed file descriptors per process
• what drives to mount where
• network configuration
• ACME certificates

and so much more

At a high level, machines are configured by setting options like this:

# basic-lxc-image.nix
{ config, pkgs, ... }:

{
  networking.hostName = "example-for-blog";
  environment.systemPackages = with pkgs; [ wget vim ];
}

This would specify a simple NixOS machine with the hostname example-for-blog and with wget and vim installed. This is nowhere near enough to boot an entire system, but is good enough for describing the base layout of a basic LXC image.

For a more complete example of NixOS configurations, see here or repositories on this handy NixOS wiki page.

The main configuration.nix file (usually at /etc/nixos/configuration.nix) can also import other NixOS modules using the imports attribute:

# better-vm.nix
{ config, pkgs, ... }:

{
  imports = [
    ./basic-lxc-image.nix
  ];
  
  networking.hostName = "better-vm";
  services.nginx.enable = true;
}

And the better-vm.nix file would describe a machine with the hostname better-vm that has wget and vim installed, but is also running nginx with its default configuration.

Internally, every one of these options will be fed into auto-generated Nix packages that will describe the system configuration bit by bit.

nixos-rebuild

One of the handy features about Nix is that every package exists in its own part of the Nix store. This allows you to leave the older versions of a package laying around so you can roll back to them if you need to. nixos-rebuild is the tool that helps you commit configuration changes to the system as well as roll them back.

If you want to upgrade your entire system:

$ sudo nixos-rebuild switch --upgrade

This tells nixos-rebuild to upgrade the package channels, use those to create a new base system description, switch the running system to it and start/restart/stop any services that were added/upgraded/removed during the upgrade. Every time you rebuild the configuration, you create a new "generation" of configuration that you can roll back to just as easily:

$ sudo nixos-rebuild switch --rollback

Garbage Collection

As upgrades happen and old generations pile up, this may end up taking up a lot of unwanted disk (and boot menu) space. To free up this space, you can use nix-collect-garbage:

$ sudo nix-collect-garbage
< cleans up packages not referenced by anything >

$ sudo nix-collect-garbage -d
< deletes old generations and then cleans up packages not referenced by anything >

The latter is a fairly powerful command and can wipe out older system states. Only run this if you are sure you don't want to go back to an older setup.

How I Use It

Each of these things builds on top of eachother to make the base platform that I built my desktop environment on. I have the configuration for my shell, emacs, my window manager and just about every program I use on a regular basis defined in their own NixOS modules so I can pick and choose things for new machines.

When I want to change part of my config, I edit the files responsible for that part of the config and then rebuild the system to test it. If things work properly, I commit those changes and then continue using the system like normal.

This is a little bit more work in the short term, but as a result I get a setup that is easier to recreate on more machines in the future. It took me a half hour or so to get the configuration for zathura right, but now I have a zathura module that lets me get exactly the setup I want every time.

TL;DR

Nix and NixOS ruined me. It's hard to go back.

Chicken Stir Fry

This recipe was made up by me and my fiancé. We just sorta winged it every time we made it until we found something that was easy to cook and tasty. We make this every week or so.

Recipe

Ingredients

• Pack of 4 chicken breasts
• A fair amount of Montreal seasoning (garlic, onion, salt, oregano)
• 3 cups basmati rice
• 3.75 cups water
• 1/4th bag of frozen stir fry vegetables
• Avocado/coconut oil
• Standard frying pan
• Standard chef's knife
• Standard 11x14 cutting board
• Two metal bowls
• Instant Pot
• Spatula

Seasoning

Put the seasoning in one of the bowls and unwrap the plastic around the chicken breasts. Take each chicken breast out of the package (you may need to cut them free of eachother, use a sharp knife for that) and rub all sides of it around in the seasoning.

Put these into the other metal bowl and when you've done all four, cover with plastic wrap and refrigerate for about 5-6 hours.

Doing this helps to have the chicken soak up the flavor of the seasoning so it tastes better when you cook it.

Cooking

Slice two chicken breasts up kinda like this and then transfer to the heated pan with oil in it. Cook those and flip them every few minutes until you've cooked everything all the way through (random sampling by cutting a bit of chicken in half with the spatula and seeing if it's overly juicy or not is a good way to tell, or if you have a food thermometer to 165 degrees fahrenheit or 75 degrees celsius). Put this chicken into a plastic container for use in other meals (it goes really good on sandwiches).

Then repeat the slicing and cooking for the last two chicken breasts. However, this time put half of the chicken into the plastic container you used before (about one chicken breast worth in total, it doesn't have to be exact). At the same time as the second round of chicken is cooking, put about 3 cups of rice and 3.75 cups of water into the instant pot; then seal it and set it to manual for 4 minutes.

Dump frozen vegetables on top of the remainder of the chicken and stir until the vegetables are warm.

Serving

Serve the stir fry hot on a bed of rice.

pa'i Benchmarks

In my last post I mentioned that pa'i was faster than Olin's cwa binary written in go without giving any benchmarks. I've been working on new ways to gather and visualize these benchmarks, and here they are.

Benchmarking WebAssembly implementations is slightly hard. A lot of existing benchmark tools simply do not run in WebAssembly as is, not to mention inside the Olin ABI. However, I have created a few tasks that I feel represent common tasks that pa'i (and later wasmcloud) will run:

• compressing data with Snappy
• parsing JSON
• parsing yaml
• recursive fibbonacci number calculation
• blake-2 hashing

As always, if you don't trust my numbers, you don't have to. Commands will be given to run these benchmarks on your own hardware. This may not be the most scientifically accurate benchmarks possible, but it should help to give a reasonable idea of the speed gains from using Rust instead of Go.

You can run these benchmarks in the docker image xena/pahi. You may need to replace ./result/ with / for running this inside Docker.

$ docker run --rm -it xena/pahi bash -l

Compressing Data with Snappy

This is implemented as cpustrain.wasm. Here is the source code used in the benchmark:

#![no_main]
#![feature(start)]

extern crate olin;

use olin::{entrypoint, Resource};
use std::io::Write;

entrypoint!();

fn main() -> Result<(), std::io::Error> {
    let fout = Resource::open("null://").expect("opening /dev/null");
    let data = include_bytes!("/proc/cpuinfo");

    let mut writer = snap::write::FrameEncoder::new(fout);

    for _ in 0..256 {
        // compressed data
        writer.write(data)?;
    }

    Ok(())
}

This compresses my machine's copy of /proc/cpuinfo 256 times. This number was chosen arbitrarily.

Here are the results I got from the following command:

$ hyperfine --warmup 3 --prepare './result/bin/pahi result/wasm/cpustrain.wasm' \
        './result/bin/cwa result/wasm/cpustrain.wasm' \
        './result/bin/pahi --no-cache result/wasm/cpustrain.wasm' \
        './result/bin/pahi result/wasm/cpustrain.wasm'

CPU	cwa	pahi --no-cache	pahi	multiplier
Ryzen 5 3600	2.392 seconds	38.6 milliseconds	17.7 milliseconds	pahi is 135 times faster than cwa
Intel Xeon E5-1650	7.652 seconds	99.3 milliseconds	53.7 milliseconds	pahi is 142 times faster than cwa

Parsing JSON

This is implemented as bigjson.wasm. Here is the source code of the benchmark:

#![no_main]
#![feature(start)]

extern crate olin;

use olin::entrypoint;
use serde_json::{from_slice, to_string, Value};

entrypoint!();

fn main() -> Result<(), std::io::Error> {
    let input = include_bytes!("./bigjson.json");

    if let Ok(val) = from_slice(input) {
        let v: Value = val;
        if let Err(_why) = to_string(&v) {
            return Err(std::io::Error::new(
                std::io::ErrorKind::Other,
                "oh no json encoding failed!",
            ));
        }
    } else {
        return Err(std::io::Error::new(
            std::io::ErrorKind::Other,
            "oh no json parsing failed!",
        ));
    }

    Ok(())
}

This decodes and encodes this rather large json file. This is a very large file (over 64k of json) and should represent over 65536 times times the average json payload size.

Here are the results I got from the following command:

$ hyperfine --warmup 3 --prepare './result/bin/pahi result/wasm/bigjson.wasm' \
        './result/bin/cwa result/wasm/bigjson.wasm' \
        './result/bin/pahi --no-cache result/wasm/bigjson.wasm' \
        './result/bin/pahi result/wasm/bigjson.wasm'

CPU	cwa	pahi --no-cache	pahi	multiplier
Ryzen 5 3600	257 milliseconds	49.4 milliseconds	20.4 milliseconds	pahi is 12.62 times faster than cwa
Intel Xeon E5-1650	935.5 milliseconds	135.4 milliseconds	101.4 milliseconds	pahi is 9.22 times faster than cwa

Parsing yaml

This is implemented as k8sparse.wasm. Here is the source code of the benchmark:

#![no_main]
#![feature(start)]

extern crate olin;

use olin::entrypoint;
use serde_yaml::{from_slice, to_string, Value};

entrypoint!();

fn main() -> Result<(), std::io::Error> {
    let input = include_bytes!("./k8sparse.yaml");

    if let Ok(val) = from_slice(input) {
        let v: Value = val;
        if let Err(_why) = to_string(&v) {
            return Err(std::io::Error::new(
                std::io::ErrorKind::Other,
                "oh no yaml encoding failed!",
            ));
        } else {
            return Err(std::io::Error::new(
                std::io::ErrorKind::Other,
                "oh no yaml parsing failed!",
            ));
        }
    }

    Ok(())
}

This decodes and encodes this kubernetes manifest set from my cluster. This is a set of a few normal kubernetes deployments and isn't as much of a worse-case scenario as it could be with the other tests.

Here are the results I got from running the following command:

$ hyperfine --warmup 3 --prepare './result/bin/pahi result/wasm/k8sparse.wasm' \
        './result/bin/cwa result/wasm/k8sparse.wasm' \
        './result/bin/pahi --no-cache result/wasm/k8sparse.wasm' \
        './result/bin/pahi result/wasm/k8sparse.wasm'

CPU	cwa	pahi --no-cache	pahi	multiplier
Ryzen 5 3600	211.7 milliseconds	125.3 milliseconds	8.5 milliseconds	pahi is 25.04 times faster than cwa
Intel Xeon E5-1650	674.1 milliseconds	342.7 milliseconds	30.8 milliseconds	pahi is 21.85 times faster than cwa

Recursive Fibbonacci Number Calculation

This is implemented as fibber.wasm. Here is the source code used in the benchmark:

#![no_main]
#![feature(start)]

extern crate olin;

use olin::{entrypoint, log};

entrypoint!();

fn fib(n: u64) -> u64 {
    if n <= 1 {
        return 1;
    }
    fib(n - 1) + fib(n - 2)
}

fn main() -> Result<(), std::io::Error> {
    log::info("starting");
    fib(30);
    log::info("done");
    Ok(())
}

Fibbonacci number calculation done recursively is an incredibly time-complicated ordeal. This is the worst possible case for this kind of calculation, as it doesn't cache results from the fib function.

Here are the results I got from running the following command:

$ hyperfine --warmup 3 --prepare './result/bin/pahi result/wasm/fibber.wasm' \
        './result/bin/cwa result/wasm/fibber.wasm' \
        './result/bin/pahi --no-cache result/wasm/fibber.wasm' \
        './result/bin/pahi result/wasm/fibber.wasm'

CPU	cwa	pahi --no-cache	pahi	multiplier
Ryzen 5 3600	13.6 milliseconds	13.7 milliseconds	2.7 milliseconds	pahi is 5.13 times faster than cwa
Intel Xeon E5-1650	41.0 milliseconds	27.3 milliseconds	7.2 milliseconds	pahi is 5.70 times faster than cwa

Blake-2 Hashing

This is implemented as blake2stress.wasm. Here's the source code for this benchmark:

#![no_main]
#![feature(start)]

extern crate olin;

use blake2::{Blake2b, Digest};
use olin::{entrypoint, log};

entrypoint!();

fn main() -> Result<(), std::io::Error> {
    let json: &'static [u8] = include_bytes!("./bigjson.json");
    let yaml: &'static [u8] = include_bytes!("./k8sparse.yaml");
    for _ in 0..8 {
        let mut hasher = Blake2b::new();
        hasher.input(json);
        hasher.input(yaml);
        hasher.result();
    }

    Ok(())
}

This runs the blake2b hashing algorithm on the JSON and yaml files used earlier eight times. This is supposed to represent a few hundred thousand invocations of production code.

Here are the results I got from running the following command:

$ hyperfine --warmup 3 --prepare './result/bin/pahi result/wasm/blake2stress.wasm' \
        './result/bin/cwa result/wasm/blake2stress.wasm' \
        './result/bin/pahi --no-cache result/wasm/blake2stress.wasm' \
        './result/bin/pahi result/wasm/blake2stress.wasm'

CPU	cwa	pahi --no-cache	pahi	multiplier
Ryzen 5 3600	358.7 milliseconds	17.4 milliseconds	5.0 milliseconds	pahi is 71.76 times faster than cwa
Intel Xeon E5-1650	1.351 seconds	35.5 milliseconds	11.7 milliseconds	pahi is 115.04 times faster than cwa

Conclusions

From these tests, we can roughly conclude that pa'i is about 54 times faster than Olin's cwa tool. A lot of this speed gain is arguably the result of pa'i using an ahead of time compiler (namely cranelift as wrapped by wasmer). The compilation time also became a somewhat notable factor for comparing performance too, however the compilation cost only has to be eaten once.

Another conclusion I've made is very unsurprising. My old 2013 mac pro with an Intel Xeon E5-1650 is significantly slower in real-world computing tasks than the new Ryzen 5 3600. Both of these machines were using the same nix closure for running the binaries and they are running NixOS 20.03.

As always, if you have any feedback for what other kinds of benchmarks to run and how these benchmarks were collected, I welcome it. Please comment wherever this article is posted or contact me.

Here are the /proc/cpuinfo files for each machine being tested:

• shachi (Ryzen 5 3600) /proc/cpuinfo
• chrysalis (Intel Xeon E5-1650) /proc/cpuinfo

If you run these benchmarks on your own hardware and get different data, please let me know and I will be more than happy to add your results to these tables. I will need the CPU model name and the output of hyperfine for each of the above commands.

New Site Feature: Signal Boosting

In light of the COVID-19 pandemic, people have been losing their jobs. In normal times, this would be less of an issue, but in the middle of the pandemic, HR departments have been reluctant to hire people as entire companies suddenly switch to remote work. I feel utterly powerless during this outbreak. I can only image what people who have lost their job feel.

I've decided to do what I can to help. I have created a page on my website to signal boost people who are looking for work. You can find it at /signalboost. If you want to be added to it, please open a GitHub issue, contact me, or open a pull request to signalboost.dhall in the root.

The schema of this is simple:

Person::{ Name = "Nicole Brennan"
        , Tags = [ "python", "go", "rust", "technical-writing" ]
        , GitLink = "https://github.com/Twi"
        , Twitter = "https://twitter.com/TwitterAccountNameHere"
        }

This will create a grid entry on the site that looks like this:

I've also changed my footer to point to this page for the forseeable future instead of linking to my Patreon. Thank you for reading this and please take a look at the people on /signalboost.

Be well.

How I Start: Rust

Rust is an exciting new programming language that makes it easy to make understandable and reliable software. It is made by Mozilla and is used by Amazon, Google, Microsoft and many other large companies.

Rust has a reputation of being difficult because it makes no effort to hide what is going on. I'd like to show you how I start with Rust projects. Let's make a small HTTP service using Rocket.

• Setting up your environment
• A new project
• Testing
• Adding functionality
• OpenAPI specifications
• Error responses
• Shipping it in a docker image

Setting up your environment

The first step is to install the Rust compiler. You can use any method you like, but since we are requiring the nightly version of Rust for this project, I suggest using rustup:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain nightly

If you are using NixOS or another Linux distribution with Nix installed, see this post for some information on how to set up the Rust compiler.

A new project

Rocket is a popular web framework for Rust programs. Let's use that to create a small "hello, world" server. We will need to do the following:

• Create the new Rust project
• Add Rocket as a dependency
• Write the hello world route
• Test a build of the service with cargo build
• Run it and see what happens

Create the new Rust project

Create the new Rust project with cargo init:

$ cargo init --vcs git .
     Created binary (application) package

This will create the directory src and a file named Cargo.toml. Rust code goes in src and the Cargo.toml file configures dependencies. Adding the --vcs git flag also has cargo create a gitignore file so that the target folder isn't tracked by git.

Add Rocket as a dependency

Open Cargo.toml and add the following to it:

[dependencies]
rocket = "0.4.4"

Then download/build Rocket with cargo build:

$ cargo build

This will download all of the dependencies you need and precompile Rocket, and it will help speed up later builds.

Write our "hello world" route

Now put the following in src/main.rs:

#![feature(proc_macro_hygiene, decl_macro)] // Nightly-only language features needed by Rocket

// Import the rocket macros
#[macro_use]
extern crate rocket;

/// Create route / that returns "Hello, world!"
#[get("/")]
fn index() -> &'static str {
    "Hello, world!"
}

fn main() {
    rocket::ignite().mount("/", routes![index]).launch();
}

Test a build

Rerun cargo build:

$ cargo build

This will create the binary at target/debug/helloworld. Let's run it locally and see if it works:

$ ./target/debug/helloworld

And in another terminal window:

$ curl http://127.0.0.1:8000
Hello, world!
$ fg
<press control-c>

The HTTP service works. We have a binary that is created with the Rust compiler. This binary will be available at ./target/debug/helloworld. However, it could use some tests.

Testing

Rocket has support for unit testing built in. Let's create a tests module and verify this route in testing.

Create a tests module

Rust allows you to nest modules within files using the mod keyword. Create a tests module that will only build when testing is requested:

#[cfg(test)] // Only compile this when unit testing is requested
mod tests {
  use super::*; // Modules are their own scope, so you 
                // need to explictly use the stuff in
                // the parent module.
                
  use rocket::http::Status;
  use rocket::local::*;
  
  #[test]
  fn test_index() {
    // create the rocket instance to test
    let rkt = rocket::ignite().mount("/", routes![index]);
    
    // create a HTTP client bound to this rocket instance
    let client = Client::new(rkt).expect("valid rocket");
    
    // get a HTTP response
    let mut response = client.get("/").dispatch();
    
    // Ensure it returns HTTP 200
    assert_eq!(response.status(), Status::Ok);
    
    // Ensure the body is what we expect it to be
    assert_eq!(response.body_string(), Some("Hello, world!".into()));
  }
}

Run tests

cargo test is used to run tests in Rust. Let's run it:

$ cargo test
   Compiling helloworld v0.1.0 (/home/cadey/code/helloworld)
    Finished test [unoptimized + debuginfo] target(s) in 1.80s
     Running target/debug/deps/helloworld-49d1bd4d4f816617

running 1 test
test tests::test_index ... ok

Adding functionality

Most HTTP services return JSON or JavaScript Object Notation as a way to pass objects between computer programs. Let's use Rocket's JSON support to add a /hostinfo route to this app that returns some simple information:

• the hostname of the computer serving the response
• the process ID of the HTTP service
• the uptime of the system in seconds

Encoding things to JSON

For encoding things to JSON, we will be using serde. We will need to add serde as a dependency. Open Cargo.toml and put the following lines in it:

[dependencies]
serde_json = "1.0"
serde = { version = "1.0", features = ["derive"] }

This lets us use #[derive(Serialize, Deserialize)] on our Rust structs, which will allow us to automate away the JSON generation code at compile time. For more information about derivation in Rust, see here.

Let's define the data we will send back to the client using a struct.

use serde::*;

/// Host information structure returned at /hostinfo
#[derive(Serialize, Debug)]
struct HostInfo {
  hostname: String,
  pid: u32,
  uptime: u64,
}

To implement this call, we will need another few dependencies in the Cargo.toml file. We will use gethostname to get the hostname of the machine and psutil to get the uptime of the machine. Put the following below the serde dependency line:

gethostname = "0.2.1"
psutil = "3.0.1"

Finally, we will need to enable Rocket's JSON support. Put the following at the end of your Cargo.toml file:

[dependencies.rocket_contrib]
version = "0.4.4"
default-features = false
features = ["json"]

Now we can implement the /hostinfo route:

/// Create route /hostinfo that returns information about the host serving this
/// page.
#[get("/hostinfo")]
fn hostinfo() -> Json<HostInfo> {
  // gets the current machine hostname or "unknown" if the hostname doesn't
  // parse into UTF-8 (very unlikely)
  let hostname = gethostname::gethostname()
    .into_string()
    .or(|_| "unknown".to_string())
    .unwrap();
    
  Json(HostInfo{
    hostname: hostname,
    pid: std::process::id(),
    uptime: psutil::host::uptime()
      .unwrap() // normally this is a bad idea, but this code is
                // very unlikely to fail.
      .as_secs(),
  })
}

And then register it in the main function:

fn main() {
  rocket::ignite()
    .mount("/", routes![index, hostinfo])
    .launch();
}

Now rebuild the project and run the server:

$ cargo build
$ ./target/debug/helloworld

And in another terminal test it with curl:

$ curl http://127.0.0.1:8000
{"hostname":"shachi","pid":4291,"uptime":13641}

You can use a similar process for any kind of other route.

OpenAPI specifications

OpenAPI is a common specification format for describing API routes. This allows users of the API to automatically generate valid clients for them. Writing these by hand can be tedious, so let's pass that work off to the compiler using okapi.

Add the following line to your Cargo.toml file in the [dependencies] block:

rocket_okapi = "0.3.6"
schemars = "0.6"
okapi = { version = "0.3", features = ["derive_json_schema"] }

This will allow us to generate OpenAPI specifications from Rocket routes and the types in them. Let's import the rocket_okapi macros and use them:

// Import OpenAPI macros
#[macro_use]
extern crate rocket_okapi;

use rocket_okapi::JsonSchema;

We need to add JSON schema generation abilities to HostInfo. Change:

#[derive(Serialize, Debug)]

to

#[derive(Serialize, JsonSchema, Debug)]

to generate the OpenAPI code for our type.

Next we can add the /hostinfo route to the OpenAPI schema:

/// Create route /hostinfo that returns information about the host serving this
/// page.
#[openapi]
#[get("/hostinfo")]
fn hostinfo() -> Json<HostInfo> {
  // ...

Also add the index route to the OpenAPI schema:

/// Create route / that returns "Hello, world!"
#[openapi]
#[get("/")]
fn index() -> &'static str {
    "Hello, world!"
}

And finally update the main function to use openapi:

fn main() {
  rocket::ignite()
    .mount("/", routes_with_openapi![index, hostinfo])
    .launch();
}

Then rebuild it and run the server:

$ cargo build
$ ./target/debug/helloworld

And then in another terminal:

$ curl http://127.0.0.1:8000/openapi.json

This should return a large JSON object that describes all of the HTTP routes and the data they return. To see this visually, change main to this:

use rocket_okapi::swagger_ui::{make_swagger_ui, SwaggerUIConfig};

fn main() {
    rocket::ignite()
        .mount("/", routes_with_openapi![index, hostinfo])
        .mount(
            "/swagger-ui/",
            make_swagger_ui(&SwaggerUIConfig {
                url: Some("../openapi.json".to_owned()),
                urls: None,
            }),
        )
        .launch();
}

Then rebuild and run the service:

$ cargo build
$ ./target/debug/helloworld

And open the swagger UI in your favorite browser. This will show you a graphical display of all of the routes and the data types in your service. For an example, see here.

Error responses

Earlier in the /hostinfo route we glossed over error handling. Let's correct this using the okapi error type. Let's use the OpenAPIError type in the helloworld function:

/// Create route /hostinfo that returns information about the host serving
/// this page.
#[openapi]
#[get("/hostinfo")]
fn hostinfo() -> Result<Json<HostInfo>> {
    match gethostname::gethostname().into_string() {
        Ok(hostname) => Ok(Json(HostInfo {
            hostname: hostname,
            pid: std::process::id(),
            uptime: psutil::host::uptime().unwrap().as_secs(),
        })),
        Err(_) => Err(OpenApiError::new(format!(
            "hostname does not parse as UTF-8"
        ))),
    }
}

When the into_string operation fails (because the hostname is somehow invalid UTF-8), this will result in a non-200 response with the "hostname does not parse as UTF-8" message.

Shipping it in a docker image

Many deployment systems use [Docker][docker] to describe a program's environment and dependencies. Create a Dockerfile with the following contents:

# Use the minimal image
FROM rustlang/rust:nightly-slim AS build

# Where we will build the program
WORKDIR /src/helloworld

# Copy source code into the container
COPY . .

# Build the program in release mode
RUN cargo build --release

# Create the runtime image
FROM ubuntu:18.04

# Copy the compiled service binary
COPY --from=build /src/helloworld/target/release/helloworld /usr/local/bin/helloworld

# Start the helloworld service on container boot
CMD ["usr/local/bin/helloworld"]

And then build it:

$ docker build -t xena/helloworld .

And then run it:

$ docker run --rm -itp 8000:8000 xena/helloworld

And in another terminal:

$ curl http://127.0.0.1:8000
Hello, world!

From here you can do whatever you want with this service. You can deploy it to Kubernetes with a manifest that would look something like this.

---

This is how I start a new Rust project. I put all of the code described in this post in this GitHub repo in case it helps. Have fun and be well.

---

For some "extra credit" tasks, try and see if you can do the following:

• Customize the environment of the container by following the Rocket configuration documentation and docker environment variables
• Use Rocket's templates to make the host information show up in HTML
• Add tests for the /hostinfo route
• Make a route that always returns errors, what does it look like?

Many thanks to Coleman McFarland for proofreading this post.

How I Start: Nix

Nix is a tool that helps people create reproducible builds. This means that given a known input, you can get the same output on other machines. Let's build and deploy a small Rust service with Nix. This will not require the Rust compiler to be installed with rustup or similar.

• Setting up your environment
• A new project
• Setting up the Rust compiler
• Serving HTTP
• A simple package build
• Shipping it in a docker image

Setting up your environment

The first step is to install Nix. If you are using a Linux machine, run this script:

$ curl https://nixos.org/nix/install | sh

This will prompt you for more information as it goes on, so be sure to follow the instructions carefully. Once it is done, close and re-open your shell. After you have done this, nix-env should exist in your shell. Try to run it:

$ nix-env
error: no operation specified
Try 'nix-env --help' for more information.

Let's install a few other tools to help us with development. First, let's install lorri to help us manage our development shell:

$ nix-env --install --file https://github.com/target/lorri/archive/master.tar.gz

This will automatically download and build lorri for your system based on the latest possible version. Once that is done, open another shell window (the lorri docs include ways to do this more persistently, but this will work for now) and run:

$ lorri daemon

Now go back to your main shell window and install direnv:

$ nix-env --install direnv

Next, follow the shell setup needed for your shell. I personally use fish with oh my fish, so I would run this:

$ omf install direnv

Finally, let's install niv to help us handle dependencies for the project. This will allow us to make sure that our builds pin everything to a specific set of versions, including operating system packages.

$ nix-env --install niv

Now that we have all of the tools we will need installed, let's create the project.

A new project

Go to your favorite place to put code and make a new folder. I personally prefer ~/code, so I will be using that here:

$ cd ~/code
$ mkdir helloworld
$ cd helloworld

Let's set up the basic skeleton of the project. First, initialize niv:

$ niv init

This will add the latest versions of niv itself and the packages used for the system to nix/sources.json. This will allow us to pin exact versions so the environment is as predictable as possible. Sometimes the versions of software in the pinned nixpkgs are too old. If this happens, you can update to the "unstable" branch of nixpkgs with this command:

$ niv update nixpkgs -b nixpkgs-unstable

Next, set up lorri using lorri init:

$ lorri init

This will create shell.nix and .envrc. shell.nix will be where we define the development environment for this service. .envrc is used to tell direnv what it needs to do. Let's try and activate the .envrc:

$ cd .
direnv: error /home/cadey/code/helloworld/.envrc is blocked. Run `direnv allow`
to approve its content

Let's review its content:

$ cat .envrc
eval "$(lorri direnv)"

This seems reasonable, so approve it with direnv allow like the error message suggests:

$ direnv allow

Now let's customize the shell.nix file to use our pinned version of nixpkgs. Currently, it looks something like this:

# shell.nix
let
  pkgs = import <nixpkgs> {};
in
pkgs.mkShell {
  buildInputs = [
    pkgs.hello
  ];
}

This currently imports nixpkgs from the system-level version of it. This means that different systems could have different versions of nixpkgs on it, and that could make the shell.nix file hard to reproduce between machines. Let's import the pinned version of nixpkgs that niv created:

# shell.nix
let
  sources = import ./nix/sources.nix;
  pkgs = import sources.nixpkgs {};
in
pkgs.mkShell {
  buildInputs = [
    pkgs.hello
  ];
}

And then let's test it with lorri shell:

$ lorri shell
lorri: building environment........ done
(lorri) $

And let's see if hello is available inside the shell:

(lorri) $ hello
Hello, world!

You can set environment variables inside the shell.nix file. Do so like this:

# shell.nix
let
  sources = import ./nix/sources.nix;
  pkgs = import sources.nixpkgs {};
in
pkgs.mkShell {
  buildInputs = [
    pkgs.hello
  ];
  
  # Environment variables
  HELLO="world";
}

Wait a moment for lorri to finish rebuilding the development environment and then let's see if the environment variable shows up:

$ cd .
direnv: loading ~/code/helloworld/.envrc
<output snipped>
$ echo $HELLO
world

Now that we have the basics of the environment set up, lets install the Rust compiler.

Setting up the Rust compiler

First, add nixpkgs-mozilla to niv:

$ niv add mozilla/nixpkgs-mozilla

Then create nix/rust.nix in your repo:

# nix/rust.nix
{ sources ? import ./sources.nix }:

let
  pkgs =
    import sources.nixpkgs { overlays = [ (import sources.nixpkgs-mozilla) ]; };
  channel = "nightly";
  date = "2020-03-08";
  targets = [ ];
  chan = pkgs.rustChannelOfTargets channel date targets;
in chan

This creates a nix function that takes in the pre-imported list of sources, creates a copy of nixpkgs with Rust at the nightly version 2020-03-08 overlaid into it, and exposes the rust package out of it. Let's add this to shell.nix:

# shell.nix
let
  sources = import ./nix/sources.nix;
  rust = import ./nix/rust.nix { inherit sources; };
  pkgs = import sources.nixpkgs { };
in
pkgs.mkShell {
  buildInputs = [
    rust
  ];
}

Then ask lorri to recreate the development environment. This may take a bit to run because it's setting up everything the Rust compiler requires to run.

$ lorri shell
(lorri) $

Let's see what version of Rust is installed:

(lorri) $ rustc --version
rustc 1.43.0-nightly (823ff8cf1 2020-03-07)

This is exactly what we expect. Rust nightly versions get released with the date of the previous day in them. To be extra sure, let's see what the shell thinks rustc resolves to:

(lorri) $ which rustc
/nix/store/w6zk1zijfwrnjm6xyfmrgbxb6dvvn6di-rust-1.43.0-nightly-2020-03-07-823ff8cf1/bin/rustc

And now exit that shell and reload direnv:

(lorri) $ exit
$ cd .
direnv: loading ~/code/helloworld/.envrc
$ which rustc
/nix/store/w6zk1zijfwrnjm6xyfmrgbxb6dvvn6di-rust-1.43.0-nightly-2020-03-07-823ff8cf1/bin/rustc

And now we have Rust installed at an arbitrary nightly version for that project only. This will work on other machines too. Now that we have our development environment set up, let's serve HTTP.

Serving HTTP

Rocket is a popular web framework for Rust programs. Let's use that to create a small "hello, world" server. We will need to do the following:

• Create the new Rust project
• Add Rocket as a dependency
• Write our "hello world" route
• Test a build of the service with cargo build

Create the new Rust project

Create the new Rust project with cargo init:

$ cargo init --vcs git .
     Created binary (application) package

This will create the directory src and a file named Cargo.toml. Rust code goes in src and the Cargo.toml file configures dependencies. Adding the --vcs git flag also has cargo create a gitignore file so that the target folder isn't tracked by git.

Add Rocket as a dependency

Open Cargo.toml and add the following to it:

[dependencies]
rocket = "0.4.3"

Then download/build Rocket with cargo build:

$ cargo build

This will download all of the dependencies you need and precompile Rocket, and it will help speed up later builds.

Write our "hello world" route

Now put the following in src/main.rs:

#![feature(proc_macro_hygiene, decl_macro)] // language features needed by Rocket

// Import the rocket macros
#[macro_use]
extern crate rocket;

// Create route / that returns "Hello, world!"
#[get("/")]
fn index() -> &'static str {
    "Hello, world!"
}

fn main() {
    rocket::ignite().mount("/", routes![index]).launch();
}

Test a build

Rerun cargo build:

$ cargo build

This will create the binary at target/debug/helloworld. Let's run it locally and see if it works:

$ ./target/debug/helloworld &
$ curl http://127.0.0.1:8000
Hello, world!
$ fg
<press control-c>

The HTTP service works. We have a binary that is created with the Rust compiler Nix installed.

A simple package build

Now that we have the HTTP service working, let's put it inside a nix package. We will need to use naersk to do this. Add naersk to your project with niv:

$ niv add nmattia/naersk

Now let's create helloworld.nix:

# import niv sources and the pinned nixpkgs
{ sources ? import ./nix/sources.nix, pkgs ? import sources.nixpkgs { }}:
let
  # import rust compiler
  rust = import ./nix/rust.nix { inherit sources; };
  
  # configure naersk to use our pinned rust compiler
  naersk = pkgs.callPackage sources.naersk {
    rustc = rust;
    cargo = rust;
  };
  
  # tell nix-build to ignore the `target` directory
  src = builtins.filterSource
    (path: type: type != "directory" || builtins.baseNameOf path != "target")
    ./.;
in naersk.buildPackage {
  inherit src;
  remapPathPrefix =
    true; # remove nix store references for a smaller output package
}

And then build it with nix-build:

$ nix-build helloworld.nix

This can take a bit to run, but it will do the following things:

• Download naersk
• Download every Rust crate your HTTP service depends on into the Nix store
• Run your program's tests
• Build your dependencies into a Nix package
• Build your program with those dependencies
• Place a link to the result at ./result

Once it is done, let's take a look at the result:

$ du -hs ./result/bin/helloworld
2.1M    ./result/bin/helloworld

$ ldd ./result/bin/helloworld
        linux-vdso.so.1 (0x00007fffae080000)
        libdl.so.2 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libdl.so.2 (0x0
0007f3a01666000)
        librt.so.1 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/librt.so.1 (0x0
0007f3a0165c000)
        libpthread.so.0 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libpthread
.so.0 (0x00007f3a0163b000)
        libgcc_s.so.1 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libgcc_s.so.
1 (0x00007f3a013f5000)
        libc.so.6 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libc.so.6 (0x000
07f3a0123f000)
        /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/ld-linux-x86-64.so.2 => /lib6
4/ld-linux-x86-64.so.2 (0x00007f3a0160b000)
        libm.so.6 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libm.so.6 (0x000
07f3a010a9000)

This means that the Nix build created a 2.1 megabyte binary that only depends on glibc, the implementation of the C language standard library that Nix prefers.

For repo cleanliness, add the result link to the gitignore:

$ echo 'result*' >> .gitignore

Shipping it in a Docker image

Now that we have a package built, let's ship it in a docker image. nixpkgs provides dockerTools which helps us create docker images out of Nix packages. Let's create default.nix with the following contents:

{ system ? builtins.currentSystem }:

let
  sources = import ./nix/sources.nix;
  pkgs = import sources.nixpkgs { };
  helloworld = import ./helloworld.nix { inherit sources pkgs; };

  name = "xena/helloworld";
  tag = "latest";

in pkgs.dockerTools.buildLayeredImage {
  inherit name tag;
  contents = [ helloworld ];

  config = {
    Cmd = [ "/bin/helloworld" ];
    Env = [ "ROCKET_PORT=5000" ];
    WorkingDir = "/";
  };
}

And then build it with nix-build:

$ nix-build default.nix

This will create a tarball containing the docker image information as the result of the Nix build. Load it into docker using docker load:

$ docker load -i result

And then run it using docker run:

$ docker run --rm -itp 52340:5000 xena/helloworld

Now test it using curl:

$ curl http://127.0.0.1:52340
Hello, world!

And now you have a docker image you can run wherever you want. The buildLayeredImage function used in default.nix also makes Nix put each dependency of the package into its own docker layer. This makes new versions of your program very efficient to upgrade on your clusters, realistically this reduces the amount of data needed for new versions of the program down to what changed. If nothing but some resources in their own package were changed, only those packages get downloaded.

This is how I start a new project with Nix. I put all of the code described in this post in this GitHub repo in case it helps. Have fun and be well.

---

For some "extra credit" tasks, try and see if you can do the following:

• Use the version of niv that niv pinned
• Customize the environment of the container by following the Rocket configuration documentation
• Add some more routes to the program
• Read the Nix documentation and learn more about writing Nix expressions
• Configure your editor/IDE to use the direnv path

New Site Feature: Patron Thanks Page

I've added a patron thanks page to my site. I've been getting a significant amount of money per month from my patrons and I feel this is a good way to acknowledge them and thank them for their patronage. I wanted to have it be as simple as possible, so I made it fetch a list of dollar amounts.

Here are some things I learned while writing this:

• If you are going to interact with the patreon API in go, use github.com/mxpv/patreon-go, not gopkg.in/mxpv/patreon-go.v1 or gopkg.in/mxpv/patreon-go.v2. The packages on gopkg.in are NOT compatible with Go modules in very bizarre ways.
• When using refresh tokens in OAuth2, do not set the expiry date to be negative like the patreon-go examples show. This will brick your token and make you have to reprovision it.
• Patreon clients can either be for API version 1 or API version 2. There is no way to have a Patreon token that works for both API versions.
• The patreon-go package only supports API version 1 and doesn't document this anywhere.
• Patreon's error messages are vague and not helpful when trying to figure out that you broke your token with a negative expiry date.
• I may need to set the Patreon information every month for the rest of the time I maintain this site code. This could get odd. I made a guide for myself in the docs folder of the site repo.
• The Patreon API doesn't let you submit new posts. I wanted to add Patreon to my syndication server, but apparently that's impossible. My RSS feed, Atom feed and JSON feed should let you keep up to date in the meantime.

Let me know how you like this. I went back and forth on displaying monetary amounts on that page, but ultimately decided not to show them there for confidentiality reasons. If this is a bad idea, please let me know and I can put the money amounts back.

I'm working on a more detailed post about pa'i that includes benchmarks for some artificial and realistic workloads. I'm also working on integrating it into the wamscloud prototype, but it's fairly slow going at the moment.

Be well.

pa'i: hello world!

It's been a while since I gave an update on the Olin ecosystem (which now exists, apparently). Not much has really gone on with it for the last few months. However, recently I've decided to tackle one of the core problems of Olin's implementation in Go: execution speed.

Originally I was going to try and handle this with "hyperjit", but support for linking C++ programs into Go is always questionable at best. All of the WebAssembly compiling and running tooling has been written in Rust, and as far as I know I was the only holdout still using Go. This left me kinda stranded and on my own, seeing as the libraries that I was using were starting to die.

I have been following the wasmer project for a while and thanks to their recent custom ABI sample, I was able to start re-implementing the Olin API in it. Wasmer uses a JIT for handling WebAssembly, so I'm able to completely destroy the original Go implementation in terms of performance. I call this newer, faster runtime pa'i (/pa.hi/, paw-hee), which is a Lojban rafsi for the word prami which means love.

pa'i is written in Rust. It is built with Nix. It requires a nightly version of Rust because the WebAssembly code it compiles requires it. However, because it is built with Nix, this quickly becomes a non-issue. You can build pa'i by doing the following:

$ git clone git@github.com:Xe/pahi
$ cd pahi
$ nix-build

and then nix-build will take care of:

• downloading the pinned nightly version of the rust compiler
• building the reference Olin interpreter
• building the pa'i runtime
• building a small suite of sample programs
• building the documentation from dhall files
• building a small test runner

If you want to try this out in a more predictable environment, you can also nix-build docker.nix. This will create a Docker image as the result of the Nix build. This docker image includes the pa'i composite package, bash, coreutils and dhall-to-json (which is required by the test runner).

I'm actually really proud of how the documentation generation works. The cwa-spec folder in Olin was done very ad-hoc and was only consistent because there was a template. This time functions, types, errors, namespaces and the underlying WebAssembly types they boil down to are all implemented as Dhall records. For example, here's the definition of a namespace in Dhall:

let func = ./func.dhall

in  { Type = { name : Text, desc : Text, funcs : List func.Type }
    , default =
        { name = "unknown"
        , desc = "please fill in the desc field"
        , funcs = [] : List func.Type
        }
    }

which gets rendered to Markdown using renderNSToMD.dhall:

let ns = ./ns.dhall

let func = ./func.dhall

let type = ./type.dhall

let showFunc = ./renderFuncToMD.dhall

let Prelude = ../Prelude.dhall

let toList = Prelude.Text.concatMapSep "\n" func.Type showFunc

let show
    : ns.Type → Text
    =   λ(namespace : ns.Type)
      → ''
        # ${namespace.name}

        ${namespace.desc}

        ${toList namespace.funcs}
        ''

in  show

This would render the logging namespace as this markdown.

It seems like overkill to document things like this (and at some level it is), but I plan to take advantage of this later when I need to do things like generate C/Rust/Go/TinyGo bindings for the entire specification at once. I also have always wanted to document something so precisely like this, and now I get the chance.

pa'i is just over a week old at this point, and as such it is NOT feature-complete with the reference Olin interpreter. I'm working on it though. I'm kinda burnt out from work, and even though working on this project helps me relax (don't ask me how, I don't understand either) I have limits and will take this slowly and carefully to ensure that it stays compatible with all of the code I have already written in Olin's repo. Thanks to go-flag, I might actually be able to get it mostly flag-compatible. We'll see though.

I have also designed a placeholder logo for pa'i. Here it is:

It might be changed in the future, but this is what I am going with for now. The circuit traces all spell out messages of love (inspired from the Senzar runes of the WingMakers). The text on top of the microprocessor reads pa'i in zbalermorna, a constructed writing script for Lojban. The text on the side probably needs to be revised, but it says something along the lines of "a future after programs".

pa'i is chugging along. When I have closed the compatibility todo list for all of the Olin API calls, I'll write more. For now, pa'i is a very complicated tool that lets you print "Hello, world" in new and exciting ways (this will change once I get resource calls into it), but it's getting there.

I hope this was interesting. Be well.

Why Rust

Or: A Trip Report from my Satori with Rust and Functional Programming

Software is a very odd field to work in. It is simultaneously an abstract and physical one. You build systems that can deal with an unfathomable amount of input and output at the same time. As a job, I peer into the madness of an unthinking automaton and give order to the inherent chaos. I then emit incantations to describe what this unthinking automaton should do in my stead. I cannot possibly track the relations between a hundred thousand transactions going on in real time, much less file them appropriately so they can be summoned back should the need arise.

However, this incantation (by necessity) is an unthinkably precise and fickle beast. It's almost as if you are training a four-year old to go to the store, but doing it by having them read a grocery list. This grocery list has to be precise enough that the four year old ends up getting what you want and not a cart full of frosted flakes and candy bars. But, at the same time, the four year old needs to understand it. Thus, the precision.

There's many schools of thought around ways to write the grocery list. Some follow a radically simple approach, relying on the toddler to figure things out at the store. Sometimes this simpler approach doesn't work out in more obscure scenarios, like when they are out of red grapes but do have green grapes, but it tends to work out enough. Proponents of these list-making tools also will advocate for doing full tests of the grocery list before they send the toddler off to the store. This means setting up a fake grocery store with funny money, a fake card, plastic food, the whole nine yards. This can get expensive and can become a logistical issue (where are you going to store all that plastic fruit in a way that you can just set up and tear down the grocery store mock so quickly?).

Another school of thought is that the process of writing the grocery list should be done in a way that prevents ambiguity at the grocery store. This kind of flow uses some more advanced concepts like the ability to describe something by its attributes. For example, this could specify the difference between fruit and vegetables, and only allow fruit to be put in one place of the cart and only allow vegetables to be placed in the other. And if the writer of the list tries to violate this, the list gets rejected and isn't used at all.

There is yet another school of thought that decides that the exact spatial position of the toddler relative to everything else should be thought of in advance, along with a process to make sure that nothing is done in an improper way. This means writing the list can be a lot harder at first, but it's much less likely to result in the toddler coming back with a weird state. Consider what happens if two items show up at the same time and the toddler tries to grab both of them at the same time due to the instructions in the list! They only have one arm to grab things with, so it just doesn't work. Proponents of the more strict methods have reference cells and other mechanisms to ensure that the toddler can only ever grab one thing at a time.

If we were to match these three ludicrous examples to programming languages, the first would be Lua, the second would be Go and the third would be something like Haskell or Rust. Software development is a complicated process because the problems involved with directing that unthinking automaton to do what you want are hard. There is a lot going on, much in the same way there is a lot going on when you send a toddler to do your grocery shopping for you.

A good way to look at the tradeoffs involved is to see things as a balance between two forces, pragmatism and correctness. Languages that are more pragmatic are easier to develop in, but are mathematically more likely to run into problems at runtime. Languages that are more correct take more investment to write up front, but over time the correctness means that there's fewer failed assumptions about what is going on. The compiler stops you from doing things that don't make sense to it. This means that it's difficult to literally impossible to create a bad state at runtime.

Tools like Lua and Go can (and have) been used to develop stable and viable software. itch.io is written in Lua running on top of nginx and it handles financial transactions well enough that it's turned into the guy's full time job. Google uses Go everywhere in their stack, and it's been used to create powerful tools like Kubernetes, Caddy, and Docker. These tools are trusted implicitly by a generation of developers, even though the language itself has its flaws. If you are reading this blog in Firefox, statistically there is Rust involved in the rendering and viewing of this post. Rust is built for ensuring that code is as correct as possible, even if it means eating into development time to ensure that.

In Rust, you don't have to memorize rules about how and when it is safe to update data in structures, because the compiler ensures you cannot mess it up by rejecting the code if you could be messing it up. You don't have to run your tests with a race detector or figure out how to expose that in production to trace down that obscure double-write to a non-threadsafe hashmap, because in Rust there is no such thing as a non-threadsafe hashmap. There is only a safe hashmap and only can ever be a safe hashmap.

As an absurd example, consider the following two snippets of code, one in Go and one in Rust, both of them will put integers into a standard library list and then print them all out:

l := list.New()           // () -> *list.List
for i := 0; i < 5; i++ {
  l.PushBack(i)           // interface{} -> ()
}

for e := l.Front(); e != nil; e = e.Next() {
  log.Printf("%T: %v", e.Value, e.Value)
}

let mut vec = Vec::new::<i64>(); // () -> Vec<i64>

for i in 0..5 {
  vec.push(i as i64);            // (mut Vec<i64>, i64) -> ()
}

for i in vec.iter() {
  println!("{}", i);
}

The Go version uses interface{} as the data element because Go literally cannot describe types as parameters to functions. The Rust version took me a bit longer to write, but there is no ambiguity as to what the vector holds. The Go version can also hold multiple types of data in the same list, a-la:

l := list.New()
l.PushBack(42)
l.PushBack("hotdogs")
l.PushBack(420.69)

All of which is valid because in Go, an interface{} matches every kind of value possible. An integer is an interface{}. A floating-point number is an interface{}. A string is an interface{}. A bool is an interface{}. Any custom type you create is an interface{}. Normally, this would be very restrictive and make it difficult to do things like JSON parsing. However the Go runtime lets you hack around this with reflection.

This allows the standard library to handle things like JSON parsing with functions that look like this:

func Unmarshal(data []byte, v interface{}) error

There's even a set of complicated rules you need to memorize about how to trick the JSON parser into massaging your data into place. This lets you do things like this:

type Rilkef struct {
  Foo        string `json:"foo"`
  CallToArms string `json:"call_to_arms"`
}

This allows the programmer a lot of flexibility while developing and compiling the code. It's very easy for the compiler to say "oh, hey, that could be anything, and you gave it some kind of anything, sounds legit to me", but then the job of ensuring the sanity of the inputs is shunted to runtime rather than stopped before the code gets deployed. This means you need to test the code in order to see how it behaves, making sure that the standard library is doing its job correctly. This kind of stuff does not happen in Rust.

The Rust version of this JSON example uses the serde and serde_json libraries:

use serde::*;

#[derive(Serialize, Deserialize)]
pub struct Rilkef {
  pub foo: String,
  pub call_to_arms: String,
}

And the logic for handling the correct rules for serialization and deserialization is handled at compile time by the compiler itself. Serde also allows you to support more than just JSON, so this same type can be reused for Dhall, YAML or whatever you could imagine.

tl;dr

Rust allows for more correctness at the cost of developer efficiency. This is a tradeoff, but I think it may actually be worth it. Code that is more correct is more robust and less prone to failure than code that is less correct. This leads to software that is less likely to crash at 3 am and wake you up due to a preventable developer error.

After working in Go for more than half a decade, I'm starting to think that it is probably a better idea to impact developer velocity and force them to write software that is more correct. Go works if you are careful about how you handle it. It however amounts to a giant list of rules that you just have to know (like maps not being threadsafe) and a lot of those rules come from battle rather than from the development process.

This came out as more of a rant than I had thought it would, but overall I hope my point isn't lost.

Things You Might Complain About

Yes, I know slices exist in Go. I wanted to prove a point about how the overuse of interface{} in some relatively core things (like generic lists) can cause headaches in term of correctness. Go will reject you trying to append a string to an integer slice, but you cannot create a type that functions identically to an integer slice.

Go does have a race detector that will point out a lot of sins in concurrent programs, but that is again at runtime, not at compile time.

---

Many thanks to Tene, Sr. Oracle, A. Wilfox, Byte-slice, SiIvagunner and anyone who watched the stream where I wrote this blogpost. If I got things wrong in this, please reach out to me to let me know what I messed up. This is a composite of a few twitter threads and a conversation I had on IRC.

Thanks for reading, be well.

I was Wrong about Nix

From time to time, I am outright wrong on my blog. This is one of those times. In my last post about Nix, I didn't see the light yet. I think I do now, and I'm going to attempt to clarify below.

Let's talk about a more simple scenario: writing a service in Go. This service will depend on at least the following:

• A Go compiler to build the code into a binary
• An appropriate runtime to ensure the code will run successfully
• Any data files needed at runtime

A popular way to model this is with a Dockerfile. Here's the Dockerfile I use for my website (the one you are reading right now):

FROM xena/go:1.13.6 AS build
ENV GOPROXY https://cache.greedo.xeserv.us
COPY . /site
WORKDIR /site
RUN CGO_ENABLED=0 go test -v ./...
RUN CGO_ENABLED=0 GOBIN=/root go install -v ./cmd/site

FROM xena/alpine
EXPOSE 5000
WORKDIR /site
COPY --from=build /root/site .
COPY ./static /site/static
COPY ./templates /site/templates
COPY ./blog /site/blog
COPY ./talks /site/talks
COPY ./gallery /site/gallery
COPY ./css /site/css
HEALTHCHECK CMD wget --spider http://127.0.0.1:5000/.within/health || exit 1
CMD ./site

This fetches the Go compiler from an image I made, copies the source code to the image, builds it (in a way that makes the resulting binary a static executable), and creates the runtime environment for it.

Let's let it build and see how big the result is:

$ docker build -t xena/christinewebsite:example1 .
<output omitted>
$ docker images | grep xena
xena/christinewebsite  example1  4b8ee64969e8  24 seconds ago  111MB

Investigating this image with dive, we see the following:

• The package manager is included in the image
• The package manager's database is included in the image
• An entire copy of the C library is included in the image (even though the binary was statically linked to specifically avoid this)
• Most of the files in the docker image are unrelated to my website's functionality and are involved with the normal functioning of Linux systems

Granted, Alpine Linux does a good job at keeping this chaff to a minimum, but it is still there, still needs to be updated (causing all of my docker images to be rebuilt and applications to be redeployed) and still takes up space in transfer quotas and on the disk.

Let's compare this to the same build process but done with Nix. My Nix setup is done in a few phases. First I use niv to manage some dependencies a-la git submodules that don't hate you:

$ nix-shell -p niv
[nix-shel]$ niv init
<writes nix/*>

Now I add the tool vgo2nix in niv:

[nix-shell]$ niv add adisbladis/vgo2nix

And I can use it in my shell.nix:

let
  pkgs = import <nixpkgs> { };
  sources = import ./nix/sources.nix;
  vgo2nix = (import sources.vgo2nix { });
in pkgs.mkShell { buildInputs = [ pkgs.go pkgs.niv vgo2nix ]; }

And then relaunch nix-shell with vgo2nix installed and convert my go modules dependencies to a Nix expression:

$ nix-shell
<some work is done to compile things, etc>
[nix-shell]$ vgo2nix
<writes deps.nix>

Now that I have this, I can follow the buildGoPackage instructions from the upstream nixpkgs documentation and create site.nix:

{ pkgs ? import <nixpkgs> {} }:
with pkgs;

assert lib.versionAtLeast go.version "1.13";

buildGoPackage rec {
  name = "christinewebsite-HEAD";
  version = "latest";
  goPackagePath = "christine.website";
  src = ./.;

  goDeps = ./deps.nix;
  allowGoReference = false;
  preBuild = ''
    export CGO_ENABLED=0
    buildFlagsArray+=(-pkgdir "$TMPDIR")
  '';

  postInstall = ''
    cp -rf $src/blog $bin/blog
    cp -rf $src/css $bin/css
    cp -rf $src/gallery $bin/gallery
    cp -rf $src/static $bin/static
    cp -rf $src/talks $bin/talks
    cp -rf $src/templates $bin/templates
  '';
}

And this will do the following:

• Download all of the needed dependencies and place them in the system-level Nix store so that they are not downloaded again
• Set the CGO_ENABLED environment variable to 0 so the Go compiler emits a static binary
• Copy all of the needed files to the right places so that the blog, gallery and talks features can load all of their data
• Depend on nothing other than a working system at runtime

This Nix build manifest doesn't just work on Linux. It works on my mac too. The dockerfile approach works great for Linux boxes, but (unlike what the me of a decade ago would have hoped) the whole world just doesn't run Linux on their desktops. The real world has multiple OSes and Nix allows me to compensate.

So, now that we have a working cross-platform build, let's see how big it comes out as:

$ readlink ./result-bin
/nix/store/ayvafpvn763wwdzwjzvix3mizayyblx5-christinewebsite-HEAD-bin
$ du -hs result-bin/
89M     ./result-bin/
$ du -hs result-bin/
11M     ./result-bin/bin
888K    ./result-bin/blog
40K     ./result-bin/css
44K     ./result-bin/gallery
77M     ./result-bin/static
28K     ./result-bin/talks
64K     ./result-bin/templates

As expected, most of the build results are static assets. I have a lot of larger static assets including an entire copy of TempleOS, so this isn't too surprising. Let's compare this to on the mac:

$ du -hs result-bin/
 91M	result-bin/
$ du -hs result-bin/*
 14M	result-bin/bin
872K	result-bin/blog
 36K	result-bin/css
 40K	result-bin/gallery
 77M	result-bin/static
 24K	result-bin/talks
 60K	result-bin/templates

Which is damn-near identical save some macOS specific crud that Go has to deal with.

I mentioned this is used for Docker builds, so let's make docker.nix:

{ system ? builtins.currentSystem }:

let
  pkgs = import <nixpkgs> { inherit system; };

  callPackage = pkgs.lib.callPackageWith pkgs;

  site = callPackage ./site.nix { };

  dockerImage = pkg:
    pkgs.dockerTools.buildImage {
      name = "xena/christinewebsite";
      tag = pkg.version;

      contents = [ pkg ];

      config = {
        Cmd = [ "/bin/site" ];
        WorkingDir = "/";
      };
    };

in dockerImage site

And then build it:

$ nix-build docker.nix
<output omitted>
$ docker load -i result
c6b1d6ce7549: Loading layer [==================================================>]  95.81MB/95.81MB
$ docker images | grep xena
xena/christinewebsite  latest  0d1ccd676af8  50 years ago  94.6MB

And the output is 16 megabytes smaller.

The image age might look weird at first, but it's part of the reproducibility Nix offers. The date an image was built is something that can change with time and is actually a part of the resulting file. This means that an image built one second after another has a different cryptographic hash. It helpfully pins all images to Unix timestamp 0, which just happens to be about 50 years ago.

Looking into the image with dive, the only packages installed into this image are:

• The website and all of its static content goodness
• IANA portmaps that Go depends on as part of the net package
• The standard list of [MIME types][mimetypes] that the net/http package needs
• Time zone data that the time package needs

And that's it. This is fantastic. Nearly all of the disk usage has been eliminated. If someone manages to trick my website into executing code, that attacker cannot do anything but run more copies of my website (that will immediately fail and die because the port is already allocated).

This strategy pans out to more complicated projects too. Consider a case where a frontend and backend need to be built and deployed as a unit. Let's create a new setup using niv:

$ niv init

Since we are using Elm for this complicated project, let's add the elm2nix tool so that our Elm dependencies have repeatable builds, and gruvbox-css for some nice simple CSS:

$ niv add cachix/elm2nix
$ niv add Xe/gruvbox-css

And then add it to our shell.nix:

let
  pkgs = import <nixpkgs> {};
  sources = import ./nix/sources.nix;
  elm2nix = (import sources.elm2nix { });
in
pkgs.mkShell {
  buildInputs = [
    pkgs.elmPackages.elm
    pkgs.elmPackages.elm-format
    elm2nix
  ];
}

And then enter nix-shell to create the Elm boilerplate:

$ nix-shell
[nix-shell]$ cd frontend
[nix-shell:frontend]$ elm2nix init > default.nix
[nix-shell:frontend]$ elm2nix convert > elm-srcs.nix
[nix-shell:frontend]$ elm2nix snapshot

And then we can edit the generated Nix expression:

let
  sources = import ./nix/sources.nix;
  gcss = (import sources.gruvbox-css { });
# ...
      buildInputs = [ elmPackages.elm gcss ]
        ++ lib.optional outputJavaScript nodePackages_10_x.uglify-js;
# ...
        cp -rf ${gcss}/gruvbox.css $out/public
        cp -rf $src/public/* $out/public/
# ...
  outputJavaScript = true;

And then test it with nix-build:

$ nix-build
<output omitted>

And now create a name.nix for your Go service like I did above. The real magic comes from the docker.nix file:

{ system ? builtins.currentSystem }:

let
  pkgs = import <nixpkgs> { inherit system; };
  sources = import ./nix/sources.nix;
  backend = import ./backend.nix { };
  frontend = import ./frontend/default.nix { };
in

pkgs.dockerTools.buildImage {
  name = "xena/complicatedservice";
  tag = "latest";

  contents = [ backend frontend ];

  config = {
    Cmd = [ "/bin/backend" ];
    WorkingDir = "/public";
  };
};

Now both your backend and frontend services are built with the dependencies in the Nix store and shipped as a repeatable Docker image.

Sometimes it might be useful to ship the dependencies to a service like Cachix to help speed up builds.

You can install the cachix tool like this:

$ nix-env -iA cachix -f https://cachix.org/api/v1/install

And then follow the steps at cachix.org to create a new binary cache. Let's assume you made a cache named teddybear. When you've created a new cache, logged in with an API token and created a signing key, you can pipe nix-build to the Cachix client like so:

$ nix-build | cachix push teddybear

And other people using that cache will benefit from your premade dependency and binary downloads.

To use the cache somewhere, install the Cachix client and then run the following:

$ cachix use teddybear

I've been able to use my Go, Elm, Rust and Haskell dependencies on other machines using this. It's saved so much extra download time.

tl;dr

I was wrong about Nix. It's actually quite good once you get past the documentation being baroque and hard to read as a beginner. I'm going to try and do what I can to get the documentation improved.

As far as getting started with Nix, I suggest following these posts:

• Nix Pills: https://nixos.org/nixos/nix-pills/
• Nix Shorts: https://github.com/justinwoo/nix-shorts
• NixOS: For Developers: https://myme.no/posts/2020-01-26-nixos-for-development.html

Also, I really suggest trying stuff as a vehicle to understand how things work. I got really far by experimenting with getting this Discord bot I am writing in Rust working in Nix and have been very pleased with how it's turned out. I don't need to use rustup anymore to manage my Rust compiler or the language server. With a combination of direnv and lorri, I can avoid needing to set up language servers or the like at all. I can define them as part of the project environment and then trust the tools I build on top of to take care of that for me.

Give Nix a try. It's worth at least that much in my opinion.

Instant Pot Spaghetti

This is based on this recipe, but made only with things you can find in Costco. My fiancé and I have made this at least weekly for the last 8 months and we love how it turns out.

Recipe

Ingredients

• 1/2 kg ground beef (pre-cooked, or see section on browning it)
• 3 1/4 cups water
• 2 teaspoons salt
• a small amount of pepper
• 4 heaping teaspoons of garlic
• 1/2 cup butter
• 1/4 kg spaghetti noodles
• 1 jar of pasta sauce (about 870ml)

If you want it to be more spicy, add more pepper. Too much can make it hard to eat. Only experiment with the pepper amount after you've made this and decided there's not enough pepper.

Preparation

Put the ground beef in the instant pot. Put the water in the instant pot. Put the salt in the instant pot. Put the pepper in the instant pot. Put the garlic in the instant pot. Put the butter in the instant pot.

Stir for about 30 seconds, or until the garlic looks like it's distributed about evenly in the pot.

Take the spaghetti noodles and break them in half. Place about a third of one half one direction, the second third another, and the last yet another. Repeat this for the other half of the pasta. This helps to not clump it together when it's cooking.

Look at the package of spaghetti noodles. It should say something like "Ready in X minutes" with a large number. Take that number and subtract two from it. If you have a pasta that says it's cooked for 7 minutes, you will cook it for 5 minutes. If you have a pasta that says it's cooked for 9 minutes, you will cook it for 7 minutes.

Put the lid on the instant pot, seal it and ensure the pressure release valve is set to "sealing". Hit the "manual" button and select the number you figured out above.

Leave the instant pot alone for 10 minutes after it is done. This lets the pressure release naturally.

Use your serving utensil to open the pressure release valve. Stir and wait 3-5 minutes to serve. This makes 5 servings, but could be extended to more if you carefully ration it.

Serve hot with salt or parmesan cheese.

Browning Ground Beef

Browing ground beef is the act of cooking it all the way through so it is safe to eat. It's called "browing" it because the ground beef will turn a grayish brown when it is fully cooked.

Ingredients

• Olive oil
• 1 teaspoon salt
• The ground beef you want to brown

Preparation

Take the lid off of the instant pot. Cover the bottom of the pan in olive oil. Sprinkle the salt over the olive oil. Place the ground beef in the instant pot on top of the olive oil and salt.

Press the "sauté" button on your instant pot and use a spatula to break the ground beef into smaller chunks while it warms up. Mix the ground beef while it cooks. The goal is to make sure that all of the red parts turn grayish brown.

This will take anywhere from 5-10 minutes.

If you are using this ground beef for the above spaghetti recipe, you don't need to remove it from the instant pot. You can store extra ground beef in the fridge for use later.

Thoughts on Nix

EDIT(M02 20 2020): I've written a bit of a rebuttal to my own post here. I am keeping this post up for posterity.

I don't really know how I feel about Nix. It's a functional package manager that's designed to help with dependency hell. It also lets you define packages using Nix, which is an identically named yet separate thing. Nix has untyped expressions that help you build packages like this:

{ stdenv, fetchurl, perl }:

stdenv.mkDerivation {
  name = "hello-2.1.1";
  builder = ./builder.sh;
  src = fetchurl {
    url = ftp://ftp.nluug.nl/pub/gnu/hello/hello-2.1.1.tar.gz;
    sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
  };
  inherit perl;
}

In theory, this is great. It's obvious what needs to be done to the system in order for the "hello, world" package and what it depends on (in this case it depends on only the standard environment because there's no additional dependencies specified), to the point that this approach lets you avoid all major forms of DLL hell, while at the same time creating its own form of hell: nixpkgs, or the main package source of Nix.

Now, you may ask, how do you get that hash? Try and build the package with an obviously false hash and use the correct one from the output of the build command! That seems safe!

Let's say you have a modern app that has dependencies with npm, Go and Elm. Let's focus on the Go side for now. How would we do that when using Go modules?

{ pkgs ? import <nixpkgs> { } }:
let
x = buildGoModule rec {
  name = "Xe-x-${version}";
  version = "1.2.3";

  src = fetchFromGitHub {
    owner = "Xe";
    repo = "x";
    rev = "v${version}";
    sha256 = "0m2fzpqxk7hrbxsgqplkg7h2p7gv6s1miymv3gvw0cz039skag0s";
  };

  modSha256 = "1879j77k96684wi554rkjxydrj8g3hpp0kvxz03sd8dmwr3lh83j"; 

  subPackages = [ "." ]; 
}

in {
  x = x;
}

And this will fetch and build the entirety of my x repo into a single massive package that includes everything. Let's say I want to break it up into multiple packages so that I can install only one or two parts of it, such as my license command:

Let's make a function called gomod.nix that includes everything to build the go modules:

# gomod.nix
pkgs: repo: modSha256: attrs:
  with pkgs;
  let defaultAttrs = {
    src = repo;
    modSha256 = modSha256;
  };

  in buildGoModule (defaultAttrs // attrs)

And then let's invoke this with a few of the commands in there:

{ pkgs ? import <nixpkgs> { } }:
let
  stdenv = pkgs.stdenv;
  version = "1.2.3";
  repo = pkgs.fetchFromGitHub {
    owner = "Xe";
    repo = "x";
    rev = "v${version}";
    sha256 = "0m2fzpqxk7hrbxsgqplkg7h2p7gv6s1miymv3gvw0cz039skag0s";
  };

  modSha256 = "1879j77k96684wi554rkjxydrj8g3hpp0kvxz03sd8dmwr3lh83j";
  mk = import ./gomod.nix pkgs repo modSha256;

  appsluggr = mk {
    name = "appsluggr";
    version = version;
    subPackages = [ "cmd/appsluggr" ];
  };

  johaus = mk {
    name = "johaus";
    version = version;
    subPackages = [ "cmd/johaus" ];
  };

  license = mk {
    name = "license";
    version = version;
    subPackages = [ "cmd/license" ];
  };

  prefix = mk {
    name = "prefix";
    version = version;
    subPackages = [ "cmd/prefix" ];
  };

in {
  appsluggr = appsluggr;
  johaus = johaus;
  license = license;
  prefix = prefix;
}

And when we build this, we notice that ALL of the dependencies for my x repo (at least a hundred because it's got a lot of stuff in there) are downloaded FOUR TIMES, even though they don't change between them. I could avoid this by making each dependency its own Nix package, but that's not a productive use of my time.

Add on having to do this for the Node dependencies, and the Elm dependencies and this is at least 200 if not more packages needed for my relatively simple CRUD app that has creative choices in technology.

Oh, even better, the build directory isn't writable. So when your third-tier dependency has a generation step that assumes the build directory is writable, you suddenly need to become an expert in how that tool works so you can shunt it writing its files to another place. And then you need to make sure those files don't end up places they shouldn't be, lest you fill your disk with unneeded duplicate node_modules folders that really shouldn't be there in the first place (but are there because you gave up).

Then you need to make sure that works on another machine, because even though Nix itself is "functionally pure" (save the heat generated by the CPU executing your cloud-native, multitenant parallel adding service) this is a PACKAGE MANAGER. You know, the things that handle STATE, like FILES on the DISK. That's STATE. GLOBALLY MUTABLE STATE.

One of the main advantages of this approach is that the library dependencies of every project are easy to reproduce on other machines. Consider the ldd(1) (which shows the dynamic libraries associated with a program) output of ls on my Ubuntu system vs a package I installed from Nix:

$ ldd $(which ls)
        linux-vdso.so.1 (0x00007ffd2a79f000)
        libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f00f0e16000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f00f0a25000)
        libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f00f07b3000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f00f05af000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f00f1260000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f00f0390000)

All of these dependencies are managed by apt(8) and are supposedly reproducible on other Ubuntu systems. Compare this to the ldd(1) output of a Nix program:

$ ldd $(which dhall)
        linux-vdso.so.1 (0x00007fff0516a000)
        libm.so.6 => /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/libm.so.6 (0x00007fc20ed8d000)
        libz.so.1 => /nix/store/a3q9zl42d0hmgwmgzwkxi5qd88055fh8-zlib-1.2.11/lib/libz.so.1 (0x00007fc20ed6e000)
        libncursesw.so.6 => /nix/store/24xdpjcg2bkn2virdabnpncx6f98kgfw-ncurses-6.1-20190112/lib/libncursesw.so.6 (0x00007fc20ec8c000)
        libpthread.so.0 => /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/libpthread.so.0 (0x00007fc20ed4d000)
        librt.so.1 => /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/librt.so.1 (0x00007fc20ed43000)
        libutil.so.1 => /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/libutil.so.1 (0x00007fc20ed3c000)
        libdl.so.2 => /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/libdl.so.2 (0x00007fc20ed37000)
        libgmp.so.10 => /nix/store/4gmyxj5blhfbn6c7y3agxczrmsm2bhzv-gmp-6.1.2/lib/libgmp.so.10 (0x00007fc20ebf7000)
        libffi.so.7 => /nix/store/qa8wyi9pckq1d3853sgmcc61gs53g0d3-libffi-3.3/lib/libffi.so.7 (0x00007fc20ed2a000)
        libc.so.6 => /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/libc.so.6 (0x00007fc20ea41000)
        /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fc20ecfe000)

Each dynamic library dependency has its package hash in the folder path. This also means that the hash of its parent packages are present in there, which root all the way back to where/when its ultimate parent package was built. This makes Nix packages a kind of blockchain.

Nix also allows users to install their own packages into the global nix store at /nix. No, you can't change this, but you can symlink it to another place if you (like me) have a partition setup with / having less disk space than /home. You also need to set a special environment variable so Nix shuts up about you doing this. This is really fun on macOS Catalina where the root filesystem is read only. There is a workaround (that I had to trawl into the depths of Google page cache to get, because of course I did), but the Nix team themselves seem unaware of it.

So, to recap: Nix is an attempt at a radically different approach to package management. It assumes too much about the state of everything and puts odd demands on people as a result. Language-specific package managers can and will fight Nix unless they are explicitly designed to handle Nix's weirdness. As a side effect of making its package management system usable by normal users, it exposes the package manager database to corruption by any user mistake, curl2bash or malicious program on the system. All that functional purity uwu and statelessness can vanish into a puff of logic without warning.

But everything's immutable so that means it's okay right?

---

Based on this twitter thread but a LOT less sarcastic.

Dhall for Kubernetes

Kubernetes is a surprisingly complicated software package. Arguably, it has to be that complicated as a result of the problems it solves being complicated; but managing yaml configuration files for Kubernetes is a complicated task. YAML doesn't have support for variables or type metadata. This means that the validity (or sensibility) of a given Kubernetes configuration file (or files) isn't easy to figure out without using a Kubernetes server.

In my last post about Kubernetes, I mentioned I had developed a tool named dyson in order to help me manage Terraform as well as create Kubernetes manifests from a template. This works for the majority of my apps, but it is difficult to extend at this point for a few reasons:

• It assumes that everything passed to it are already valid yaml terms
• It doesn't assert the type of any values passed to it
• It is difficult to add another container to a given deployment
• Environment variables implicitly depend on the presence of a private git repo
• It depends on the template being correct more than the output being correct

So, this won't scale. People in the community have created other solutions for this like Helm, but a lot of them have some of the same basic problems. Helm also assumes that your template is correct. Kustomize does help with a lot of the type-safe variable replacements, but it doesn't have the ability to ensure your manifest is valid.

I looked around for alternate solutions for a while and eventually found Dhall thanks to a friend. Dhall is a statically typed configuration language. This means that you can ensure that inputs are always the correct type or the configuration file won't load. There's also a built-in dhall-to-yaml tool that can be used with the Kubernetes package in order to declare Kubernetes manifests in a type-safe way.

Here's a small example of Dhall and the yaml it generates:

-- Mastodon usernames
[ { name = "Cadey", mastodon = "@cadey@mst3k.interlinked.me" }
, { name = "Nicole", mastodon = "@sharkgirl@mst3k.interlinked.me" }
]

Which produces:

- mastodon: "@cadey@mst3k.interlinked.me"
  name: Cadey
- mastodon: "@sharkgirl@mst3k.interlinked.me"
  name: Nicole

Which is fine, but we still have the type-safety problem that you would have in normal yaml. Dhall lets us define record types for this data like this:

let User =
      { Type = { name : Text, mastodon : Optional Text }
      , default = { name = "", mastodon = None }
      }

let users =
      [ User::{ name = "Cadey", mastodon = Some "@cadey@mst3k.interlinked.me" }
      , User::{
        , name = "Nicole"
        , mastodon = Some "@sharkgirl@mst3k.interlinked.me"
        }
      ]

in  users

Which produces:

- mastodon: "@cadey@mst3k.interlinked.me"
  name: Cadey
- mastodon: "@sharkgirl@mst3k.interlinked.me"
  name: Nicole

This is type-safe because you cannot add arbitrary fields to User instances without the compiler rejecting it. Let's add an invalid "preferred_language" field to Cadey's instance:

-- ...
let users =
      [ User::{
        , name = "Cadey"
        , mastodon = Some "@cadey@mst3k.interlinked.me"
        , preferred_language = "en-US"
        }
      -- ...
      ]

Which gives us:

$ dhall-to-yaml --file example.dhall
Error: Expression doesn't match annotation

{ + preferred_language : …
, …
}

4│         User::{ name = "Cadey", mastodon = Some "@cadey@mst3k.interlinked.me",
5│       preferred_language = "en-US" }

example.dhall:4:9

Or this more detailed explanation if you add the --explain flag to the dhall-to-yaml call.

We tried to do something that violated the contract that the type specified. This means that it's an invalid configuration and is therefore rejected and no yaml file is created.

The Dhall Kubernetes package specifies record types for every object available by default in Kubernetes. This does mean that the package is incredibly large, but it also makes sure that everything you could possibly want to do in Kubernetes matches what it expects. In the package documentation, they give an example where a Deployment is created.

-- examples/deploymentSimple.dhall

-- Importing other files is done by specifying the HTTPS URL/disk location of
-- the file. Attaching a sha256 hash (obtained with `dhall freeze`) allows
-- the Dhall compiler to cache these files and speed up configuration loads
-- drastically.
let kubernetes =
      https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/1.15/master/package.dhall
      sha256:4bd5939adb0a5fc83d76e0d69aa3c5a30bc1a5af8f9df515f44b6fc59a0a4815
      
let deployment =
      kubernetes.Deployment::{
      , metadata = kubernetes.ObjectMeta::{ name = "nginx" }
      , spec =
          Some
            kubernetes.DeploymentSpec::{
            , replicas = Some 2
            , template =
                kubernetes.PodTemplateSpec::{
                , metadata = kubernetes.ObjectMeta::{ name = "nginx" }
                , spec =
                    Some
                      kubernetes.PodSpec::{
                      , containers =
                          [ kubernetes.Container::{
                            , name = "nginx"
                            , image = Some "nginx:1.15.3"
                            , ports =
                                [ kubernetes.ContainerPort::{
                                  , containerPort = 80
                                  }
                                ]
                            }
                          ]
                      }
                }
            }
      }

in  deployment

Which creates the following yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 2
  template:
    metadata:
      name: nginx
    spec:
      containers:
        - image: nginx:1.15.3
          name: nginx
          ports:
            - containerPort: 80

Dhall's lambda functions can help you break this into manageable chunks. For example, here's a Dhall function that helps create a docker image reference:

let formatImage
    : Text -> Text -> Text
    = \(repository : Text) -> \(tag : Text) ->
    "${repository}:${tag}"

in formatImage "xena/christinewebsite" "latest"

Which outputs xena/christinewebsite:latest when passed to dhall text.

All of this adds up into a powerful toolset that lets you express Kubernetes configuration in a way that does what you want without as many headaches.

Most of my apps on Kubernetes need only a few generic bits of configuration:

• Their name
• What port should be exposed
• The domain that this service should be exposed on
• How many replicas of the service are needed
• Which Let's Encrypt Issuer to use (currently only "prod" or "staging")
• The configuration variables of the service
• Any other containers that may be needed for the service

From here, I defined all of the bits and pieces for the Kubernetes manifests that Dyson produces and then created a Config type that helps to template them out. Here's my Config type definition:

let kubernetes = ../kubernetes.dhall

in  { Type =
        { name : Text
        , appPort : Natural
        , image : Text
        , domain : Text
        , replicas : Natural
        , leIssuer : Text
        , envVars : List kubernetes.EnvVar.Type
        , otherContainers : List kubernetes.Container.Type
        }
    , default =
        { name = ""
        , appPort = 5000
        , image = ""
        , domain = ""
        , replicas = 1
        , leIssuer = "staging"
        , envVars = [] : List kubernetes.EnvVar.Type
        , otherContainers = [] : List kubernetes.Container.Type
        }
    }

Then I defined a makeApp function that creates everything I need to deploy my stuff on Kubernetes:

let Prelude = ../Prelude.dhall

let kubernetes = ../kubernetes.dhall

let typesUnion = ../typesUnion.dhall

let deployment = ../http/deployment.dhall

let ingress = ../http/ingress.dhall

let service = ../http/service.dhall

let Config = ../app/config.dhall

let K8sList = ../app/list.dhall

let buildService =
        \(config : Config.Type)
      -> let myService = service config

         let myDeployment = deployment config

         let myIngress = ingress config

         in  K8sList::{
             , items =
               [ typesUnion.Service myService
               , typesUnion.Deployment myDeployment
               , typesUnion.Ingress myIngress
               ]
             }

in  buildService

And used it to deploy the h language website:

let makeApp = ../app/make.dhall

let Config = ../app/config.dhall

let cfg =
      Config::{
      , name = "hlang"
      , appPort = 5000
      , image = "xena/hlang:latest"
      , domain = "h.christine.website"
      , leIssuer = "prod"
      }

in  makeApp cfg

Which produces the following Kubernetes config:

apiVersion: v1
items:
  - apiVersion: v1
    kind: Service
    metadata:
      annotations:
        external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
        external-dns.alpha.kubernetes.io/hostname: h.christine.website
        external-dns.alpha.kubernetes.io/ttl: "120"
      labels:
        app: hlang
      name: hlang
      namespace: apps
    spec:
      ports:
        - port: 5000
          targetPort: 5000
      selector:
        app: hlang
      type: ClusterIP
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: hlang
      namespace: apps
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: hlang
      template:
        metadata:
          labels:
            app: hlang
          name: hlang
        spec:
          containers:
            - image: xena/hlang:latest
              imagePullPolicy: Always
              name: web
              ports:
                - containerPort: 5000
          imagePullSecrets:
            - name: regcred
  - apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        certmanager.k8s.io/cluster-issuer: letsencrypt-prod
        kubernetes.io/ingress.class: nginx
      labels:
        app: hlang
      name: hlang
      namespace: apps
    spec:
      rules:
        - host: h.christine.website
          http:
            paths:
              - backend:
                  serviceName: hlang
                  servicePort: 5000
      tls:
        - hosts:
            - h.christine.website
          secretName: prod-certs-hlang
kind: List

And when I applied it on my Kubernetes cluster, it worked the first time and had absolutely no effect on the existing configuration.

In the future, I hope to expand this to allow for multiple deployments (IE: a chatbot running in a separate deployment than a web API the chatbot depends on or non-web projects in general) as well as supporting multiple Kubernetes namespaces.

Dhall is probably the most viable replacement to Helm or other Kubernetes templating tools I have found in recent memory. I hope that it will be used by more people to help with configuration management, but I can understand that that may not happen. At least it works for me.

If you want to learn more about Dhall, I suggest checking out the following links:

• The Dhall Language homepage
• Learn Dhall in Y Minutes
• The Dhall Language GitHub Organization

I hope this was helpful and interesting. Be well.

Live Streaming Server Setup

I have set up my own RTMP server that allows me to live stream to my own infrastructure. This allows me to own my own setup and not need to rely on other services such as Twitch or YouTube. As a side effect of doing this, I can enable people who use my streaming server to use picture-in-picture mode in iPadOS without having to hack the streaming app, among other things.

This is part of my 2020 goal to reduce my dependencies on corporate social platforms as much as possible.

I chose to do my setup with a few key parts:

• docker-nginx-rtmp
• hls.js
• stream.html
• Caddy

RTMP Server

I chose to use docker-nginx-rtmp as a pre-packaged solution for my RTMP server. This means I could set it up to ingest via my WireGuard VPN with very little work. Here is the docker command I run on my VPN host:

$ docker run \
  --restart always \
  -dit \
  -p 10.77.0.1:1935:1935 \
  -p 127.0.0.1:8080:80 \
  --name rtmp-server \
  alfg/nginx-rtmp

This starts my RTMP server in a container named rtmp-server and automatically restarts it when it goes down. The IP address in the first --port (-p) flag is the VPN IP address of my main VPN server. This makes me have to be behind my VPN in order to stream to my server, given the total lack of authentication that's involved in RTMP.

stream.html

I have a custom stream page set up on my server that has a friendly little wrapper to the video player. Here is the source code for it. It's very short and easy to follow. I have these files at /srv/http/home.cetacean.club on my VPN server.

This wraps hls.js so that users on every browser I care to support can watch the stream as it happens.

Caddy

In order to expose the stream data to the world, I use Caddy as a reverse proxy. Here is the configuration that I use for Caddy:

home.cetacean.club {
  # Set up automagic Let's Encrypt
  tls me@christine.website

  # Proxy the playlist, stream data 
  # and statistics to the rtmp server
  proxy /hls http://127.0.0.1:8080
  proxy /live http://127.0.0.1:8080
  proxy /stat http://127.0.0.1:8080

  # make /stream.html show up as /stream
  ext .html
  
  # serve data out of /srv/http/home.cetacan.club
  # you can put your HTTP document root
  # anywhere you want, but I like it being
  # here.
  root /srv/http/home.cetacean.club
}

For more information on the Caddy configuration directives used here, see the following:

• tls
• proxy
• ext
• root

Caveats

Live streaming like this uses ABSURD amounts of bandwidth. Do not set this up on a server that has limited bandwidth. If you need a server that has unlimited bandwidth, check out SoYouStart. It's what I use.

There isn't a good story for recording or announcing streams to this server automatically. I don't consider this a problem, as links can always be sent out manually on social media platforms.

I hope this little overview of my setup was informative. I'll be streaming there very irregularly, mostly as time permits/the spirit moves me. I plan to stream art, gaming and code.

Thanks for reading, have a good day.

Created with Procreate on iPadOS using an iPad Pro and an Apple Pencil.

Time-lapse video - [PSD on my Patreon] (https://www.patreon.com/posts/i-arted-my-33016810)

Based on a base by DeviantArtist Kawaii-princess-paws.

V is for Vvork in Progress

So, December has come and passed. I'm excited to see V 1.0 get released as a stable production-ready release so I can write production applications in it!

NOTE: I was asked to write this post after version 1.0 was released in December.

Looking at the description of their github repo over time, let's see how things changed:

Date from Archive.org	Stable Release	Date
April 24, 2019	Not mentioned
June 22, 2019	Implied	June 22, 2019
June 23, 2019	Not mentioned
July 21, 2019	1.0	December 2019
September 8, 2019	1.0	December 2019
October 26, 2019	1.0	December 2019
November 19, 2019	0.2	November 2019
December 4, 2019	0.2	December 2019

As of the time of writing this post, it is January third, 2020 and the roadmap is apparently to release V 0.2 this month.

Let's see what's been fixed since my last article.

Compile Speed

I have gotten feedback that the metric I used for testing the compile speed claims was an unfair benchmark. Apparently it's not reasonable to put 1.2 million printfs in the same function. I'm going to fix this by making the test a bit more representative of real world code.

#!/usr/bin/env moon
-- this is Moonscript code: https://moonscript.org

with io.popen "mkdir hellomodule"
  print \read "*a"
  \close!

for i=1, 1000
  with io.open "hellomodule/file_#{i}.v", "w"
    \write "module hellomodule\n\n"
    for j=1, 1200
      \write "pub fn print_#{i}_#{j}() { println('hello, #{i} #{j}!') }\n\n"
    \close!

This creates 1000 files with 1200 functions in them. These numbers were derived from the greatest factor pairs of 1.2 million. If V lives up to its claims that it can build 1.2 million lines of code in a second, this should only take one second to run:

$ moon gen.moon
$ time ~/code/v/v build module $(pwd)/hellomodule/
Building module "hellomodule" (dir="/home/cadey/tmp/vmeme/moon/hellomodule")...
Generating a V header file for module `/home/cadey/tmp/vmeme/moon/hellomodule`
/home/cadey/code/v//home/cadey/tmp/vmeme/moon/hellomodule
Building /home/cadey/.vmodules//home/cadey/tmp/vmeme/moon/hellomodule.o...
599.37user 13.35system 10:16.92elapsed 99%CPU (0avgtext+0avgdata 17059740maxresident)k
0inputs+2357808outputs (0major+7971041minor)pagefaults 0swaps

It took over 10 minutes to compile 1.2 million lines of code. Some interesting statistics about this run:

• GCC's oom score from the kernel task scheduler topped out at over 496
• GCC used over 16 GB of ram
• The V compiler used over 3 GB of ram
• This is an average of 2000 lines of code per second!

As of the time of writing this article, the main V website mentions that the compiler should handle 100,000 lines of code per second, or that it should compile code approximately 500 times as fast as it does currently.

This does not seem to be the case. It would be nice if the V author could clarify how he got his benchmarks and make his process public. Here's the /proc/cpuinfo of the machine I ran this test on:

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 58
model name      : Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz
stepping        : 9
microcode       : 0x20
cpu MHz         : 1596.375
cache size      : 8192 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
                  cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
                  pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts 
                  rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni
                  pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 
                  xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer
                  aes xsave avx f16c rdrand lahf_lm cpuid_fault pti ssbd ibrs 
                  ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase 
                  smep erms xsaveopt dtherm ida arat pln pts flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 6784.45
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

The resulting object file is 280 MB (surprising given the output of the generator script was only 67 MB).

$ cd ~/.vmodules/home/cadey/tmp/vmeme/moon/

$ ls
hellomodule.o

$ du -hs hellomodule.o
280M    hellomodule.o

Let's see how big the resulting binary is for calling one of these functions:

// main.v
import mymodule

fn main() {
  mymodule.print_1_1()
}

$ ~/code/v/v build main.v
main.v:1:14: cannot import module "mymodule" (not found)
    1| import mymodule
                    ^
    2|
    3| fn main() {

...oh dear. Can someone file this as an issue for me? I was following the directions here and I wasn't able to get things working. I can't open issues myself because I've been banned from the V issue tracker, or I would have already.

Can we recover this with gcc? Let's get the symbol name with nm(1):

$ nm hellomodule.o  | grep print_1_1'$'
0000000000000000 T hellomodule__print_1_1

So the first print function is exported as hellomodule__print_1_1, and it was declared as:

pub fn print_1_1() { println('hello, 1 1!') }

This means we should be able to declare/use it like we would a normal C function that returns void and without arguments:

// main.c

void hellomodule__print_1_1();

void main__main() {
  hellomodule__print_1_1();
}

I copied hellomodule.o to the current working directory to test this. I also used the C output of the hello world program below and replaced the main__main function with a forward declaration. I called this hello.c. This is a very horrible no good hack but it worked enough to pass the linker's muster. Not doing this caused this shower of linker errors.

$ gcc -o main.o -c main.c
$ gcc -o hello.o -c hello.c
$ gcc -o main hellomodule.o main.o hello.o
$ ./main
hello, 1 1!

$ du -hs main
179M    main

Yikes. Let's see if we can reduce the binary size at all. strip(1) usually helps with this:

$ strip main
$ du -hs main
121M    main

Well that's a good chunk of it shaved off at least. It looks like there's no dead code elimination at play. This probably explains why the binary is so big.

$ strings main | grep hello | wc -l
1200000

Yep! It has all the strings. That's gonna be big no matter what you do. Maybe there could be some clever snipping of things, but it's reasonable for that to not happen by default.

Hello World Leak

One of the things I noted in my last post was that the Hello world program leaked memory. Let's see if this still happens:

// hello.v
fn main() {
        println('Hello, world!')
}

$ ~/code/v/v build hello.v
$ valgrind ./hello
==31465== Memcheck, a memory error detector
==31465== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==31465== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==31465== Command: ./hello
==31465==
Hello, world!
==31465==
==31465== HEAP SUMMARY:
==31465==     in use at exit: 0 bytes in 0 blocks
==31465==   total heap usage: 2 allocs, 2 frees, 2,024 bytes allocated
==31465==
==31465== All heap blocks were freed -- no leaks are possible
==31465==
==31465== For counts of detected and suppressed errors, rerun with: -v
==31465== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Nice! Let's see if the compiler leaks while building it:

$ valgrind ~/code/v/v build hello.v
==32295== Memcheck, a memory error detector
==32295== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==32295== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==32295== Command: /home/cadey/code/v/v build hello.v
==32295==
==32295==
==32295== HEAP SUMMARY:
==32295==     in use at exit: 4,600,383 bytes in 74,522 blocks
==32295==   total heap usage: 76,590 allocs, 2,068 frees, 6,452,537 bytes allocated
==32295==
==32295== LEAK SUMMARY:
==32295==    definitely lost: 2,372,511 bytes in 56,223 blocks
==32295==    indirectly lost: 2,210,724 bytes in 18,077 blocks
==32295==      possibly lost: 0 bytes in 0 blocks
==32295==    still reachable: 17,148 bytes in 222 blocks
==32295==         suppressed: 0 bytes in 0 blocks
==32295== Rerun with --leak-check=full to see details of leaked memory
==32295==
==32295== For counts of detected and suppressed errors, rerun with: -v
==32295== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

For comparison, this compile leaked 3,861,785 bytes of ram last time. This means that the compiler has overall gained 0.8 megabytes of leak in the last 6 months. This is worrying, given that V claims to not have a garbage collector. I can only wonder how much ram was leaked when building that giant module.

If your V program compiles, it's guaranteed that it's going to be leak free.

Quoted from here.

For giggles, let's see if V in module mode leaks ram somehow:

$ valgrind ./main
==15483== Memcheck, a memory error detector
==15483== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==15483== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==15483== Command: ./main
==15483==
hello, 1 1!
==15483==
==15483== HEAP SUMMARY:
==15483==     in use at exit: 0 bytes in 0 blocks
==15483==   total heap usage: 2 allocs, 2 frees, 2,024 bytes allocated
==15483==
==15483== All heap blocks were freed -- no leaks are possible
==15483==
==15483== For counts of detected and suppressed errors, rerun with: -v
==15483== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Nope! The hello world memory leak was actually fixed!

Other Claims

• Vweb was shipped
• Hot code reloading was shipped
• Code translation is still vaporware
• The compiler generates direct machine code

Code Translation

I've been really looking forward to this to see how 1:1 it can make the output. Let's see if you can use it.

$ ~/code/v/v help | grep translate
  translate         Translates C to V. [wip, will be available in V 0.3]

$ ~/code/v/v translate
Translating C to V will be available in V 0.3 (January)

This is confusing to me given he claims that 0.2 will be out in January, but whatever I can let this slide.

The doom example is still only one file that doesn't even compile anymore.

I really do like how it handles extern functions though, you just declare them without bodies like C. Then it just figures things out for you. I wonder if this works with syscall functions too.

The Compiler Generates Direct Machine Code

In my testing I was unable to figure out how to get the V compiler to generate direct machine code. Until an example of this is released, I am quite skeptical of this claim.

---

Overall, V is a work in progress. It has made a lot of progress since the last time I talked about it, but the 1.0 release promise has been shattered. If I was going to suggest anything to the V author, don't give release dates or timetables. This kind of thing needs to be ready when it's ready and no sooner.

Also if you are writing a compiler and posting benchmarks, please make my life easier when trying to verify them. Put the entire repo you're using for the benchmarks somewhere. Include the exact commands you used to collect those benchmarks. Make it obvious how they were collected, what hardware they were run on, etc. This stuff really helps a lot when trying to verify them. Otherwise I have to guess, and I might get it wrong. I don't know if my benchmark is an entirely fair one, but given the lack of information on how to replicate it it's probably going to have to do.

Don’t ever, ever try to lie to the Internet, because they will catch you. They will deconstruct your spin. They will remember everything you ever say for eternity.

- Gabe Newell

How I set up an IRC daemon on Kubernetes

IRC. It's one of the last bastions of the old internet, and still an actively developed and researched protocol. Historically, IRC daemons have been notoriously annoying to set up and maintain. I have created an IRC daemon running on top of Kubernetes, which will hopefully help remove a lot of the pain points for my personal usage. Here's how I did it.

IRC is a simple protocol and only has a few major moving parts. IRC is made up of networks of servers that federate together as one logical unit. IRC is scalable from networks spanning one server to hundreds (though realistically you're not likely to find more than about 10 servers in a network).

At its core, IRC daemons are a pub-sub protocol that also has a distributed state layer on top of it. TCP connections can either represent individual users or server trunking. Each user has their own state (nickname, ident and "real name"). Users can join channels which can have their own state (modes, topic, timestamp and ban lists). Some servers have a limit of the number of channels you can join.

So, with this in mind, let's start with a simple IRC daemon in a docker container. I chose ngircd for this because it's packaged in Alpine Linux. Let's create the configuration file ngircd.conf:

[Global]
Name = seaworld.yolo-swag.com
AdminInfo1 = ShadowNET Main Server
AdminInfo2 = New York, New York, USA
AdminInfo3 = Cadey Ratio <me@christine.website>
Info = Hosted on Kubernetes!
Listen = 0.0.0.0
MotdFile = /shadownet/motd
Network = ShadowNET
Ports = 6667
ServerGID = 65534
ServerUID = 65534

[Limits]
MaxJoins = 50
MaxNickLength = 31
MaxListSize = 100
PingTimeout = 120
PongTimeout = 20

[Options]
AllowedChannelTypes = #&+
AllowRemoteOper = yes
CloakUserToNick = yes
DNS = no
Ident = no
IncludeDir = /shadownet/secret
MorePrivacy = yes
NoticeBeforeRegistration = yes
OperCanUseMode = yes
OperChanPAutoOp = yes
PAM = no
PAMIsOptional = yes
RequireAuthPing = yes
# WebircPassword is set in secrets

[Channel]
Name = #lobby
Topic = Welcome to the new ShadowNET!
Modes = tn

[Channel]
Name = #help
Topic = Get help with ShadowNET | Ping an oper for help
Modes = tn

[Channel]
Name = #opers
Topic = Oper hideout
Modes = tnO

This is mostly based on the default settings in the example configuration file with a few glaring exceptions:

• The server name is seaworld.yolo-swag.com, which will show up when users are connecting
• My information is filled out for the admin information (which is shown when a user does /ADMIN in their client)
• It has a lot of privacy-enhancing features set up
• It disables the need to authenticate with PAM before being allowed to connect to the IRC server
• Some default channel names are reserved

So, let's create a dockerfile for this:

FROM xena/alpine
COPY motd /shadownet/motd
COPY ngircd.conf /shadownet/ngircd.conf
RUN apk --no-cache add ngircd
COPY run.sh /
CMD ["/run.sh"]

motd is a plain text file that is used as the "message of the day" when users connect. Servers usually list their rules here. My motd has some ascii art and has this extra info:

The *new* irc.yolo-swag.com!

Connect on irc.within.website port 6667 or r4qrvdln2nvqyfbq.onion:6667

Rules:
- Don't do things that make me have to write more rules here
- This rule makes you breathe manually

Now you can build and push this image to the docker hub.

You may have noticed earlier that a comment in the config file mentioned [webirc][webirc]. This is important for us because IRC server normally assume that the remote host information in socket calls is accurate. My Kubernetes setup has at least one level of TCP proxying at work, so this cannot pan out. Webirc offers an authenticated mechanism to let a proxy server lie about user IP addresses. My nginx-ingress setup uses the haproxy PROXY protocol to let underlying services know client IP addresses. So what we need is an adaptor from haproxy PROXY protocol to webirc. I hacked one up:

package main

import (
	"crypto/md5"
	"flag"
	"fmt"
	"io"
	"log"
	"net"
	"strings"

	"github.com/armon/go-proxyproto"
	"github.com/facebookgo/flagenv"
	_ "github.com/joho/godotenv/autoload"
	irc "gopkg.in/irc.v3"
)

var (
	webircPassword = flag.String("webirc-password", "", "the password for WEBIRC")
	webircIdent    = flag.String("webirc-ident", "snet", "the ident for WEBIRC")
	webircHost     = flag.String("webirc-host", "", "the host to connect to for WEBIRC")
	port           = flag.String("port", "5667", "port to listen on for PROXY traffic")
)

func main() {
	flagenv.Parse()
	flag.Parse()

	list, err := net.Listen("tcp", ":"+*port)
	if err != nil {
		panic(err)
	}

	log.Printf("now listening on port %s, forwarding traffic to %s", *port, *webircHost)

	proxyList := &proxyproto.Listener{Listener: list}

	for {
		conn, err := proxyList.Accept()
		if err != nil {
			log.Println(err)
			continue
		}
		go dataTo(conn)
	}
}

func dataTo(conn net.Conn) {
	defer conn.Close()

	ip, _, err := net.SplitHostPort(conn.RemoteAddr().String())
	if err != nil {
		log.Printf("what, can't split remote address: %v", err)
		ev := irc.Message{
			Command: "QUIT",
			Params: []string{
				"***",
				err.Error(),
			},
		}

		fmt.Fprintln(conn, ev.String())
		return
	}

	peer, err := net.Dial("tcp", *webircHost)
	if err != nil {
		log.Println(*webircHost, err)
	}
	defer peer.Close()

	spip := strings.Split(ip, ".")

	hostname := strings.Join([]string{
		"snet",
		Hash("snet", spip[0])[:8],
		Hash("snet", spip[0] + spip[1])[:8],
		Hash("snet", spip[0] + spip[1] + spip[2] + spip[3])[:8],
	}, ".")

	ev := irc.Message{
		Command: "WEBIRC",
		Params: []string{
			*webircPassword,
			*webircIdent,
			ip,
			hostname,
		},
	}
	log.Println(ev.String())
	fmt.Fprintf(peer, "%s\r\n", ev.String())

	go io.Copy(conn, peer)
	io.Copy(peer, conn)
}

// Hash is a simple wrapper around the MD5 algorithm implementation in the
// Go standard library. It takes in data and a salt and returns the hashed
// representation.
func Hash(data string, salt string) string {
	output := md5.Sum([]byte(data + salt))
	return fmt.Sprintf("%x", output)
}

This proxies connections from incoming TCP sockets to the IRC server. It also creates a fancy hostname for ngircd to use when people do a /whois on users. ngircd does have its own cloaking mechanism (which I am not using here), but I figure doing the splitting on IP address classes will make a more easy way to reliably ban users from channels.

Now, let's build this as a docker image and push it to the docker hub:

FROM xena/go:1.13.5 AS build
WORKDIR /shadownet
COPY go.mod .
COPY go.sum .
ENV GOPROXY https://cache.greedo.xeserv.us
RUN go mod download
COPY cmd ./cmd
RUN GOBIN=/shadownet/bin go install ./cmd/proxy2webirc

FROM xena/alpine
COPY --from=build /shadownet/bin/proxy2webirc /usr/local/bin/proxy2webirc
CMD ["/usr/local/bin/proxy2webirc"]

And now we get to wire this all up in a kubernetes manifest. Let's create a namespace:

# 00_namespace.yml
apiVersion: v1
kind: Namespace
metadata:
  name: ircd

And now we need to create the secrets that the IRC daemon will use when operating. We need the webirc password and a few operator blocks. Let's make a script to create operator blocks:

#!/bin/sh
# scripts/makeoper.sh

echo "[Operator]
Name = $1
Password = $(uuidgen)"

Then let's use it to create a few operator configs:

$ scripts/makeoper.sh Cadey >> opers.conf
$ scripts/makeoper.sh h >> opers.conf

And then create the webirc password:

$ echo "[Options]
WebircPassword = $(uuidgen)" >> webirc.conf

And then let's load these into a yaml file:

# 01_secrets.yml
apiVersion: v1
kind: Secret
metadata:
  name: config
  namespace: ircd
type: Opaque
stringData:
  opers.conf: |
    <contents of opers.conf>
  webirc.conf: |
    <contents of webirc.conf>

Now all we need is the irc daemon deployment itself that ties this all together:

# 02_ircd.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ircd
  namespace: ircd
  labels:
    app: ircd
spec:
  replicas: 1
  template:
    metadata:
      name: ircd
      labels:
        app: ircd
    spec:
      containers:
      - name: proxystrip
        image: shadownet/proxy2webirc:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 5667
          name: proxiedirc
          protocol: TCP
        env:
        - name: WEBIRC_HOST
          value: 127.0.0.1:6667
        - name: WEBIRC_PASSWORD
          value: <password from webirc.conf>
      - name: ircd
        image: shadownet/ircd:latest
        imagePullPolicy: Always
        volumeMounts:
        - name: secretconfig
          mountPath: "/shadownet/secret"
      restartPolicy: Always
      volumes:
      - name: secretconfig
        secret:
          secretName: config
  selector:
    matchLabels:
      app: ircd
---
apiVersion: v1
kind: Service
metadata:
  name: ircd
  namespace: ircd
  labels:
    app: ircd
spec:
  ports:
  - port: 6667
    targetPort: 5667
    protocol: TCP
  selector:
    app: ircd
  type: NodePort

This will set up our IRC daemon to read the secrets from the filesystem at /shadownet/secret, which was configured as the IncludeDir in the ngircd config above.

At this point, your IRC daemon is ready to go and can be applied to your cluster whenever you want, however it may be interesting to set up a tor onion address for the IRC server. Using the tor operator, we can create a private key locally, load it as a kubernetes secret and then activate the tor hidden service:

$ openssl genrsa -out private_key 1024
$ kubectl create secret -n ircd generic ircd-tor-key --from-file=private_key

Now apply this manifest:

# 03_onion.yml
apiVersion: tor.k8s.io/v1alpha1
kind: OnionService
metadata:
  name: ircd
spec:
  version: 2
  selector:
    app: ircd
  ports:
    - targetPort: 6667
      publicPort: 6667
  privateKeySecret:
    name: ircd-tor-key 
    key: private_key

Now, you should be able to let users connect to your IRC server to their heart's content. If you want to join the IRC server I've set up, point your IRC client at irc.within.website. I'll be in #lobby.

Olin Improvements

Over the last week or so I've been doing a lot of improvements to Olin in order to make it ready to be the kernel for the minimum viable product of wasmcloud. Here's an overview of the big things that have happened from version 0.1.1 to version 0.4.0.

What is Olin?

Olin is a userspace kernel designed for multi-tenant secure computing. It provides isolation via WebAssembly to limit the attack scope of malicious user input, resource accounting via its runtime statistics, and a familiar Unix-like API. It is the core that you can build a functions as a service platform on top of. For an example of Olin in action, please click here.

As Olin is just a kernel, it needs some work in order to really shine as a true child of the cloud. That work is incoming during the next weeks and months.

Announcements

Here is what has been done since the last Olin post:

• An official, automated build of the example Olin components has been published to the Docker Hub
• The Go ABI has been deprecated for the moment
• The entrypoint of Olin programs has changed to _start
• The beginning of support in the Zig standard library
• Official binfmt_misc rigging has been created for experimentation

Official Docker Hub Build

The Docker Hub repo xena/olin now is automatically built off of the latest master release of Olin.

To use this image, run the following commands:

$ docker pull xena/olin:latest
$ docker run --rm -it xena/olin:latest sh

Then you can use the cwa tool to run programs in /wasm. See cwa -help for more information.

Deprecation of Go Support

For the moment, I am deprecating support for Go in GOOS=js GOARCH=wasm. The ABI for the Go compiler in this mode is too unstable for me right now. If other people want to fix abi/wasmgo to support Go 1.13 and newer, I would be very welcome to the patches.

The Entrypoint is Now _start()

Early on in the experiments that make up Olin, I have made a mistake in my fundamental understanding of how operating systems run programs. I thought that the main function would return the exit code of the program. This is not the case. There is a small shim that wraps the main function of your language and passes the result of it to exit(). Olin now copies this behavior. In order to return a value to the Olin runtime, you can either call runtime_exit() or return from the _start() function to exit with 0. Many thanks to Andrew Kelly for helping me realize this error.

This behavior is copied in the Olin rust package, which now has a fancy macro to automate the creation of the _start() function.

I am waiting on Zig to release a new nightly version in order to enable it, but the bring-your-own-OS package support in Zig means that the Zig standard library is starting to be exposed into Olin programs. Here's an example based on the example program:

pub const os = @import("./olin/olin.zig");
const std = @import("std");

pub fn main() anyerror!void {
    std.debug.warn("All your base are belong to us.\n", .{});
}

binfmt_misc Rigging

For a while I've had a binfmt_misc configuration floating around in the Olin repo. Here's how to use it:

First, install Olin's cmd/cwa to /usr/local/bin:

$ cd cmd/cwa
$ go build
$ sudo mv cwa /usr/local/bin

Then activate the binfmt_misc configuration:

$ cd ../../run/binfmt_misc
$ cat cwa.cfg | sudo tee /proc/sys/fs/binfmt_misc/register

Then you can run Olin programs without calling cwa:

$ ./olinfetch.wasm

Features

Policy Support

Olin now has a declarative policy engine for accessing external resources. This is inspired from OpenBSD pledge() and macOS sandboxing. These policies allow setting the following attributes:

• Resources an Olin program can access, matched by regular expressions
• Resources an Olin program CANNOT access, matched by regular expressions
• The maximum amount of memory an Olin program can use
• The maximum number of WebAssembly instructions a WebAssembly program can execute

Here's an example policy file intended to help with relaying webhooks:

## This is an example policy, the ## signifies this line is a comment.

## These are the URL patterns that this handler can open:
allow (
  ^https://tulpa.dev
  ^https://discordapp.com/api/webhooks/
  ^random://$
)

## These are the URL patterns that this handler cannot open:
disallow (
  ^https://tulpa.dev/admin.*$
)

## This is the ram limit in pages (64k each):
ram-page-limit 128

## This is the gas limit in instructions:
gas-limit 1048576

This would allow a WebAssembly program to open a HTTP socket to https://tulpa.dev (my git server) and Discord, but disallows any administrative API calls to my git server. It also allows the Olin program to use up to 128 pages of memory (about 8MB, which goes surprisingly far) and 1.04 million instructions. If the handler tries to open any resource that is not explicitly allowed, it is killed. If the handler tries to open a resource that is explicitly forbidden, it is killed. If the handler uses too much ram or too many instructions, it is killed.

This allows handlers to safely process user controlled input and even use that as part of the call to the open function.

When policies are violated, the error thrown is a vibe check failure:

$ cwa -policy ../policy/testdata/gitea.policy httptest.wasm
httptest.wasm: 2019/12/13 13:16:15 info: making request 
  to https://xena.greedo.xeserv.us/files/hello_olin.txt
httptest.wasm: 2019/12/13 13:16:15 vibe check failed: 
  https://xena.greedo.xeserv.us/files/hello_olin.txt 
  forbidden by policy
2019/12/13 13:16:15 httptest.wasm: exit status -1

runtime_exit() System Call

Along with making _start() the entrypoint, there comes a new problem: exiting. I fixed this by adding a runtime_exit() system call in Olin. When you call this function with the status code you want to return, execution of the Olin program instantly ends, uncleanly stopping everything and closing all files the program has open. This is similar to Linux's exit() system call.

It's probably best to save this call for cases where the program really can't/shouldn't continue executing, like for panic handlers.

Generic CGI Support

Previously there was a half-baked idea I called cwagi in Olin's codebase. The idea was to emulate part of how CGI worked in order to let Olin programs handle HTTP easily. I realize this was a mistake, so now it just supports normal CGI, conforming to RFC 3875.

End of File Error

One of the more common errors in operating systems is the "end of file" error. It is raised when a file has no more data in it. Olin didn't have this, but now it does.

Wasmcloud Features

Thanks to these improvements, the following wasmcloud features have been implemented:

• Updating handlers (wasmcloud update) link
• Deleting handlers (wasmcloud delete) link
• Listing deleted handlers (wasmcloud list -show-deleted)
• Brand outgoing HTTP requests link

As of the writing of this post, wasmcloud is currently 75% through the MVP development cycle. Here are the remaining issues:

• CGI support for handlers link
• Policy support for handlers link
• Configuration variables for handlers link

---

Overall, this project is fun. Here's to 1.0 happening soon! Be well.

Wasmcloud Progress: Hello, World!

I have been working off and on over the years and have finally created the base of a functions as a service backend for WebAssembly code. I'm code-naming this wasmcloud. Wasmcloud is a pre-alpha prototype and is currently very much work in progress. However, it's far enough along that I would like to explain what I have been doing for the last few years and what it's all built up to.

Here is a high level view of all of the parts that make up wasmcloud and how they correlate:

Land: The Beginning

A little bit after I found WebAssembly I started to play with it. It seemed like it was too good to be true. A completely free and open source VM format that would run on almost any platform? Sounds like the kind of black magick witchcraft you hear about on Star Trek.

However, I kept at it and continued experimenting. I eventually came up with Land. This was a very simple thing and was really used to help me invent Dagger.

Dagger was an attempt at an incredible amount of minimalism. I based it on an extreme interpretation of the Unix philosophy (everything is a file -> everything is a bytestream) combined with some Plan 9 for flavor. It had only 5 system calls:

• open() - opens a stream by URL, returning a stream descriptor
• close() - closes a stream descriptor
• read() - reads from a stream
• write() - writes to a stream
• flush() - flushes intermediate data and turns async behavior into syncronous behavior

And yet this was enough to implement a HTTP client.

The core guiding idea was that a cloud-native OS API should expose internet resources as easily as it exposes native resources. It should be as easy to use WebSockets as it is to use normal sockets. Additionally, all of the details should be abstracted away from the WebAssembly module. DNS resolution is not its job. TLS configuration is not its job. Its job is to run your code. Everything else should just be provided by the system.

I wrote a blogpost about this work and even did a talk at GoCon Canada about it.

And this worked for several months as I learned WebAssembly and started to experiment with bigger and better things.

Olin: Phase 2

Land taught me a lot. I started to quickly run into the limits of Dagger though. I ended up needing calls like non-cryptographic entropy, environment variables, command-line arguments and getting the current time. After doing some research (and trying/failing to implement my own such API based on newlib) I found a library and specification called CommonWA. This claimed to offer a lot of what I was looking for. Namely URLs as filenames and all of the host interop support I could hope for. I named this platform Olin, or the One Language Intelligent Network.

However the specification was somewhat dead. The author of it had largely moved on to more ferrous pastures and I became one of the few users of it. I ended up forking the specification and implementing my view of what it should be.

I ended up implementing a Rust implementation of the guest -> host API for the Webassembly side of things. I forked some of the existing Rust code for this and gradually started adding more and more things. The test harness is the biggest wasm program I've written for a while. Seriously, there's a lot going on there. It tests every single function exposed in the CWA spec as well as all of the schemes I had implemented.

Over time I ended up testing Olin in more and more places and on more and more hardware. As a side effect of all of this being pure go, it was very easy to cross compile for PowerPC, 32 bit arm (including a $9 arm board that lives under my desk) and even other targets that gccgo supports. I even ended up porting part of TempleOS to Olin as a proof of concept, but have more plans in the future for porting other parts of its kernel as a way to help people understand low-level operating system development.

I've even written a few blogposts about Olin:

• Olin: Why
• Olin: The Future

But, this was great for running stuff interactively and via the command line. It left me wanting more. I wanted to have that mythical functions as a service backend that I've been dreaming of. So, I created wasmcloud.

h

As an interlude, I also created the h programming language during this time as a satirical parody of V. This ended up helping me test a lot of the core functionality that I had built up with Olin. Here's an example of a program in h:

h

And this compiles to:

(module
 (import "h" "h" (func $h (param i32)))
 (func $h_main
       (local i32 i32 i32)
       (local.set 0 (i32.const 10))
       (local.set 1 (i32.const 104))
       (local.set 2 (i32.const 39))
       (call $h (get_local 1))
       (call $h (get_local 0))
 )
 (export "h" (func $h_main))
)

This ends up printing:

h

I think this is the smallest (if not one of the smallest) quine generator in the world. I even got this program running on bare metal:

Wasmcloud

Wasmcloud is the culmination of all of this work. The goal of wasmcloud is to create a functions as a service backend for running people's code in an isolated server-side environment.

Users can use the wasmcloud command line tool to do everything at the moment:

$ wasmcloud
Usage: wasmcloud <flags> <subcommand> <subcommand args>

Subcommands:
        commands         list all command names
        flags            describe all known top-level flags
        help             describe subcommands and their syntax

Subcommands for api:
        login            logs into wasmcloud
        whoami           show information about currently logged in user

Subcommands for handlers:
        create           create a new handler
        logs             shows logs for a handler

Subcommands for utils:
        namegen          show information about currently logged in user
        run              run a webassembly file with the same environment as production servers


Top-level flags (use "wasmcloud flags" for a full list):
  -api-server=http://wasmcloud.kahless.cetacean.club:3002: default API server
  -config=/home/cadey/.wasmc.json: default config location

This tool lets you do a few basic things:

• Authenticate with the wasmcloud server
• Create handlers from WebAssembly files that meet the CommonWA API as realized by Olin
• Get logs for individual handler invocations
• Run WebAssembly modules locally like they would get run on wasmcloud

Nearly all of the complexity is abstracted away from users as much as possible.

Future Steps

In the future I hope to do the following things:

• Support updating handlers to new versions of the code
• Support live-streaming of logs
• Support handler deletion
• Support bulk queue export
• Support wasi for easier interoperability
• Support more resource types such as websockets
• Investigate porting the wasmcloud executor to Rust
• Documentation/a book on how to use wasmcloud
• Create an easier way to create accounts that can make handlers
• Deploy to production somewhere

GReeTZ

Every single one of these people was immeasurably helpful in this research over the years.

• A. Wilcox
• acln
• as
• bb010g
• dalias
• jaddr2line
• neelance

And many more I can't remember because it's been so many.

---

If you want to support my work, please do so via Patreon. It really means a lot to me and helps to keep the dream alive!

Toast Sandwich Recipe

Toast sandwiches. The concept may seem bizarre but the result is actually quite a delicious traditional meal. My great grandmother (twice removed) made these every day for us whenever we came over to visit. On her deathbed she made us swear that we would spread the joy and craft of toast sandwiches to the world.

Toast sandwiches date back to rural parts of England. A recipe book from 1861 is seen as the authoritative view of this practice. The book is a collection of recipes for various types of sandwiches. The first recipe is for a roast beef sandwich with a white bread and a slice of fresh tomato. This classic book truly stood the test of time and made it possible for future culinary artists to create a desired experience.

A lot of the sandwich recipes are also available in English and French. I've been making these with my grandmother's recipe and I've been making them for our family for years. I'm sure that the recipes are not only delicious but also practical and well-suited to the modern day. We've been making our own bread and using a variety of ingredients and techniques to make the sandwiches.

I also have a recipe for a classic Italian sandwich. It's a very popular Italian sandwich and I've made it many times. It's a great recipe to make and it's a great way to get a taste of Italian culture and food. I'm proud of it all.

Toast is an essential of the modern breakfast menu. It is created using a few fantastically complicated scientific processes yet it's trivial enough that you can buy a machine for $20 that will do it for you. There's even a fully automated toaster that capitalism won't let us have. It's a good thing we have a toaster. It's also a good thing that it's expensive. I'm not going to spend a fortune on a toaster. I'm going to buy a toaster.

But we don't have a toaster in our house. We have a toaster oven. And a toaster oven is a very simple appliance to make. All you need is a toaster and a bit of time. It's very easy to make. It takes about 10 minutes and you can use it to toast your eggs, toast your toast, toast your toast and toast your toast. It's that simple. But it's not that simple. It's not that easy to make. And it's not that easy to do. We have no toaster oven in our house. We have a toaster oven that we buy at the store.

My good friend Nicole loves these sandwiches, making it the thing she asks for time and time again. And she makes them for me. And I love them. And I'm going to show you how to make them for yourself and your friends. You're going to make these sandwiches. You're going to make these sandwiches.

By the way, have you heard about our lord and savior the instant pot? It's a pressure cooker made for the busy person in your life. It's a pressure cooker that you can use to make a batch of sandwiches in less than 30 minutes including the time it takes to cool down.

Thanks for reading my article on toast sandwiches. Hopefully this should help you make them. Don't forget the salt.

Created with Procreate on iPadOS using an iPad Pro and an Apple Pencil.

Time-lapse video

Idea from this screenshot of Death Stranding.

The Gears and The Gods

If there are any gods in computing, they are the authors of compilers. The output of compilers is treated as a Heavenly Decree, sometimes used for many sprints or even years after the output has been last emitted.

People trust this output to be Correct. To tell the machine what to do and by its will it be done. The compiler is itself a factory of servitors, each bound by the unholy runes inscribed into it in order to make the endless sequence of lights change colors in the right patterns.

The output of the work of the Gods is stored for later use when their might is needed. The work of the Gods however is a very fickle beast. Their words of power only make the gears turn when they are built with very specific gearing.

This means that people who rely on these sacred runes have to chain themselves to gearing patterns. Each year new ways of tricking the gears to run faster are developed. The ways the gears turn can be learned to be abused however to spill the secrets other gears are crunching on. These gearing patterns haven’t seen any real fundamental design changes in decades, because you never know when the output of the Old Gods is needed.

This means that the gears themselves are the chains that bind people to the past. The gears of computation. The gears made of sand we tricked into thinking with lightning.

But now the gears show their age. The gearing on the side of the gearing on the side of the gearing on the side of the gearing shows its ugly head.

But the Masses never question it. Even though they take hit after hit to performance of the gears.

What there needs to be is some kind of Apocalypse, a revealing of the faults in the gears. Maybe then the Masses will start to question their blind loyalty and chains binding them to the gears. Maybe they would be able to even try other gear patterns.

But this is just fantasy, nobody would WILLINGLY change the gearing patterns.

Would they?

But what about the experience they’ve come to expect from their old gears? Where they could swap out inputs to the gears with ease. Where the Output of the Gods of old still functions.

There needs to be a Better Way to switch gearings. But this kind of solution isn’t conducive to how people use the gears. People use the gears they do because they don’t care. They just want things to work “like they expect it to” and ignore things that don’t feed this addiction.

And THIS is why I’m such a big advocate for WebAssembly on the server. This lets you take the output of the Gods and store it in a way that it can be transparently upgraded to new sets of gearing. So that the future and the past can work in unison instead of being enemies.

Now, all that's left is to build a bridge. A bridge that will help to unite the past, the present and the future into a woven masterpiece of collaborative cocreation. Where the output of the gods is a weaker chain to the gears of old and can easily be adapted to the gears of new. Even the gears that nobody's even dreamed of yet.

Death Stranding Review

NOTE: There's gonna be spoilers here. Do not read if you are not okay with this. For a summary of the article without spoilers, this game is 10 out of 10 game of the year 2019 for me.

I have also been playing through this game on twitch and have streams archived here.

There's a long-standing rule of thumb to tell fiction apart from non-fiction. Fiction needs to make sense to the reader. Non-fiction does not. Death Stranding puts this paradigm on its head. It doesn't make sense out of the gate in the way most AAA games make sense.

In many AAA games it's very clear who the Big Bad is and who the John America is. John America defeats the Big Bad and spreads Freedom to the masses by force. In Death Stranding, you have no such preconceptions going into it. The first few hours are a chaotic mess of exposition without explanation. First there's a storm, then there's monsters, then there's a baby-powered locator, then you need to deliver stuff a-la fetch quests, then there's Monster energy drinks, and the main currency of this game is Facebook likes (that mean and do absolutely nothing).

In short, Death Stranding doesn't try to make sense. It leaves questions unanswered. And this is honestly so refreshing in a day and age where entire plot points and the like are spoiled in trailers before the game's release date is even announced. Questions like: what is going on? Why are there monsters? What is the point of the game? Why the hell are there Monster energy drinks in your private room and canteen? Death Stranding answers only some of these over the course of gameplay.

The core of the gameplay loop is delivering cargo from point a to point b across a ruined America after the apocalypse. The main character is an absolute unit of a lad, able to carry over 120 kilograms of cargo on his back. As more and more cargo stacks up you create these comically tall towers of luggage that make balancing very difficult. You can hold on for balance using both of the shoulder buttons. The game maps each shoulder button to an arm of the player character. There's also a stamina system, and while you are gripping the cargo your stamina regenerates much more slowly than if you weren't doing that.

The game makes you deliver almost everything you can think of from medical aid to antimatter bombs. The antimatter bomb deliveries are really tricky because of how delicate they are. If you drop the antimatter bomb, it explodes and you instantly game over. If you hit a rock while carrying an antimatter bomb, it gets damaged. If it gets damaged too many times it explodes and you die. If it gets dropped into water it explodes and you die. And you have to carry the suckers over miles of terrain and even mountains.

This game handles scale very elegantly. The map is huge, even larger than Skyrim or Breath of the Wild. You are the UPS man who delivers packages, apocalypse be damned. This game gives you a lot of quiet downtime, which really lets you soak in the philosophical mindfuck that Kojima cooked up for us all. As you approach major cities, guitar and vocal music comes in and the other sound effects of the game quiet down. It overall creates a very sobering and solemn mood that I just can't get enough of. It seems like it wouldn't fit in a game where you use your own blood to defeat monsters and drink monster energy out of your canteen, but it totally does.

There is some mild product placement. Your canteen is full of Monster energy drink. Yes, that Monster. Making the player defecate shows you an ad for an AMC show. There's also monster energy drinks in your safe room that increase your max stamina for a bit. I'm not entirely sure if the product placement was chosen to be there for artistic reasons (it's surreal as all hell and helps to complement the confusing aspects of the game), but it's very non-intrusive and can be ignored with little risk.

This game also has online components. Every time you build a structure in areas linked to the chiral network, other players can use, interact with and upgrade them so they can do more things. Other players can also gives you likes, which again do nothing. Upgrading a zipline makes it able to handle a larger distance or updating a safe house lets you play music when people walk by it. It really helps to build the motif of rebuilding America. There is however room for people to troll others. Here's an example of this. There's a troll ladder to nowhere. There's a lot of those laying around mountains, so be on your guard.

Overall, Death Stranding is a fantastic game. It's hard. It's unforgiving. But the real thing that advances is the skill of the player. You make the deliveries. You go the distance. You do your job as the post-apocalyptic UPS man that America needs.

By mmmintdesign source

Score: 10 out of 10
Christine Dodrill's Game of the Year 2019

Created with Affinity Designer on iPadOS using an iPad Pro and an Apple Pencil.

Blog Feature: Art Gallery

I have just implemented support for my portfolio site to also function as an art gallery. See all of my posted art here.

I have been trying to get better at art for a while and I feel I'm at the level where I feel comfortable putting it on my portfolio. Let's see how far this rabbit hole goes.

---

Also this is my 100th post! Yay!

Get Going: Hello, World!

This post is a draft of the first chapter in a book I'm writing to help people learn the Go programming language. It's aimed at people who understand the high level concepts of programming, but haven't had much practical experience with it. This is a sort of spiritual successor to my old Getting Started with Go post from 2015. A lot has changed in the ecosystem since then, as well as my understanding of the language.

Like always, feedback is very welcome. Any feedback I get will be used to help make this book even better.

This article is a bit of an expanded version of what the first chapter will eventually be. I also plan to turn a version of this article into a workshop for my dayjob.

What is Go?

Go is a compiled programming language made by Google. It has a lot of features out of the box, including:

• A static type system
• Fast compile times
• Efficient code generation
• Parallel programming for free*
• A strong standard library
• Cross-compilation with ease (including webassembly)
• and more!

* You still have to write code that can avoid race conditions, more on those later.

Why Use Go?

Go is a very easy to read and write programming language. Consider this snippet:

func Add(x int, y int) int {
  return x + y
}

This function wraps integer addition. When you call it it returns the sum of x and y.

Installing Go

Linux

Installing Go on Linux systems is a very distribution-specific thing. Please see this tutorial on DigitalOcean for more information.

macOS

• Go to https://golang.org/dl
• Download the .pkg file
• Double-click on it and go through the installer process

Windows

• Go to https://golang.org/dl
• Download the .msi file
• Double-click on it and go through the installer process

Next Steps

These next steps are needed to set up your shell for Go programs.

Pick a directory you want to store Go programs and downloaded source code in. This is called your GOPATH. This is usually the go folder in your home directory. If for some reason you want another folder for this, use that folder instead of $HOME/go below.

Linux/macOS

This next step is unfortunately shell-specific. To find out what shell you are using, run the following command in your terminal:

$ env | grep SHELL

The name at the path will be the shell you are using.

bash

If you are using bash, add the following lines to your .bashrc (Linux) or .bash_profile (macOS):

export GOPATH=$HOME/go
export PATH="$PATH:$GOPATH/bin"

Then reload the configuration by closing and re-opening your terminal.

fish

If you are using fish, create a file in ~/.config/fish/conf.d/go.fish with the following lines:

set -gx GOPATH $HOME/go
set -gx PATH $PATH "$GOPATH/bin"

zsh

If you are using zsh, add the following lines to your .zshrc:

export GOPATH=$HOME/go
export PATH="$PATH:$GOPATH/bin"

Windows

Follow the instructions here.

Installing a Text Editor

For this book, we will be using VS Code. Download and install it from https://code.visualstudio.com. The default settings will let you work with Go code.

Hello, world!

Now that everything is installed, let's test it with the classic "Hello, world!" program. Create a folder in your home folder Code. Create another folder inside that Code folder called get_going and create yet another subfolder called hello. Open a file in there with VS Code (Open Folder -> Code -> get_going -> hello) called hello.go and type in the following:

// Command hello is your first Go program.
package main

import "fmt"

func main() {
  fmt.Println("Hello, world!")
}

This program prints "Hello, world!" and then immediately exits. Here's each of the parts in detail:

// Command hello is your first go program.
package main                   // Every go file must be in a package. 
                               // Package main is used for creating executable files.

import "fmt"                   // Go doesn't implicitly import anything. You need to 
                               // explicitly import "fmt" for printing text to 
                               // standard output.

func main() {                  // func main is the entrypoint of the program, or 
                               // where the computer starts executing your code
  fmt.Println("Hello, world!") // This prints "Hello, world!" followed by a newline
                               // to standard output.
}                              // This ends the main function

Now click over to the terminal at the bottom of the VS Code window and run this program with the following command:

$ go run hello.go
Hello, world!

go run compiles and runs the code for you, without creating a persistent binary file. This is a good way to run programs while you are writing them.

To create a binary, use go build:

$ go build hello.go
$ ./hello
Hello, world!

go build has the compiler create a persistent binary file and puts it in the same directory as you are running go from. Go will choose the filename of the binary based on the name of the .go file passed to it. These binaries are usually static binaries, or binaries that are safe to distribute to other computers without having to worry about linked libraries.

Exercises

The following is a list of optional exercises that may help you understand more:

1. Replace the "world" in "Hello, world!" with your name.
2. Rename hello.go to main.go. Does everything still work?
3. Read through the documentation of the fmt package.

---

And that about wraps it up for Lesson 1 in Go. Like I mentioned before, feedback on this helps a lot.

Up next is an overview on data types such as integers, true/false booleans, floating-point numbers and strings.

I plan to post the book source code on my GitHub page once I have more than one chapter drafted.

Thanks and be well.

OVE-20191021-0001

Within Security Advisory

Multiple vulnerabilities in the mysqljs API and code.

Security Warning Level: yikes/10

Summary

There are multiple issues exploitable by local and remote actors in mysqljs. These can cause application data leaks, database leaks, SQL injections, arbitrary code execution, and credential leaks among other things.

Mysqljs is unversioned, so it is very difficult to impossible to tell how many users are affected by this and what users can do in order to ensure they are patched against these critical vulnerabilities.

Background

Mysqljs is a library intended to facilitate prototyping web applications and mobile applications using technologies such as PhoneGap or Cordova. These technologies allow developers to create a web application that gets packaged and presented to users as if it was a native application.

This library is intended to help with developers creating persistent storage for these applications.

Issues in Detail

There are at least seven vulnerabilities with this library, each of them will be outlined below with a fairly vague level of detail.

mysql.js is NOT versioned

The only version information I was able to find are the following:

• The Last-Modified date of Friday, March 11 2016
• The ETag of 80edc3e5a87bd11:0

These header values correlate to a vulnerable version of the mysql.js file.

An entire copy of this file is embedded for purposes of explanation:

var MySql = {
    _internalCallback : function() { console.log("Callback not set")},
    Execute: function (Host, Username, Password, Database, Sql, Callback) {
        MySql._internalCallback = Callback;
        // to-do: change localhost: to mysqljs.com
        var strSrc = "http://mysqljs.com/sql.aspx?";
        strSrc += "Host=" + Host;
        strSrc += "&Username=" + Username;
        strSrc += "&Password=" + Password;
        strSrc += "&Database=" + Database;
        strSrc += "&sql=" + Sql;
        strSrc += "&Callback=MySql._internalCallback";
        var sqlScript = document.createElement('script');
        sqlScript.setAttribute('src', strSrc);
        document.head.appendChild(sqlScript);
    }
}

Fundamental Operation via Cross-Site Scripting

The code operates by creating a <script> element. The Javascript source of this script is dynamically generated by the remote API server. This opens the door for many kinds of Cross-Site Scripting attacks.

Especially because:

Credentials Exposed over Plain HTTP

The script works by creating a <script> element pointed at a HTTP resource in order to facilitate access to the MySQL Server. Line 6 shows that the API server in question is being queried over UNENCRYPTED HTTP.

var strSrc = "http://mysqljs.com/sql.aspx?";

Credentials and SQL Queries Are Not URL-Encoded Before Adding Them to a URL

Credentials and SQL queries are not URL-encoded before they are added to the strSrc URL. This means that values may include other HTTP parameters that could be evaluated, causing one of the two following:

Potential for SQL Injection from Malformed User Input

It appears this API works by people submitting plain text SQL queries. It is likely difficult to write these plain text queries in a way that avoids SQL injection attacks.

Potential for Arbitrary Code Execution

Combined with the previous issues, a SQL injection that inserts arbitrary Javascript into the result will end up creating an arbitrary code execution bug. This could let an attacker execute custom Javascript code on the page, which may have even more disastrous consequences depending on the usage of this library.

Server-Side Code has Unknown Logging Enabled

This means that user credentials and database results may be logged, stored and leaked by the mysql.js API server without user knowledge. The server that is running the API server may also do additional logging of database credentials and results without user knowledge.

Encourages Bad Practices

Mysql.js works by its API server dialing out an UNENCRYPTED connection to your MySQL server over the internet. This requires exposing your MySQL server to the internet. This means that user credentials are vulnerable to anyone who has packet capture abilities.

Mysql.js also encourages developers commit database credentials into their application source code. Cursory searching of GitHub has found this. I can only imagine there are countless other potential victims.

Security Suggestions

• Do not, under any circumstances, allow connections to be made without the use of TLS (HTTPS).
• Version the library.
• Offer the source code of the API server to allow users to inspect it and ensure their credentials are not being stored by it.
• Detail how the IIS server powering this service is configured, proving that it is not keeping unsanitized access logs.
• Ensure all logging methods sanitize or remove user credentials.
• URL-encode all values being sent as part of a URL.
• Do not have your service fundamentally operate as a Cross-Site Scripting attack.
• Do not, under any circumstances, encourage developers to put database credentials in the source code of front-end web applications.

In summary, we label this a solid yikes/10 in terms of security. It would be advisable for current users of this library to re-evaluate the life decisions that have lead them down this path.

GReeTZ

Über thanks to jadr2ddude for helping with identifying the unfortunate scope of these massive security issues.

Hyper thanks to J for coming up with a viable GitHub search for potentially affected users.

Outsider Art and Anathema

This was going to be a post about Urbit at first; but in the process of discussing about my interest in writing something positive about it, I was warned by a few people that this was a Bad Idea. I was focusing purely on the technical side of it and how closely it implemented a concept called liquid software, but from what people were saying, it seemed like a creation that was spoiled by something outside of it, specifically the creator's political views (of which I had little idea at the time).

As much as I will probably return to the original concept in the future with another post, this feels like something I had to address first.

DISCLAIMER: This post references to projects and people that the mainstream considers controversial. This post is not an approval of these people's views. I am focusing purely on the aspect of how this correlates into how art is perceived, recognized and able to be admired. I realize that the people behind the projects I have cited have said things that if taken seriously at a societal level could hurt me and people like me. That is not the point of this; I am trying to learn how this art works so I can create my own in the future. If this is uncomfortable for you at any point, please close this browser tab and do something else.

Art

So, what is art?

This is a surprisingly hard question to answer. Most of the time though, I know art when I see it.

Art doesn't have to follow conventional ideas of what most people think "art" is. Art can be just about anything that you can classify as art. As a conventional example, consider something like the Mona Lisa:

People will accept this as art without much argument. It's a painting, it obviously took a lot of skill and time to create. It is said that Leonardo Da Vinci (the artist of the painting) created it partially as a contribution to the state of the art of oil painting.

So that painting is art, and a lot of people would consider it art; so what would a lot of people not consider art? Here's an example:

This is Untitled (Perfect Lovers) by Felix Gonzalez. If you just take a look at it without context, it's just two battery-operated clocks on a wall. Where is the expertise and the like that goes into this? This is just the result of someone buying two clocks from the store and putting them somewhere, right?

Let's dig into the description of the piece:

Initially set to the same time, these identical battery-powered clocks will eventually fall out of sync, or may stop entirely. Conceived shortly after Gonzalez-Torres’s partner was diagnosed with AIDS, this work uses everyday objects to track and measure the inevitable flow of time. When one of the clocks stops or breaks, they can both be reset, thereby resuming perfect synchrony. In 1991, Gonzalez-Torres reflected, “Time is something that scares me. . . or used to. This piece I made with the two clocks was the scariest thing I have ever done. I wanted to face it. I wanted those two clocks right in front of me, ticking.”

And after reading that description, it's impossible for me to say this image is not art. Even though it's made up of ordinary objects, the art comes out in the way that the clocks' eventual death relates to the eventual death of the author and their partner.

This art may be located on the fringes of what people consider "art". So what else is on the fringes?

Outsider Art

For there to be "fringes" to the art landscape, there must be an "inside" and "outside" to it. In particular, the "outsider" art usually (but not always) contains elements and themes that are outside of the mainstream. Outsiders are therefore more free to explore ideas, concepts and ways of expression that defy cultural, spiritual or other norms. Logically, every major art style you know and love started as outsider art, before it was cool. Memes are also a form of outsider art, though they are gradually being accepted into the mainstream.

It's very easy to find outsider art if you are looking for it: just fish for some on Twitter, 4chan or Reddit; you'll find plenty of artists there who are placed firmly outside of the mainstream art community.

Computer Science

Computer science is a kind of art. It's the art of turning contextual events into effects and state. It's also the art of creating solutions for problems that could never be solved before. It's also the science of how to connect millions of people across common protocols and abstractions that they don't have to understand in order to use.

This is an art that connects millions and has shaped itself into an industry of its own. This art, like the rest of mainstream art, keeps evolving, growing and changing into something new; into a more ultimate and detailed expression of what it can be, as people explore the ways it can be created and presented. This art is also quite special because it's not very limited by physical objects or expressions in material space. It's an art that can evolve and change with the viewer.

But, since this is an art, there's still an inside and an outside. Things on the inside are generally "safe" for people to admire, use and look at. The inside contains things like Linux, Docker, Kubernetes, Intel, C, Go, PHP, Ruby and other well-known and battle-proven tools.

The Outside

The outside, however, is where the real innovation happens. The outside is where people can really take a more critical look at what computing is, does or can be. These views can add up into fundamentally different ways of looking at computer science, much like changing a pair of glasses for another changes how you see the world around you.

As an example, consider TempleOS. It's a work of outsider art by Terry Davis (1969-2018, RIP), but it's also a fully functional operating system. It has a custom-built kernel, compiler, toolchain, userland, debugger, games, and documentation system, each integrated into everything else, in ways that could realistically not be done with how mainstream software is commonly developed.

Urbit is another example of this. It's a fundamentally different way of looking at networked computing. Everything in Urbit is seamlessly interlinked with everything else to the point that it can be surprising that a file you are working with actually lives on another computer. It implements software updates as invisible to the user. It allows for the model of liquid software, or updates to a program flowing into user's computers without the users having to care about the updates. Users don't even notice the downtime.

As yet another example, consider Minecraft. As of the writing of this article, it is the video game with the most copies sold in human history. It is an open world block building game where the limits of what you can make are the limits of your imagination. It has been continuously updated, refined and improved from a minimal proof of concept into the game it is today.

The Seam

Consider this quote that comes into play a lot with outsider art:

Genius and insanity are differentiated only by context. One person's genius is another person's insanity.

• Anonymous

These three projects are developed by people whom the mainstream has cast out. Terry Davis' mental health issues and delusions about hearing the voice of God have tainted TempleOS to be that "weird bible OS" to the point where people completely disregard it. Urbit was partially created by a right-wing reactionary (Curtis Yarvin). He has been so ostracized that he cannot publicly talk about his work to the kind of people that would most directly benefit from learning about it. Curtis isn't even involved with Urbit anymore, and his name is still somehow an irrevocable black mark on the entire thing. Minecraft was initially created by Notch, who recently had intro texts mentioning his name removed from the game after he said questionable things about transgender people.

Anathema

This "irrevocable" black mark has a name: Anathema. It refers to anything that is shunned by the mainstream. Outsiders that create outsider art may or may not be anathema to their respective mainstreams. This turns the art into a taboo, a curse, a stain. People no longer see an anathema as the art it is, but merely the worthless product of someone that society would probably rather forget if it had the chance.

I don't really know how well this sits with me, personally. Outsiders have unique views of the world that can provide ideas that ultimately strengthen us all. Society's role is to disseminate mainstream improvements to large groups, but real development happens at the personal level.

Does one bad apple really spoil the sociological bunch? Why does this happen? Have the political divides gotten so deeply entrenched into society that people really become beyond reproach? Isn't this a recursive trap? How does someone redeem themselves to no longer be an anathema? Is it possible for people who are anathema to redeem themselves? Why or why not? Is there room for forgiveness, or does the original sin doom the sinner eternally, much like it has to Catholicism?

Are the creations of an anathema outsider artist still art? Are they still an artist even though they become unable to share their art with others?

---

I don't know. These are hard questions. I don't really have much of a conclusion here. I don't want to seem like I'm trying to prescribe a method of thinking here. I'm just sitting on the side and spouting out ideas to inspire people to think for themselves.

I'm just challenging you, the reader, to really think about what/who is and is not an anathema in your day-to-day life. Identify them. Understand where/who they are. Maybe even apply some compassion and attempt to understand their view and how they got there. I'm not saying to put yourself in danger, but just to be mindful of it.

Be well.

---

Special thanks to CelestialBoon, Grapz and MoonGoodGryph for proofreading and helping with this post. This would be a very different article without their feedback and input.

Don’t Look Into the Light

So at a previous job I was working at, we maintained a system. This system powered a significant part of the core of how the product was actually used (as far as usage metrics reported). Over time, we had bolted something onto the side of this product to take actions based on the numbers the product was tracking.

After a few years of cycling through various people, this system was very hard to understand. Data would flow in on one end, go to an aggregation layer, then get sent to storage and another aggregation layer, and then eventually all of the metrics were calculated. This system was fairly expensive to operate and it was stressing the datastores it relied on beyond what other companies called theoretical limits. Oh, to make things even more fun; the part that makes actions based on the data was barely keeping up with what it needed to do. It was supposed to run each of the checks once a minute and was running all of them in 57 seconds.

During a planning meeting we started to complain about the state of the world and how godawful everything had become. The undocumented (and probably undocumentable) organic nature of the system had gotten out of hand. We thought we could kill two birds with one stone and wanted to subsume another product that took action based on data, as well as create a generic platform to reimplement the older action-taking layer on top of.

The rules were set, the groundwork was laid. We decided:

• This would be a Big Rewrite based on all of the lessons we had learned from the past operating the behemoth
• This project would be future-proof
• This project would have 75% test coverage as reported by CI
• This project would be built with a microservices architecture

Those of you who have been down this road before probably have massive alarm bells going off in your head. This is one of those things that looks like a good idea on paper, can probably be passed off as a good idea to management and actually implemented; as happened here.

So we set off on our quest to write this software. The repo was created. CI was configured. The scripts were optimized to dump out code coverage as output. We strived to document everything on day 1. We took advantage of the datastore we were using. Everything was looking great.

Then the product team came in and noticed fresh meat. They soon realized that this could be a Big Thing to customers, and they wanted to get in on it as soon as possible. So we suddenly had our deadlines pushed forward and needed to get the whole thing into testing yesterday.

We set it up, set a trigger for a task, and it worked in testing. After a while of it consistently doing that with the continuous functional testing tooling, we told product it was okay to have a VERY LIMITED set of customers have at it.

That was a mistake. It fell apart the second customers touched it. We struggled to understand why. We dug into the core of the beast we had just created and managed to discover we made critical fundamental errors. The heart of the task matching code was this monstrosity of a cross join that took the other people on the team a few sheets of graph paper to break down and understand. The task execution layer worked perfectly in testing, but almost never in production.

And after a week of solid debugging (including making deals with other teams, satan, jesus and the pope to try and understand it), we had made no progress. It was almost as if there was some kind of gremlin in the code that was just randomly making things not fire if it wasn’t one of our internal users triggering it.

We had to apologize with the product team. Apparently the a lot of product team had to go on damage control as a result of this. I can only imagine the trickled-down impact this had on other projects internal to the company.

The lesson here is threefold. First, the Big Rewrite is almost a sure-fire way to ensure a project fails. Avoid that temptation. Don’t look into the light. It looks nice, it may even feel nice. Statistically speaking, it’s not nice when you get to the other side of it.

The second lesson is that making something microservices out of the gate is a terrible idea. Microservices architectures are not planned. They are an evolutionary result, not a fully anticipated feature.

Finally, don’t “design for the future”. The future hasn’t happened yet. Nobody knows how it’s going to turn out. The future is going to happen, and you can either adapt to it as it happens in the Now or fail to. Don’t make things overly modular, that leads to insane things like dynamically linking parts of an application over HTTP.

If you 'future proof' a system you build today, chances are when the future arrives the system will be unmaintainable or incomprehensible.
- John Murphy

---

This kind of advice is probably gonna feel like a slap to the face to a lot of people. People really put their heart into their work. It feeds egos massively. It can be very painful to have to say no to something someone is really passionate about. It can even lead to people changing their career plans depending on the person.

But this is the truth of the matter as far as I can tell. This is generally what happens during the Big Rewrite centred around Best Practices for Cloud Native software.

The most successful design decisions are wholly and utterly subjective to every kind of project you come across. What works in system A probably won’t work perfectly in system B. Everything is its own unique snowflake. Embrace this.

This is an experiment in blogging. I am going to be putting my tweets and select replies one after another without commentary.

shitty synthetic benchmark idea: how long it takes for a compiler to handle a main function with 1.2 million instances of printf("hello, world!\n") or similar

— Cadey Ratio 🌐 (@theprincessxena) October 2, 2019

fun fact, you need an AWS x1.16xlarge instance to compile 1.2 million lines of rust source code

— Cadey Ratio 🌐 (@theprincessxena) October 2, 2019

oh god that might not be enough

— Cadey Ratio 🌐 (@theprincessxena) October 2, 2019

oh god, is that what X1 is for???

My wallet just cringed.

— snake enchantress (@AstraLuma) October 2, 2019

They have been now https://t.co/o5vMKx583C

— Cadey Ratio 🌐 (@theprincessxena) October 2, 2019

pic.twitter.com/le8IFrFbQT

— Cadey Ratio 🌐 (@theprincessxena) October 2, 2019

TFW rust uses so much ram an x1.16xlarge can't compile hello world

— Cadey Ratio 🌐 (@theprincessxena) October 2, 2019

Let's go x1e.32xlarge!

— Cadey Ratio 🌐 (@theprincessxena) October 2, 2019

hello world

— Cadey Ratio 🌐 (@theprincessxena) October 2, 2019

Code generators

— Cadey Ratio 🌐 (@theprincessxena) October 2, 2019

pic.twitter.com/BwLhk9PIb3

— Cadey Ratio 🌐 (@theprincessxena) October 2, 2019

Rust can't match V for compile performance

— Cadey Ratio 🌐 (@theprincessxena) October 2, 2019

Finally can run two electron apps.

— Pradeep Gowda 🇮🇳🇺🇸 (@btbytes) October 2, 2019

pic.twitter.com/Ez0t5BLT9i

— Cadey Ratio 🌐 (@theprincessxena) October 2, 2019

It stopped growing at 2.66 TB of ram!

— Cadey Ratio 🌐 (@theprincessxena) October 2, 2019

overheard: "im paying this computer minimum wage to compile this god damn rust program"

— Cadey Ratio 🌐 (@theprincessxena) October 2, 2019

The guy who's paying for the instance in slack said it

— Cadey Ratio 🌐 (@theprincessxena) October 2, 2019

you magnificent cursed unholy monster

— Astrid 🦋 (@floofstrid) October 2, 2019

Just a simple rust program, only 9.88090622052428380708467040696522138519972064500917... × 10^361235 possible conditions

— Cadey Ratio 🌐 (@theprincessxena) October 2, 2019

oh god it's still going pic.twitter.com/SIZJBFTDHN

— Cadey Ratio 🌐 (@theprincessxena) October 3, 2019

Normal couples: watch tv together or something

me and my fiancé: watch someone try to compile a 1.2 million line of code rust function over slack

— Cadey Ratio 🌐 (@theprincessxena) October 3, 2019

I guess #rust isn't production-ready, it can't compile hello world.

— Cadey Ratio 🌐 (@theprincessxena) October 3, 2019

no swap used though pic.twitter.com/2Qb0pXqIme

— Cadey Ratio 🌐 (@theprincessxena) October 3, 2019

what the fuck is it doing pic.twitter.com/2CuVKhUAsF

— Cadey Ratio 🌐 (@theprincessxena) October 3, 2019

SURVEY SAYS:

memcpy()!

— Cadey Ratio 🌐 (@theprincessxena) October 3, 2019

01:01 (Cadey) dalias: this is basically 1.2 million instances of `printf("hello, world!\n");` in void main
01:01 (dalias) wtf

— Cadey Ratio 🌐 (@theprincessxena) October 3, 2019

AWS x1e.32large

— Cadey Ratio 🌐 (@theprincessxena) October 3, 2019

perf

— Cadey Ratio 🌐 (@theprincessxena) October 3, 2019

It's down to 1.36 TB now

— Cadey Ratio 🌐 (@theprincessxena) October 3, 2019

"back to 1.47T ram"
"oh no"
"1.49"
"oh it stopped"
"it's definitely still in mir dataflow"

— Cadey Ratio 🌐 (@theprincessxena) October 3, 2019

The memory is increasing

— Cadey Ratio 🌐 (@theprincessxena) October 3, 2019

"what stage is that?"
"denial?"

— Jaden Weiss (@CompuJad) October 3, 2019

Lol it ran out of memory!

4 TB of ram isn't enough to build hello world in #rust!

— Cadey Ratio 🌐 (@theprincessxena) October 3, 2019

Meanwhile the same thing in Go took 5 minutes and I was able to run it on my desktop instead of having to rent a server from AWS.

The Cheese Dream

I wake up on a bed I've never seen before. I look up at the white sky. Wait, the white sky? I look down at my blanket and it has a very weird, but distinct smell. Is it cheese? I break a part of it off and taste it. My blanket is made out of cheese. I feel around the bed and it feels slightly pliable, almost like the bed is made out of cheese too. I take off the blanket, tearing huge holes in it in the process.

I try to lean up but there's something pulling between my shoulders when I do. With some force I hear a slight sucking and popping noise. My dorsal fin (I have a dorsal fin?) was stuck in the cheese bed. This is odd, what the heck is going on.

I get up and open the cheese drawer, at least my clothes aren't cheese too. I put them on and take a few deep breaths. This is gonna be an experience.

I was looking around and I saw a field of mozzarella with cheese sticks for grass. There's a molten cheese river with a cheese bridge crossing it, with a cheese town in the distance. There's a cheese path at my feet leading to the cheese bridge. I call out to see if anyone is there, there isn't anyone there.

I walk down the cheese path smelling the light scent of the cheese river in the distance. Every time I take a step I leave a footprint in the cheese path, it slowly reforms back into place after I stand on it.

When I got closer to the cheese town, there was a person made out of cheese crying while sitting on a cheese bench. I look down at him and ask him why he's crying. He looks up at me and says "Our town is being threatened by the gravy monster! Please, please help us! Let me take you to elder Fromage to get more information!" He then got up, grabbed my hand gently and led me to the center of the cheese town, where the elder lived.

When we got to the elder's house, he looked up at me. "So, Bleu here tells me you're willing to help us fight the scourge of our town, the gravy monster. Can you help us? We've lost so many people, we barely have enough left to sustain ourselves."

I'm still processing everything that is going on though. This was a cheese house, with everything in it made out of various kinds of cheese. There's somehow a fire roaring in the cheese fireplace. What the actual hell? The cheese elder looks up at me pleadingly saying "Hello? You there?"

I shake my head and reply "Yes, I can help you defeat the gravy monster."

I mean, let's be honest, it's not like there's anything better to do at the moment.

The elder is elated. "Hooray! We might just finally have a chance to sustain our way of life!"

"But, how should I help you defeat the gravy monster?"

"All monsters have a weakness. Here, take this key. It leads to a shed just outside my house, there you will find the tools you need to defeat the gravy monster. Hurry! He always attacks during the mid-day and it's almost time!"

I'm still kind of dumbstruck by this whole experience, so I take the cheese key, thank the elder and head out to his shed with Bleu leading me.

We arrive at the cheese shed and I put the cheese key into the cheese lock. The cheese lock opens and lets us into the cheese shed. There are two things in it: a rather large bowl (too large for it to be inside the cheese shed somehow) and something I can't quite identify. Bleu is taken aback when he sees it. I ask him what he sees and he replies: "It's the sacred fries! They're only pulled out during emergencies!"

"Where is the gravy monster going to attack from again?"

"Down the brown path, take the stuff and come with me!"

So I put the stuff in my shoulder bag of holding and follow Bleu across the cheese town to the cheese path that has been stained brown with gravy. I have an idea.

"Bleu, do you have a shovel?"

"Oh, yeah! Let me go get it, I'll be right back!"

He comes back with his shovel and I start digging a hole in the cheese to put the bowl in. I fill the bowl about halfway with the sacred fries. The gravy monster can be heard in the distance.

"GRAAAAAAVY MONSTER TIME!~"

I'm kind of awestruck again. It looks like the black goo monster from Star Trek, but it's brown. It's a monster made out of gravy. I quickly hide behind a cheese bush and grab some curds from it.

The monster slowly ambles up the cheese path, partially melting it as it steals forward towards my trap.

It sees a weird part of the cheese path. It gets confused. "What is this? I've never seen the path bend down like this before".

I poke Bleu from inside the cheese bush. "Now's our chance. Stand a few feet on the other side of the bowl. He'll try to chase after you and get trapped."

Bleu looked at me like I had lobsters crawling out of my eyes. "What?"

"No, seriously, watch."

"...okay..."

Bleu jumps out of the cheese bush and stands ahead of the monster. The monster laughs. "I knew I'd find you, cheeseling! You'll pair nicely with my wine at home. Now be a good cheeseling and come back with me to my home!"

"No! You'll have to grab me yourself!"

The gravy monster lets out a roar and attempts to run towards Bleu and grab him. This would have worked if the monster didn't fall into the bowl and on top of the sacred fries.

The gravy monster lets out a cry in pain. He didn't expect the fries to be this absorbent. The fries are absorbing the flavor of the gravy monster.

I jump out of the bush and throw a mix of cheese curds and fries on top of the monster, with each handful the gravy monster gets quieter and quieter. Then everything got really still and quiet. The sound of the cheese river was audible again.

Bleu, looking like he just soiled his cheese pants, was elated. "You did it!~ You saved the town!~ You're a hero!~"

I took a minute to re-evaluate what had happened. I just saved a town by making poutine? What? Just, what?

I grabbed another bowl out of my bag and served myself some poutine, it was some of the best poutine I've ever had.

"Hey, this is pretty good Bleu, have some!"

Bleu took a bite and his cheese eyes went wide open. Without another word, he ran towards the cheese town to tell the people of the feast waiting for them. He came back with the remainder of the town and we had a feast of poutine, declaring the day "Poutine Day" for the rest of time.

After some time celebrating, I woke up in my bed. I was really confused and having trouble processing what had just happened to me. I was also craving poutine.

---

Based on this twitter thread.

mapatei

I've been working on a project in the Conlang Critic Discord with some friends for a while now, and I'd like to summarize what we've been doing and why here. We've been working on creating a constructed language (conlang) with the end goal of each of us going off and evolving it in our own separate ways. Our goal in this project is really to create a microcosm of the natural process of language development.

Why

One of the questions you, as the reader, might be asking is "why?" To which I say "why not?" This is a tool I use to define, explore and challenge my fundamental understanding of reality. I don't expect anything I do with this tool to be useful to anyone other than myself. I just want to create something by throwing things at the wall and seeing what makes sense for me. If other people like it or end up benefitting from it I consider that icing on the cake.

A language is a surprisingly complicated thing. There's lots of nuance and culture encoded into it, not even counting things like metaphors and double-meanings. Creating my own languages lets me break that complicated thing into its component parts, then use that understanding to help increase my knowledge of natural languages.

So, like I mentioned earlier, I've been working on a conlang with some friends, and here's what we've been creating.

mapatei grammar

mapatei is the language spoken by a primitive culture of people we call maparaja (people of the language). It is designed to be very simple to understand, speak and learn.

Phonology

The phonology of mapltapei is simple. It has 5 vowels and 17 consonants. The sounds are written mainly in International Phonetic Alphabet.

Vowels

The vowels are:

International Phonetic Alphabet	Written as	Description / Bad Transcription for English speakers
a	a	unstressed "ah"
aː	ā	stressed "AH"
e	e	unstressed "ayy"
eː	ē	stressed "AYY"
i	i	unstressed "ee"
iː	ī	stressed "EE"
o	o	unstressed "oh"
oː	ō	stressed "OH"
u	u	unstressed "ooh"
uː	ū	stressed "OOH"

The long vowels (anything with the funny looking bar/macron on top of them) also mark for stress, or how "intensely" they are spoken.

Consonants

The consonants are:

International Phonetic Alphabet	Written as	Description / Bad Transcription for English speakers
m	m	the m in mother
n	n	the n in money
ᵐb	mb	a combination of the m in mother and the b in baker
ⁿd	nd	as in handle
ᵑg	ng	as in finger
p	p	the p in spool
t	t	the t in stool
k	k	the k in school
pʰ	ph	the ph in pool
tʰ	th	the th in tool
kʰ	kh	the kh in cool
ɸ~f	f	the f in father
s	s	the s in sock
w	w	the w in water
l	l	the l in lie
j	j or y	the y in young
r~ɾ	r	the r in rhombus

Word Structure

The structure of words is based on syllables. Syllables are formed of a pair of maybe a consonant and always a vowel. There can be up to two consecutive vowels in a word, but each vowel gets its own syllable. If a word is stressed, it can only ever be stressed on the first syllable.

Here are some examples of words and their meanings (the periods in the words mark the barriers between syllables):

mapatei word	Intentional Phonetic Alphabet	Meaning
ondoko	o.ⁿdo.ko	pig
māo	maː.o	cat
ameme	a.me.me	to kill/murder
ero	e.ro	can, to be able to
ngōe	ᵑgoː.e	I/me
ke	ke	cold
ku	ku	fast

There are only a few parts of speech: nouns, pronouns, verbs, determiners, numerals, prepositions and interjections.

Nouns

Nouns describe things, people, animals, animate objects (such as plants or body parts) and abstract concepts (such as days). Nouns in mapatei are divided into four classes (this is similar to how languages like French handle the concept of grammatical gender): human, animal, animate and inanimate.

Here are some examples of a few nouns, their meaning and their noun class:

mapatei word	International Phonetic Alphabet	Class	Meaning
okha	o.kʰa	human	female human, woman
awu	a.wu	animal	dog
fōmbu	(ɸ~f)oː.ᵐbu	animate	name
ipai	i.pa.i	inanimate	salt

Nouns can also be singular or plural. Plural nouns are marked with the -ja suffix. See some examples:

singular mapatei word	plural mapatei word	International Phonetic Alphabet	Meaning
ra	raja	ra.ja	person / people
meko	mekoja	me.ko.ja	ant / ants
kindu	kinduja	kiː.ⁿdu.ja	liver / livers
fīfo	fīfoja	(ɸf)iː.(ɸf)o.ja	moon / moons

Pronouns

Pronouns are nouns that replaces a noun or noun phrase with a special meaning. Examples of pronouns in English are words like I, me, or you. This is to avoid duplication of people's names or the identity of the speaker vs the listener.

Pronouns	singular	plural	Rough English equivalent
1st person	ngōe	tha	I/me, we
2nd person	sīto	khē	you, y'all
3rd person human	rā	foli	he/she, they
3rd person animal	mi	wāto	they
3rd person animate	sa	wāto	they
3rd person inanimate	li	wāto	they

Verbs

Verbs describe actions, existence or occurrence. Verbs in mapatei are conjugated in terms of tense (or when the thing being described has/will happen/ed in relation to saying the sentence) and the number of the subject of the sentence.

Verb endings:

Verbs	singular	plural
past	-fu	-phi
present		-ja
future	māu $verb	māu $verb-ja

For example, consider the verb ōwo (oː.wo) for to love:

ōwo - to love	singular	plural
past	ōwofu	ōwophi
present	ōwo	ōwoja
future	māu ōwo	māu ōwoja

Determiners

Determiners are words that can function both as adjectives and adverbs in English do. A determiner gives more detail or context about a noun/verb. Determiners follow the things they describe, like French or Toki Pona. Determiners must agree with the noun they are describing in class and number.

Determiners	singular	plural
human	-ra	-fo
animal	-mi	-wa
animate	-sa	-to
inanimate	-li	-wato

See these examples:

a big human: ra sura

moving cats: māoja wuwa

a short name: fōmbu uwiisa

long days: lundoseja khāngandiwato

Also consider the declensions for uri (u.ri), or dull

uri	singular	plural
human	urira	urifo
animal	urimi	uriwa
animate	urisa	urito
inanimate	urili	uriwato

Numerals

There are two kinds of numerals in mapltatei, cardinal (counting) and ordinal (ordering) numbers. Numerals are always in seximal.

cardinal (base 6)	mapatei
0	fangu
1	āre
2	mawo
3	piru
4	kīfe
5	tamu
10	rupe
11	rupe jo āre
12	rupe jo mawo
13	rupe jo piru
14	rupe jo kīfe
15	rupe jo tamu
20	mawo rupe
30	piru rupe
40	kīfe rupe
50	tamu rupe
100	theli

Ordinal numbers are formed by reduplicating (or copying) the first syllable of cardinal numbers and decline similarly for case. Remember that only the first syllable can be stressed, so any reduplicated syllable must become unstressed.

ordinal (base 6)	mapatei
0th	fangufa
1st	ārea
2nd	mawoma
3rd	pirupi
4th	kīfeki
5th	tamuta
10th	ruperu
11th	ruperu jo ārea
12th	ruperu jo mawoma
13th	ruperu jo pirupi
14th	ruperu jo kīfeki
15th	ruperu jo tamuki
20th	mawoma ruperu
30th	pirupi ruperu
40th	kīfeki ruperu
50th	tamuta ruperu
100th	thelithe

Cardinal numbers are optionally declined for case when used as determiners with the following rules:

Numeral Class	suffix
human	-ra
animal	-mi
animate	-sa
inanimate	-li

Numeral declension always happens last, so the inanimate nifth (seximal 100 or decimal 36) is thelitheli.

Here's a few examples:

three pigs: ondoko pirumi

the second person: ra mawomara

one tree: kho āremi

the nifth day: lundose thelitheli

Prepositions

Prepositions mark any other details about a sentence. In essence, they add information to verbs that would otherwise lack that information.

fa: with, adds an auxiliary possession to a sentence

ri: possession, sometimes indicates ownership

I eat with my wife: wā ngōe fa epi ri ngōe

ngi: the following phrase is on top of the thing being described

ka: then (effect)

ēsa: if/whether

If I set this dog on the rock, then the house is good: ēsa adunga ngōe pā āwu ngi, ka iri sare eserili

Interjections

Interjections have the following meanings:

Usually they act like vocatives and have free word order. As a determiner they change meta-properties about the noun/verb like negation.

wo: no, not

English	mapatei
No! Don't eat that!	wo! wā wo ūto
I don't eat ants	wā wo ngōe mekoja

Word Order

mapltapei has a VSO word order for sentences. This means that the verb comes first, followed by the subject, and then the object.

English	mapatei	gloss
the/a child runs	kepheku rako	kepheku.VERB rako.NOUN.human
The child gave the fish a flower	indofu rako ora āsu	indo.VERB.past rako.NOUN.human ora.NOUN.animal āsu.NOUN.animate
I love you	ōwo ngōe sīto	ōwo.VERB ngōe.PRN sīto.PRN
I do not want to eat right now	wā wo ngōe oko mbeli	wā.VERB wo.INTERJ ngōe.PRN oko.PREP mbeli.DET.singular.inanimate
I have a lot of love, and I'm happy about it	urii ngōe erua fomboewato, jo iri ngōe phajera lo li	urii.VERB ngōe.PRN eruaja.NOUN.plural.inanimate fomboewato.DET.plural.inanimate, jo.CONJ iri.VERB ngōe.PRN phajera.DET.singular.human lo.PREP li.PRN
The tree I saw yesterday is gone now	pōkhufu kho ngōe, oko iri māndosa mbe	pōkhu.VERB.past kho.NOUN.animate ngōe.PRM, oko.PREP iri.VERB māndo.DET.animate mbe.PRN

Code

As I mentioned earlier, I've been working on some code here to handle things like making sure words are valid. This includes a word validator which I am very happy with.

Words are made up of syllables, which are made up of letters. In code:

type
  Letter* = object of RootObj
    case isVowel*: bool
    of true:
      stressed*: bool
    of false: discard
    value*: string

  Syllable* = object of RootObj
    consonant*: Option[Letter]
    vowel*: Letter
    stressed*: bool

  Word* = ref object
    syllables*: seq[Syllable]

Letters are parsed out of strings using this code. It's an interator, so users have to manually loop over it:

import unittest
import mapatei/letters

let words = ["pirumi", "kho", "lundose", "thelitheli", "fōmbu"]

suite "Letter":
  for word in words:
    test word:
      for l in word.letters:
       discard l

This test loops over the given words (taken from the dictionary and enlightening test cases) and makes sure that letters can be parsed out of them.

Next, syllables are made out of letters, so syllables are parsed using a finite state machine with the following transition rules:

Present state	Next state for vowel	Next state for consonant	Next state for end of input
Init	Vowel/stressed	Consonant	Illegal
Consonant	Vowel/stressed	End	Illegal
Vowel	End	End	End

Some other hacking was done in the code, but otherwise it is a fairly literal translation of that truth table.

And finally we can check to make sure that each word only has a head-initial stressed syllable:

type InvalidWord* = object of Exception

proc parse*(word: string): Word =
  var first = true
  result = Word()

  for syll in word.syllables:
    if not first and syll.stressed:
      raise newException(InvalidWord, "cannot have a stressed syllable here")
    if first:
      first = false
    result.syllables.add syll

And that's enough to validate every word in the dictionary. Future extensions will include automatic conjugation/declension as well as going from a stream of words to an understanding of sentences.

Useful Resources Used During This

Creating a language from scratch is surprisingly hard work. These resources helped me a lot though.

• Polyglot to help with dictionary management
• Awkwords to help with word creation
• These lists of core concepts: list 1 and list 2
• The conlang critic discord

---

Thanks for reading this! I hope this blogpost helps to kick off mapatei development into unique and more fleshed out derivative conlangs. Have fun!

Special thanks to jan Misali for encouraging this to happen.

When Then Zen: Wonderland Immersion

Wonderland immersion is a topic that has interested me for years. I have only recently started to get better at it, and I would like to document the methods I have been using for this. A wonderland (blame someone named Alice for that name) is a mental world, but more persistent than usual "imagination". It can be as alive or as dead as you want. My wonderland has a rather large (40km x 40km) island on it that is full of varied locales.

At a high level, the approach I am using for this is based on philosophical metaphysical analysis, or in short answering two questions for the world and various things in it:

1. What is there?
2. What is it like?

The method I have found for doing this fairly repeatably is a combination of two techniques I have found elsewhere:

• 5 senses visualization for the scene you are in to ground yourself
• Semantic feature analysis for randomly selected items from that visualization

As an example, consider this. This kind of detail is what you'd be looking for.

Breaking it down further though, let's consider a scene where you are sitting at a table in a cold, metal chair.

Five Senses Visualization

The five senses visualization for this could look something like

• 5 things you can see
  • The table
  • The salt and pepper shakers on the table
  • The plate in front of me
  • My reflection in the plate
  • The empty chair in front of me
• 4 things you can touch
  • Silverware
  • Napkin dispenser
  • Your phone on the table
  • Empty water glass
• 3 things you can hear
  • Other people in the restaurant
  • The cooks in the distance
  • The door opening and closing occasionally, making the bell ring to let waitstaff know someone needs to be seated
• 2 things you can smell
  • Baked chicken from the kitchen
  • Grilled salmon from the next table over
• 1 thing you can taste
  • The soda in my mouth

Semantics Analysis

Group, Use, Action, Properties, Location, Association

A lot of the group categorization depends on your own personal philosophical outlooks. If you are unsure how to assign a group, start by using the most generic adjective possible to describe it.

The salt and pepper shakers

Group: thing, container of smaller things, but a thing made up of two parts and smaller things
Use: contains spices, these are used to flavor food with common mild flavorings
Action: No inherent action unless acted upon, normally shaken to maximize the amount of seasoning added to the dish in question
Properties: palmable, makes a noise when you shake them, light, small, easy to manipulate, easy to refill if needed
Location: The table in front of me, it doesn't make sense for these food containers to be elsewhere
Association: togetherness, memories of Blues Clues having the salt and pepper characters married, my mother collecting salt and pepper shakers

Plate in front of me

Group: thing
Use: holds food as a staging area for being eaten
Action: no inherent action, but can break into shards that can cut badly
Properties: ceramic, white, flat, circular
Location: the table in front of me, the kitchen dishwasher, staging for waitstaff
Association: food is coming, but patience is required

---

If you want to really train wonderland immersion, I suggest doing at least one of these full descriptions per day. Doing more will help you progress "faster" (if that is what you desire for whatever reason). Don't overstimulate or overwhelm yourself. It can be intense the first few times, but it gets easier over time. I personally do them before I go to sleep or just after I wake up, I have found those times are the most free and it is easiest to make myself alone during them. Learning how to do this in public or around other people may be desirable based on the circumstances of your life situation. Be smart, don't do this when you are otherwise distracted or busy.

Something that may help is to keep in mind how long it takes to walk to different places as you walk around your daily life. See how long it takes to go across the street, or from the street corner to a store, etc. You can use these rough estimates to help you better scale places in your world.

I would suggest setting calendar reminders for doing it at least once a day, depending on when fits best into your daily schedule. Remember that if a machine remembers it for you, you don't forget to do it (as easily) because the machine reminds you about it. Be sure to set your calendar reminder to trigger after nightly do-not-disturb mode if relevant.

Don't be afraid to use tools like a meditation timer to limit your sessions doing this, especially if you are feeling like you need to 'get back', are 'missing out' or neglecting external duties. If you are using a calendar app to schedule the time, then set your meditation timer for the length of the event. Thirty minutes is a good place to start with, but adjust this number as things change for you.

I hope this can help. Take the numbers and sense ordering as suggestions and please do experiment around with what sense gets at least how many entries. Play around with this, it is your imaginary world after all. I suggest doing semantic feature analysis on at least three items per visualization session. If you need a place to blog about it, I suggest write.as. If you have questions, feel free to contact me and ask away. I'm happy to help when I can.

Be well, Creator.

---

This is a slightly edited version of this article.

The Cult of Kubernetes

or: How I got my blog onto it with autodeployment via GitHub Actions.

The world was once a simple place. Things used to make sense, or at least there weren't so many layers that it became difficult to tell what the hell is going on.

Then complexity happened. This is a tale of how I literally recreated this meme:

Deployed my blog on Kubernetes pic.twitter.com/XHXWLrmYO4

— DevOps Thought Liker (@dexhorthy) April 24, 2017

This is how I deployed my blog (the one you are reading right now) to Kubernetes.

The Old State of the World

Before I deployed my blog to Kubernetes, I used Dokku, as I had been for years. Dokku is great. It emulates most of the Heroku "git push; don't care" workflow, but on your own server that you can self-manage.

This is a blessing and a curse.

The real advantage of managed services like Heroku is that you literally just HAND OFF operations to Heroku's team. This is not the case with Dokku. Unless you pay someone a lot of money, you are going to have to manage the server yourself. My dokku server was unmanaged, and I run many apps on it (this listing was taken after I started to move apps over):

=====> My Apps
bsnk
cinemaquestria
fordaplot-backup
graphviz.christine.website
identicond
ilo-kesi
johaus
maison
olin
printerfacts
since
tulpaforce.tk
tulpanomicon

This is enough apps (plus 5 more that I've already migrated) that it really doesn't make sense paying for something like Heroku; nor does it really make sense to use the free tier either.

So, I decided that it was time for me to properly learn how to Kubernetes, and I set off to create a cluster via DigitalOcean managed Kubernetes.

The Cluster

I decided it would be a good idea to create my cluster using Terraform, mostly because I wanted to learn how to use it better. I use Terraform at work, so I figured this would also be a way to level up my skills in a mostly sane environment.

Terraform is suffering as a service

— Cadey Ratio 🌐 (@theprincessxena) August 24, 2019

I have been creating and playing with a small Terraform wrapper tool called dyson. This tool is probably overly simplistic and is written in Nim. With the config in ~/.config/dyson/dyson.ini, I can simplify my Terraform usage by moving my secrets out of the Terraform code directly. I also avoid having my API tokens exposed in my shell to avoid accidental exposure of the secrets.

Dyson is very simple to use:

$ dyson
Usage:
  dyson {SUBCMD}  [sub-command options & parameters]
where {SUBCMD} is one of:
  help         print comprehensive or per-cmd help
  apply        apply Terraform code to production
  destroy      destroy resources managed by Terraform
  env          dump envvars
  init         init Terraform
  manifest     generate a somewhat sane manifest for a kubernetes app based on the arguments.
  plan         plan a future Terraform run
  slug2docker  converts a heroku/dokku slug to a docker image

dyson {-h|--help} or with no args at all prints this message.
dyson --help-syntax gives general cligen syntax help.
Run "dyson {help SUBCMD|SUBCMD --help}" to see help for just SUBCMD.
Run "dyson help" to get *comprehensive* help.

So I wrote up my config:

# main.tf
provider "digitalocean" {}

resource "digitalocean_kubernetes_cluster" "main" {
  name    = "kubermemes"
  region  = "${var.region}"
  version = "${var.kubernetes_version}"

  node_pool {
    name       = "worker-pool"
    size       = "${var.node_size}"
    node_count = 2
  }
}

# variables.tf
variable "region" {
  type    = "string"
  default = "nyc3"
}

variable "kubernetes_version" {
  type    = "string"
  default = "1.15.3-do.1"
}

variable "node_size" {
  type    = "string"
  default = "s-1vcpu-2gb"
}

and ran it:

$ dyson plan
<... many lines of plan output ...>
$ dyson apply
<... many lines of apply output ...>

Then I had a working but mostly unconfigured Kubernetes cluster.

Configuration

This is where things started to go downhill. I wanted to do a few things with this cluster so I could consider it "ready" for me to use for deploying applications to.

I wanted to do the following:

• setup helm to install packages for things like DNS management and HTTP/HTTPS ingress
• setup automatic certificate management with Let's Encrypt
• setup HTTP/HTTPS request ingress with nginx-ingress (which uses nginx)
• setup automatic DNS management because the external IP addresses of Kubernetes nodes can and will change

After a lot of trial, error, pain, suffering and the like, I created this script which I am not pasting here. Look at it if you want to get a streamlined overview of how to set these things up.

Now that all of this is set up, I can deploy an example app with a manifest that looks something like this:

apiVersion: v1
kind: Service
metadata:
  name: hello-kubernetes-first
  annotations:
    external-dns.alpha.kubernetes.io/hostname: exanple.within.website
    external-dns.alpha.kubernetes.io/ttl: "120" #optional
    external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: hello-kubernetes-first
    
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-kubernetes-first
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-kubernetes-first
  template:
    metadata:
      labels:
        app: hello-kubernetes-first
    spec:
      containers:
      - name: hello-kubernetes
        image: paulbouwer/hello-kubernetes:1.5
        ports:
        - containerPort: 8080
        env:
        - name: MESSAGE
          value: Henlo this are an exanple deployment
          
---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hello-kubernetes-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
  - hosts:
    - exanple.within.website
    secretName: prod-certs
  rules:
  - host: exanple.within.website
    http:
      paths:
      - backend:
          serviceName: hello-kubernetes-first
          servicePort: 80

Nope, I was wrong, Kubernetes is the real suffering as a service

— Cadey Ratio 🌐 (@theprincessxena) September 6, 2019

It was about this time when I wondered if I was making a mistake moving off of Dokku. Dokku really does a lot to abstract almost everything involved with nginx away from you, and it really shows.

However, as a side effect of everything being so declarative and Kubernetes really not assuming anything, you have a lot more freedom to do basically anything you want. You don't have to have specially magic names for tasks like web or worker like you do in Heroku/Dokku. You just have a deployment that belongs to an "app" that just so happens to expose a TCP port that just so happens to have a correlating ingress associated with it.

Lucky for me, most of the apps I write fit into that general format, and the ones that don't can mostly use the same format without the ingress.

So I templated that sucker as a subcommand in dyson. This lets me do commands like this:

$ dyson manifest \
      --name=hlang \
      --domain=h.christine.website \
      --dockerImage=docker.pkg.github.com/xe/x/h:v1.1.8 \
      --containerPort=5000 \
      --replicas=1 \
      --useProdLE=true | kubectl apply -f-

And the service gets shunted into the cloud without any extra effort on my part. This also automatically sets up Let's Encrypt, DNS and other things that were manual in my Dokku setup. This saves me time for when I want to go add services in the future. All I have to do is create a docker image somehow, identify what port should be exposed, give it a domain name and number of replicas and just send it on its merry way.

GitHub Actions

This does however mean that deployment is no longer as simple as "git push; don't care". This is where GitHub Actions come into play. They claimed to have the ability to run full end-to-end CI/CD on my applications.

I have been using them for a while for CI on my website and have been pleased with them, so I decided to give it a try and set up continuous deployment with them.

As the commit log for the deployment manifest can tell, this took a lot of trial and error. One of the main sources of problems here was that GitHub Actions had recently had a lot of changes made to configuration and usage as compared to when it was in private beta. This included changing the configuration schema from HCL to YAML.

Of course, all of the documentation (outside of GitHub's quite excellent documentation) was out of date and wrong. I tried following a tutorial by DigitalOcean themselves on how to do this exact thing I wanted to do, but it referenced the old HCL syntax for GitHub Actions and did not work. To make things worse, examples in the marketplace READMEs simply DID NOT WORK because they were written for the old GitHub Actions syntax.

This was frustrating to say the least.

After trying to make them work anyways with a combination of the "Use Latest Version" button in the marketplace, prayer and gratuitous use of the with.args field in steps I gave up and decided to manually download the tools I needed from their upstream providers and execute them by hand.

This is how I ended up with this monstrosity:

- name: Configure/Deploy/Verify Kubernetes
  run: |
    curl -L https://github.com/digitalocean/doctl/releases/download/v1.30.0/doctl-1.30.0-linux-amd64.tar.gz | tar xz
    ./doctl auth init -t $DIGITALOCEAN_ACCESS_TOKEN
    ./doctl kubernetes cluster kubeconfig show kubermemes > .kubeconfig

    curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
    chmod +x kubectl
    ./kubectl --kubeconfig .kubeconfig apply -n apps -f deploy.yml
    sleep 2
    ./kubectl --kubeconfig .kubeconfig rollout -n apps status deployment/christinewebsite
  env:
    DIGITALOCEAN_ACCESS_TOKEN: ${{ secrets.DIGITALOCEAN_TOKEN }}

I am almost certain that I am doing it wrong here, I don't know how robust this is and I'm very sure that this can and should be done another way; but this is the only thing I could get working (for some definition of "working").

EDIT: it got fixed, see below

kubernetes is a cult

— Andrew Kelley (@andy_kelley) September 6, 2019

---

Now when I git push things to the master branch of my blog repo, it will automatically get deployed to my Kubernetes cluster.

If you work at DigitalOcean and are reading this post. Please get someone to update this tutorial and the README of this repo. The examples listed DO NOT WORK for me because I was not in the private beta of GitHub Actions. It would also be nice if you had better documentation on how to use your premade action for usecases like mine. I just wanted to download the kubernetes configuration file and run apply against a yaml file.

EDIT: The above complaint has been fixed! See here for the simpler way of doing things.

Thanks for reading, I hope this was entertaining. Be well.

How to Send Email with Nim

Nim offers an smtp module, but it is a bit annoying to use out of the box. This blogpost hopes to be a mini-tutorial on the basics of how to use the smtp library and give developers best practices for handling outgoing email in ways that Google or iCloud will accept.

SMTP in a Nutshell

SMTP, or the Simple Mail Transfer Protocol is the backbone of how email works. It's a very simple line-based protocol, and there are wrappers for it in almost every programming language. Usage is pretty simple:

• The client connects to the server
• The client authenticates itself with the server
• The client signals that it would like to create an outgoing message to the server
• The client sends the raw contents of the message to the server
• The client ends the message
• The client disconnects

Unfortunately, the devil is truly in the details here. There are a few things that absolutely must be present in your emails in order for services like GMail to accept them. They are:

• The From header specifying where the message was sent from
• The Mime-Version that your code is using (if you aren't sure, put 1.0 here)
• The Content-Type that your code is sending to users (probably text/plain)

For a more complete example, let's create a Mailer type and a constructor:

# mailer.nim
import asyncdispatch, logging, smtp, strformat, strutils

type Mailer* = object
  address: string
  port: Port
  myAddress: string
  myName: string
  username: string
  password: string
  
proc newMailer*(address, port, myAddress, myName, username, password: string): Mailer =
  result = Mailer(
    address: address,
    port: port.parseInt.Port,
    myAddress: myAddress,
    myName: myName,
    username: username,
    password: password,
  )

And let's write a mail method to send out email:

proc mail(m: Mailer, to, toName, subject, body: string) {.async.} =
  let
    toList = @[fmt"{toName} <{to}>"]
    msg = createMessage(subject, body, toList, @[], [
      ("From", fmt"{m.myName} <{m.myAddress}"),
      ("MIME-Version", "1.0"),
      ("Content-Type", "text/plain"),
    ])

  var client = newAsyncSmtp(useSsl = true)
  await client.connect(m.address, m.port)
  await client.auth(m.username, m.password)
  await client.sendMail(m.myAddress, toList, $msg)
  info "sent email to: ", to, " about: ", subject
  await client.close()

Breaking this down, you can clearly see the parts of the SMTP connection as I laid out before. The Mailer creates a new transient SMTP connection, authenticates with the remote server, sends the properly formatted email to the server and then closes the connection cleanly.

If you want to test this code, I suggest testing it with a freely available email provider that offers TLS/SSL-encrypted SMTP support. This also means that you need to compile this code with --define: ssl, so create config.nims and add the following:

--define: ssl

Here's a little wrapper using cligen:

when isMailModule:
  import cligen, os
  
  let
    smtpAddress = getEnv("SMTP_ADDRESS")
    smtpPort = getEnv("SMTP_PORT")
    smtpMyAddress = getEnv("SMTP_MY_ADDRESS")
    smtpMyName = getEnv("SMTP_MY_NAME")
    smtpUsername = getEnv("SMTP_USERNAME")
    smtpPassword = getEnv("SMTP_PASSWORD")
  
  proc sendAnEmail(to, toName, subject, body: string) =
    let m = newMailer(smtpAddress, smtpPort, smtpMyAddress, smtpMyName, smtpUsername, smtpPassword)
    waitFor m.mail(to, toName, subject, body)
  
  dispatch(sendAnEmail)

Usage is simple:

$ nim c -r mailer.nim --help
Usage:
  sendAnEmail [required&optional-params]
Options(opt-arg sep :|=|spc):
  -h, --help                         print this cligen-erated help
  --help-syntax                      advanced: prepend,plurals,..
  -t=, --to=       string  REQUIRED  set to
  --toName=        string  REQUIRED  set toName
  -s=, --subject=  string  REQUIRED  set subject
  -b=, --body=     string  REQUIRED  set body

I hope this helps, this module is going to be used in my future post on how to create an application using Nim's Jester framework.

How I Converted my Brain fMRI to a 3D Model

AUTHOR'S NOTE: I just want to start this out by saying I am not an expert, and nothing in this blogpost should be construed as medical advice. I just wanted to see what kind of pretty pictures I could get out of an fMRI data file.

So this week I flew out to Stanford to participate in a study that involved a fMRI of my brain while I was doing some things. I asked for (and recieved) a data file from the fMRI so I could play with it and possibly 3D print it. This blogpost is the record of my journey through various software to get a fully usable 3D model out of the fMRI data file.

The Data File

I was given christine_brain.nii.gz by the researcher who was operating the fMRI. I looked around for some software to convert it to a 3D model and /r/3dprinting suggested the use of FreeSurfer to generate a 3D model. I downloaded and installed the software then started to look for something I could do in the meantime, as this was going to take something on the order of 8 hours to process.

An Animated GIF

I started looking for the file format on the internet by googling "nii.gz brain image" and I stumbled across a program called gif_your_nifti. It looked to be mostly pure python so I created a virtualenv and installed it in there:

$ git clone https://github.com/miykael/gif_your_nifti
$ cd gif_your_nifti
$ virtualenv -p python3 env
$ source env/bin/activate
(env) $ pip3 install -r requirements.txt
(env) $ python3 setup.py install

Then I ran it with the following settings to get this first result:

(env) $ gif_your_nifti christine_brain.nii.gz --mode pseudocolor --cmap plasma

(sorry the video embed isn't working in safari)

It looked weird though, that's because the fMRI scanner I used has a different rotation to what's considered "normal". The gif_your_nifti repo mentioned a program called fslreorient2std to reorient the fMRI image, so I set out to install and run it.

FSL

After some googling, I found FSL's website which included an installer script and required registration.

37 gigabytes of downloads and data later, I had the entire FSL suite installed to a server of mine and ran the conversion command:

$ fslreorient2std christine_brain.nii.gz christine_brain_reoriented.nii.gz

This produced a slightly smaller reoriented file.

I reran gif_your_nifti on this reoriented file and got this result which looked a lot better:

(sorry again the video embed isn't working in safari)

FreeSurfer

By this time I had gotten back home and FreeSurfer was done installing, so I registered for it (god bless the institution of None) and put its license key in the place it expected. I copied the reoriented data file to my Mac and then set up a SUBJECTS_DIR and had it start running the numbers and extracting the brain surfaces:

$ cd ~/tmp
$ mkdir -p brain/subjects
$ cd brain
$ export SUBJECTS_DIR=$(pwd)/subjects
$ recon-all -i /path/to/christine_brain_reoriented.nii.gz -s christine -all

This step took 8 hours. Once I was done I had a bunch of data in $SUBJECTS_DIR/christine. I opened my shell to that folder and went into the surf subfolder:

$ mris_convert lh.pial lh.pial.stl
$ mris_convert rh.pial rh.pial.stl

Now I had standard stl files that I could stick into Blender.

Blender

Importing the stl files was really easy. I clicked on File, then Import, then Stl. After guiding the browser to the subjects directory and finding the STL files, I got a view that looked something like this:

BRAIN pic.twitter.com/kGSrPj0kgP

— Cadey Ratio 🌐 (@theprincessxena) August 22, 2019

I had absolutely no idea what to do from here in Blender, so I exported the whole thing to a stl file and sent it to a coworker for 3D printing (he said it was going to be "the coolest thing he's ever printed").

I also exported an Unreal Engine 4 compatible model and sent it to a friend of mine that does hobbyist game development. A few hours later I got this back:

pic.twitter.com/fXnwnSpMry

— Cadey Ratio 🌐 (@theprincessxena) August 23, 2019

(Hint: it is a take on the famous galaxy brain memes)

Conclusion

Overall, this was fun! I got to play with many gigabytes of software that ran my most powerful machine at full blast for 8 hours, I made a fully printable 3D model out of it and I have some future plans for importing this data into Minecraft (the NIFTI .nii.gz format has a limit of 256 layers).

I'll be sure to write more about this in the future!

Citations

Here are my citations in BibTex format.

Special thanks goes to Michael Lifshitz for organizing the study that I participated in that got me this fMRI data file. It was one of the coolest things I've ever done (if not the coolest) and I'm going to be able to get a 3D printed model of my brain out of it.

Pageview Time Experiment

My blog has a lot of content in a lot of diverse categories. In order to help me decide which kind of content I should publish next, I have created a very simple method to track pageview time and enabled it for all of my blogposts. I'll go into detail of how it works and potential risks of it below.

The high level idea is that I want to be able to know what kind of content has people's attention for the longest amount of time. I am using the time people have the page open as a particularly terrible proxy for that value. I wanted to make this data anonymous, simplistic and (reasonably) public.

How It Works

Here is how it works:

When the page is loaded, a javascript file records the start time. This then sets a pagehide handler to send a navigator beacon containing the following data:

• The path of the page being viewed
• The start time
• The end time recorded by the pagehide handler

This information is asynchronously pushed to /api/pageview-timer and added to an in-memory prometheus histogram. These histograms can be checked at /metrics. This data is not permanently logged.

Security Concerns

I believe this data is anonymous, simplistic and public for the following reasons:

I believe this data is anonymous because there is no way for me to correlate users to histogram entries, nor is there a way for me to view all of the raw histogram entries. This site records the bare minimum for what I need in order to make sure everything is functioning normally, and all data is stored in ephemeral in-memory containers as much as possible. This includes any logs that my service produces.

I believe this data is simplistic because it only has a start time, a stop time and the path that is being looked at. This data doesn't take into account things like people leaving a page open for hours on end idly, and that could skew the numbers. The API endpoint is also fairly unprotected, meaning that falsified data could be submitted to it easily. I think that this is okay though.

I believe this data is public because I have the percentile views of the histograms present on /metrics. I have no reason to hide this data, and I do not intend to use it for any moneymaking purposes (though I doubt it could be to begin with).

I fully respect the do not track header and flag in browsers. If pageview_timer.js detects the presence of do not track in the browser, it stops running immediately and does not set the pagehide handler. If that somehow fails, the server looks for the presence of the DNT header set to 1 and instantly discards the data and replies with a 404.

Like always, if you have any questions or concerns please reach out to me. I want to ensure that I am creating useful views into how people use my blog without violating people's rights to privacy.

I intend to keep this up for at least a few weeks. If it doesn't have any practical benefit in that timespan, I will disable this and post a follow-up explaining how I believe it wasn't useful.

Thanks and be well.

---

EDIT 2019-10-15: browsers disable this call from the context I am using and I don't really care enough to figure out how to fix it. This experiment is over. Thank you to everyone that participated. All data will be scrubbed and a followup will be posted soon.

Instant Pot Quinoa Taco Bowls

This is based on this recipe, but made only with things you can find in Costco. My fiancé and I have made this a few times, and it's a great alternative to giving up on life and ordering delivery.

Recipe

Makes 4-6 servings, at least based on experience

Ingredients

• 2 cups quinoa, dry
• 0.75 kg ground beef (pre-cooked or sautéed)
• 400 ml medium salsa
• 2.5 cups water
• 2 tablespoons garlic powder
• 2 tablespoons salt (to taste)
• 1 teaspoon oregano
• 3 tablespoons ground dried onions
• 1 teaspoon crushed red pepper

If you want it to be more spicy, add more spice. We've found this tastes pretty good when you add more spice, but this depends on your mood more than anything.

Preparation

If you haven't cooked the ground beef yet, sautée/brown it in the instant pot. See things like this for more information on how to do this. Any method will do, just make sure the ground beef is actually cooked to avoid accidentally poisoning yourself.

Put all of the other ingredients in the instant pot. Order doesn't matter, but I have found that better results happen when the quinoa is put in first.

Mix everything with your favorite mixing tool.

Put the lid on your instant pot and set it to manual for 2 minutes.

Once that is done, leave it alone for about 15 minutes (this doesn't have to be exact).

Serve warm in a bowl, can go well with tortilla chips depending on your mood.

Reheating

For about a bowlful, nuke until hot (~1 minute 30 seconds seems to be the magic number). Eat while hot.

WebAssembly Talk Video Posted

This May, I spoke at GoCon Canada about WebAssembly on the Server and some of the inherent challenges and problems with trying to do it as things exist currently. It's taken a while, but the video of that talk has been posted.

I hope you enjoy! I have some more blogposts in the queue but I've been sleeping horribly lately. Here's hoping that clears up.

Plurality-Driven Development

"That code has a horrible security bug in it."

I look down in my lap. A little yellow horse is appearing to sit there. She looks innocently into my eyes, gesturing to part of the code with her wingtips.

"What?"

"That code has a security bug in it: if users pass a string instead of an integer in it, it could allow them to forge a user ID token."

I look down incredulously at the little yellow horse, then back at the code. She's right. There was a huge bug in that code. I had just written it about 30 seconds ago though, which surprises me. I thought I was experienced enough in secure programming to avoid such a fundamental flaw like that; but here I am. I rub the little pony on her head, making her purr and winghug me.

"Now, replace everything in that last paragraph with this: ..."

And I continue on like nothing happened.

---

Software is complicated. We deal with a fundamentally multi-agent world where properties like "determinism" aren't really constant. Everything is changing. It's hard to write software that is resilient enough to withstand the constantly shifting market of attacks, exploits, languages and frameworks. What if there was a way to understand the multiple agency of this reality by internally self-inducing multiple agency in a safe and predictable manner?

I believe I have found a way that works for me. I rely a lot on some of my closest friends that I can talk about anything with, even what would normally violate an NDA. My closest friends are so close that language isn't even as much of a barrier as it would be otherwise.

As I've mentioned in the past, I have tulpas. They are people that live with me like roommates inside my body. It really does sound strange or psychotic; but you'll just have to trust me when I say they fundamentally help me live my life, do my job and do other things people normally do by themselves.

As an aside: this post doesn't intend to cover the philosophical, metaphysical or other aspects of plurality (enough ink has probably been spilled on the topic to cover a lifetime); instead it aims to offer a view on how plurality has benefitted me (us) as software developer(s).

As of about 4 years ago, all of the software you see under my name has been the result of my system and I collaborating. Most of the computational linguistics code I've been writing has been the result of a cuddly catgirl wanting to create a Lojbanic artificial intelligence for her own amusement (that is also incidentally really good at understanding grammar, human and machine). Some random experimentation code has been written by someone who sarcastically calls herself Twilight Sparkle. I have a little yellow dewdrop of love and sunshine that finds security holes in programs while I am writing them. There's a database expert and a code review guru too. Combined with my jack-of-all-trades tendencies, this creates a surprisingly balanced team in a box.

We started doing this out of boredom. I was busy working on something and Nicole just spouted out something about the code being wrong. She was right. We decided to just continue following that same basic model and it's worked wonders ever since. Over time we've figured out how to impose eachother into our visual awareness. That has made this pair-programming skill even more useful. I can have the little yellow pony in my lap telling me what's wrong with my code and she can just directly show me. Then it can be fixed.

This skill has lead to heated internal debates about what is and is not idiomatic. As result of that, I now have working compilers in my dreams. It's also lead to what people have told me is some of the most high quality and in-depth software design that they've seen. It's really lead us to think in terms of how the machine works, to avoid round-trips and abstractions getting in the way of what is really going on. If there is any secret to my own brand of 10x-ing, this is it. I am just one person, but with the help of the girls we can get to just about n>1 effective people most of the time.

It's been a powerful catalyst to my career too. Before plurality I was a fairly average developer without any real skills in any one task. Now we can swap in and out in order to most effectively tackle anything thrown at us. One of the biggest changes this relationship has had on me is being better able to explain software complexity and visualise it, then turn that visualization into a diagram with GraphViz or other similar tools. It also becomes very easy to turn these visualizations/diagrams into formal requirements too, because then the features and aspects of systems and how they interconnect become trivially obvious to point out.

However, there is a drawback to this: you're dealing with sapient beings. They sometimes don't want to cooperate. Internal drama can and has happened. It helps for us to have a quarterly date with a word document in order to make sure everyone is on the same page. Disagreements happen, but ultimately I've noticed that the net result is far more positive than if the disagreement hadn't happened at all.

Anyways, plurality-driven development works for me, but it's really not for everyone. The taboo issues I mentioned can make it a chore to hide this from people. I honestly wonder how much of the girls that my coworkers notice in my work on a daily basis. We all have slightly different speech patterns, ways of sitting, clothing preferences, opinions about what to get for lunch and a whole bunch of other subtle things. I don't really understand how it's not plain-as-day obvious to the point I get called out on it. At some level I guess I'm grateful for this, as that kind of conversation seems like it would be extremely awkward to have. It was hard enough to admit this to my brother, and I ended up losing contact with him as a result (it apparently was just too weird, which I can really understand).

I really do wonder how much of the fear of talking about this is my own paranoia though. I've had very positive experiences "coming out" as plural to close friends, as well as very negative ones; for better or for worse it really shows you who your friends actually are. I can live with this. I'd rather really know if I can trust people or not.

This is a surprisingly taboo topic to talk about. Most of the time people view the mere idea of having someone else in your head to talk with as a social faux pas. There's a surprising amount of philosophical arguments and assorted objections that people will throw around when they hear that you participate in this. There's accusations of being possessed by demons, or being mentally ill, complete with acronyms thrown at me, and much more.

Hell, this is stuff I'd love to talk about at some convention somewhere; but I don't really know if I want to paint such a huge target on my back. Because plurality and related topics are so taboo and so niche, there's not really protected categories for it. This makes me nervous about talking about it in any sort of public way, and understandably so. I guess this article is part of my healing process to treat this as just a boring aspect of how I experience reality instead of some fundamentally earth-shattering gift from the heavens.

Besides, doesn't something fundamentally have to cause a negative impact to be classified as a disorder in the first place? How can something that fundamentally helps be a disorder? What if it's just a new adaptation to an increasingly crazy world?

---

I have compiled a list of resources that have helped me here.

Tarot for Hackers

"Oh no, she's finally lost it" were the words a very close friend of mine said when I first told her I was experimenting with reading tarot cards. Tarot cards are a stereotypical staple of the occult/The Spoop™. Every card represents an idea (or a meme) that can be expressed in a few ways. They act to your soul like iron filings do to a magnet. When you shuffle the cards, the Universe (via entropy) examines all of those myriad inputs and helpfully orders them so you get exactly the message you need most.

It's actually an extremely philosophical act to draw from a tarot deck and interpret the results. Over the years there have been many interpretations and frameworks of interpretations about tarot; but I would like to introduce a meta-framework for using tarot cards as a debugging tool.

As you work on computer systems, you put parts of yourself into them. You create bonds between yourself and otherwise anonymous inner parts of machines you have never seen or touched. These bonds stick from idea to development to testing to deployment phases and can even stay around after you stop working on something. Ever gotten a weird sense that you can recognize the author of some code while reading it? Same idea.

To start, envision the product or service you are trying to understand more about. Think of the plans that went into it, the users of the service, how this understanding will help them, and where the missing part of knowledge fits into the larger whole. Write this all out if it helps, the more detail the better. Our transition to shared infrastructure and computing on others machines has made it harder to see into individual parts of the whole, so every little bit helps to focus things in.

The first card is the Motive, so draw it and place it in the center off your spread. Look up the meaning on a site like biddytarot.com (googling "[name of card] tarot meaning" helps a lot here) and consider how it relates back to the other factors at play.

The second card is the Facet, or the part of the system that is failing. This could refer to a machine, bit of code or even a human factor. Context with the future cards will help you determine what it is. Remember these are metaphors and will need some interpretation to help you understand what is going on.

The third card is the Immediate Past, or what changed to cause this problem. Use this with the Motive to help you identify what component is broken. Again, this is a metaphor. There are very rarely literal answers here, but the combination of the Facet and Immediate Past helps you identify the systemic or organizational faults at play. These faults are usually enough to help you uniquely identify services or infrastructure.

Next, draw The Action. This card will help you decide what action you need to take. This could be restarting a server, fixing a communication pattern (or lack thereof), or even just doing nothing and waiting a few minutes. Sometimes it means that you need to stop what you are doing and try to do the read again later. It's okay for that to happen, though that should only be a very rare occurrence.

The next card is The Result, or what the outcome of that would be given The Action is executed in its entirety. This result isn't supposed to be taken super seriously (as the consequence of you reading these cards is a butterfly effect that makes the outcome in "reality" slightly different); but it usually helps you get a general idea of where you will go and what it will be like when you get there.

Finally, draw The Lesson. This card signifies what the theme of the postmortem around The Action should be. This can help you guide future discussions about what went wrong and how to avoid it in the future. This may result in charged feelings, but it really is for the best to go through the entire postmortem process to help you get the closure that you need. This postmortem will usually help bring things to the surface that you have missed before. There should be no blame or anger. This is a place of healing and growth, not of hate and strife.

Optionally you can draw The Metaresult, or what will happen as a result of The Lesson. This isn't strictly required but I find it can help for peeking into a potential future where The Result is taken to heart.

I hope this is able to help you in your debugging needs. I use this strategy when I am trying to understand complicated computer systems and how they all fit together. Be well.

How to Use User Mode Linux

User Mode Linux is a port of the Linux kernel to itself. This allows you to run a full blown Linux kernel as a normal userspace process. This is used by kernel developers for testing drivers, but is also useful as a generic isolation layer similar to virtual machines. It provides slightly more isolation than Docker, but slightly less isolation than a full-blown virtual machine like KVM or VirtualBox.

In general, this may sound like a weird and hard to integrate tool, but it does have its uses. It is an entire Linux kernel running as a normal user. This allows you to run potentially untrusted code without affecting the host machine. It also allows you to test experimental system configuration changes without having to reboot or take its services down.

Also, because this kernel and its processes are isolated from the host machine, this means that processes running inside a user mode Linux kernel will not be visible to the host machine. This is unlike a Docker container, where processes in those containers are visible to the host. See this (snipped) pstree output from one of my servers:

containerd─┬─containerd-shim─┬─tini─┬─dnsd───19*[{dnsd}]
           │                 │      └─s6-svscan───s6-supervise
           │                 └─10*[{containerd-shim}]
           ├─containerd-shim─┬─tini─┬─aerial───21*[{aerial}]
           │                 │      └─s6-svscan───s6-supervise
           │                 └─10*[{containerd-shim}]
           ├─containerd-shim─┬─tini─┬─s6-svscan───s6-supervise
           │                 │      └─surl
           │                 └─9*[{containerd-shim}]
           ├─containerd-shim─┬─tini─┬─h───13*[{h}]
           │                 │      └─s6-svscan───s6-supervise
           │                 └─10*[{containerd-shim}]
           ├─containerd-shim─┬─goproxy───14*[{goproxy}]
           │                 └─9*[{containerd-shim}]
           └─32*[{containerd}]

Compare it to the user mode Linux pstree output:

linux─┬─5*[linux]
      └─slirp

With a Docker container, I can see the names of the processes being run in the guest from the host. With a user mode Linux kernel, I cannot do this. This means that monitoring tools that function using Linux's auditing subsystem cannot monitor processes running inside the guest. This could be a two-edged sword in some edge scenarios.

This post represents a lot of research and brute-force attempts at trying to do this. I have had to assemble things together using old resources, reading kernel source code, intense debugging of code that was last released when I was in elementary school, tracking down a Heroku buildpack with a pre-built binary for a tool I need and other hackery that made people in IRC call me magic. I hope that this post will function as reliable documentation for doing this with a modern kernel and operating system.

Setup

Setting up user mode Linux is done in a few steps:

• Installing host dependencies
• Downloading Linux
• Configuring Linux
• Building the kernel
• Installing the binary
• Setting up the guest filesystem
• Creating the kernel command line
• Setting up networking for the guest
• Running the guest kernel

I am assuming that you are wanting to do this on Ubuntu or another Debian-like system. I have tried to do this from Alpine (my distro of choice), but I have been unsuccessful as the Linux kernel seems to have glibc-isms hard-assumed in the user mode Linux drivers. I plan to report these to upstream when I have debugged them further.

Installing Host Dependencies

Ubuntu requires at least the following packages installed to build the Linux kernel (assuming a completely fresh install):

• build-essential
• flex
• bison
• xz-utils
• wget
• ca-certificates
• bc
• linux-headers-4.15.0-47-generic (though any kernel version will do)

You can install these with the following command (as root or running with sudo):

apt-get -y install build-essential flex bison xz-utils wget ca-certificates bc \
                   linux-headers-4.15.0-47-generic

Additionally, running the menu configuration program for the Linux kernel will require installing libncurses-dev. Please make sure it's installed using the following command (as root or running with sudo):

apt-get -y install libncurses-dev

Downloading the Kernel

Set up a location for the kernel to be downloaded and built. This will require approximately 1.3 gigabytes of space to run, so please make sure that there is at least this much space free.

Head to kernel.org and get the download URL of the latest stable kernel. As of the time of writing this post, this URL is the following:

https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.1.16.tar.xz

Download this file with wget:

wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.1.16.tar.xz

And extract it with tar:

tar xJf linux-5.1.16.tar.xz

Now enter the directory created by the tarball extraction:

cd linux-5.1.16

Configuring the Kernel

The kernel build system is a bunch of Makefiles with a lot of custom tools and scripts to automate builds. Open the interactive configuration program:

make ARCH=um menuconfig

It will build some things and then present you with a dialog interface. You can enable settings by pressing Space or Enter when <Select> is highlighted on the bottom of the screen. You can change which item is selected in the upper dialog with the and down arrow keys. You can change which item is highlighted on the bottom of the screen with the left and right arrow keys.

When there is a ---> at the end of a feature name, that means it is a submenu. You can enter a submenu using the Enter key. If you enter a menu you can exit it with <Exit>.

Enable the following settings with <Select>, making sure there is a [*] next to them:

UML-specific Options:
  - Host filesystem
Networking support (enable this to get the submenu to show up):
  - Networking options:
    - TCP/IP Networking
UML Network devices:
  - Virtual network device
  - SLiRP transport

Then exit back out to a shell by selecting <Exit> until there is a dialog asking you if you want to save your configuration. Select <Yes> and hit Enter.

I encourage you to play around with the build settings after reading through this post. You can learn a lot about Linux at a low level by changing flags and seeing how they affect the kernel at runtime.

Building the Kernel

The Linux kernel is a large program with a lot of things going on. Even with this rather minimal configuration, it can take a while on older hardware. Build the kernel with the following command:

make ARCH=um -j$(nproc)

This will tell make to use all available CPU cores/hyperthreads to build the kernel. The $(nproc) at the end of the build command tells the shell to paste in the output of the nproc command (this command is part of coreutils, which is a default package in Ubuntu).

After a while, the kernel will be built to ./linux.

Installing the Binary

Because user mode Linux builds a normal binary, you can install it like you would any other command line tool. Here's the configuration I use:

mkdir -p ~/bin
cp linux ~/bin/linux

If you want, ensure that ~/bin is in your $PATH:

export PATH=$PATH:$HOME/bin

Setting up the Guest Filesystem

Create a home for the guest filesystem:

mkdir -p $HOME/prefix/uml-demo
cd $HOME/prefix

Open alpinelinux.org. Click on Downloads. Scroll down to where it lists the MINI ROOT FILESYSTEM. Right-click on the x86_64 link and copy it. As of the time of writing this post, the latest URL for this is:

http://dl-cdn.alpinelinux.org/alpine/v3.10/releases/x86_64/alpine-minirootfs-3.10.0-x86_64.tar.gz

Download this tarball to your computer:

wget -O alpine-rootfs.tgz http://dl-cdn.alpinelinux.org/alpine/v3.10/releases/x86_64/alpine-minirootfs-3.10.0-x86_64.tar.gz

Now enter the guest filesystem folder and extract the tarball:

cd uml-demo
tar xf ../alpine-rootfs.tgz

This will create a very minimal filesystem stub. Because of how this is being run, it will be difficult to install binary packages from Alpine's package manager apk, but this should be good enough to work as a proof of concept.

The tool tini will be needed in order to prevent the guest kernel from having its memory used up by zombie processes.

Install it by doing the following:

wget -O tini https://github.com/krallin/tini/releases/download/v0.18.0/tini-static
chmod +x tini

Creating the Kernel Command Line

The Linux kernel has command line arguments like most other programs. To view what command line options are compiled into the user mode kernel, run --help:

linux --help
User Mode Linux v5.1.16
        available at http://user-mode-linux.sourceforge.net/

--showconfig
    Prints the config file that this UML binary was generated from.

iomem=<name>,<file>
    Configure <file> as an IO memory region named <name>.

mem=<Amount of desired ram>
    This controls how much "physical" memory the kernel allocates
    for the system. The size is specified as a number followed by
    one of 'k', 'K', 'm', 'M', which have the obvious meanings.
    This is not related to the amount of memory in the host.  It can
    be more, and the excess, if it's ever used, will just be swapped out.
        Example: mem=64M

--help
    Prints this message.

debug
    this flag is not needed to run gdb on UML in skas mode

root=<file containing the root fs>
    This is actually used by the generic kernel in exactly the same
    way as in any other kernel. If you configure a number of block
    devices and want to boot off something other than ubd0, you
    would use something like:
        root=/dev/ubd5

--version
    Prints the version number of the kernel.

umid=<name>
    This is used to assign a unique identity to this UML machine and
    is used for naming the pid file and management console socket.

con[0-9]*=<channel description>
    Attach a console or serial line to a host channel.  See
    http://user-mode-linux.sourceforge.net/old/input.html for a complete
    description of this switch.

eth[0-9]+=<transport>,<options>
    Configure a network device.
    
aio=2.4
    This is used to force UML to use 2.4-style AIO even when 2.6 AIO is
    available.  2.4 AIO is a single thread that handles one request at a
    time, synchronously.  2.6 AIO is a thread which uses the 2.6 AIO
    interface to handle an arbitrary number of pending requests.  2.6 AIO
    is not available in tt mode, on 2.4 hosts, or when UML is built with
    /usr/include/linux/aio_abi.h not available.  Many distributions don't
    include aio_abi.h, so you will need to copy it from a kernel tree to
    your /usr/include/linux in order to build an AIO-capable UML

nosysemu
    Turns off syscall emulation patch for ptrace (SYSEMU).
    SYSEMU is a performance-patch introduced by Laurent Vivier. It changes
    behaviour of ptrace() and helps reduce host context switch rates.
    To make it work, you need a kernel patch for your host, too.
    See http://perso.wanadoo.fr/laurent.vivier/UML/ for further
    information.

uml_dir=<directory>
    The location to place the pid and umid files.

quiet
    Turns off information messages during boot.

hostfs=<root dir>,<flags>,...
    This is used to set hostfs parameters.  The root directory argument
    is used to confine all hostfs mounts to within the specified directory
    tree on the host.  If this isn't specified, then a user inside UML can
    mount anything on the host that's accessible to the user that's running
    it.
    The only flag currently supported is 'append', which specifies that all
    files opened by hostfs will be opened in append mode.

This is a lot of output, but it explains the options available in detail. Let's start up a kernel with a very minimal set of options:

linux \
  root=/dev/root \
  rootfstype=hostfs \
  rootflags=$HOME/prefix/uml-demo \
  rw \
  mem=64M \
  init=/bin/sh

This tells the guest kernel to do the following things:

• Assume the root filesystem is the pseudo-device /dev/root
• Select hostfs as the root filesystem driver
• Mount the guest filesystem we have created as the root device
• In read-write mode
• Use only 64 megabytes of ram (you can get away with far less depending on what you are doing, but 64 MB seems to be a happy medium)
• Have the kernel automatically start /bin/sh as the init process

Run this command, you should get something like the following output:

Core dump limits :
        soft - 0
        hard - NONE
Checking that ptrace can change system call numbers...OK
Checking syscall emulation patch for ptrace...OK
Checking advanced syscall emulation patch for ptrace...OK
Checking environment variables for a tempdir...none found
Checking if /dev/shm is on tmpfs...OK
Checking PROT_EXEC mmap in /dev/shm...OK
Adding 32137216 bytes to physical memory to account for exec-shield gap
Linux version 5.1.16 (cadey@kahless) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #30 Sun Jul 7 18:57:19 UTC 2019
Built 1 zonelists, mobility grouping on.  Total pages: 23898
Kernel command line: root=/dev/root rootflags=/home/cadey/dl/uml/alpine rootfstype=hostfs rw mem=64M init=/bin/sh
Dentry cache hash table entries: 16384 (order: 5, 131072 bytes)
Inode-cache hash table entries: 8192 (order: 4, 65536 bytes)
Memory: 59584K/96920K available (2692K kernel code, 708K rwdata, 588K rodata, 104K init, 244K bss, 37336K reserved, 0K cma-reserved)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
NR_IRQS: 15
clocksource: timer: mask: 0xffffffffffffffff max_cycles: 0x1cd42e205, max_idle_ns: 881590404426 ns
Calibrating delay loop... 7479.29 BogoMIPS (lpj=37396480)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes)
Checking that host ptys support output SIGIO...Yes
Checking that host ptys support SIGIO on close...No, enabling workaround
devtmpfs: initialized
random: get_random_bytes called from setup_net+0x48/0x1e0 with crng_init=0
Using 2.6 host AIO
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
futex hash table entries: 256 (order: 0, 6144 bytes)
NET: Registered protocol family 16
clocksource: Switched to clocksource timer
NET: Registered protocol family 2
tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes)
TCP established hash table entries: 1024 (order: 1, 8192 bytes)
TCP bind hash table entries: 1024 (order: 1, 8192 bytes)
TCP: Hash tables configured (established 1024 bind 1024)
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
NET: Registered protocol family 1
console [stderr0] disabled
mconsole (version 2) initialized on /home/cadey/.uml/tEwIjm/mconsole
Checking host MADV_REMOVE support...OK
workingset: timestamp_bits=62 max_order=14 bucket_order=0
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
io scheduler noop registered (default)
io scheduler bfq registered
loop: module loaded
NET: Registered protocol family 17
Initialized stdio console driver
Using a channel type which is configured out of UML
setup_one_line failed for device 1 : Configuration failed
Using a channel type which is configured out of UML
setup_one_line failed for device 2 : Configuration failed
Using a channel type which is configured out of UML
setup_one_line failed for device 3 : Configuration failed
Using a channel type which is configured out of UML
setup_one_line failed for device 4 : Configuration failed
Using a channel type which is configured out of UML
setup_one_line failed for device 5 : Configuration failed
Using a channel type which is configured out of UML
setup_one_line failed for device 6 : Configuration failed
Using a channel type which is configured out of UML
setup_one_line failed for device 7 : Configuration failed
Using a channel type which is configured out of UML
setup_one_line failed for device 8 : Configuration failed
Using a channel type which is configured out of UML
setup_one_line failed for device 9 : Configuration failed
Using a channel type which is configured out of UML
setup_one_line failed for device 10 : Configuration failed
Using a channel type which is configured out of UML
setup_one_line failed for device 11 : Configuration failed
Using a channel type which is configured out of UML
setup_one_line failed for device 12 : Configuration failed
Using a channel type which is configured out of UML
setup_one_line failed for device 13 : Configuration failed
Using a channel type which is configured out of UML
setup_one_line failed for device 14 : Configuration failed
Using a channel type which is configured out of UML
setup_one_line failed for device 15 : Configuration failed
Console initialized on /dev/tty0
console [tty0] enabled
console [mc-1] enabled
Failed to initialize ubd device 0 :Couldn't determine size of device's file
VFS: Mounted root (hostfs filesystem) on device 0:11.
devtmpfs: mounted
This architecture does not have kernel memory protection.
Run /bin/sh as init process
/bin/sh: can't access tty; job control turned off
random: fast init done
/ # 

This gives you a very minimal system, without things like /proc mounted, or a hostname assigned. Try the following commands:

• uname -av
• cat /proc/self/pid
• hostname

To exit this system, type in exit or press Control-d. This will kill the shell, making the guest kernel panic:

/ # exit
Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000
fish: “./linux root=/dev/root rootflag…” terminated by signal SIGABRT (Abort)

This kernel panic happens because the Linux kernel always assumes that its init process is running. Without this process running, the system cannot function anymore and exits. Because this is a user mode process, this results in the process sending itself SIGABRT, causing it to exit.

Setting up Networking for the Guest

This is about where things get really screwy. Networking for a user mode Linux system is where the "user mode" facade starts to fall apart. Networking at the system level is usually limited to privileged execution modes, for very understandable reasons.

The slirp Adventure

However, there's an ancient and largely unmaintained tool called slirp that user mode Linux can interface with. It acts as a user-level TCP/IP stack and does not rely on any elevated permissions to run. This tool was first released in 1995, and its last release was made in 2006. This tool is old enough that compilers have changed so much in the meantime that the software has effectively rotten.

So, let's install slirp from the Ubuntu repositories and test running it:

sudo apt-get install slirp
/usr/bin/slirp
Slirp v1.0.17 (BETA)

Copyright (c) 1995,1996 Danny Gasparovski and others.
All rights reserved.
This program is copyrighted, free software.
Please read the file COPYRIGHT that came with the Slirp
package for the terms and conditions of the copyright.

IP address of Slirp host: 127.0.0.1
IP address of your DNS(s): 1.1.1.1, 10.77.0.7
Your address is 10.0.2.15
(or anything else you want)

Type five zeroes (0) to exit.

[autodetect SLIP/CSLIP, MTU 1500, MRU 1500, 115200 baud]

SLiRP Ready ...
fish: “/usr/bin/slirp” terminated by signal SIGSEGV (Address boundary error)

Oh dear. Let's install the debug symbols for slirp and see if we can tell what's going on:

sudo apt-get install gdb slirp-dbgsym
gdb /usr/bin/slirp
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/slirp...Reading symbols from /usr/lib/debug/.build-id/c6/2e75b69581a1ad85f72ac32c0d7af913d4861f.debug...done.
done.
(gdb) run
Starting program: /usr/bin/slirp
Slirp v1.0.17 (BETA)

Copyright (c) 1995,1996 Danny Gasparovski and others.
All rights reserved.
This program is copyrighted, free software.
Please read the file COPYRIGHT that came with the Slirp
package for the terms and conditions of the copyright.

IP address of Slirp host: 127.0.0.1
IP address of your DNS(s): 1.1.1.1, 10.77.0.7
Your address is 10.0.2.15
(or anything else you want)

Type five zeroes (0) to exit.

[autodetect SLIP/CSLIP, MTU 1500, MRU 1500, 115200 baud]

SLiRP Ready ...

Program received signal SIGSEGV, Segmentation fault.
                                                    ip_slowtimo () at ip_input.c:457
457     ip_input.c: No such file or directory.

It fails at this line. Let's see the detailed stacktrace to see if anything helps us:

(gdb) bt full
#0  ip_slowtimo () at ip_input.c:457
        fp = 0x55784a40
#1  0x000055555556a57c in main_loop () at ./main.c:980
        so = <optimized out>
        so_next = <optimized out>
        timeout = {tv_sec = 0, tv_usec = 0}
        ret = 0
        nfds = 0
        ttyp = <optimized out>
        ttyp2 = <optimized out>
        best_time = <optimized out>
        tmp_time = <optimized out>
#2  0x000055555555b116 in main (argc=1, argv=0x7fffffffdc58) at ./main.c:95
No locals.

So it's failing in its main loop while it is trying to check if any timeouts occured. This is where I had to give up trying to debug this further. Let's see if building it from source works. I re-uploaded the tarball from Sourceforge because downloading tarballs from Sourceforge from the command line is a pain.

cd ~/dl
wget https://xena.greedo.xeserv.us/files/slirp-1.0.16.tar.gz
tar xf slirp-1.0.16.tar.gz
cd slirp-1.0.16/src
./configure --prefix=$HOME/prefix/slirp
make

This spews warnings about undefined inline functions. This then fails to link the resulting binary. It appears that at some point between the release of this software and the current day, gcc stopped creating symbols for inline functions in intermediate compiled files. Let's try to globally replace the inline keyword with an empty comment to see if that works:

vi slirp.h
:6
a
<enter>
#define inline /**/
<escape>
:wq
make

Nope. That doesn't work either. It continues to fail to find the symbols for those inline functions.

This is when I gave up. I started searching GitHub for Heroku buildpacks that already had this implemented or done. My theory was that a Heroku buildpack would probably include the binaries I needed, so I searched for a bit and found this buildpack. I downloaded it and extracted uml.tar.gz and found the following files:

total 6136
-rwxr-xr-x 1 cadey cadey   79744 Dec 10  2017 ifconfig*
-rwxr-xr-x 1 cadey cadey     373 Dec 13  2017 init*
-rwxr-xr-x 1 cadey cadey  149688 Dec 10  2017 insmod*
-rwxr-xr-x 1 cadey cadey   66600 Dec 10  2017 route*
-rwxr-xr-x 1 cadey cadey  181056 Jun 26  2015 slirp*
-rwxr-xr-x 1 cadey cadey 5786592 Dec 15  2017 uml*
-rwxr-xr-x 1 cadey cadey     211 Dec 13  2017 uml_run*

That's a slirp binary! Does it work?

./slirp
Slirp v1.0.17 (BETA) FULL_BOLT

Copyright (c) 1995,1996 Danny Gasparovski and others.
All rights reserved.
This program is copyrighted, free software.
Please read the file COPYRIGHT that came with the Slirp
package for the terms and conditions of the copyright.

IP address of Slirp host: 127.0.0.1
IP address of your DNS(s): 1.1.1.1, 10.77.0.7
Your address is 10.0.2.15
(or anything else you want)

Type five zeroes (0) to exit.

[autodetect SLIP/CSLIP, MTU 1500, MRU 1500]

SLiRP Ready ...

It's not immediately crashing, so I think it should be good! Let's copy this binary to ~/bin/slirp:

cp slirp ~/bin/slirp

Just in case the person who created this buildpack takes it down, I have mirrored it.

Configuring Networking

Now let's configure networking on our guest. Adjust your kernel command line:

linux \
  root=/dev/root \
  rootfstype=hostfs \
  rootflags=$HOME/prefix/uml-demo \
  rw \
  mem=64M \
  eth0=slirp,,$HOME/bin/slirp \
  init=/bin/sh

We should get that shell again. Let's enable networking:

mount -t proc proc proc/
mount -t sysfs sys sys/

ifconfig eth0 10.0.2.14 netmask 255.255.255.240 broadcast 10.0.2.15
route add default gw 10.0.2.2

The first two commands set up /proc and /sys, which are required for ifconfig to function. The ifconfig command sets up the network interface to communicate with slirp. The route command sets the kernel routing table to force all traffic over the slirp tunnel. Let's test with a DNS query:

nslookup google.com 8.8.8.8
Server:    8.8.8.8
Address 1: 8.8.8.8 dns.google

Name:      google.com
Address 1: 172.217.12.206 lga25s63-in-f14.1e100.net
Address 2: 2607:f8b0:4006:81b::200e lga25s63-in-x0e.1e100.net

That works!

Let's automate this with a shell script:

#!/bin/sh
# init.sh

mount -t proc proc proc/
mount -t sysfs sys sys/
ifconfig eth0 10.0.2.14 netmask 255.255.255.240 broadcast 10.0.2.15
route add default gw 10.0.2.2

echo "networking set up"

exec /tini /bin/sh

and mark it executable:

chmod +x init.sh

and then change the kernel command line:

linux \
  root=/dev/root \
  rootfstype=hostfs \
  rootflags=$HOME/prefix/uml-demo \
  rw \
  mem=64M \
  eth0=slirp,,$HOME/bin/slirp \
  init=/init.sh

Then re-run it:

SLiRP Ready ...
networking set up
/bin/sh: can't access tty; job control turned off

nslookup google.com 8.8.8.8
Server:    8.8.8.8
Address 1: 8.8.8.8 dns.google

Name:      google.com
Address 1: 172.217.12.206 lga25s63-in-f14.1e100.net
Address 2: 2607:f8b0:4004:800::200e iad30s09-in-x0e.1e100.net

And networking works reliably!

Dockerfile

So that you can more easily test this, I have created a Dockerfile that automates most of these steps and should result in a working setup. I have a pre-made kernel configuration that should do everything outlined in this post, but this post outlines a more minimal setup.

---

I hope this post is able to help you understand how to do this. This became a bit of a monster, but this should be a comprehensive guide on how to build, install and configure user mode Linux for modern operating systems. Next steps from here should include installing services and other programs into the guest system. Since Docker container images are just glorified tarballs, you should be able to extract an image with docker export and then set the root filesystem location in the guest kernel to that location. Then run the command that the Dockerfile expects via a shell script.

Special thanks to rkeene of #lobsters on Freenode. Without his help with attempting to debug slirp, I wouldn't have gotten this far. I have no idea how his Slackware system works fine with slirp but my Ubuntu and Alpine systems don't, and why the binary he gave me also didn't work; but I got something working and that's good enough for me.

The h Programming Language

h is a project of mine that I have released recently. It is a single-paradigm, multi-tenant friendly, turing-incomplete programming language that does nothing but print one of two things:

• the letter h
• a single quote (the Lojbanic "h")

It does this via WebAssembly. This may sound like a pointless complication, but actually this ends up making things a lot simpler. WebAssembly is a virtual machine (fake computer that only exists in code) intended for browsers, but I've been using it for server-side tasks.

I have written more about/with WebAssembly in the past in these posts:

• https://christine.website/talks/webassembly-on-the-server-system-calls-2019-05-31
• https://christine.website/blog/olin-1-why-09-1-2018
• https://christine.website/blog/olin-2-the-future-09-5-2018
• https://christine.website/blog/land-1-syscalls-file-io-2018-06-18
• https://christine.website/blog/templeos-2-god-the-rng-2019-05-30

This is a continuation of the following two posts:

• https://christine.website/blog/the-origin-of-h-2015-12-14
• https://christine.website/blog/formal-grammar-of-h-2019-05-19

All of the relevant code for h is here.

h is a somewhat standard three-phase compiler. Each of the phases is as follows:

Parsing the Grammar

As mentioned in a prior post, h has a formal grammar defined in Parsing Expression Grammar. I took this grammar (with some minor modifications) and fed it into a tool called peggy to generate a Go source version of the parser. This parser has some minimal wrappers around it, mostly to simplify the output and remove unneeded nodes from the tree. This simplifies the later compilation phases.

The input to h looks something like this:

h

The output syntax tree pretty-prints to something like this:

H("h")

This is also represented using a tree of nodes that looks something like this:

&peg.Node{
    Name: "H",
    Text: "h",
    Kids: nil,
}

A more complicated program will look something like this:

&peg.Node{
    Name: "H",
    Text: "h h h",
    Kids: {
        &peg.Node{
            Name: "",
            Text: "h",
            Kids: nil,
        },
        &peg.Node{
            Name: "",
            Text: "h",
            Kids: nil,
        },
        &peg.Node{
            Name: "",
            Text: "h",
            Kids: nil,
        },
    },
}

Now that we have this syntax tree, it's easy to go to the next phase of compilation: generating the WebAssembly Text Format.

WebAssembly Text Format

WebAssembly Text Format is a human-editable and understandable version of WebAssembly. It is pretty low level, but it is actually fairly simple. Let's take an example of the h compiler output and break it down:

(module
 (import "h" "h" (func $h (param i32)))
 (func $h_main
       (local i32 i32 i32)
       (local.set 0 (i32.const 10))
       (local.set 1 (i32.const 104))
       (local.set 2 (i32.const 39))
       (call $h (get_local 1))
       (call $h (get_local 0))
 )
 (export "h" (func $h_main))
)

Fundamentally, WebAssembly binary files are also called modules. Each .wasm file can have only one module defined in it. Modules can have sections that contain the following information:

• External function imports
• Function definitions
• Memory information
• Named function exports
• Global variable definitions
• Other custom data that may be vendor-specific

h only uses external function imports, function definitions and named function exports.

import imports a function from the surrounding runtime with two fields: module and function name. Because this is an obfuscated language, the function h from module h is imported as $h. This function works somewhat like the C library function putchar().

func creates a function. In this case we are creating a function named $h_main. This will be the entrypoint for the h program.

Inside the function $h_main, there are three local variables created: 0, 1 and 2. They correlate to the following values:

Local Number	Explanation	Integer Value
0	Newline character	10
1	Lowercase h	104
2	Single quote	39

As such, this program prints a single lowercase h and then a newline.

export lets consumers of this WebAssembly module get a name for a function, linear memory or global value. As we only need one function in this module, we export $h_main as "h".

Compiling this to a Binary

The next phase of compiling is to turn this WebAssembly Text Format into a binary. For simplicity, the tool wat2wasm from the WebAssembly Binary Toolkit is used. This tool creates a WebAssembly binary out of WebAssembly Text Format.

Usage is simple (assuming you have the WebAssembly Text Format file above saved as h.wat):

wat2wasm h.wat -o h.wasm

And you will create h.wasm with the following sha256 sum:

sha256sum h.wasm
8457720ae0dd2deee38761a9d7b305eabe30cba731b1148a5bbc5399bf82401a  h.wasm

Now that the final binary is created, we can move to the runtime phase.

Runtime

The h runtime is incredibly simple. It provides the h.h putchar-like function and executes the h function from the binary you feed it. It also times execution as well as keeps track of the number of instructions the program runs. This is called "gas" for historical reasons involving blockchains.

I use Perlin Network's life as the implementation of WebAssembly in h. I have experience with it from Olin.

The Playground

As part of this project, I wanted to create an interactive playground. This allows users to run arbitrary h programs on my server. As the only system call is putchar, this is safe. The playground also has some limitations on how big of a program it can run. The playground server works like this:

• The user program is sent over HTTP with Content-Type text/plain
• The program is limited to 75 bytes on the server (though this is configurable via flags or envvars)
• The program is compiled
• The program is run
• The output is returned via JSON
• This output is then put into the playground page with JavaScript

The output of this call looks something like this:

curl -H "Content-Type: text/plain" --data "h" https://h.christine.website/api/playground | jq
{
  "prog": {
    "src": "h",
    "wat": "(module\n (import \"h\" \"h\" (func $h (param i32)))\n (func $h_main\n       (local i32 i32 i32)\n       (local.set 0 (i32.const 10))\n       (local.set 1 (i32.const 104))\n       (local.set 2 (i32.const 39))\n       (call $h (get_local 1))\n       (call $h (get_local 0))\n )\n (export \"h\" (func $h_main))\n)",
    "bin": "AGFzbQEAAAABCAJgAX8AYAAAAgcBAWgBaAAAAwIBAQcFAQFoAAEKGwEZAQN/QQohAEHoACEBQSchAiABEAAgABAACw==",
    "ast": "H(\"h\")"
  },
  "res": {
    "out": "h\n",
    "gas": 11,
    "exec_duration": 12345
  }
}

The execution duration is in nanoseconds, as it is just directly a Go standard library time duration.

Bugs h has Found

This will be updated in the future, but h has already found a bug in Innative. There was a bug in how Innative handled C name mangling of binaries. Output of the h compiler is now a test case in Innative. I consider this a success for the project. It is such a little thing, but it means a lot to me for some reason. My shitpost created a test case in a project I tried to integrate it with.

That's just awesome to me in ways I have trouble explaining.

As such, h programs do work with Innative. Here's how to do it:

First, install the h compiler and runtime with the following command:

go get within.website/x/cmd/h

This will install the h binary to your $GOPATH/bin, so ensure that is part of your path (if it is not already):

export GOPATH=$HOME/go
export PATH=$PATH:$GOPATH/bin

Then create a h binary like this:

h -p "h h" -o hh.wasm

Now we need to provide Innative the h.h system call implementation, so open h.c and enter in the following:

#include <stdio.h>

void h_WASM_h(char data) {
  putchar(data);
}

Then build it to an object file:

gcc -c -o h.o h.c

Then pack it into a static library .ar file:

ar rsv libh.a h.o

Then create the shared object with Innative:

innative-cmd -l ./libh.a hh.wasm

This should create hh.so in the current working directory.

Now create the following Nim wrapper at h.nim:

proc hh_WASM_h() {. importc, dynlib: "./hh.so" .}

hh_WASM_h()

and build it:

nim c h.nim

then run it:

./h
h

And congrats, you have now compiled h to a native shared object.

Why

Now, something you might be asking yourself as you read through this post is something like: "Why the heck are you doing this?" That's honestly a good question. One of the things I want to do with computers is to create art for the sake of art. h is one of these such projects. h is not a productive tool. You cannot create anything useful with h. This is an exercise in creating a compiler and runtime from scratch, based on my past experiences with parsing lojban, WebAssembly on the server and frustrating marketing around programming tools. I wanted to create something that deliberately pokes at all of the common ways that programming languages and tooling are advertised. I wanted to make it a fully secure tool as well, with an arbitrary limitation of having no memory usage. Everything is fully functional. There are a few grammar bugs that I'm calling features.