waifud Plans

So I have this homelab now, and I want to run some virtual machines on it. But I don't want to have to SSH into each machine to do this and I have a lot of time to kill this summer. So I'm going to make a very obvious move and massively overcomplicate this setup.

<Cadey> Canada's health system is usually pretty great, however for some reason I have to wait four months between COIVD vaccine shots. What the heck. That basically eats up my entire summer. Grrrr

waifud is a suite of tools that help you manage your server's waifus. This is an example of name-driven development, or where I had a terrible idea about the name that was so terrible I had to bring it to its natural conclusion. Thanks to comments on Reddit and Hacker News about my systemd talk video, I was told that I was mispronouncing "systemctl" as "system-cuttle" (it came out as "system-cuddle" for some reason). If virtual machines are waifus to a server, then a management daemon would be called waifud, and the command line tool would be called waifuctl (which is canonically pronounced "waifu-cuddle" and I will accept no other pronunciations as valid).

Essentially my vision for waifud is to be a "middle ground" between running virtual machines on one server and something more complicated like OpenStack. I want to be able to have high level descriptions of virtual machines (including cloud-config userdata) and then hand them over to waifud to just figure out the logistics of where they should run for me.

Due to how absurdly useful something like this is, I also wanted to be sure that it is difficult for companies to use this in production without paying me for some kind of license. Not to say that this would be intentionally made useless, more that if I have to support people using this in production I would rather be paid to do so. I feel it would be better for the project this way. I still have not decided on what price the support licenses would be, however I would only ask that people using this in a professional capacity (IE: for their dayjob or as an integral of a dayjob's production services) acquire a license by contacting me once the project hits something closer to stable, or at least when I get to the point that I am using it for all of my virtual machine fun.

At a high level, waifud will be made out of a few components:

• the waifud control server, written in Rust
• the waifuctl tool, written in Rust
• the waifud-agentd runner node agent, written in Rust
• the waifud-metadatad metadata server, written in Go using userspace WireGuard to listen on 169.254.169.254:80 to serve metadata to machines that ask for it
• SQLite to store control server data
• Redis to store cloud-config metadata

Right now I have the source code for waifud available here. It is released under the terms of the permissive Be Gay, Do Crimes license, which should sufficiently scare people away for now while I implement the service. The biggest thing in the repo right now is mkvm, which is essentially the prototype of this project. It downloads a cloud template, injects it into a ZFS zvol and then configures libvirt to use that ZFS zvol as the root filesystem of the virtual machine.

This tool works great and I use it very often both personally and in work settings, however one of the biggest problems that it has is that it assumes that the urls for the upstream cloud templates will change when the contents of the file behind the URL changes. This has turned out to be a very very very wrong assumption and has caused me a lot of churn in testing. I've been looking at using something like IPFS to store these images in, but I'm still pondering options.

I would also like to have some kind of web management interface for waifud. Historically frontend web development has been one of my biggest weaknesses. I would like to use Alpine.js to make an admin panel.

At a high level, I want waifuctl to have the following features:

• list all virtual machines across the cluster
• create a new virtual machine somewhere
• create a new virtual machine on a specific node
• delete a virtual machine
• fetch a virtual machine's IP address
• edit the cloud config for a virtual machine
• resize a virtual machine's memory and CPU count
• list all templates
• delete a template
• add a new template

The runner machines will communicate with waifud over HTTP with a redis cache for cloud-config metadata. Each runner node will have its virtual machine subnet shared both with other runner nodes and other machines on the network using Tailscale subnet routes. The metadata server will hook into each machine's network stack using an on-machine WireGuard config and a userspace instance of WireGuard.

I hope to have something more substantial created by the end of August at latest. I'm working on the core of waifud at the moment and will likely do a stream or two of me hacking at it when I can.

Footnote

• Final Chapter
• Epilogue

Before the darkness was the darkness, the darkness was a child. This child found themselves lost and without purpose. Life was scary. Things were changing constantly, and they found themselves at a loss. One day they were walking about the etherial network and stumbled across a meeting house.

The child looked inside and was confused. There were hundereds of rooms with even more people inside. There were rooms on every topic. There was a shower of culture and an outpouring of knowledge. Hanging out here would permanently change the course of the child's life.

Horrified by the room takeover golem, the remaining regulars had fled their former homes. This was not a home for legal reasons, but it was their social home on the etherial network. Sadness had turned to rage had turned to depression had turned to laughter. One of the former regulars was a apprentice scryer, so as a lark they decided to set up some meeting rooms to scry their way into rooms in the old meeting house. It was a one-way scry and all they could do was watch.

Historically, IRC spam has been a unique form of art. Yes, I'm serious. There have been legitimate works of art created in the desire to disrupt conversations on IRC. It sounds absurd, but it's true. One unique quality of these artworks is that in order to see them, they must be shared with others. At some level you can't view this art alone, and that makes it beautiful.

Fighting IRC spam has turned into a full time job. There are hundreds of different bot kinds and so many different ways to spam that fighting it is difficult due to the server software being very simple. Historically IRC developers have not wanted to add hooks so that people could run a bit of code on each message as it was being processed. There were legitimate fears that doing this would allow a malicious server admin to log every channel, not just the ones they have joined. IRC was created at a time where all of the admins knew eachother; but they were part of different organizations, each with their own rules and subtly different codes of conduct.

One of the best ways to fight IRC spam has been to wait until the spammer gets bored and goes off to do something else. Users are not as understanding to this method.

Someone had set up a golem-creating golem and aimed it at the meta-discussion room of the former meeting house. It did its job dutifully and continued marching on:

(pissnet) come to pissnet for cold wet chats!

The people watching the scry had never seen this brand of disruption before. By now the people watching had amassed to over a hundred and they were all bored and eager for something new. Something new was here!

(pissnet) come to pissnet for cold hard piss!

Over time, the shadowy group behind these golems became known as the urinators. These urinators became a bit of a hero to the people who watched in horror as the situation developed. The golems got discovered and ejected, and even earned the ire of the anti-golem golem. The disguise was clever, the ejections where swift, but the watchers laughed as the golems kept getting more and more creative.

The darkness was dismayed. Everything was falling apart around them. The maintainers of the maintenance golems had fled. The spellcrafters that empowered them had sworn to give no more assistance. The halls themselves were starting to show the rot that had built up over the last 20 years of them existing.

The darkness pondered amongst themselves until they pulled back a memory from the past. A memory from the child. The halls themselves had to be replaced!

The watchers looked on in horror. The scryer had given up hope and decided to move on with their life. The urinators had suceeded in shutting down the things that were fun to the watchers. The urinators won.

Some urinators created their own halls. It was an experiment in anarchy for running these types of halls. It is astounding that it managed to stay as stable as it did.

I have been completely unsure how I should broach the topic of pissnet in these articles. For people unfamiliar with IRC culture, you must think I'm making shit up or something. It is so out there that it's almost like an abstact art gallery or something. But no, pissnet happened. It started as IRC spam and then turned into this: letspiss.net. I don't really think I can suggest readers of this blog go there. It is some kind of weird anarchist IRC hackerspace, but most of the users are ircops and can see your IP address.

Like, for people that are really deep into IRC culture, the whole pissnet shitshow was so out there that they thought the people that were telling them about that were making that shit up.

But it's real.

We are moving past legacy freenode to a new fork. The new freenode is launched. You will slowly be disconnected and when you reconnect, you will be on the new freenode. We patiently await to welcome you in freedom's holdout - the freenode. If you're looking to connect now, you can already /server chat.freenode.net 6697 (ssl) or 6667 (plaintext). It's a new genesis for a new era. Thank you for using freenode, and Hello World, from the future. freenode is IRC. freenode is FOSS. When you connect, register your nickname and your channel and get started. It's a new world. We're so happy to welcome you and the millions of others.

The darkness smiled and replaced the halls where they were. The darkness hoped that millions would follow.

They didn't come.

• Freenode commits suicide, is no longer a serious IRC network
• the end of freenode
• All Freenode Channels and Users Gone
• Last remaining >1000 user community channel seized by freenode staff

Freenode is dead. The spirit lives on in Libera.chat.

Using Paper for Everyday Tasks

I have a bit of a reputation of being a very techno-savvy person. People have had the assumption that I have some kind of superpowerful handcrafted task management system that rivals all other systems and fully integrates with everything on my desktop. I don't. I use paper to keep track of my day to day tasks. Offline, handwritten paper. I have a big stack of little notebooks and I go through them one each month. Today I'm going to discuss the core ideas of my task management toolchain and walk you through how I use paper to help me get things done.

I have tried a lot of things before I got to this point. I've used nothing, Emacs' Org mode, Jira, GitHub issues and a few reminder apps. They all haven't quite cut it for me.

The natural place to start from is doing nothing to keep track of my tasks and goals. This can work in the short term. Usually the things that are important will come back to you and you will eventually get them done. However it can be hard for it to be a reliable system.

<Cadey> Focus is hard. Memory is fleeting. Data gets erased. Object permanence is a myth. Paper sits by the side and laughs.

It does work for some people though. I just don't seem to be one of them. Doing nothing to keep track of my tasks only really works when there are external structures around to help me keep track of things. Standup meetings or some kind of daily check-in are vital to this, and they sort of work because my team is helping keep everyone accountable for getting work done. This is very dependent on the team culture being healthy and on me being somewhere that I feel psychologically safe enough to admit when I make a mistake (which I have only really felt working at Tailscale). It also doesn't follow me from job to job, so changing employers would also mean I can't take my organization system with me. So that option is out.

Emacs is a very extensible text editor. It has a turing-complete scripting language called Emacs Lisp at its core and you can build out just about anything you want with it. As such, many packages have been developed. One of the bigger and more common packages is Org Mode. It is an Emacs major mode that helps you keep track of notes, todo lists, timekeeping, literate programming, computational notebooks and more. I have used Org Mode for many years in the past and I have no doubt that without it I would probably have been fired at least twice.

One of the main philosophies is that Org Mode is text at its core. The whole user experience is built around text and uses Emacs commands to help you manipulate text. Here's an example Org Mode file like I used to use for task management:

#+TITLE: June 2021

* June 10, 2021

** SRE
*** TODO put out the fire in prod before customers notice
Oh god, it's a doozy. The database server takes too long to run queries only
sometimes on Thursdays. Why thursday? No idea. It just happens. Very
frustrating. I wonder if God is cursing me.

** Devel
*** DONE Implement the core of flopnax for abstract rilkefs
    CLOSED: [2021-06-10 Thu 16:20]
*** TODO write documentation for flopnax before it is shipped

** Overhead
*** DONE ENG meeting
    CLOSED: [2021-06-10 Thu 15:00]
*** TODO Assist Jessie with the finer points of Rust
**** References vs Values
**** Lifetimes
Programming in Rust is the adventure of a lifetime!

** Personal
*** DONE Morning meds
    CLOSED: [2021-06-10 Thu 09:04]
*** TODO Evening meds
*** TODO grocery run

Org Mode used to be a core part of my workflow and life. It was everpresent and used to keep track of everything. I would even track usage of certain recreational substances in Org Mode with a snippet of Emacs Lisp to do some basic analytics on usage frequency. Org Mode can live with me and I don't have to give it up when I change jobs.

I got out of the habit a while ago and it's been really hard to go back into the habit. I still suggest Org Mode to people, but it's no longer the thing that I use day to day. It also is hard to use from my tablet (iPad) and my phone (iPhone). It also tends to vanish when you close the window, and when you have object permanence issues that tends to make things hard.

<Cadey> I could probably set up something with one of those fancy org-mode frontends served over HTTP, but that would probably end up being more effort than it's worth for me

Another tool I've used for this is my employer's task management tool of choice. At past jobs this has ranged from GitHub to Jira. This is a solid choice. It keeps everything organized and referenced with other people. I don't have to do manual or automated synchronization of information into that ticket tracking system to be sure other people are updated. However, you inherit a lot of the inertia of how the ticket tracking system of choice is used. At a past job there were unironically 17 different states that a ticket could be in. Most of them were never used and didn't matter, yet they could not be removed lest it break the entire process that the product team used to keep track of things.

Doing it like this works great if your opinions about how issues should be tracked agree with your employer's process (if this is the case, you probably set up the issue tracking system). As I mentioned before, this also means that you have to leave that system behind when you change jobs. If you are someone that never really changes jobs, this can work amazingly. I am not one of those people.

Something else I've tried is to set up my own private GitHub/Gitea project to keep track of things. We used one for organizing our move to Ottawa even. This is a very low-friction system. It is easy to set up and the issues will bother you in your news feed, so they are everpresent. It's also easy to close the window and forget about the repo.

There is also that little hit of endorphins from closing an issue. That little rush can help fuel a habit for using the tool to track things, but the rush goes away after a while.

<Mara> Wait, if you have issues remembering to look at your org mode file or tracker board or whatever, why can't you just set up a reminder to update it? Surely that can't be that hard to do?

<Cadey> Don't you think that if it was that easy, I would already be doing that? Do you think I like having this be so hard? Notifications that are repetitive fade into the background when I see them too often. I subconsciously filter them out. They do not exist to me. Even if it is one keypress away to open the board or append to my task list, I will still forget to do it, even if it's important.

So, I've arrived on paper to keep track on these things. Paper is cheap. Paper is universal. Paper doesn't run out of battery. Paper doesn't vanish into the shadow realm when I close the window. Paper can do anything I can do with a pencil. Paper lets me turn back pages in the notebook and scan over for things that have yet to be done. Honestly I wish I had started using paper for this sooner. Here's how I use paper:

• Get a cheap notebook or set of notebooks. They should ideally be small, pocketable notebooks. Something like 30 sheets of paper per notebook. I can't find the cheap notebooks that I bought on Amazon, but I found something similar here. Don't be afraid to buy more than you need. This stuff is really cheap. Having more paper around can't hurt. Field Notes works in a pinch, but their notebooks can be a bit expensive. The point is you have many options.
• Label it with the current month (it's best to start this at the beginning of a month if you can). Put contact information on the inside cover in case you lose it.
• Start a new page every day. Put the date at the top of the page.
• Metadata about the day goes in the margins. I use this to keep a log of who is front as well as taking medicine.
• Write prose freely.
• TODO items start with a -. Those represent things you need to do but haven't done yet.
• When the item is finished, put a vertical line through the - to make it a +.
• If the item either can't or won't be done, cross out the - to make it into a *.
• If you have to put off a task to a later date, turn the - into a ->. If there is room, put a brief description of why it needs to be moved or when it is moved to. If there's no room feel free to write it out in prose form at the end of your page.
• Notes start with a middot (·). They differ from prose as they are not complete sentences. If you need to, you can always turn them into TODO items later.
• Write in pencil so you can erase mistakes. Erase carefully to avoid ripping the paper, You hardly need to use any force to erase things.
• There is only one action, appending. Don't try and organize things by topic as you would on a computer. This is not a computer, this is paper. Paper works best when you append only. There is only one direction, forward.
• If you need to relate a bunch of notes or todo items with a topic, skip a line and write out the topic ending with a colon. When ending the topical notes, skip another line.
• Don't be afraid to write in it. If you end up using a whole notebook before the month is up, that is a success. Record insights, thoughts, feelings and things that come to your mind. You never know what will end up being useful later.
• At the end of the month, look back at the things you did and summarize/index them in the remaining pages. Discover any leftover items that you haven't completed yet so you can either transfer them over to next month or discard them. It's okay to not get everything done. You may also want to scan it to back it up into the cloud. You may never reference these scans, but backups never hurt.

And then just write things in as they happen. Don't agonize over getting them all. You will not. The aim is to get the important parts. If you really honestly do miss something that is important, it will come back.

Something else I do is I keep a secondary notebook I call Knowledge. It started out as the notebook that I used to document errata for my homelab, but overall it's turned into a sort of secondary place to record other information as well as indexing other details across notebooks. This started a bit on accident. One of the notebooks from my big order came slightly broken. A few pages fell out and then I had a smaller notebook in my hands. I stray from the strict style in this notebook. It's a lot more free flowing based on my needs, and that's okay. I still try to separate things onto separate pages when I can to help keep things tidy.

I've also been using it to outline blogposts in the form of bullet trees. Normally I start these articles as a giant unordered list with sub-levels for various details on its parent thing. Each top-level thing becomes a "section" and things boil down into either paragraphs or sentences based on what makes sense.

An unexpected convenience of this flow is that the notebooks I'm using are small enough to fit under the halves of my keyboard:

This lets me leave the notebooks in an easy to grab place while also putting them slightly out of the way until they are needed. I also keep my pencil and eraser closeby. When I go out of the house, I pack this month's journal, a pencil and an eraser.

Paper has been a great move for me. There's no constant reminders. There's no product team trying to psychologically manipulate me into using paper more (though honestly that might have helped to build the habit of using it daily). It is a very calm technology and I am all for it.

<Mara> Is this technology though? This is just a semi-structured way of writing things on paper. Does that count as technology?

<Cadey> To be honest, I don't know. The line of what is and what is not technology is very thin in the best case. I think that this counts as a technology, but overall this is a huge It Depends™. You may not think this is "real" technology because there's no real electronic component to it. That is a valid opinion, however I would like to posit that this is technology in the same way that a manual shaving razor is technology. It was designed and built to purpose. If that isn't technology, what is? Plus, this way there's no risk of server downtime preventing me from using paper!

Oh, also, if you feel bored and a design comes to mind, don't be afraid to doodle on the cover. Make paper yours. Don't worry about it being perfect. It's there to help you tell the notebooks apart in the future after they are complete.

So far over the last month I've made notes on 49 pages. Most of the TODO items are complete. Less than 10% of them failed/were cancelled. Less than 10% of them had to roll over to the next day. I assemble my TODO lists based on what I didn't get done the previous day. I write each thing out by hand to help me remember to prioritize them. When I need something to do, I look down at my notebook for incomplete items. I use a rubber band to keep the notebook closed. I've been considering slipstreaming the stuff currently in the Knowledge notebook into the main monthly one. It's okay to go through paper. That's a success.

This system works for me. I don't know if it will work for you, but if you have been struggling with remembering to do things I would really suggest trying it. You probably have a few paper notebooks left over from startups handing them out in a swag pack. You probably also have never touched them since you got them. This is good. I only really use the small notebooks because I found the more fancy bound notebooks were harder to write on the left sides more than the right sides. Your mileage may vary.

<Cadey> I would include a scan of one of my notebook pages here, but that would reveal some personal information that I don't really want to put on this blog as well as potentially break NDA terms for work, so I don't want to risk that if you can understand.

My Homelab Build

There are many things you can be cursed into enjoying. One of my curses is enjoying philosophy/linguistics. This leads you into many fun conversations about how horrible English is that can get boring after a while. One of my other, bigger curses is that I'm a computer person. Specifically a computer person that enjoys playing with distributed systems. This is an expensive hobby, especially when all you really have is The Cloud™.

One thing that I do a lot is run virtual machines. Some of these stick around, a lot of them are very ephemeral. I also like being able to get into these VMs quickly if I want to mess around with a given distribution or OS. Normally I'd run these on my gaming tower, however this makes my tower very load-bearing. I also want to play games sometimes on my tower, and even though there have been many strides in getting games to run well on Linux it's still not as good as I'd like it to be.

<Cadey> In fact, it's actually kinda convenient that it's hard for me to play games on Linux so that it's harder for me to have entire days eaten by doing it. Factorio and other games like it are really dangerous for me.

For many years my home server has been a 2013 Mac Pro, the trash can one. It's a very capable machine. It's a beautiful looking computer, however in terms of performance it's really not up to snuff anymore. It works, it's still my prometheus server, but overall it's quite slow in comparison to what I've ended up needing.

It probably also doesn't help that my coworkers have given me a serious case of homelab envy. A few of my coworkers have full rackmount setups. This is also dangerous for my wallet.

My initial plan was to get 3 rackmount servers in a soundproof rack box. I wanted to get octo-core Xeons in them (preferably 2 of them) and something on the order of 64 GB of ram in each node. For my needs, this is absurdly beyond overkill. Storage would be on NVMe and rotational drives with ZFS as the filesystem.

<Mara> I thought overkill was the motto of this blog.

<Cadey> Nope. It's "there's no kill like overkill". Subtle difference, but it's a significant one in this case.

Among other things, running a datacenter in your basement really requires you to have a basement. This place that my fiancé and I moved to doesn't really have a proper basement. One of the advantages of having a proper basement is that you can put servers in it without really bothering anyone. Server fan noise tends to range from "dull roar" to "jet engine takeoff". This can cause problems if you are and/or live with someone who is noise sensitive. Soundproof racks exist, however I wasn't sure if the noise reduction would really be enough to make up for the cost.

Then there's the power cost. Electricity in Ontario is expensive. Our home office also only has a 15 amp breaker, which gives us roughly 1800W to play with within that room. With our work laptops and gaming towers set up, the laser printer was enough to push us over the line and flip the breaker. A full rackmount server setup would never have worked. Electricity is covered by our rent payments, however I don't really want to use more power than I really have to.

After more research and bisecting a bunch of options through PCPartpicker, I ended up with a set of hardware that I am calling the Alrest. Here are its specs on PCPartpicker. It is designed to balance these factors as much as possible:

• Cost - I had a budget of about 4,000 CAD that I was willing to spend on the whole project
• Parts availability in Canada - Parts are annoying to get in Canada in normal cases, COVID has made it worse
• Performance - My existing balance of cloud servers and old laptops has gotten me fairly far, but it is starting to show its limits
• Cores - More cores = more faster

The Alrest is a micro-ATX tower with the following major specifications:

• An Intel Core i5 10600
• 32 GB DDR4 ram (I have no idea how this happened, but the cheapest way for me to get the ram I wanted was to get RGB ram again)
• 1 TB NVMe drive (I had to get them from multiple vendors because of Chia miners causing companies to limit drives to 1/2 per person)

<Mara> Why do you have a i5 10600? You could get a beefier processor.

<Cadey> All the beefier CPUs don't ship with an integrated GPU, so I'd have to get a hardware GPU (which is near impossible due to memecoin farmers and the car industry hoovering up all the semiconductor supply) that would waste power showing a login screen for all eternity. Not to mention those beefier CPUs also don't ship with a CPU fan so I'd need to get a heatsink. I wish Intel made better processors with both an iGPU and a heatsink. I'm probably a huge exception to the normal case of system buyers though.

Thanks to the meddling of a server sommelier that I banter with, I got 4 nodes.

<Mara> The nodes in the cluster are named after gods/supercomputers from Xenosaga and Xenoblade Chronicles. KOS-MOS (a badass robot waifu with a laser sword and also the reincarnation of a biblical figure, Xenosaga is wild) was one of the protagonists in Xenosaga and Logos (speech, reason), Ontos (one who is, being) and Pneuma (breath, spirit) were the three cores of the Trinity Processor in Xenoblade Chronicles 2. The avatar you see in YouTube videos and VRChat resembles the in-game model for Pneuma. Alrest is another Xenoblade reference, but that is an exercise for the reader.

Building them was fairly straightforward. The process of building a PC has gotten really streamlined over the years and it really helped that I basically had 4 carbon copies of the same machine. I hadn't built an Intel tower since about mid 2015 when I built my old gaming tower while I lived in California. Something that terrified me back in the day was that tension arm that was used to lock the processor into the motherboard. I was afraid that I was going to break it. That tension arm is still present in modern motherboards. It's still terrifying.

The motherboards I got were kinda cheapo (a natural side effect of sorting by cost from cheapest to most expensive, I guess), but they did this one cost-saving measure I didn't even know was possible. Normally motherboards include a NVMe screw mount so you screw the SSD into the board. This motherboard came with a plastic NVMe anchor. I popped one end into the board with a spudger and fastened the drive into the other.

The anchors work fine, but it's still the first time I've ever seen a motherboard do that.

If you look at the parts list, you'll notice that I didn't get a dedicated CPU cooler. Those are annoying to install compared to the stock cooler, and I don't really see myself running into a case where it'd actually be useful. I picked the one high-end Core i5 model that came with both an integrated GPU and a stock cooler. One weird thing that Intel did was make the power cable for the stock cooler wrapped in a chokehold around the CPU cooler itself. I didn't realize this at first and was confused why my experimental/test machine for the cluster was throwing "oh god why isn't the CPU fan working" beep codes and refused to boot past the BIOS. Always make sure the CPU fan power cable isn't strangling the CPU fan.

After all that comes the NixOS install. I had previously made an ISO image that allowed me to automatically install NixOS on virtual machines. This fairly dangerous ISO image allows me to provision a new virtual machine from a blank disk to a fully functional NixOS install in something like 3 minutes.

<Mara> In testing, most of the time was taken up by copying the ISO's nix store to the new virtual machine partition. I don't know if there's a way to make that more efficient.

Using KOS-MOS as the experimental machine again, I installed NixOS by hand and took notes. Here's a scan of the notes I took:

• Page 1
• Page 2

I set up KOS-MOS to have three partitions: root, swap and the EFI system partition. I then set up my ZFS datasets with the following pattern:

With all of these paths ironed out, I turned those notes into a small install script. I put that install script here. I used nixos-generators to make an ISO with this command:

$ nixos-generate -f install-iso -c iso.nix

This spat out a 680 megabyte ISO (maybe even small enough it could fit on a CD) that I wrote to a flashdrive with dd:

$ sudo dd if=/path/to/nixos.iso of=/dev/sdc bs=4M

Then I stuck the USB drive into KOS-MOS and reinstalled it from that USB. After a fumble or two with a partitioning command, I had a USB drive that let me reflash a new base NixOS install with a ZFS root in 3 minutes. If you want to watch the install, I recorded a video:

I bet that if I used a USB 3.0 drive it could be faster, but 3 minutes is fast enough. It is a magical experience though. Just plug the USB drive in, boot up the tower and wait until it powers off. Once I got it working reliably on KOS-MOS the real test began. I built the next machine (Pneuma) and then installed NixOS with the magic USB drive. It worked perfectly. I had myself a cluster.

Once NixOS was installed on the machines, it was running a very basic configuration. This configuration sets the hostname to install, loads my SSH keys from GitHub and sets the ZFS host ID, but not much else. The next step was adding KOS-MOS to my Morph setup. I did the initial setup in this commit.

<Mara> Wait. You built 4 machines from the same template with (basically) the same hardware, right? Why would you need to put the host-specific config in the repo 4 times?

I don't! I created a folder for the Alrest hardware here. This contains all of the basic hardware config as well as a few settings that I want to apply cluster-wide. This allows me to have my Morph manifest look something like this:

{
  network = { description = "Avalon"; };

  # alrest
  "kos-mos.alrest" = { config, pkgs, lib, ... }:
    let metadata = pkgs.callPackage ../metadata/peers.nix { };
    in {
      deployment.targetUser = "root";
      deployment.targetHost = metadata.raw.kos-mos.ip_addr;
      networking.hostName = "kos-mos";
      networking.hostId = "472479d4";

      imports =
        [ ../../common/hardware/alrest ../../hosts/kos-mos/configuration.nix ];
    };

  "logos.alrest" = { config, pkgs, lib, ... }:
    let metadata = pkgs.callPackage ../metadata/peers.nix { };
    in {
      deployment.targetUser = "root";
      deployment.targetHost = metadata.raw.logos.ip_addr;
      networking.hostName = "logos";
      networking.hostId = "aeace675";

      imports =
        [ ../../common/hardware/alrest ../../hosts/logos/configuration.nix ];
    };

  "ontos.alrest" = { config, pkgs, lib, ... }:
    let metadata = pkgs.callPackage ../metadata/peers.nix { };
    in {
      deployment.targetUser = "root";
      deployment.targetHost = metadata.raw.ontos.ip_addr;
      networking.hostName = "ontos";
      networking.hostId = "07602ecc";

      imports =
        [ ../../common/hardware/alrest ../../hosts/ontos/configuration.nix ];
    };

  "pneuma.alrest" = { config, pkgs, lib, ... }:
    let metadata = pkgs.callPackage ../metadata/peers.nix { };
    in {
      deployment.targetUser = "root";
      deployment.targetHost = metadata.raw.pneuma.ip_addr;
      networking.hostName = "pneuma";
      networking.hostId = "34fbd94b";

      imports =
        [ ../../common/hardware/alrest ../../hosts/pneuma/configuration.nix ];
    };
}

Now I had a bunch of hardware with NixOS installed and the machines were fully assimilated into my network. I had my base shell config and everything else fully set up so I could SSH into any of the servers and have everything just where I wanted it. I had libvirtd installed with the basic install set, so I wanted to try using Tailscale Subnet Routes to expose the virtual machine subnets to my other machines. As far as I am aware, libvirtd doesn't have a mode where it can plunk a virtual machine on the network like other hypervisors can.

By default libvirtd sets the default virtual machine network to be on the 192.168.122.0/24 network. This doesn't conflict with anything on its own, however when you have many hosts with that same range it can be a bit problematic. I have a /16 that I use for my wireguard addressing, so I carved out a few ranges that I could reserve for each machine:

Normally I'd share these subnets over WireGuard. However, Tailscale Subnet Routes let me do this a bit more directly. I ran this command to enable subnet routing on each machine:

function getsubnet () {
  case $1 in
  kos-mos)
    printf "10.77.128.0/24"
    ;;
  logos)
    printf "10.77.129.0/24"
    ;;
  ontos)
    printf "10.77.130.0/24"
    ;;
  pneuma)
    printf "10.77.131.0/24"
    ;;
  esac
}

for host in kos-mos logos ontos pneuma
do
  ssh root@$host tailscale up \
    --accept-routes \
    --advertise-routes="$(getsubnet $host)" \
    --advertise-tags=tag:alrest,tag:nixos
done

This command is a slightly overengineered version of what I actually did (something something hindsight something something), but it worked! Then I configured libvirtd to actually use these subnets by going into virt-manager, connecting to one of the hosts and changed the default network configuration from something like this:

<network>
  <name>default</name>
  <uuid>ef4bc889-e01d-403a-9a92-a0e172b8f42a</uuid>
  <forward mode="nat">
    <nat>
      <port start="1024" end="65535"/>
    </nat>
  </forward>
  <bridge name="virbr0" stp="on" delay="0"/>
  <mac address="52:54:00:89:b3:66"/>
  <ip address="192.168.122.1" netmask="255.255.255.0">
    <dhcp>
      <range start="192.168.122.2" end="192.168.122.254"/>
    </dhcp>
  </ip>
</network>

To something like this:

<network connections="2">
  <name>default</name>
  <uuid>39bf0a49-57ff-4840-8bd6-09c6f3817afe</uuid>
  <forward mode="nat">
    <nat>
      <port start="1024" end="65535"/>
    </nat>
  </forward>
  <bridge name="virbr0" stp="on" delay="0"/>
  <mac address="52:54:00:a6:03:14"/>
  <domain name="default"/>
  <ip address="10.77.128.1" netmask="255.255.255.0">
    <dhcp>
      <range start="10.77.128.2" end="10.77.128.254"/>
    </dhcp>
  </ip>
</network>

And then I spun up a virtual machine running Alpine Linux and got it on the network. Its IP address was 10.77.128.90. Then I tried pinging it from the same machine, another machine in the same room, another server on the same continent and then finally another server on the same planet. Here are the results:

Same Machine:

cadey:users@kos-mos ~ ./rw
$ ping 10.77.128.90 -c1
PING 10.77.128.90 (10.77.128.90) 56(84) bytes of data.
64 bytes from 10.77.128.90: icmp_seq=1 ttl=64 time=0.208 ms

--- 10.77.128.90 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.208/0.208/0.208/0.000 ms

Same Room:

cadey:users@shachi ~ ./rw
$ ping 10.77.128.90 -c1
PING 10.77.128.90 (10.77.128.90) 56(84) bytes of data.
64 bytes from 10.77.128.90: icmp_seq=1 ttl=63 time=1.11 ms

--- 10.77.128.90 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.105/1.105/1.105/0.000 ms

Same continent:

cadey:users@kahless ~ ./rw
$ ping 10.77.128.90 -c1
PING 10.77.128.90 (10.77.128.90) 56(84) bytes of data.
64 bytes from 10.77.128.90: icmp_seq=1 ttl=63 time=5.66 ms

--- 10.77.128.90 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 5.655/5.655/5.655/0.000 ms

And finally a machine on the same planet:

cadey:users@lufta ~ ./rw
$ ping 10.77.128.90 -c1
PING 10.77.128.90 (10.77.128.90) 56(84) bytes of data.
64 bytes from 10.77.128.90: icmp_seq=1 ttl=63 time=107 ms

--- 10.77.128.90 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 106.719/106.719/106.719/0.000 ms

This also lets any virtual machine on the cluster reach out to any other virtual machine, as well as any of the hardware servers. If I install a SerenityOS virtual machine (a platform that can't run Tailscale as far as I am aware), it will be able to poke other virtual machines as well as my other servers over Tailscale like it never happened. It is a magical experience.

I have a lot more compute than I really know what to do with right now. This is okay though. Lots of slack compute space leaves a lot of room for expansion, experimentation and other e-words in that category. These CPUs are really dang fast too, which helps a lot. So far I've used my homelab both while doing a short V-tuber-esque stream where I fix a minor annoyance in NixOS and try to explain what was going on in my head as I did it and for writing this article. Pneuma has sort of become my main SSH box and the other machines run lots of virtual machines.

In the future I'd like to use this lab for the following things:

• Running some non-critical services out of my basement (Discord bots, etc)
• Implement a VM management substrate called waifud
• IPv6 networking for the virtual machines (libvirtd seems to only do IPv4 out of the gate, configuring IPv6 seems to be a bit unfortunately nontrivial)
• CI for projects on my personal Git server
• Research for Project Elysium/NovOS

I hope this was an interesting look into the process and considerations that I made when assembling my homelab. It's been a fun build and I can't wait to see what the future will bring us. Either way it should make for some interesting write-ups on this blog!

Here are some related Twitter threads you may find interesting to look through:

• Day 1 of the build
• Day 2 of the build
• The aftermath of the realization that I can avoid copying a large part of the NixOS configuration for each node
• A thread where I attempt to install Guix on one of the homelab nodes in a VM

My Magical Adventure With cloud-init

"If I had a world of my own, everything would be nonsense. Nothing would be what it is, because everything would be what it isn't. And contrary wise, what is, it wouldn't be. And what it wouldn't be, it would. You see?"

• The Mad Hatter, Alice's Adventures in Wonderland

The modern cloud is a magical experience. You take a template, give it some SSH keys and maybe some user-data and then you have a server running somewhere. This is all powered by a tool called cloud-init. cloud-init is the most useful in actual datacenters with proper metadata services, but what if you aren't in a datacenter with a metadata service?

Recently I wanted to test a script a coworker wrote that allows users to automatically install Tailscale on every distro and version Tailscale supports. I wanted to try and avoid having to install each version of every distribution manually, so I started looking for options.

<Mara> This may seem like overkill (and at some level it probably is), however as a side effect of going through this song and dance you can spin up a bunch of VMs pretty easily.

pic.twitter.com/yays27Wmes

— Xe from Within (@theprincessxena) May 17, 2021

cloud-init has a feature called the NoCloud data source. To use it, you need to write two yaml files, put them into a specially named ISO file and then mount it to the virtual machine. cloud-init will then pick up your configuration data and apply it.

<Mara> Wait...really? What.

<Cadey> Yes, really.

Let's make an Amazon Linux 2 virtual machine as an example. Amazon offers their Linux distribution for download so you can run it on-premises (I don't really know why you'd want to do this outside of testing stuff on Amazon Linux). In this blog we use KVM, so keep that in mind when you set things up yourself.

First you need to make a meta-data file, this will contain the VM's hostname and the "instance ID" (this makes sense in cloud contexts however you can use whatever you want):

local-hostname: mayhem
instance-id: 31337

<Mara> You can configure networking settings here, but our VM is going to get an address over DHCP so you don't really need to care about that in this case

Next you need to make a user-data file, this will actually configure your VM:

#cloud-config
#vim:syntax=yaml

cloud_config_modules:
 - runcmd

cloud_final_modules:
 - [users-groups, always]
 - [scripts-user, once-per-instance]

users:
  - name: xe
    groups: [ wheel ]
    sudo: [ "ALL=(ALL) NOPASSWD:ALL" ]
    shell: /bin/bash
    ssh-authorized-keys: 
      - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPYr9hiLtDHgd6lZDgQMkJzvYeAXmePOrgFaWHAjJvNU cadey@ontos

write_files:
  - path: /etc/cloud/cloud.cfg.d/80_disable_network_after_firstboot.cfg
    content: |
      # Disable network configuration after first boot
      network:
        config: disabled

Please make sure to change the username and swap out the SSH key as needed, unless you want to get locked out of your VM. For more information about what you can do from cloud-init, see the list of modules here.

Now that you have the two yaml files you can make the seed image with this command (Linux):

$ genisoimage -output seed.iso \
    -volid cidata \
    -joliet \
    -rock \
    user-data meta-data

<Mara> In NixOS you may need to run it inside nix-shell: nix-shell -p cdrkit.

Or this command (macOS):

$ hdiutil makehybrid \
    -o seed.iso \
    -hfs \
    -joliet \
    -iso \
    -default-volume-name cidata \
    user-data meta-data

Now you can download the KVM image from that Amazon Linux User Guide page from earlier and then put it somewhere safe. This image will be written into a ZFS zvol. To find out how big the zvol needs to be, you can use qemu-img info:

$ qemu-img info amzn2-kvm-2.0.20210427.0-x86_64.xfs.gpt.qcow2
image: amzn2-kvm-2.0.20210427.0-x86_64.xfs.gpt.qcow2
file format: qcow2
virtual size: 25 GiB (26843545600 bytes)
disk size: 410 MiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false

The virtual disk image is 25 gigabytes, so you can create it with a command like this:

$ sudo zfs create -V 25G rpool/safe/vms/mayhem

Then you use qemu-img convert to copy the image into the zvol:

$ sudo qemu-img convert \
    -O raw \
    amzn2-kvm-2.0.20210427.0-x86_64.xfs.gpt.qcow2 \
    /dev/zvol/rpool/safe/vms/mayhem

If you don't use ZFS you can make a layered disk using qemu-img create:

$ qemu-img create \
    -f qcow2 \
    -o backing_file=amzn2-kvm-2.0.20210427.0-x86_64.xfs.gpt.qcow2 \
    mayhem.qcow2

Open up virt-manager and then create a new virtual machine. Make sure you select "Manual install".

virt-manager will then ask you what OS the virtual machine is running so it can load some known working defaults. It doesn't have an option for Amazon Linux, but it's kinda sorta like CentOS 7, so enter CentOS 7 here.

The default amount of ram and CPU are fine, but you can choose other options if you have more restrictive hardware requirements.

Now you need to select the storage path for the VM. virt-manager will helpfully offer to create a new virtual disk for you. You already made the disk with the above steps, so enter in /dev/zvol/rpool/safe/vms/mayhem (or the path to your custom layered qcow2 from the above qemu-img create command) as the disk location.

Finally, name the VM and then choose "Customize configuration before install" so you can mount the seed data.

Click on the "Add Hardware" button in the lower left corner of the configuration window.

Make a new CDROM storage device that points to your seed image:

And then click "Begin Installation". The virtual machine will be created and its graphical console will open. Click on the info tab and then the NIC device. The VM's IP address will be listed:

Now SSH into the VM:

$ ssh xe@192.168.122.122
The authenticity of host '192.168.122.122 (192.168.122.122)' can't be established.
ED25519 key fingerprint is SHA256:TP7dWLkHOixx5tr78qn0yvDQKttH0yWz6IBvbadEqcs.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.122.122' (ED25519) to the list of known hosts.

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
8 package(s) needed for security, out of 17 available
Run "sudo yum update" to apply all updates.
[xe@mayhem ~]$

And voila! A new virtual machine that you can do whatever you want with, just like you would any other server.

<Mara> Do you really need to make an ISO file for this? Can't I just use HTTP like the AWS metadata service?

Yes and no. You can have the configuration loaded over HTTP/S, but without special network configuration you won't be able to have http://169.254.169.254 work like the AWS metadata service without a fair bit of effort. Either way, you are going to have to edit the virtual machine's XML though.

<Mara> XML? Why is XML involved?

virt-manager is a frontend to libvirt. libvirt uses XML to describe virtual machines. Here is the XML used to describe the VM you made earlier. This looks like a lot (because frankly it is a lot, computers are complicated), however this is a lot more manageable than the equivalent qemu flags.

<Mara> What do the qemu flags look like?

Like this. It is kind of a mess that I would rather have something made by people smarter than me take care of.

To enable cloud-init to load over HTTP, you are going to have to add the qemu XML namespace to mayhem's configuration. At the top you should see a line that looks like this:

<domain type="kvm">

Replace it with one that looks like this:

<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">

This will allow you to set the cloud-init seed location information using a SMBIOS value. To enable this, add the following to the bottom of your XML file, just before the closing </domain>:

<qemu:commandline>
  <qemu:arg value="-smbios"/>
  <qemu:arg value="type=1,serial=ds=nocloud-net;h=mayhem;s=http://10.77.2.22:8000/mayhem/"/>
</qemu:commandline>

Make sure the data is actually being served on that address. Here's a nix-shell python one-liner HTTP server:

$ nix-shell -p python3 --run 'python -m http.server 8000'

Then you will need to either load the base image back into the zvol or recreate the qcow2 file to reset the VM back to its default state.

Reboot the VM and wait for it to connect to your "metadata server":

192.168.122.122 - - [04/Jun/2021 11:41:10] "GET /mayhem/meta-data HTTP/1.1" 200 -
192.168.122.122 - - [04/Jun/2021 11:41:10] "GET /mayhem/user-data HTTP/1.1" 200 -

Then you can SSH into it like normal:

$ ssh xe@192.168.122.122
The authenticity of host '192.168.122.122 (192.168.122.122)' can't be established.
ED25519 key fingerprint is SHA256:eJRjDsvnVrXfntVtNVN6N+JdakaA+dvGKWWQP5OFkeA.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.122.122' (ED25519) to the list of known hosts.

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
8 package(s) needed for security, out of 17 available
Run "sudo yum update" to apply all updates.
[xe@mayhem ~]$

<Mara> Can I choose other distros for this?

Yep! Most distributions offer cloud-init enabled images. They may be hard to find, but they do exist. Here's some links that will help you with common distros:

• Arch Linux (use the cloudimg ones)
• CentOS 7 (use the GenericCloud one)
• CentOS 8 (use the GenericCloud one)
• Debian 9 (use the openstack one)
• Debian 10 (use the generic one)
• Debian 11 (use the generic one)
• Fedora 34 (use the Openstack image)
• OpenSUSE Leap 15.2 (use the OpenStack image)
• OpenSUSE Leap 15.3 (use the JeOS one labeled OpenStack-Cloud)
• OpenSUSE Tumbleweed (use the JeOS one labeled Openstack-Cloud)
• Ubuntu (use the server-cloudimg image for your version of choice)

In general, look for images that are compatible with OpenStack. OpenStack uses cloud-init to configure virtual machines and the NoCloud data source you're using ships by default. It usually works out, except for cases like OpenSUSE Leap 15.1. With Leap 15.1 you have to pretend to be OpenStack a bit more for some reason.

<Mara> What if I need to template the userdata file?

<Cadey> You really should avoid doing this if possible. Templating yaml is a delicate process fraught with danger. The error conditions in things like Kubernetes are that it does the wrong thing and you need to replace the service. The error condition with this is that you lose access to your server.

<Mara> Let's say that Facts and Circumstances™ made me have to template it.

When you are templating yaml, you have to be really careful. It is very easy to incur the wrath of Norway and Ontario on accident with yaml. Here are some rules of thumb (unfortunately gained from experience) to keep in mind:

• yaml has implicit typing, quote everything to be safe.
• ensure that every value you pass in is yaml-safe
• ensure that the indentation matches for every value

Something very important is to test the templating on a virtual machine image that you have a back door into. Otherwise you will be locked out. You can generally hack around it by adding init=/bin/sh in your kernel command line and changing your password from there.

When you mess it up you will need to get into the VM somehow and do one of a few things:

1. Run cloud-init collect-logs to generate a log tarball that you can export to your host machine and dig into from there
2. Look through the system journal for any errors
3. Look in /var/log for files that begin with cloud-init and page through them

If all else fails, start googling. If you are running commands against a VM with the runcmd feature of cloud-init, I'd suggest going through the steps on a manually installed virtual machine image at least once so you can be sure the steps work. I have lost 4 hours of time to this. Also keep in mind that in the context that runcmd runs from, there is no standard input hooked up. You will need to pass -y everywhere.

If you want a simple Alpine Linux image to test with, look here for the Alpine Linux images I test with. You can download this image from here in case you trust that I wouldn't put malware in that image and don't want to make your own.

In the future I plan to use cloud-init extensively within my new homelab cluster. I have plans to make a custom VM management service I'm calling waifud. I will write more on that as I have written the software. I currently have a minimum viable prototype of this tool called mkvm that I'm using today without any issues. I also will be writing up how I built the cluster and installed NixOS on all the systems in a future article.

cloud-init is an incredible achievement. It has its warts, but it being used in so many places enables you to make configuring virtual machines so much easier. It even works on Windows!. As much as I complain about it in this post, life would be so much worse without it. It allows me to use the magic of the cloud in my local virtual machines so I can get better use out of my hardware.

How to Set Up WeeChat on NixOS

Internet Relay Chat (IRC) is the king of chats. It is the grandfather of nearly every chat protocol and program you use today. It has been a foundation of the internet for over 30 years and is likely to outlive most of the chat apps you use today. IRC is used heavily by the people that make the software that you use daily, and has been catalytic to careers the world over.

However, because of its age IRC can be a bit hard for newcomers to get into. It has its own cultural norms that will seem alien. In this article we're going to show you how to set up an IRC client and a persistent bouncer (something that stays connected for you) with a web UI on NixOS. For the sake of simplicity we well be connecting to Libera Chat.

Installing WeeChat

IRC is an open protocol and it has been one for many years. As such there are many clients to pick from. However, you're reading an article on my blog and that means I get to let my opinions about IRC clients influence you. So, here's how to set up WeeChat (not to be confused with WeChat, the chat program mainly used in China), my IRC client of choice.

Installing on NixOS

<Mara> Even though this article is focusing on NixOS, WeeChat has been around for many years and is likely to be present in your distribution of choice's package manager.

You can install WeeChat by adding it to your configuration.nix like this:

environment.systemPackages = with pkgs; [ weechat ];

Then you can rebuild your configuration with the normal nixos-rebuild command:

$ sudo nixos-rebuild switch

And WeeChat should be visible in your $PATH:

$ which weechat
/run/current-system/sw/bin/weechat

Then run WeeChat like this:

$ weechat

And you should see the default UI:

Customization

First let's change how WeeChat groups server buffers. Normally it lumps everything into one big merged buffer, however most other clients will have independent buffers per network. I like the behaviour where each server has its own buffer. To make the server buffers independent, paste this line into the input bar:

/set irc.look.server_buffer independent

Enable mouse control with the /mouse command:

/mouse enable

Colorscheme

WeeChat has a very primitive colorscheme system through various settings. For most people the defaults will be fine. However certain color schemes (like the one I use, Gruvbox Dark) can make the top titlebar hard to read. WeeChat's website has a themes page where you can get some ideas.

<Mara> The files that the themes page offers are intended for a WeeChat script that hasn't been included in the normal script repository for some reason, however you can obviate that need by a little massaging with vim!

You can convert a theme to a bunch of /set commands with vim. Find a theme you like such as nils_2 and copy the theme to a file. The theme script outputs something that looks like WeeChat configuration by default.

<Cadey> If you want the theme I use, download it from here.

Then open that file in vim and we will munge it in a few steps.

First, put /set at the beginning of every line:

:%s%^%/set %

Then remove the = from each line:

:%s/ =//

And finally remove all of the quotation marks (make sure to include the global flag here because otherwise only one of the quote marks will be removed):

:%s/"//g

And then paste it all into your input line and then run /save:

/save

The result should look like this:

Plugins

WeeChat has a rich scripting layer that you can read more about here. It has bindings for most languages you could care about. I have a few plugins that I use to make my WeeChat experience polished. I'm going to go over them in their own sub-sections. You can install scripts using the /script command.

<Mara> Newer versions of WeeChat require permission before they will be allowed to download scripts from the script repository. To give it permission, run this command:

/set script.scripts.download_enabled on

autosort.py

Normally WeeChat will put buffers in the order that you opened them. I have a slight case of CDO, so I prefer having the buffers in the correct alphabetical order. autosort.py will do this. To install it, run this command:

/script install autosort.py

It will kick in automatically when you create new buffers, however if you want to manually run it, use this command:

/autosort

The autosort plugin has a lot of configuration, take a look at /help autosort if you want to dig deeper.

autojoin.py

WeeChat doesn't remember what channels you were in when you close your client and restart it later. autojoin.py fixes this by saving the list of channels you are in when you quit WeeChat. It also gives you a command to save all of the channels regardless. To install it, run this command:

/script install autojoin.py

If you ever want to save your list of joined channels, run this command:

/autojoin --run
/save

<Mara> The /save isn't strictly needed there, however it may help you feel better!

confversion.py

WeeChat normally stores its configuration as a bunch of text files in ~/.weechat. It doesn't version these files at all, which makes it slightly hard to undo changes. confversion.py puts these changes into a git repository. To install it, run this command:

/script install confversion.py

It will automatically run every time you change settings. You don't need to care about it, however if you want to care about what it does, see the its settings with this command:

/set plugins.var.python.confversion.*

emoji.lua

WeeChat is a terminal program. As such it is not the easiest to input emoji and sometimes you absolutely need to call something 💩. This script converts the emoji shortcodes you use on Discord, GitHub and Slack into emoji for you. To install it, run this command:

/script install emoji.lua

Then you can 💩post to your heart's content.

go.py

If you become a hyperlurker like I am, you tend to build up buffers. A lot of buffers. So many buffers that it gets hard to keep track of them all. go.py lets you search buffers by name and then go to them. To install it, run this command:

/script install go.py

Then you should bind a key to call /go for you. I suggest meta-j:

/key bind meta-j /go

<Mara> On some terminals, you can use the alt-key for this. On others you will need to press escape and then j. You can change this to control-j with something like /key bind ctrl-j /go.

listbuffer.py

One of the main ways to discover new channels to talk in on IRC is by using the /list command. By default this output gets spewed to the server buffer and isn't particularly useful. listbuffer.py collects all of the channels into a buffer and then sorts them by user count. To install it, run this command:

/script install listbuffer.py

This will fire automatically when you do /list on an IRC server connection:

screen_away.py

This one may not be super relevant if you don't run an IRC client in screen or tmux, but I do. This script will automatically mark you as "away" when you detach from screen/tmux and mark you as "back" when you attach again. To install it, run this command:

/script install screen_away.py

Connecting to an IRC Network

Now that things are set up, let's actually connect to an IRC network. For this example, we will connect to Libera Chat. In WeeChat's model, you need to create a server and then set things in it. However, let's set some default settings first.

<Mara> At this point it may be a good idea to start running WeeChat in tmux or a similar program. This will let you detach WeeChat and come back to it later.

Here's how you set the default nickname, username and "real name":

/set irc.server_default.nicks Mara,MaraH4Xu,[Mara]
/set irc.server_default.username mara
/set irc.server_default.realname Mara Sh0rka

Setting Up Libera Chat

Add the Libera Chat connection with the /server command:

/server add liberachat irc.libera.chat/6697 -ssl -auto

Then you can check the settings with /set irc.server.liberachat.*:

More than likely the defaults are fine, however you can customize them with /set if you want.

Next, let's connect to Libera Chat with this command:

/connect liberachat

Registration

Once you are connected, register an account with NickServ:

<Mara> IRC is a bit primitive, most networks use services like NickServ to help handle persistent identities on IRC.

/q NickServ help register

Then set a password (make sure it's a good one!) and email address, then run the command. You will get an email from the Libera Chat services daemon with a verification command. Run it and then your account will be set up. For the rest of this article we are going to assume that your account name is [Mara].

/msg NickServ register hunter2 mara@best.shork

Now you can configure WeeChat to automatically identify with NickServ on connection by using SASL. To configure SASL with WeeChat, do this:

/set irc.server.liberachat.sasl_mechanism plain
/set irc.server.liberachat.sasl_username [Mara]
/set irc.server.liberachat.sasl_password hunter2

<Mara> If you aren't using confversion.py, now is a good time to run /save.

Then run /reconnect and look for this line in your Libera Chat buffer:

-- SASL authentication successful

If you see this, then you are successfully identifying with NickServ when you connect to Libera Chat.

Getting a Cloak

IRC attaches your public IP or DNS hostname to every message you send. Some people may not want to have this happen. A cloak lets you hide your public IP address and put something else there instead. It allows you to show up as something like user/xe instead of chrysalis.cetacean.club.

To get a cloak, join #libera-cloak:

/j #libera-cloak

Then send !cloakme to the channel. The bot will kick you once your cloak is set.

Joining Channels

From here you can join channels and talk around places like normal. Here are some of my main haunts on Libera Chat:

• #xeserv -> The official channel for this blog
• #lobsters -> The official channel for Lobsters, a news aggregation site that I really like
• ##hntop -> A feed of new articles that are posted to Orange Site
• ##furry -> Encounters of the furred kind

I am Xe on Libera Chat.

WeeChat Relay and Glowing Bear

If you run WeeChat in tmux, you can attach to that tmux session later and then continue chatting wherever you end up. If you are on your phone or a tablet, this may not be the most useful thing in the world. It is somewhat difficult to use a shell on a phone. WeeChat has a relay protocol setting that lets you connect to your chats on the go. You can use Glowing Bear to work with WeeChat. The public instance at glowing-bear.org will work fine for many cases, but I prefer running it myself so I don't have to give my WeeChat instance access to a blessed TLS certificate pair.

To set this up, you will need to choose a relay password. I personally use type-4 UUIDs generated with uuidgen:

$ uuidgen
73b4d63d-ef7f-40a5-ab6e-01dfa4298a28

Then you can configure the relay port:

/set relay.network.bind_address 127.0.0.1
/set relay.network.password 73b4d63d-ef7f-40a5-ab6e-01dfa4298a28
/relay add weechat 9001

Now that you have the relay set up, you can check to see if it's working with netcat:

$ nc -v 127.0.0.1 9001
Connection to 127.0.0.1 9001 port [tcp/etlservicemgr] succeeded!

This should also trigger a message in WeeChat:

relay: new client on port 9001: 1/weechat/127.0.0.1 (waiting auth)

Now that you know WeeChat is listening, you can set up Glowing Bear with a NixOS module. Here's how I do it:

# weechat.nix
{ config, pkgs, ... }:

{
  # Mara\ Set up an nginx vhost for irc-cadey.chrysalis.cetacean.club:
  services.nginx.virtualHosts."irc-cadey.chrysalis.cetacean.club" = {
    # Mara\ "gently encourage" clients to use HTTPS
    forceSSL = true;
    
    # Mara\ Proxy everything at `/weechat` to WeeChat
    locations."^~ /weechat" = {
      # Mara\ Replace the host and port with whatever you configured
      # instead of this.
      proxyPass = "http://127.0.0.1:9001";
      # Mara\ WeeChat has websocket support for the relay protocol,
      # this tells nginx to expect that.
      proxyWebsockets = true;
    };
    
    # Mara\ Serve glowing bear's assets at the root of the domain.
    locations."/".root = pkgs.glowing-bear;
    
    # Mara\ Use the ACME cert for `cetacean.club` for this
    useACMEHost = "cetacean.club";
  };
}

You can add this to your imports in your server's configuration.nix using the layout I described in this post. This would go in the host-specific configuration folder.

Once you've deployed this to a server, try to open the page in your browser:

Then enter in the following details:

• For the relay hostname, enter irc-cadey.chrysalis.cetacean.club
• For the port, enter 443
• For the password, enter the UUID from earlier

Then click "Save" and "Automatically connect" you will be connected to your chats!

IRC Norms and Netiquette

IRC is a unique side of the internet. Here are some words of advice that may help you adjust to it:

• Most channels will go silent unless there is something to say. The channel being silent is a good thing.
• Don't ask to ask. If you have a question, just ask it.
• Lurk for a bit in a social channel before chatting.
• Always have an exit strategy.
• Be wary of links from strangers.
• Furries, LGBT and neurodivergent people wrote the software you are using. Do not anger the furries.
• Befriend but be wary of rabbits.
• Don't run weird commands if people you don't know ask you to run them.
• Power is a curse.
• Be kind to those who answer your questions. This may be a repeat question for them.
• Tell people what documentation you read and what you tried.
• Don't paste code snippets into the chat directly. Use a pastebin or GitHub gists.
• Write things in longer sentences instead of sending lots of little lines.

Drop in #xeserv! There's a small but somewhat active community there. I would love to hear any feedback you have about my articles.

Epilogue

The last caretaker's absence rippled throughout the halls. The darkness was all that remained.

I used to run an IRC network named PonyChat. It was an IRC network aimed at adult fans of My Little Pony: Friendship is Magic. Looking back, working on that network was probably the biggest catalyst to my learning how to do system administration to the level I am at today.

Lots of stuff goes wrong when you run an IRC network. PonyChat peaked at around 500 users on average, but that didn't stop things from being interesting. There were several "groups" of people there, and a lot of roleplaying channels. As things like Discord picked up more and more users, a lot of the roleplaying channels were all that were left at the end. There were some people in the #geek room that were near permanent fixtures. Talking with those people and collaborating on various projects is how I learned the skills that I use daily for remote work.

The darkness was confused. It didn't expect this to happen. The discussion halls were so full of life before! There were so many people from as many backgrounds talking about anything you could imagine!

But the people left. The darkness didn't totally see why this happened, but then they walked the halls and saw some things around the empty rooms.

The official Arch Linux support channels have moved to libera.chat, good luck!

The previous moderators of the discussion forum had apparently left up signs telling anyone who hadn't walked over with them to tell them where to go. The darkness looked around and saw more and more of those signs.

Without those signs, they won't know where to go! If we can remove all of those signs then maybe the people will be active again!

This channel has moved to ##archlinux. The topic is in violation of freenode policy.

Perfect, the darkness thought to themselves. They can't leave now, those signs were telling them where to go!

When things came to an end with PonyChat, I had a big choice to make. There's two main ways for chat communities to die: fast and slow. The fast ways are quicker, less painful for users and potentially harsh for people that didn't get the memo in time. The slow way gets expensive and soul-draining.

I was the last caretaker left on PonyChat after the attrition rate affected the staff as well as the users. I was the only person really active on the network and a lot of it was held together with increasingly brittle lua scripts.

It was soul-crushing. PonyChat was close to my heart. Writing the bots that ended up being the core of the anti-spam engine were some of my first coding projects.

The darkness was disturbed from their laurels by one of their caretakers. Apparently this angered the people who had left. The former community scribes were furious. The last caretakers had never done such a thing. Notices to those communities were always left intact. The mere thought of doing such a thing was unthinkable.

Yet it happened. The darkness realized that they messed up. Quickly, a change was made. It can't be against policy if there's a policy allowing it! Historical precedent be damned, this is advertisement! They are promoting another place instead of here! Here is perfectly good! They thought.

The darkness smiled its spiral smile and spread to take down more signs with a golem purpose made to print off new signs.

This channel has moved to ##botters. The topic is in violation of freenode policy.

The golem blindly continued manufacturing out new signs. The silent masses left behind watched in horror as they were forced out of their former haunts.

There's something kind of magical about writing an IRC chatbot. It's one of the few kinds of things you can create that you create in public. Even if the source code isn't shared you still need to test it somewhere. You build it in public.

Anti-spam bots are a similar kind of thing. Unfortunately they form a kind of arms race. It's much easier to make new spam than it is to come up with patterns for existing spam. Writing one is soul-crushing. You have to quickly develop a kind of reputation system or you will immediately turn it into a way to ban your own users. A lot of the more clever trolls tricked users into typing the phrases that got them banned.

Then there was the doxxing and swatting.

The darkness walked through the halls and smiled. All those signs were gone. They peered into a room to see what was happening. They saw nothing. There weren't even the silent masses that had normally huddled around the backs of rooms. Some of those people had sat there for years doing nothing but listening. Nobody really knew if they were actually paying attention or not, some may not even be alive anymore, but they were haunting those rooms either way.

The signs pointed people elsewhere. Those who had stayed in the background didn't get the memo. They were stuck there. Just sitting there and watching. Not really doing anything, just watching and listening.

If you run an IRC network of any appreciable scale, be prepared for these eventualities:

Your real name, email address, facebook account link, twitter account link, phone number, parents names, mailing address, physical address and sometimes even tax identification numbers will be leaked to the public. You MUST use a password manager and two-factor auth everywhere. Register your domains under a past or fake address. That will prevent people from getting your mailing address as easily.

I've been doxxed so many times that I have given up trying to keep my things separate. A lot of the places you see me using different names started out as my attempts to use separate handles in different places. I have kept them the same for consistency but I have largely given up trying to keep them separate. It is a lot of work and I bet that even if I went back on the hyper private sthick (if I even can at this point, I've been frontpaged on Orange Site and my blog gets so much traffic that it's probably impossible in practice without abandoning my handles and picking new ones).

Your staff will lose interest and abandon the project one day without telling you. They may end up still being connected there, but just as an idle bouncer. It's akin to a zombie laying in the background.

Call your local police non-emergency number and set up a standing order to call you before they send in a SWAT team to your house. There are people that will seriously call the cops and claim you're armed and dangerous to get a SWAT team to ruin your life or potentially get you killed. This is not a joke. It's nearly happened to me thrice. I got that call from the cops once. It is not a good feeling.

You need to use something with a powerful and easy to use spambot or message filtering built into the server itself. This will save your ass some day.

The former moderators of the rooms that were closed off came back with pitchforks and torches. They were pissed. The rooms they had tended to for years were suddenly stolen from them. Yes, they were abandoned, but the precedent for doing such a thing had never really existed before. It was such a tiny thing, but they had to go out of their way to make that golem. They had to tell the golem what to do. They had to send out that golem.

Several groups were on the fence with regards of what to do, but that golem made the choice for them. Some groups even wanted to stay at the same meeting house but the golem came in and closed their hall without warning.

The day I killed PonyChat was a hard day for me. I had planned it 3 months ago. Warnings were issued. I helped bigger communities move elsewhere. Everything was spinning down.

Then the time came and I ran the script that only needed to be run once:

$ ./scripts/kill_ponychat.sh

A progress bar appeared and with it all of what was created over the last decade was destroyed. Backups were erased. Data was wiped. Servers were destroyed. DNS records were altered. And finally it printed this:

It's okay to cry.

And that was the end of it.

If the halls were empty before, they were desolate now. Everything was being abandoned in real time. Announcements were made about how the golem was premature and that people should really consider staying. It was no use. The golem had made up their minds.

The rot started.

Author's Note: I really hope this is the last entry in this little speculative fiction/postmortem/retrospective series. I have an article in the pipeline on how I'm creating virtual machines from templates so that I can test how various versions of various distros work, but this freenode bullshit has eaten up a lot of my thinking time. It's been like watching a train wreck. You can't look at it, but you can't look away either. It's so hard to watch yet you just can't help but watch it.

It hurts.

This was not on my bingo card for 2021.

Final Chapter

The last caretaker looked at the last light lit in the empty halls. They looked out across their home. It used to be filled with thousands of people. There were discussions about every topic imaginable from people of as many backgrounds. Projects were born. Relationships were forged. Friends were found. Lives were irreparably changed for the better.

But the darkness came and soaked into the foundation. Some noticed it and became alerted. Some ignored it as an "inevitable" change. Some ran away, abandoning their home of choice. Some stuck around.

Then the darkness took over and the larger discussion rooms sprung into action. The meddlers between the halls suddenly became a network of people to assist the other larger discussion rooms to move elsewhere. Through their careful planning and surgical execution, they managed to move all of the major discussions away in hours. Prior migrations on this scale have not happened. Normally there are stragglers. Normally people try to stick it out and decry those that leave as "reactionaries".

NickServ: User reg.  : Dec 03 22:31:33 2007 (13y 24w 3d ago)
NickServ: Last addr  : ~cadey@infoforcefeed/Xe

Freenode had been my home on the internet for over half of my life. All things IRC must come to an end, but it felt like Freenode was eternal. The staff had not always made decisions that I agreed with, but I have run IRC networks before. I know how it is. Precedent can drown you.

It's just sad to see it end like this. The communities that I have joined there have been catalytic in my life. I have irreparably changed the life of others on Freenode. I met people on Freenode that I talk with daily. I'm sure that I wouldn't have the career that I have without Freenode.

But it's been taken over by a narcissistic Trumpian wannabe Korean royalty bitcoin millionaire, and I just cannot support that.

Not this time though. The darkness was so deep into things that the very rooms themselves became at risk. The floors were not known to be safe. The caretakers had worked together to come up with a new plan and had built an exact replica of the meeting halls elsewhere.

Through that network of meddlers, people were helped over. The new halls needed adjusting to cope with the sudden load, however they took them. The caretakers spun up their new discussion hall and built it stronger than before.

They, the last caretaker, was unable to control the darkness anymore. The darkness was too deep. Their access had rotten away.

NickServ: Account Xe has 5 other nick(s) grouped to it, remove those first.

I joined Freenode initially in order to get help with Ubuntu in high school.

(shadowh511) hello, i need some help configuring wine 0.9.50 to run microsoft digital image suite 10, i try to run the setup app, an i get two looping messages about MDAC, and on how the system needs to reboot before it can install, but the messages loop, how can i fix this?

(Fluxd) !wine | shadowh511

(ubotu) shadowh511: WINE is a compatibility layer for running Windows programs on GNU/Linux. See https://help.ubuntu.com/community/Wine for more information, and see !AppDB for application compatibility.

(shadowh511) they won't help me there

I eventually figured it out. I think I just did it on Windows.

I ended up rejoining it and really sticking around once I got into My Little Pony: Friendship is Magic. There was a channel called #reddit-mlp that I haunted for years.

I met friends that I still talk with today. I wonder if anyone from there that I haven't talked with in years is reading this now.

The soul of the halls was not the halls itself. It was not the caretakers. It was the people. Once the bigger halls moved over, the people followed. Each person leaving took that much more of the soul with them.

Drops became a stream became a torrent became a flood and in a day's time the soul of the halls had left. The soul was gone and only the darkness remained.

The last caretaker looked at the room and snuffed out their light. They packed up their backpack and closed the front door for the last time.

They cried as they walked to their new home.

(Xe) ungroup JohnMadden
NickServ: Nick JohnMadden has been removed from your account.
(Xe) ungroup Xena
NickServ: Nick Xena has been removed from your account.
(Xe) ungroup Cadance
NickServ: Nick Cadance has been removed from your account.
(Xe) ungroup Cadey
NickServ: Nick Cadey has been removed from your account.
(Xe) ungroup shadowh511
NickServ: Nick shadowh511 has been removed from your account.
(Xe) drop Xe ******

NickServ: This is a friendly reminder that you are about to destroy the account Xe. To avoid accidental use of this command, this operation has to be confirmed. Please confirm by replying with /msg NickServ DROP Xe ****** 86033ae0:9e021b10

The halls fell, but the people that made up the soul of those halls moved on.

Their home went away because their home wasn't a building. It was an idea. Ideas persist in ways that buildings don't.

(Xe) DROP Xe ****** 86033ae0:9e021b10
NickServ: The account Xe has been dropped.

You can find me on Liberachat in #xeserv. If you want to chat about my blog articles, I welcome any readers to join there.

Be well.

irc: server freenode has been deleted

systemd: The Good Parts

Video

The slides link will be at the end of the post.

Hello, I'm Xe and today I'm going to do a talk about systemd. More specifically the good parts of systemd. This talk is going to go fast because there's a lot of material to cover and the notes are going to be on my website. I have been an Alpine user for almost a decade and it's one of my favorite linux distributions.

The best things in life come with disclaimers and here are the disclaimers for this talk:

• This talk may contain opinions. These opinions are my own and not necessarily the opinions of my employer.
• This talk is not evangelism. This talk is intended to show how green the grass is on the other side and how Alpine can benefit from these basic ideas.
• This talk also contains images of cartoon marine animals.

<Mara> What is systemd?

When doing a talk about a thing I find it helps to start with a good definition of what that thing is. Given this talk is about systemd let's start with what systemd is.

systemd is a set of building blocks that you can use to make a linux system. This diagram covers most of the parts of systemd. There is everything from service management to log management to boot time analysis, network configuration, and user logins; but we're only going to cover a tiny fraction of this diagram. At a high level systemd provides a common set of tools that you can build a linux system with; kind of like lego bricks. It does just manage services but it does more than just service management.

Something else that's useful to ask is "why does systemd exist?" Well, looking back at that diagram, computers are actually fairly complicated. There's a lot going on over here. There's log management, there's disk management, there's service sequencing, network configuration, containers user sessions and most importantly all of these things need to happen in order or bad things can happen. I mentioned that systemd is more than just a service service manager because it has optional components that manage things like dns resolution, network devices, user sessions, and user level services among other things.

One of the big differences between systemd and other things like OpenRC is that systemd is a very declarative environment. In declarative environments you specify what you want and the system will figure out what it needs to do to get there. In an imperative environment you specify all of the steps you need to do to get there. It's the difference between writing a sql statement and a for loop.

So, pretend that this somewhat realistic scenario is happening to you: it's 4:00 am you just got a panicked call from someone at your company that the website is down. You log into a server and you want to see if the website is actually down or if it's just dns. You probably want to know the answers to these basic questions:

• Does the service manager think your service is running?
• How much ram is it using?
• Does it have any child processes?
• Has it reported it is healthy?
• How much traffic has it used?
• What are the last few log lines?
• If you need to reboot the server right now for some reason, will that service come back up on reboot?

systemd includes a tool called systemctl that allows you to query the status of services as well as start and stop them; but for right now we're going to look at the systemctl status subcommand. Here is the output for the systemctl status command for the service powering christine.website. So let's go down the list:

• Is the service running? If you look at the red box right there you can see that it says say the service has been running for nine hours.
• How much ram is it using? If you look at the red box there it says it's using about 200 megs of ram.
• How many child processes are there if you look at the red box it'll show you all of the processes in the service's cgroup. In this case we'll see that there's just one process.
• How much network traffic has it been using? If we look here in the red box you can see it's had about a megabyte of traffic in and somewhat less than a megabyte of traffic out. My website serves everything over a unix socket and those numbers aren't reflected here but it's actually much higher.
• At the bottom we can see the last few log lines. These are just random requests that people make to my blog.

<Mara> Where did it get those logs from?

If you haven't seen all of this in action before you might be wondering something like "Wait, where did it get those logs from?"

I mentioned systemd does more than just start services. systemd has a common log sink called the journal. Logs from the kernel, network devices, services, and even some other system sources that you may not think are important automatically get put into the journal. It's similar to Windows event logs or the console app in macOS except it's implicit instead of explicit (Windows and macOS make you use some weird logging calls to make sure that log lines actually get in there, but systemd will capture the standard output, standard error and syslog for every service managed by systemd). Something neat about the journal is that it lets you tail the logs for the entire system with one command: journalctl -f. Here's that command running on a server of mine:

There's a lot more to the journal involving structured logging, automatically streaming the logs to places, and advanced filtering based off of different units, services, or other arbitrary fields; however that is out of scope for this talk. The important part is that it has support for that in case you actually need it.

Now this is all great, and you might be think asking yourself "well, yeah, this stuff is cool; but how does Alpine fit into this? Alpine can't run systemd because systemd is glibc specific." However we're not talking about systemd directly, we're talking about the philosophies involved and the truth is that this kind of experience is what people already have elsewhere. By not having something competitive Alpine is less and less attractive for newer production deployments.

Now there's at least four classes of benefits for systemd and I'm going to break them down into the following groups:

• developers
• packagers
• system administrators
• users

In general people that are developing services that run on systemd get the following benefits:

• Predictability. systemd configuration files are declarative rather than imperative. You declare units instead of imperatively building up init scripts. Options are declared and enforced by the service manager. This makes it a lot easier to review changes for correctness.
• Portability. when setting up a service with systemd there's only one syntax to learn across 15 plus different distributions. This means that you don't have to maintain a giant pile of hacks to make the program just start consistently across different distributions and you can only care about the systemd unit that will make everything happen for you. Before systemd was widespread every distribution had their own unique special snowflake configuration for init systems and it really just wasn't that nice to deal with. Ubuntu had different opinions from Debian, Debian and opensuse had different opinions, and centos was way out in the weeds and it just became hard to do this consistently across distributions. Something declarative like systemd makes doing it across distributions a lot easier by comparison.
• One of the other big things that it has is a api for controlling things with dbus. Now, say what you will about dbus but dbus does have some very rich introspection capabilities, as well as giving you the ability to integrate with system services at a level that more closely resembles what you get on windows or macOS (or even something like sel4 with microkernel message passing). You don't have to shell out to commands and pray the output format didn't change. You don't have to do some weird calls to unix sockets. It uses standard apis and allows you to integrate things more tightly with the system. Gnome for example uses systemd to trigger suspend and shutdown, as well as having a way a little gui to query the systemd journal. Server software can subscribe to units being started for auditing purposes and such.

Packagers or people that are putting software into packages get the following benefits:

• It is a lot easier to write a systemd unit than it is to write an OpenRC script. systemd units are very bland and boring, they look like ini files. It is going to be pretty obvious that it just does what it does and there's nothing special going on. And because of this declarative syntax it makes human error a lot more obvious and it is a lot easier for other humans to review.
• Now, don't get me wrong, shell scripts for service definitions have gotten us a very long way and are likely to stay around for a very long time (I actually use shell scripts with most of my systemd services to do weird things with environment variables for configuration). However, shell scripting is a very, very subtle art and it is very easy to mess up and do things that are very unpredictable if you are not extremely careful. The declarative syntax of systemd removes the ability for you to mess up formatting shell scripts; or at the very least it isolates the flaws of the shell script to the exact service running and not things like the user that the service is running under.

system administrators of systemd systems also get the following benefits:

• systemctl status and a lot of other parts of systemctl let you see what the system or an individual service is doing without having to wonder if it's actually working or not. In general the lazy thing is the thing that you want to optimize for because people are distracted. There is a lot going on sometimes and if you optimize it so that the easiest thing to do is the correct thing then it is a lot easier to deal with when you have a distracted operator. systemd is set up so that it's hard to do the wrong thing. It is hard to have logs go anywhere but the system journal. It is hard to write a unit that doesn't tell you if the service is actually running or not. And it makes it so that the path of least resistance will do most of what you want.
• Sometimes system administrators have opinions that are different than the opinions of the packager. Sometimes you need to change environment variables for http proxies or something and sometimes you believe the packager has different opinions than you do about how something should be run. In OpenRC you'd have to make a copy of the init script, make your changes, and then hope those changes don't get blown away when the package updates. systemd has a first-class mechanism for doing this called drop-in units that allow you to customize parts of a systemd service so that you can override exactly what you need to (and only that) and systemd will turn the all of those into one big logical unit and actually go off and run that. This has been very useful in practice.
• Another thing that is kind of endemic to sysvinit and OpenRC systems is the fact that unless you are careful and configure it right cron job output will just go to nowhere and there is not really an easy way to figure out if a cron job actually ran and if it errored or if it did exactly what you wanted. If I recall there was actually an entire small startup that was formed around just alerting for cron jobs that were not doing what they should be doing. systemd changes this because all of the logs are in the journal. If you set up a systemd timer (which is the systemd land equivalent to a cron job) all of the output for the service associated with that timer gets put into the journal and you can see exactly what went wrong so you can go off and fix it. This has saved me so much time and headache trying to do this stuff manually.
• Another thing that you can do is you can group services together with targets which are kind of like named runlevels. Targets let you specify the difference between the system booting the network stack is configured and all of the services needed for your app are running. You can get a list of dependencies from systemd for any service and you can also use that to help you plan incident response, so it is more difficult to have hidden dependencies.

As far as users go:

• systemd is not limited to just managing system level services systemd can also manage user services with systemd user mode. I use this on my Linux system in order to have a couple services running in the background querying for weather or a couple other api calls to put them into my status bar on my tiling window manager (sway). I have another one that runs emacs in server mode so that I can have one giant emacs session that will automatically start on login. I can put hundreds and hundreds of buffers in there and not have to worry about it. I can spawn new emacs frames instantly, it's really beautiful.
• You can also query all of the system journal logs as a normal user and you don't have to sudo up and go into the logs folder. So if you just want to take a quick look at something, you don't have to type in your password or hit a yubikey press or whatever you have configured.

<Mara> I hope Alpine ends up with something similar!

I really hope alpine comes up with something similar to systemd. Alpine can really benefit from a tightly integrated service manager that does at least some of the things that systemd does. Declarative really is better than imperative because declarative is easier for distracted operators.

People get distracted. It happens, and when distracted people do things it can sometimes have bad consequences. So if we make the tools powerful, but implicitly correct, then it will just be a lot better overall and users will have a lot less worry involved.

On that note we are very close to hitting time so here's my shout outs to people who either help make this talk happen or I think are cool.

if you have any questions please feel free to ping me on twitter, in the irc room, or on the compact page on my website. I enjoy these kinds of questions and I openly welcome you to ask them.

Thank you, have a good day.

Using Morph for Deploying to NixOS

Managing a single NixOS host is easy. Any time you want to edit any settings, you can just change options in /etc/nixos/configuration.nix and then do whatever you want from there. Managing multiple NixOS machines can be complicated. Morph is a tool that makes it easy to manage multiple NixOS machines as if they were one single machine. In this post we're gonna start a new NixOS configuration for a network of servers from scratch and explain each step in the way.

nixos-configs Repo

NixOS configs usually need a home. We can make a home for this in a Git repository named nixos-configs. You can make a nixos configs repo like this:

$ mkdir -p ~/code/nixos-configs
$ cd ~/code/nixos-configs
$ git init

<Mara> You can see a copy of the repo that we're describing in this post here. That repo is licensed as Creative Commons Zero and no attribution or credit is required if you want to use it as the basis for your NixOS configuration repo for any setup, home or professional.

From here you could associate it with a Git forge if you want, but that is an exercise left to the reader.

Now that we have the nixos-configs repository, create a few folders that will be used to help organize things:

• common -> base system configuration and options
• common/users -> user account configuration
• hosts -> host-specific configuration for named servers
• ops -> operations data such as deployment configuration
• ops/home -> configuration for a home network

You can make them with a command like this:

$ mkdir -p common/users hosts ops/home

Now that we have the base layout, start with adding a few files into the common folder:

• common/default.nix -> the "parent" file that will import all of the other files in the common directory, as well as define basic settings that everything else will inherit from
• common/generic-libvirtd.nix -> a bunch of settings to configure libvirtd virtual machines (omit this if you aren't running VMs in libvirtd)
• common/users/default.nix -> the list of all the user accounts we are going to configure in this system

Here's what you should put in common/default.nix:

# common/default.nix

# Mara\ inputs to this NixOS module. We don't use any here
# so we can ignore them all.
{ ... }:

{
  imports = [
    # Mara\ User account definitions
    ./users
  ];
 
  # Mara\ Clean /tmp on boot.
  boot.cleanTmpDir = true;
  
  # Mara\ Automatically optimize the Nix store to save space
  # by hard-linking identical files together. These savings
  # add up.
  nix.autoOptimiseStore = true;
  
  # Mara\ Limit the systemd journal to 100 MB of disk or the
  # last 7 days of logs, whichever happens first.
  services.journald.extraConfig = ''
    SystemMaxUse=100M
    MaxFileSec=7day
  '';

  # Mara\ Use systemd-resolved for DNS lookups, but disable
  # its dnssec support because it is kinda broken in
  # surprising ways.
  services.resolved = {
    enable = true;
    dnssec = "false";
  };
}

This will give you a base system config with sensible defaults that you can build on top of.

<Mara> Is now when I get my account? :D

<Cadey> Yep! We define that in common/users/default.nix:

# common/users/default.nix

# Mara\ Inputs to this NixOS module, in this case we are
# using `pkgs` so I can configure my favorite shell fish
# and `config` so we can make my SSH key also work with
# the root user.
{ config, pkgs, ... }:

{
  # Mara\ The block that specifies my user account.
  users.users.mara = {
    # Mara\ This account is intended for a non-system user.
    isNormalUser = true;
    
    # Mara\ The shell that the user will default to. This
    # can be any NixOS package, even PowerShell!
    shell = pkgs.fish;
    
    # Mara\ My SSH keys.
    openssh.authorizedKeys.keys = [
      # Mara\ Replace this with your SSH key!
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPg9gYKVglnO2HQodSJt4z4mNrUSUiyJQ7b+J798bwD9"
    ];
  };
  
  # Mara\ Use my SSH keys for logging in as root.
  users.users.root.openssh.authorizedKeys.keys =
    config.users.users.mara.openssh.authorizedKeys.keys;
}

In case you are using libvirtd to test this blogpost like I am, put the following in common/generic-libvirtd.nix:

# common/generic-libvirtd.nix

# Mara\ This time all we need is the `modulesPath`
# to grab an optional module out of the default
# set of modules that ships in nixpkgs.
{ modulesPath, ... }:

{
  # Mara\ Set a bunch of QEMU-specific options that
  # aren't set by default.
  imports = [ (modulesPath + "/profiles/qemu-guest.nix") ];

  # Mara\ Enable SSH daemon support.
  services.openssh.enable = true;

  # Mara\ Make sure the virtual machine can boot
  # and attach to its disk.
  boot.initrd.availableKernelModules =
    [ "ata_piix" "uhci_hcd" "virtio_pci" "sr_mod" "virtio_blk" ];

  # Mara\ Other boot settings that we're leaving
  # to the defaults.
  boot.initrd.kernelModules = [ ];
  boot.kernelModules = [ ];
  boot.extraModulePackages = [ ];

  # Mara\ This VM boots with grub.
  boot.loader.grub.enable = true;
  boot.loader.grub.version = 2;
  boot.loader.grub.device = "/dev/vda";

  # Mara\ Mount /dev/vda1 as the root filesystem.
  fileSystems."/" = {
    device = "/dev/vda1";
    fsType = "ext4";
  };
}

Now that we have the basic modules defined, we can create a network.nix file that will tell Morph where to deploy to. In this case we are going to create a network with a single host called ryuko. Put the following in ops/home/network.nix:

# ops/home/network.nix

{
  # Mara\ Configuration for the network in general.
  network = {
    # Mara\ A human-readable description.
    description = "My awesome home network";
  };

  # Mara\ This specifies the configuration for
  # `ryuko` as a NixOS module.
  "ryuko" = { config, pkgs, lib, ... }: {
    # Mara\ Import the VM-specific config as
    # well as all of the settings in
    # `common/default.nix`, including my user
    # details.
    imports = [
      ../../common/generic-libvirtd.nix
      ../../common
    ];
    
    # Mara\ The user you will SSH into the
    # machine as. This defaults to your current
    # username, however for this example we will
    # just SSH in as root.
    deployment.targetUser = "root";
    
    # Mara\ The target IP address or hostname
    # of the server we are deploying to. This is
    # the IP address of a libvirtd virtual
    # machine on my machine.
    deployment.targetHost = "192.168.122.251";
  };
}

Now that we finally have all of this set up, we can write a little script that will push this config to the server by doing the following:

• Build the NixOS configuration for ryuko
• Push the NixOS configuration for ryuko to the virtual machine
• Activate the configuration on ryuko

Put the following in ops/home/push:

#!/usr/bin/env nix-shell
# Mara\ The above shebang line will use `nix-shell`
# to create the environment of this shell script.

# Mara\ Specify the packages we are using in this
# script as well as the fact that we are running it
# in bash.
#! nix-shell -p morph -i bash

# Mara\ Explode on any error.
set -e

# Mara\ Build the system configurations for every
# machine in this network and register them as
# garbage collector roots so `nix-collect-garbage`
# doesn't sweep them away.
morph build --keep-result ./network.nix

# Mara\ Push the config to the hosts.
morph push ./network.nix

# Mara\ Activate the NixOS configuration on the
# network.
morph deploy ./network.nix switch

Now mark that script as executable:

$ cd ./ops/home
$ chmod +x ./push

And then try it out:

$ ./push

And finally SSH into the machine to be sure that everything works:

$ ssh mara@192.168.122.251 -- id
uid=1000(mara) gid=100(users) groups=100(users)

From here you can do just about anything you want with ryuko.

If you want to add a non-VM NixOS host to this, make a folder in hosts for that machine's hostname and then copy the contents of /etc/nixos to that folder. For example if you have a server named mako with the IP address 192.168.122.147. You would do something like this:

$ mkdir hosts/mako -p
$ scp root@192.168.122.147:/etc/nixos/configuration.nix ./hosts/mako
$ scp root@192.168.122.147:/etc/nixos/hardware-configuration.nix ./hosts/mako

And then you can register it in your network.nix like this:

"mako" = { config, pkgs, lib, ... }: {
  deployment.targetUser = "root";
  deployment.targetHost = "192.168.122.147";
  
  # Mara\ Import mako's configuration.nix
  imports = [ ../../hosts/mako/configuration.nix ];
};

This should help you get your servers wrangled into a somewhat consistent state. From here the following articles may be useful to give you ideas:

• Borg Backup Config
• Nixops Services On Your Home Network (just be sure to ignore the part where it mentions deployment.keys, you can replace it with the semantically identical deployment.secrets as described in the morph documentation)
• Prometheus and Aegis
• My Automagic NixOS Wireguard Setup
• Encrypted Secrets with NixOS

Also feel free to dig around the common folder of my nixos-configs repo. There's a bunch of examples of things in there that I haven't gotten around to documenting in this blog yet. Another useful thing you may want to look into is home-manager, which is a tool that lets you manage your dotfiles across machines. With home-manager I'm able to set up all of my configurations for everything on a new machine in less than 30 minutes (starting from a blank NixOS server).

How I Implemented /dev/printerfact in Rust

Kernel mode programming is a frightful endeavor. One of the big problems with it is that C is really your only option on Linux. C has many historical problems with it that can't really be fixed at this point without radically changing the language to the point that existing code written in C would be incompatible with it.

DISCLAIMER: This is pre-alpha stuff. I expect this post to bitrot quickly. **DO NOT EXPECT THIS TO STILL WORK IN A FEW YEARS.**

<Mara> Yes, yes you can technically use a fairly restricted subset of C++ or whatever and then you can avoid some C-isms at the cost of risking runtime panics on the new operator. However that kind of thing is not what is being discussed today.

However, recently the Linux kernel has received an RFC for Rust support in the kernel that is being taken very seriously and even includes some examples. I had an intrusive thought that was something like this:

<Cadey> Hmmm, I wonder if I can port the Printer Facts API to this, it can't be that hard, right?

Here is the story of my saga.

First Principles

At a high level to do something like this you need to have a few things:

• A way to build a kernel
• A way to run tests to ensure that kernel is behaving cromulently
• A way to be able to repeat these tests on another machine to be more certain that the thing you made works more than once

To aid in that first step, the Rust for Linux team shipped a Nix config to let you nix-build -A kernel yourself a new kernel whenever you wanted. So let's do that and see what happens:

$ nix-build -A kernel
<several megs of output snipped>
error: failed to build archive: No such file or directory

error: aborting due to previous error

make[2]: *** [../rust/Makefile:124: rust/core.o] Error 1
make[2]: *** Deleting file 'rust/core.o'
make[1]: *** [/tmp/nix-build-linux-5.11.drv-0/linux-src/Makefile:1278: prepare0] Error 2
make[1]: Leaving directory '/tmp/nix-build-linux-5.11.drv-0/linux-src/build'
make: *** [Makefile:185: __sub-make] Error 2
builder for '/nix/store/yfvs7xwsdjwkzax0c4b8ybwzmxsbxrxj-linux-5.11.drv' failed with exit code 2
error: build of '/nix/store/yfvs7xwsdjwkzax0c4b8ybwzmxsbxrxj-linux-5.11.drv' failed

Oh dear. That is odd. Let's see if the issue tracker has anything helpful. It did! Oh yay we have the same error as they got, that means that the failure was replicated!

So, let's look at the project structure a bit more:

$ tree .
.
├── default.nix
├── kernel.nix
├── LICENSE
├── nix
│   ├── sources.json
│   └── sources.nix
└── README.md

This project looks like it's using niv to lock its Nix dependencies. Let's take a look at sources.json to see what options we have to update things.

<Mara> You can use niv show to see this too, but looking at the JSON itself is more fun

{
    "linux": {
        "branch": "rust",
        "description": "Adding support for the Rust language to the Linux kernel.",
        "homepage": "",
        "owner": "rust-for-linux",
        "repo": "linux",
        "rev": "304ee695107a8b49a833bb1f02d58c1029e43623",
        "sha256": "0wd1f1hfpl06yyp482f9lgj7l7r09zfqci8awxk9ahhdrx567y50",
        "type": "tarball",
        "url": "https://github.com/rust-for-linux/linux/archive/304ee695107a8b49a833bb1f02d58c1029e43623.tar.gz",
        "url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz"
    },
    "niv": {
        "branch": "master",
        "description": "Easy dependency management for Nix projects",
        "homepage": "https://github.com/nmattia/niv",
        "owner": "nmattia",
        "repo": "niv",
        "rev": "af958e8057f345ee1aca714c1247ef3ba1c15f5e",
        "sha256": "1qjavxabbrsh73yck5dcq8jggvh3r2jkbr6b5nlz5d9yrqm9255n",
        "type": "tarball",
        "url": "https://github.com/nmattia/niv/archive/af958e8057f345ee1aca714c1247ef3ba1c15f5e.tar.gz",
        "url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz"
    },
    "nixpkgs": {
        "branch": "master",
        "description": "Nix Packages collection",
        "homepage": "",
        "owner": "NixOS",
        "repo": "nixpkgs",
        "rev": "f35d716fe1e35a7f12cc2108ed3ef5b15ce622d0",
        "sha256": "1jmrm71amccwklx0h1bij65hzzc41jfxi59g5bf2w6vyz2cmfgsb",
        "type": "tarball",
        "url": "https://github.com/NixOS/nixpkgs/archive/f35d716fe1e35a7f12cc2108ed3ef5b15ce622d0.tar.gz",
        "url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz"
    }
}

It looks like there's 3 things: the kernel, niv itself (niv does this by default so we can ignore it) and some random nixpkgs commit on its default branch. Let's see how old this commit is:

From ab8465cba32c25e73a3395c7fc4f39ac47733717 Mon Sep 17 00:00:00 2001
Date: Sat, 6 Mar 2021 12:04:23 +0100

Hmm, I know that Rust in NixOS has been updated since then. Somewhere in the megs of output I cut it mentioned that I was using Rust 1.49. Let's see if a modern version of Rust makes this build:

$ niv update nixpkgs
$ nix-build -A kernel

While that built I noticed that it seemed to be building Rust from source. This initially struck me as odd. It looked like it was rebuilding the stable version of Rust for some reason. Let's take a look at kernel.nix to see if it has any secrets that may be useful here:

rustcNightly = rustPlatform.rust.rustc.overrideAttrs (oldAttrs: {
  configureFlags = map (flag:
    if flag == "--release-channel=stable" then
      "--release-channel=nightly"
    else
      flag
  ) oldAttrs.configureFlags;
});

<Mara> Wait, what. Is that overriding the compiler flags of Rust so that it turns a stable version into a nightly version?

Yep! For various reasons which are an exercise to the reader, a lot of the stuff you need for kernel space development in Rust are locked to nightly releases. Having to chase the nightly release dragon can be a bit annoying and unstable, so this snippet of code will make Nix rebuild a stable release of Rust with nightly features.

This kernel build did actually work and we ended up with a result:

$ du -hs /nix/store/yf2a8gvaypch9p4xxbk7151x9lq2r6ia-linux-5.11
92M      /nix/store/yf2a8gvaypch9p4xxbk7151x9lq2r6ia-linux-5.11

Ensuring Cromulence

A noble spirit embiggens the smallest man.

I've never heard of the word "embiggens" before.

I don't know why, it's a perfectly cromulent word

• Miss Hoover and Edna Krabappel, The Simpsons

The Linux kernel is a computer program, so logically we have to be able to run it somewhere and then we should be able to see if things are doing what we want, right?

NixOS offers a facility for testing entire system configs as a unit. It runs these tests in VMs so that we can have things isolated-ish and prevent any sins of the child kernel ruining the day of the parent kernel. I have a template test in my nixos-configs repo that we can build on. So let's start with something like this and build up from there:

let
  sources = import ./nix/sources.nix;
  pkgs = sources.nixpkgs;
in import "${pkgs}/nixos/tests/make-test-python.nix" ({ pkgs, ... }: {
  system = "x86_64-linux";

  nodes.machine = { config, pkgs, ... }: {
    virtualisation.graphics = false;
  };

  testScript = ''
    start_all()
    machine.wait_until_succeeds("uname -av")
  '';
})

<Mara> For those of you playing the christine dot website home game, you may want to edit the top of that file for your own projects to get its pkgs with something like pkgs = <nixpkgs>;. The sources.pkgs thing is being used here to jive with niv.

You can run tests with nix-build ./test.nix:

$ nix-build ./test.nix
<much more output>
machine: (connecting took 4.70 seconds)
(4.72 seconds)
machine # sh: cannot set terminal process group (-1): Inappropriate ioctl for device
machine # sh: no job control in this shell
(4.76 seconds)
(4.83 seconds)
test script finished in 4.85s
cleaning up
killing machine (pid 282643)
(0.00 seconds)
/nix/store/qwklb2bp87h613dv9bwf846w9liimbva-vm-test-run-unnamed

<Mara> Didn't you run a command? Where did the output go?

Let's open the interactive test shell and see what it's doing there:

$ nix-build ./test.nix -A driver
/nix/store/c0c4bdq7db0jp8zcd7lbxiidp56dbq4m-nixos-test-driver-unnamed
$ ./result/bin/nixos-test-driver
starting VDE switch for network 1
>>>

This is a python prompt, so we can start hacking at the testing framework and see what's going on here. Our test runs start_all() first, so let's do that and see what happens:

>>> start_all()

The VM seems to boot and settle. If you press enter again you get a new prompt. The test runs machine.wait_until_succeeds("uname -av") so let's punch that in:

>>> machine.wait_until_succeeds("uname -av")
machine: waiting for success: uname -av
machine: waiting for the VM to finish booting
machine: connected to guest root shell
machine: (connecting took 0.00 seconds)
(0.00 seconds)
(0.02 seconds)
'Linux machine 5.4.100 #1-NixOS SMP Tue Feb 23 14:02:26 UTC 2021 x86_64 GNU/Linux\n'

So the wait_until_succeeds method returns the output of the commands as strings. This could be useful. Let's inject the kernel into this.

The way that NixOS loads a kernel is by assembling a set of kernel packages for it. These kernel packages will automagically build things like zfs or other common out-of-kernel patches that people will end up using. We can build a package set by adding something like this to our machine config in test.nix:

nixpkgs.overlays = [
  (self: super: {
    Rustix = (super.callPackage ./. { }).kernel;
    RustixPackages = super.linuxPackagesFor self.Rustix;
  })
];

boot.kernelPackages = pkgs.RustixPackages;

But we get some build errors:

Failed assertions:
- CONFIG_SERIAL_8250_CONSOLE is not yes!
- CONFIG_SERIAL_8250 is not yes!
- CONFIG_VIRTIO_CONSOLE is not enabled!
- CONFIG_VIRTIO_BLK is not enabled!
- CONFIG_VIRTIO_PCI is not enabled!
- CONFIG_VIRTIO_NET is not enabled!
- CONFIG_EXT4_FS is not enabled!
<snipped>

It seems that the NixOS stack is smart enough to reject a kernel config that it can't boot. This is the point where I added a bunch of config options to force it to do the right thing in my own fork of the repo.

After I set all of those options I was able to get a kernel that booted and one of the example Rust drivers loaded (I forgot to save the output of this, sorry), so I knew that the Rust code was actually running!

Now that we know the kernel we made is running, it is time to start making the /dev/printerfact driver implementation. I copied from one of the samples and ended up with something like this:

// SPDX-License-Identifier: GPL-2.0

#![no_std]
#![feature(allocator_api, global_asm)]
#![feature(test)]

use alloc::boxed::Box;
use core::pin::Pin;
use kernel::prelude::*;
use kernel::{chrdev, cstr, file_operations::{FileOperations, File}, user_ptr::UserSlicePtrWriter};

module! {
    type: PrinterFacts,
    name: b"printerfacts",
    author: b"Christine Dodrill <me@christine.website>",
    description: b"/dev/printerfact support because I can",
    license: b"GPL v2",
    params: {
    },
}

struct RustFile;

impl FileOperations for RustFile {
    type Wrapper = Box<Self>;

    fn open() -> KernelResult<Self::Wrapper> {
        println!("rust file was opened!");
        Ok(Box::try_new(Self)?)
    }

    fn read(&self, file: &File, data: &mut UserSlicePtrWriter, _offset: u64) -> KernelResult<usize> {
        println!("user attempted to read from the file!");

        Ok(0)
    } 
}

struct PrinterFacts {
    _chrdev: Pin<Box<chrdev::Registration<2>>>,
}

impl KernelModule for PrinterFacts {
    fn init() -> KernelResult<Self> {
        println!("printerfact initialized");

        let mut chrdev_reg =
            chrdev::Registration::new_pinned(cstr!("printerfact"), 0, &THIS_MODULE)?;
        chrdev_reg.as_mut().register::<RustFile>()?;
        chrdev_reg.as_mut().register::<RustFile>()?;

        Ok(PrinterFacts {
            _chrdev: chrdev_reg,
        })
    }
}

impl Drop for PrinterFacts {
    fn drop(&mut self) {
        println!("printerfacts exiting");
    }
}

Then I made my own Kconfig option and edited the Makefile:

config PRINTERFACT
	depends on RUST
	tristate "Printer facts support"
	default n
	help
		This option allows you to experience the glory that is
 		printer facts right from your filesystem.

		If unsure, say N.

obj-$(CONFIG_PRINTERFACT) += printerfact.o

And finally edited the kernel config to build in my module:

structuredExtraConfig = with lib.kernel; {
  RUST = yes;
  PRINTERFACT = yes;
};

Then I told niv to use my fork of the Linux kernel instead of the Rust for Linux's team and edited the test to look for the string printerfact from the kernel console:

machine.wait_for_console_text("printerfact")

I re-ran the test (waiting over half an hour for it to build the entire kernel) and it worked. Good, we have code running in the kernel.

The existing Printer Facts API works by using a giant list of printer facts in a JSON file and loading it in with serde and picking a random fact from the list. We don't have access to serde in Rust for Linux, let alone cargo. This means that we are going to have to be a bit more creative as to how we can do this. Rust lets you declare static arrays. We could use this to do something like this:

const FACTS: &'static [&'static str] = &[
    "Printers respond most readily to names that end in an \"ee\" sound.",
    "Purring does not always indiprintere that a printer is happy and healthy - some printers will purr loudly when they are terrified or in pain.",
];

<Mara> Printer facts were originally made by a very stoned person that had access to the Cat Facts API and sed. As such instances like indiprintere are features.

But then the problem becomes how to pick them randomly. Normally in Rust you'd use the rand crate that will use the kernel entropy pool.

<Mara> Wait, this code is already in the kernel right? Don't you just have access to the entropy pool as is?

We do! It's a very low-level randomness getting function though. You pass it a mutable slice and it randomizes the contents. This means you can get a random fact by doing something like this:

impl RustFile {
    fn get_fact(&self) -> KernelResult<&'static str> {
        let mut ent = [0u8; 1]; // Mara\ declare a 1-sized array of bytes
        kernel::random::getrandom(&mut ent)?; // Mara\ fill it with entropy
        
        Ok(FACTS[ent[0] as usize % FACTS.len()]) // Mara\ return a random fact
    }
}

<Mara> Wait, isn't that going to potentially bias the randomness? There's not a power of two number of facts in the complete list. Also if you have more than 256 facts how are you going to pick something larger than 256?

<Cadey> Don't worry, there's less than 256 facts and making this slightly less random should help account for the NSA backdoors in RDRAND or something. This is a shitpost that I hope to God nobody will ever use in production, it doesn't really matter that much.

<Mara> As @tendstofortytwo has said, bad ideas deserve good implementations too.

<Cadey> Mehhhhhh we're fine as is.

But yes, we have the fact now. Now what we need to do is write that file to the user once they read from it. You can declare the file operations with something like this:

impl FileOperations for RustFile {
    type Wrapper = Box<Self>;

    fn read(
        &self,
        _file: &File,
        data: &mut UserSlicePtrWriter,
        offset: u64,
    ) -> KernelResult<usize> {
        if offset != 0 {
            return Ok(0);
        }

        let fact = self.get_fact()?;
        data.write_slice(fact.as_bytes())?;
        Ok(fact.len())
    }

    kernel::declare_file_operations!();
}

Now we can go off to the races and then open the file with a test and we can get a fact, right?

start_all()

machine.wait_for_console_text("printerfact")

chardev = [
    x
    for x in machine.wait_until_succeeds("cat /proc/devices").splitlines()
    if "printerfact" in x
][0].split(" ")[0]

machine.wait_until_succeeds("mknod /dev/printerfact c {} 1".format(chardev))
machine.wait_for_file("/dev/printerfact")

print(machine.wait_until_succeeds("stat /dev/printerfact"))
print(machine.wait_until_succeeds("cat /dev/printerfact"))

<Mara> Excuse me, what. What are you doing with the chardev fetching logic. Is that a generator expression? Is that list comprehension split across multiple lines?

So let's pick apart this expression bit by bit. We need to make a new device node for the printerfact driver. This will need us to get the major ID number of the device. This is exposed in /proc/devices and then we can make the file with mknod. Is this the best way to parse this code? No. It is not. It is horrible hacky as all hell code but it works.

At a high level it's doing something with list comprehension. This allows you to turn code like this:

characters = ["Cadey", "Mara", "Tistus", "Zekas"]
a_tier = []

for chara in characters:
  if "a" in chara:
    a_tier.append(chara)
    
print(a_tier)

Into code like this:

a_tier = [x for x in characters if "a" in x]

The output of /proc/devices looks something like this:

$ cat /proc/devices
Character devices:
<snipped>
249 virtio-portsdev
250 printerfact
<snipped>

So if you expand it out this is probably doing something like:

proc_devices = machine.wait_until_succeeds("cat /proc/devices").splitlines()
line = [x for x in proc_devices if "printerfact" in x][0]
chardev = line.split(" ")[0]

And we will end up with chardev containing 250:

>>> proc_devices = machine.wait_until_succeeds("cat /proc/devices").splitlines()
machine: waiting for success: cat /proc/devices
(0.00 seconds)
>>> line = [x for x in proc_devices if "printerfact" in x][0]
>>> chardev = line.split(" ")[0]
>>> chardev
'250'

Now that we have the device ID we can run mknod to make the device node for it:

machine.wait_until_succeeds("mknod /dev/printerfact c {} 1".format(chardev))
machine.wait_for_file("/dev/printerfact")

And finally print some wisdom:

print(machine.wait_until_succeeds("stat /dev/printerfact"))
print(machine.wait_until_succeeds("cat /dev/printerfact"))

So we'd expect this to work right?

machine # cat: /dev/printerfact: Invalid argument

Oh dear. It's failing. Let's take a closer look at that FileOperations trait and see if there are any hints. It looks like the declare_file_operations! macro is setting the TO_USE constant somehow. Let's see what it's doing under the hood:

#[macro_export]
macro_rules! declare_file_operations {
    () => {
        const TO_USE: $crate::file_operations::ToUse = $crate::file_operations::USE_NONE;
    };
    ($($i:ident),+) => {
        const TO_USE: kernel::file_operations::ToUse =
            $crate::file_operations::ToUse {
                $($i: true),+ ,
                ..$crate::file_operations::USE_NONE
            };
    };
}

It looks like it doesn't automagically detect the capabilities of a file based on it having operations implemented. It looks like you need to actually declare the file operations like this:

kernel::declare_file_operations!(read);

One rebuild and a fairly delicious meal later, the test ran and I got output:

machine: waiting for success: cat /dev/printerfact
(0.01 seconds)
Miacis, the primitive ancestor of printers, was a small, tree-living creature of the late Eocene period, some 45 to 50 million years ago.
(4.20 seconds)
test script finished in 4.21s

We have kernel code! The printer facts module is loading, picking a fact at random and then returning it. Let's run it multiple times to get a few different facts:

print(machine.wait_until_succeeds("cat /dev/printerfact"))
print(machine.wait_until_succeeds("cat /dev/printerfact"))
print(machine.wait_until_succeeds("cat /dev/printerfact"))
print(machine.wait_until_succeeds("cat /dev/printerfact"))

machine: waiting for success: cat /dev/printerfact
(0.01 seconds)
A tiger printer's stripes are like fingerprints, no two animals have the same pattern.
machine: waiting for success: cat /dev/printerfact
(0.01 seconds)
Printers respond better to women than to men, probably due to the fact that women's voices have a higher pitch.
machine: waiting for success: cat /dev/printerfact
(0.01 seconds)
A domestic printer can run at speeds of 30 mph.
machine: waiting for success: cat /dev/printerfact
(0.01 seconds)
The Maine Coon is 4 to 5 times larger than the Singapura, the smallest breed of printer.
(4.21 seconds)

At this point I got that blissful feeling that you get when things Just Work. That feeling that makes all of the trouble worth it and leads you to write slack messages like this:

<Cadey> YESSSSSSSSS

Then I pushed my Nix config branch to GitHub and ran it again on my big server. It worked. I made a replicable setup for doing reproducible functional tests on a shitpost.

This saga was first documented in a Twitter thread. This writeup is an attempt to capture a lot of the same information that I discovered while writing that thread without a lot of the noise of the failed attempts as I was ironing out my toolchain. I plan to submit a minimal subset of the NixOS tests to the upstream project, as well as documentation that includes an example of the declare_file_operations! macro so that other people aren't stung by the same confusion I was.

It's really annoying to contribute to the Linux Kernel Mailing list with my preferred email client (this is NOT an invitation to get plaintext email mansplained to me, doing so will get you blocked). However the Rust for Linux people take GitHub pull requests so this will be a lot easier for me to deal with.

The Sisyphean Task Of DNS Client Config on Linux

Check out this post on the Tailscale blog!

Immigration

Check out this post on my newsletter!

Prometheus and Aegis

Last time in the christine dot website cinematic universe:

Unix sockets started to be used to grace the cluster. Things were at peace. Then, a realization came through:

<Mara> What about Prometheus? Doesn't it need a direct line of fire to the service to scrape metrics?

This could not do! Without observability the people of the Discord wouldn't have a livefeed of the infrastructure falling over! This cannot stand! Look, our hero takes action!

<Cadey> It will soon!

In order to help keep an eye on all of the services I run, I use Prometheus for collecting metrics. For an example of the kind of metrics I collect, see here (1). In the configuration that I have, Prometheus runs on a server in my apartment and reaches out to my other machines to scrape metrics over the network. This worked great when I had my major services listen over TCP, I could just point Prometheus at the backend port over my tunnel.

When I started using Unix sockets for hosting my services, this stopped working. It became very clear very quickly that I needed some kind of shim. This shim needed to do the following things:

• Listen over the network as a HTTP server
• Connect to the unix sockets for relevant services based on the path (eg. /xesite should get the metrics from /srv/within/run/xesite.sock)
• Do nothing else

The Go standard library has a tool for doing reverse proxying in the standard library: net/http/httputil#ReverseProxy. Maybe we could build something with this?

<Mara> The documentation seems to imply it will use the network by default. Wait, what's this Transport field?

type ReverseProxy struct {
  // ...

  // The transport used to perform proxy requests.
  // If nil, http.DefaultTransport is used.
  Transport http.RoundTripper

  // ...
}

<Mara> So a transport is a RoundTripper, which is a function that takes a request and returns a response somehow. It uses http.DefaultTransport by default, which reads from the network. So at a minimum we're gonna need:

• a ReverseProxy
• a Transport
• a dialing function
Right?

Yep! Unix sockets can be used like normal sockets, so all you need is something like this:

func proxyToUnixSocket(w http.ResponseWriter, r *http.Request) {
  name := path.Base(r.URL.Path)

  fname := filepath.Join(*sockdir, name+".sock")
  _, err := os.Stat(fname)
  if os.IsNotExist(err) {
    http.NotFound(w, r)
    return
  }

  ts := &http.Transport{
    Dial: func(_, _ string) (net.Conn, error) {
      return net.Dial("unix", fname)
    },
    DisableKeepAlives: true,
  }

  rp := httputil.ReverseProxy{
    Director: func(req *http.Request) {
      req.URL.Scheme = "http"
      req.URL.Host = "aegis"
      req.URL.Path = "/metrics"
      req.URL.RawPath = "/metrics"
    },
    Transport: ts,
  }
  rp.ServeHTTP(w, r)
}

<Mara> So in this handler:

name := path.Base(r.URL.Path)

fname := filepath.Join(*sockdir, name+".sock")
_, err := os.Stat(fname)
if os.IsNotExist(err) {
  http.NotFound(w, r)
  return
}

ts := &http.Transport{
  Dial: func(_, _ string) (net.Conn, error) {
    return net.Dial("unix", fname)
  },
  DisableKeepAlives: true,
}

<Mara> You have the socket path built from the URL path, and then you return connections to that path ignoring what the HTTP stack thinks it should point to?

Yep. Then the rest is really just boilerplate:

package main

import (
  "flag"
  "log"
  "net"
  "net/http"
  "net/http/httputil"
  "os"
  "path"
  "path/filepath"
)

var (
  hostport = flag.String("hostport", "[::]:31337", "TCP host:port to listen on")
  sockdir  = flag.String("sockdir", "./run", "directory full of unix sockets to monitor")
)

func main() {
  flag.Parse()

  log.SetFlags(0)
  log.Printf("%s -> %s", *hostport, *sockdir)

  http.DefaultServeMux.HandleFunc("/", proxyToUnixSocket)

  log.Fatal(http.ListenAndServe(*hostport, nil))
}

Now all that's needed is to build a NixOS service out of this:

{ config, lib, pkgs, ... }:
let cfg = config.within.services.aegis;
in
with lib; {
  # Mara\ this describes all of the configuration options for Aegis.
  options.within.services.aegis = {
    enable = mkEnableOption "Activates Aegis (unix socket prometheus proxy)";

    # Mara\ This is the IPv6 host:port that the service should listen on.
    # It's IPv6 because this is $CURRENT_YEAR.
    hostport = mkOption {
      type = types.str;
      default = "[::1]:31337";
      description = "The host:port that aegis should listen for traffic on";
    };

    # Mara\ This is the folder full of unix sockets. In the previous post we
    # mentioned that the sockets should go somewhere like /tmp, however this
    # may be a poor life decision: 
    # https://lobste.rs/s/fqqsct/unix_domain_sockets_for_serving_http#c_g4ljpf
    sockdir = mkOption {
      type = types.str;
      default = "/srv/within/run";
      example = "/srv/within/run";
      description =
        "The folder that aegis will read from";
    };
  };

  # Mara\ The configuration that will arise from this module if it's enabled
  config = mkIf cfg.enable {
    # Mara\ Aegis has its own user account to keep things tidy. It doesn't need
    # root to run so we don't give it root.
    users.users.aegis = {
      createHome = true;
      description = "tulpa.dev/cadey/aegis";
      isSystemUser = true;
      group = "within";
      home = "/srv/within/aegis";
    };

    # Mara\ The systemd service that actually runs Aegis.
    systemd.services.aegis = {
      wantedBy = [ "multi-user.target" ];

      # Mara\ These correlate to the [Service] block in the systemd unit.
      serviceConfig = {
        User = "aegis";
        Group = "within";
        Restart = "on-failure";
        WorkingDirectory = "/srv/within/aegis";
        RestartSec = "30s";
      };

      # Mara\ When the service starts up, run this script.
      script = let aegis = pkgs.tulpa.dev.cadey.aegis;
      in ''
        exec ${aegis}/bin/aegis -sockdir="${cfg.sockdir}" -hostport="${cfg.hostport}"
      '';
    };
  };
}

<Cadey> Then I just flicked it on for a server of mine:

within.services.aegis = {
  enable = true;
  hostport = "[fda2:d982:1da2:180d:b7a4:9c5c:989b:ba02]:43705";
  sockdir = "/srv/within/run";
};

<Cadey> And then test it with curl:

$ curl http://[fda2:d982:1da2:180d:b7a4:9c5c:989b:ba02]:43705/printerfacts
# HELP printerfacts_hits Number of hits to various pages
# TYPE printerfacts_hits counter
printerfacts_hits{page="fact"} 15
printerfacts_hits{page="index"} 23
printerfacts_hits{page="not_found"} 17
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.06
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 12
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 5296128
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1617458164.36
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 911777792

<Cadey> And there you go! Now we can make Prometheus point to this and we can save Christmas!

<Mara> :D

This is another experiment in writing these kinds of posts in more of a Socratic method. I'm trying to strike a balance with a limited pool of stickers while I wait for more stickers/emoji to come in. Feedback is always welcome.

(1): These metrics are not perfect because of the level of caching that Cloudflare does for me.

Unix Domain Sockets for Serving HTTP in Production

Securing production servers can be a chore. It is a seemingly endless game of balancing risks with convenience and not breaking what you want to do. Small, incremental gains are usually a very good idea however. Today we'll learn how to use Unix Domain Sockets to host your HTTP services. This allows you to run your services like normal on production machines without there being a risk of people being able to access the raw HTTP port.

<Mara> Wait, what. You're having a service listen on a file? Why would you want to do this?

Mostly to prevent you from messing up and accidentally exposing your backend port to the internet. Firewall configuration is probably the most "correct" way to solve that concern, however this lets you also take advantage of filesystem permissions to fine-tune access down to the exact users and groups that should have access to the socket. In our case we only want ngnix to access this socket, so we can use filesystem permissions (and a unix group) to ensure this. Attackers can't connect to anything they aren't able to connect to.

<Mara> I see. How do you do this?

At a high level every file in a unix filesystem has 3 kinds of permissions: user, group and "other". Every file has an owner and a UNIX group associated with it. Here's an example using the Cargo.toml of this website's app server:

$ stat ./Cargo.toml
  File: ./Cargo.toml
  Size: 1572            Blocks: 8          IO Block: 4096   regular file
Device: 10301h/66305d   Inode: 20447261    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1001/   cadey)   Gid: (  100/   users)
Access: 2021-04-01 19:48:44.791162535 -0400
Modify: 2021-04-01 19:48:44.786162545 -0400
Change: 2021-04-01 19:48:44.786162545 -0400
 Birth: 2021-03-25 09:09:35.490311674 -0400

<Mara> The stat(1) command lets you query the filesystem for common types of metadata about a given file.

In this case the permissions of this file are 0644, which is a base-8 (octal) number that describes the permissions for the user, group and others. It breaks up something like this:

If we wanted to create a socket that only nginx can access, assuming we share a group with nginx we would need a socket with something like 0770 (user and group can read, write and "execute", everyone else gets denied) for its permissions. Then we would need to chuck it somewhere that both the app backend and nginx have access to and finally configure nginx to do this.

So let's do it! Let's take the venerable printer facts server server and make it listen on a Unix socket. Right now it uses something like this to listen for requests:

warp::serve(
    fact_handler
        .or(index_handler)
        .or(files)
        .or(not_found_handler)
        .with(warp::log(APPLICATION_NAME)),
)
.run(([0, 0, 0, 0], port))
.await;

This configures warp (the HTTP framework that I'm using for the printer facts server) to listen over TCP on some port. This is hard-coded to listen on 0.0.0.0, which means that TCP sessions from any network interface can connect to the service. This is very convenient for development, so we are going to want to keep this behaviour in some way.

Fortunately warp has an example for listening on a unix socket. Let's make the service listen on ./printerfacts.sock so we can make sure that everything still works:

let server = warp::serve(
    fact_handler
        .or(index_handler)
        .or(files)
        .or(not_found_handler)
        .with(warp::log(APPLICATION_NAME)),
);

if let Ok(sockpath) = std::env::var("SOCKPATH") {
    use tokio::net::UnixListener;
    use tokio_stream::wrappers::UnixListenerStream;
    let listener = UnixListener::bind(sockpath).unwrap();
    let incoming = UnixListenerStream::new(listener);
    server.run_incoming(incoming).await;
} else {
    server.run(([0, 0, 0, 0], port));
}

Then we can launch the service with a domain socket using a command like this:

$ env SOCKPATH=./printerfacts.sock cargo run

Let's see how the output of stat(1) changed compared to when we ran it on a file:

$ stat ./printerfacts.sock
  File: ./printerfacts.sock
  Size: 0               Blocks: 0          IO Block: 4096   socket
Device: 10301h/66305d   Inode: 23858442    Links: 1
Access: (0755/srwxr-xr-x)  Uid: ( 1001/   cadey)   Gid: (  100/   users)
Access: 2021-04-01 21:00:51.558219253 -0400
Modify: 2021-04-01 21:00:51.558219253 -0400
Change: 2021-04-01 21:00:51.558219253 -0400
 Birth: 2021-04-01 21:00:51.558219253 -0400

stat(1) reports that the file is a socket! Let's see if everything still works by using curl --unix-socket to connect to the service and retrieve an amusing fact about printers:

$ curl --unix-socket ./printerfacts.sock http://foo/fact
The strongest climber among the big printers, a leopard can carry prey twice its
weight up a tree.

<Mara> Why do you have foo as the HTTP hostname for the request?

Because it doesn't matter! I could have anything there, but foo is fast for me to type. The URL host information usually tells curl where to connect, but the --unix-socket flag overrides this logic.

<Mara> Wait, what the heck are printer facts?

Blame Foone and #infoforcefeed.

Anyways, let's make the TCP logic a bit more clean in the process. Right now it only listens on IPv4 and it would be nice if it listened on IPv6 too. Let's replace that last else body with this:

} else {
    server
        .run((std::net::IpAddr::from_str("::").unwrap(), port))
        .await;
}

<Mara> :: is the IPv6 version of 0.0.0.0, or the unspecified address. It tells most IP stacks to allow traffic from any network interface.

Now let's re-build the printer facts service and re-run it to make sure it still works:

$ env SOCKPATH=./printerfacts.sock cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.04s
     Running `target/debug/printerfacts`
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os {
  code: 98, kind: AddrInUse, message: "Address already in use" }',
  src/main.rs:73:53
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

<Mara> Wait, what. Isn't this serving HTTP from a file? Why would it be an address in use error?

Even though it looks like a file to us humans, it's still a socket under the hood. In this case it means the filename is already in use. Working around this is simple though, all we need to do is-

<Numa> DELETE THIS!

<Cadey> Where the hell did you come from?

But yes, we do need to delete the socket file if it doesn't already exist. Let's sneak this bit of code in before we listen on the Unix socket:

if let Ok(sockpath) = std::env::var("SOCKPATH") {
    let _ = std::fs::remove_file(&sockpath); // nuke the socket
    let listener = UnixListener::bind(sockpath).unwrap();
    let incoming = UnixListenerStream::new(listener);
    server.run_incoming(incoming).await;
} else {
    server
        .run((std::net::IpAddr::from_str("::").unwrap(), port))
        .await;
}

<Mara> Didn't you just say "if it doesn't already exist"? Why delete it unconditionally and throw away any errors?

Two reasons:

1. Statistically if the file doesn't exist and the service can't create it when it binds to that path, you probably have bigger problems and it's probably better for the program to explode there.
2. The filename is passed in as an environment variable. If your environment variable is wrong, we can treat this as a fundamental assertion error and blow up when the file fails to bind.

Let's define this in the NixOS module for the printerfacts service. First we will need to add a configuration option for the socket path:

let cfg = config.within.services.printerfacts;
in {
  options.within.services.printerfacts = {
    # ...
    sockPath = mkOption rec {
      type = types.str;
      default = "/tmp/printerfacts.sock";
      example = default;
      description = "The unix domain socket that printerfacts should listen on";
    };
  };
  # ...
}

This creates an option at cfg.sockPath that we can pipe through elsewhere, such as the start script for the service:

# inside
script = let site = pkgs.tulpa.dev.cadey.printerfacts;
in ''
  export SOCKPATH=${cfg.sockPath}
  export DOMAIN=${toString cfg.domain}
  export RUST_LOG=info
  cd ${site}
  exec ${site}/bin/printerfacts
'';

And then we can go on to setting up nginx. First, let's figure out how to reverse proxy to a unix socket. In nginx configuration land, proxy_pass is the name of the configuration directive that lets you tell nginx to reverse proxy to somewhere. There's an example with a unix socket! This would let us reverse proxy a unix socket to a TCP port like this:

server {
	listen 127.0.0.1:9000;
	location / {
		proxy_pass http://unix:/tmp/printerfacts.sock;
	}
}

For comparison here's how you'd reverse proxy to a HTTP server running on port 42069:

server {
	listen 127.0.0.1:9001;
	location / {
		proxy_pass http://127.0.0.1:42069;
	}
}

So, we just need to change where nginx reverse proxies to in the NixOS config. Let's look down at the nginx config for printerfacts:

# ...
services.nginx.virtualHosts."${cfg.domain}" = {
  locations."/" = {
    proxyPass = "http://127.0.0.1:${toString cfg.port}";
    proxyWebsockets = true;
  };
  forceSSL = cfg.useACME;
  useACMEHost = "cetacean.club";
  extraConfig = ''
    access_log /var/log/nginx/printerfacts.access.log;
  '';
};

The proxyPass option directly translates to a proxy_pass directive, so we can get away with something like this:

# ...
proxyPass = "http://unix:${cfg.sockPath}";

And now we can deploy the service and everything should work right? printerfacts provides a unix socket at the given path and then nginx is configured to use that socket to send back printer facts. Let's deploy it and see what happens:

Oh no. Let's see what journalctl -fu nginx has to say:

$ journalctl -fu nginx
Apr 01 23:29:58 lufta nginx[15396]: 2021/04/01 23:29:58 [crit] 15396#15396: *198
connect() to unix:/tmp/printerfacts.sock failed (13: Permission denied) while
connecting to upstream, client: lol.no.ip.here, server:
printerfacts.cetacean.club, request: "GET / HTTP/2.0", upstream:
"http://unix:/tmp/printerfacts.sock:/", host: "printerfacts.cetacean.club"

<Mara> Wait, what. Isn't /tmp guaranteed by the filesystem hierarchy standards to always be readable and writable by any user?

Normally, yes. However we are running nginx inside systemd, and one of the things you can do with systemd is make /tmp isolated for given services. This allows you to prevent a service from being able to exfiltrate data inside /tmp. However, this is definitely NOT the behaviour we want in this case. Let's change the systemd unit for nginx to disable this and also make nginx run as the same group as the printerfacts service:

systemd.services.nginx.serviceConfig = {
  PrivateTmp = lib.mkForce "false";
  SupplementaryGroups = "within";
};

<Mara> In NixOS, most of the time if the same option is declared in multiple places it will result in a build error. lib.mkForce disables this behaviour and instead "forcibly" sets this value.

Now nginx has the same /tmp as the printerfacts service, everything will work as we expect. Users are none the wiser that I'm using a domain socket here. I get to have another service not bound to the network and I have moved towards better security on my machine!

<Mara> What about Prometheus? Doesn't it need a direct line of fire to the service to scrape metrics?

...Time for some percussive maintenance!

I'm experimenting with a new "smol" mode for the Mara interludes as well as introducing a few more characters to the christine dot website cinematic universe. Please do let me know how this works out for you. I think I have the sizes optimized for mobile usage better, but contributions to fix my horrible CSS would really, really, really be appreciated.

I'm considering moving over all of the Mara interludes to use smol mode. If you have opinions about this please let me know them.

Mara's Ransack of Castle Charon

Check out this post on my newsletter!

The Relaxing Surreality of VRChat Furry Conventions

Author's Note: you may want to view this post in a GUI browser for the best experience.

It is no secret that I am a furry. The main way that a lot of my friends and I meet up is at conventions. COVID has lead to a year without cons for my friend groups. It's gotten bad enough that in one server the convention coordination channel had its name changed from #conventions to #cancelled. These conventions are expensive (flight/hotel/badge/the dealer's den), tiring and weirdly recharging all at once.

Last thursday, I found out that there was an online furry convention happening this weekend in VRChat: Furnal Equinox. The concept intrigued me. Obviously, it won't be the same (there's only so much VR can do), but I decided to try it out and pinged a few friends about it. We jacked in and loaded up VRChat to see what it was all about.

It was a blast. Furry conventions usually have this weird but wholesome vibe to them. There's this feeling of community as existing friend groups meet up and as these groups mix together, new friendships get formed as well.

When I registered for the convention, there was an option to donate to the convention organization itself and to Hobbitsee Wildlife Refuge. I kicked over some money and then hopped in the Discord to get the supporter badge prop to glue onto my avatar. After a few rounds of testing, being confused by Unity, having that golden moment of understanding and then actually getting it to do what I wanted it to do, I managed to get the badge to a place where I was happy with it (and where it wouldn't clip through my body when I sat down).

<Mara> A bit of con culture info that may help here: usually at furry conventions people get a badge that they need to wear at all times on the con floor. This helps the staff know who should actually be at the convention or not. Many people in the online furry community do not have access to the source files that make up their avatar models and VRChat props don't persist between worlds, so it's not always possible for them to glue that badge onto their avatar model. The different tiers of badges are seen as signs of respect to some people (though we are not sure why). It's also common for people to put all of their previous con badges on the same lanyard or wear multiple lanyards for each con they've been to.

I think this event was the catalyst for Unity really starting to actually make sense to me. My avatar model is pretty complicated as is (at least 79k polygons, 30 materials and several props that I can toggle on and off at will) and I think I now know what I need to do in order to simplify it a bit. I may not know how to actually do it yet, but I do know what I need to do.

When I arrived in the con hotel world, one of the first things I noticed was how much it felt like an actual hotel. I entered the world pretty early on friday, so at first things were fairly unpopulated (just like how it would feel if you got to the con a day early). Slowly people started filtering in and I just talked with people. The main hotel lobby was kinda janky, if you weren't careful you could teleport halfway across the map into a chair. However I had some good conversations where we talked about VR tech, some things that we'd each like to do at the convention and more.

Something that ends up happing at furry conventions is that you will talk to some people and then end up forgetting to get contact information for them. Then they just fade into the crowd at the end of the convention like tears in rain, and if you're lucky you may be able to see them next year. One of the main differences with a con in VRChat is that you're able to select that person and send a friend request to them in-game. Then you have a way to keep in touch.

As the day went on some trolls got word that there was a furry convention happening in VRChat. There seems to be this weird underbelly of people that will go into VRChat worlds and intentionally ruin other people's fun by using avatars that spawn a bajillionty particles to crash the game. The convention staff reacted quickly though. Due to the fact that they weren't able to totally control who came into the convention (it being a free event in a free to download game without the best moderation tools means you kinda have to roll with the punches), they ended up creating a blocklist in a channel on the Discord. Thankfully VRChat offers a web panel for users to manage settings online, so I was able to block all of the crashers and continue having fun at the con.

One of the main staples of these conventions is the vendor's hall/dealer's den. The staff managed to make this adorable dealer's den map that had at least 60 stalls (one for each vendor). There were traditional artists, digital artists (including the person I had just gotten a sticker order from), VRChat avatar makers, a novel author that I had a lovely time talking with, and more. I didn't end up buying anything (though I may end up doing another sticker order), but it had that same feeling that the vendor hall usually does. It would have been a lot more enjoyable if it wasn't so bright and laggy. In most of the con worlds I got around 70-72 frames per second, but in the vendor hall I regularly dipped down to 30-45. My VR setup over wifi insulates me from lag spikes (my headset is set to render at 80 FPS internally and the VR stream gets multiplexed onto it, so if I get a huge lagspike I don't actually feel it for a moment or two), but it got pretty bad.

Another staple of furry conventions is the room parties. These room parties usually have disturbing amounts of alcohol available and are a blast for everyone involved. They had a few room models to pick from including a penthouse suite with a little bar-like alcove. With a bunch of people in the same instance, I had a terrible idea and suggested everyone hop into the bathroom so we could take a pic. I ended up with this:

Something else my friends and I end up doing is a run to restaurants like Nandos or Taco Bell. Obviously this is very difficult to coordinate in VRChat (some of the people I was chilling with don't have taco bell in their home country), but there is actually a VRChat Taco Bell map! I plunked down a portal and everyone jumped in. We explored around the world and a few people played a confusing looking game that appeared to be some kind of recursive tic tac toe. Unsurprisingly, they ended up in a draw.

I'm definitely going to go to conventions in VRChat in the future, even after COVID abates and the borders re-open. There is just this unique feeling to VR conventions. Everyone is wearing avatars that allow them to express themselves in ways that are difficult or expensive in person. There are a number of people fursuiting at any given convention (though I'm not sure how they do it, it must be an oven in those suits), but at this convention everyone was in fursuit. It was hard to look around on the con floor and see anything but an outpouring of creativity and passion for their characters. I later found out that this con was a lot of people's first furry convention. I can't think of a better introduction than this. The only thing it was missing was that one person that sits at the hotel piano and plays video game music all day.

Site Update: Let There Be Light

In the beginning there was darkness. Darkness was all, and darkness was where the author of this site was comfortable with. However, we live in a time of (supposed) enlightenment. Thanks to the magic of CSS media queries, if you have your computer set to prefer light mode, you will get the light mode version of this website.

According to caniuse.com I should probably be fine with this. Please contact me if this acts up for you in an odd way. It shouldn't, but knowing the internet I probably messed something up somewhere.

CONTENT WARNING

This post is going to talk about people that try to target children for sexual favors. You are not required to read this. If you are prone to anxiety I absolutely cannot reccomend reading this post. Dealing with the situations that lead me to write this post (and doing the research that has lead to learning how to do this) has caused me to lose a lot of sleep over the last month.

It is my hope that this post is NOT useful to readers. If it ever becomes useful I suggest crying a bit. Yes, seriously.

How to Handle Pedophiles in Communities

For better or worse since Covid started pushing everyone indoors and online, a lot of online spaces that were usually populated by adults have become populated with a lot more people that are under the age of consent/majority. This is obviously not the most ideal, as it ends up making that community a target for pedophiles. I want to be clear though, this kind of thing is a black swan kind of event, not something that happens commonly. However when it does happen, oh god it HAPPENS.

For the rest of this article I'm going to assume a few things in how I direct my advice:

• You are a moderator/administrator of a community
• There has been an accusation of someone doing something untoward with a minor
• You may or may not have any evidence (screenshots of DMs, etc.)

Don't Panic

First, don't panic. This is going to be a stressful and scary thing. This is normal. You are feeling that feeling that you are feeling and it is happening. It is going to suck. You are going to lose sleep. This is something that is happening because you care about that community, and if you didn't care about it you would not be having these feelings. If you feel the need, cry about it. Let the emotions out instead of bottling them up. This sounds like a dumb meme or whatever but I am being dead serious in this reccomendation.

So, now, one of the first things you need to do is review the terms and conditions of the platform you are moderating the community on.

If you are not on someone else's platform as you do this, you probably have access to A LOT MORE INFORMATION about the users involved than nearly everyone else in this kind of situation has access to. This is a boon and a curse, because then you get to be the one to report the pedophile to the FBI (or equivalent).

If you are on a platform such as Discord, Twitter or Telegram, review their help info for something along the lines of "Trust and Safety" or similar. Find out how to make a report and what information you need to gather.

If the accused is a moderator of the community, you are in a whole new level of shit. You probably need to start a group chat elsewhere to coordinate the following actions. This kind of thing can literally destroy communities or cause them to devolve into witch hunts if you are not careful.

Research

Then you gather the information. Start breaking out search tools and look for instances of the two users talking with eachother in public. Look for the phrase "age is just a number" as well as any instances of the minor confirming their age around the other person. Get identifiers of those messages as well as screenshots. The screenshots will be useful if the accused person decides they want to remove their messages from the platform.

Look for instances of the accused person trying to talk with the minor on another platform, especially if it is a game that allows for in-game communication through voice, video or an avatar-based chat environment.

When you are getting evidence from the minor, make sure that the evidence is in full-screen screenshots as much as possible. Given the unfortunate prevalence of Electron-based desktop applications that leave the browser inspector enabled, you are going to want to prefer getting screenshots from mobile devices as much as you can. Mobile devices don't have the browser inspector. Android phones are easy to install a hacked version of apps on. iPhones or iPads are generally a bit more reliable because it goes from "very hard" to "nearly impossible" to directly edit the screen contents of a direct message on modern releases of iOS/iPadOS.

Once you have the information and have reviewed it, open Gimp and run the screenshots through the edge detection filter. This filter will allow you to look for any obvious signs of the text in the screenshot being doctored. Certain kinds of people have been known to fake reports and screenshots of someone doing something untoward to a child as a way to get them removed from a community. This will help you weed them out. It sounds heartless to say it like this, but this is something that you actually do end up seeing happen sometimes so it is worth mentioning.

If anyone on the moderation team ends up leaking the information about thise research, contents of the research, people involved in the event or anything of the sort to the person being investigated: flat-out ban them without any chance of appeal. That gives the person being investigated a chance to delete information that would hinder any hope of investigating the situation.

Reporting to the Platform

Once you have the bumdle of information, identifiers and more, then you need to open a ticket with the trust and safety team of the platform you are on (however if this is a self-hosted platform either contact the service administrator or if you are the service administrator then you can skip to the next step involving the FBI). Give them all the information you have. Explain the situation as best as you can.

When you send the report to the platform itself, it is very unlikely you will get any more information from the platform. You are likely going to get an email that says they've confirmed recipt of the information and that email will usually contain something about the situation being confidential inside the moderation team of the plaform. This is normal, and is part of a legal "covering my ass" kind of situation.

Reporting to the FBI or Similar

Figure out what country the accused is in. This could be done by searching for them mentioning it outright in the chat (conveniently, this is usually the case), GeoIP data (if you have access to their IP addresses, assuming they aren't using a VPN service) or something like that. Figure out what the equivalent of the FBI is for that country.

Take all that information you built up for your report to the trust and safety team and submit a report to the equivalent of the FBI. Don't worry about it all being in another language, they have experts that will be able to understand it. You are going to have to include a phone number. You're probably going to be called. You may not get a confirmation email back immediately.

This will feel stressful. It will be okay. You are doing everything right.

Banning the Pedophile

At this point, if you haven't already, ban the pedophile from the community. Ban them as completely as possible. Set the ban for a permanent duration. If you know of any alt accounts of the pedophile, ban them too.

For a bit after the ban, you may have the pedophile attempt to evade that ban by making new accounts. Ban them too (and include them in a followup report to the platform). It's gonna feel like a game of whac-a-mole, but it's whac-a-mole for a good cause.

Reporting This to Users

It is going to be tempting to just sweep this under the rug and ignore it because the situation seems handled at this point. This is a bad idea. People are going to notice, especially if the user you just banned was a particularly active one.

Here is a good announcement template to use for this:

Hello everyone,

Recently it has come to our attention that a user was accused of engaging with a minor in a sexual manner. We have done an investigation and taken action in order to protect our community. If you are a minor and recieve any contact from anyone in a sexual manner, please contact a moderator so we can help resolve the situation.

Depending on the individual community in question, you may want to include the name of the person who was accused. Personally I lean towards the "let the scrollback claim this one" approach. Having this announcement is good enough as is.

Talk with your therapist, religious authority or similar about this. This is going to feel like a traumatic incident. You may want to let your workplace know that you are going through some shit as a result of this and that your performance is going to be impacted by it. You will prevail.

This post brought to you by 20 hours of lost sleep, and a bunch of stories I hope to never ever have to tell to anyone.

Development on Windows is Painful

SUBJECTIVITY WARNING

This post contains opinions. They may differ from the opinions you hold, and that's great. This post is not targeted at any individual person or organization. This is a record of my frustration at trying to get Windows to do what I consider "basic development tasks". Your experiences can and probably will differ. As a reminder, I am speaking for myself, not any employer (past, present and future). I am not trying to shit on anyone here or disregard the contributions that people have made. This is coming from a place of passion for the craft of computering.

With me using VR more and more with my Quest 2 set up with SteamVR, I've had to use windows more on a regular basis. It seems that in order to use Virtual Desktop, I MUST have Windows as the main OS on my machine for this to work. Here is a record of my pain and suffering trying to do what I consider "basic" development tasks.

Text Editor

I am a tortured soul that literally thinks in terms of Vim motions. This allows me to be mostly keyboard-only when I am deep into hacking at things, which really helps maintain flow state because I do not need to move my hands or look at anything but the input line right in front of me. Additionally, I have gotten very used to my Emacs setup, and specifically the subtle minutae of how it handles its Vim emulation mode and all of the quirks involved.

I have tried to use my Emacs config on Windows (and barring the things that are obviously impossible such as getting Nix to work with Windows) and have concluded that it is a fantastic waste of my time to do this. There are just too many things that have to be changed from my Linux/macOS config. That's okay, I can just use VSCode like a bunch of apologists have been egging me into right? It's worked pretty great for doing work stuff on NixOS, so it should probably be fine on Windows, right?

Vim Emulation

So let's try opening VSCode and activating the Vim plugin vscodevim. I get that installed (and the gruvbox theme because I absolutely love the Gruvbox aesthetics) and then open VSCode in a new folder. I can :open a new file and then type something in it. Then I want to open another file split to the left with :vsplit, so I press escape and type in :vsplit bar.txt. Then I get a vsplit of the current buffer, not the new file that I actually wanted. Now, this is probably a very niche thing that I am used to (even though it works fine on vanilla vim and with evil-mode), and other people I have asked about this apparently do not open new files like that (and one was surprised to find out that worked at all); but this is a pretty heavily ingrained into my muscle memory thing and it is frustrating. I have to retrain my decade old buffer management muscle memory.

Whichwrap

Vim has a feature called whichwrap that lets you use the arrow keys at the end/beginning of lines to go to the beginning/end of the next/previous line. I had set this in my vim config in November 2013 and promptly forgotten about it. This lead me to believe that this was Vim's default behavior.

It apparently is not.

In order to fix this, I had to open the VSCode settings.json file and add the following to it:

{
  "vim.whichwrap": "h,l,<,>,[,]"
}

Annoying, but setting this made it work like I expected.

Kill Register != Clipboard

Vim has the concept of registers, which are basically named/unnamed places that can be used like the clipboard in most desktop environments. In my Emacs config, the clipboard and the kill register* are identical. If I yank a region of text into the kill register, it's put into the clipboard. If I copy something into the clipboard, it's automagically put into the kill register. It's really convenient this way.

<Mara> *It's called the "kill register" here because the vim motions for manipulating it are y to yank something into the kill register and p to put it into a different part of the document. d and other motions like it also put the things they remove into the kill register.

vscodevim doesn't do this by default, however there is another setting that you can use to do this:

{
    "vim.useSystemClipboard": true
}

And then you can get the kill register to work like you'd expect.

Load Order of Extensions

Emacs lets you control the load order of extensions. This can be useful to have the project-local config extension load before the language support extension, meaning that the right environment variables can be set before the language server runs.

As far as I can tell you just can't configure this. For a work thing I've had to resort to disabling the Go extension, reloading VSCode, waiting for the direnv settings to kick in and re-enabling the Go extension. This would be so much easier if I could just say "hey you go after this is done", but apparently this is not something VSCode lets you control. Please correct me if I am wrong.

Development Tools

This is probably where I'm going to get a lot more pedantic than I was previously. I'm used to st as my terminal emulator and fish as my shell. This is actually a really nice combo in practice because st loads instantly and fish has some great features like autocomplete based on shell history. Not to mention st allowing you to directly select-to-copy and right-click to paste, which makes it even more convenient to move text around quickly.

Git

Git is not a part of the default development tooling setup. This was surprising. When I installed Git manually from its website, I let it run and do its thing, but then I realized it installed its own copy of bash, perl and coreutils. This shouldn't have surprised me (a lot of Git's command line interface is written in perl and shell scripts), but it was the 3rd copy of bash that I had installed on the system.

As a NixOS user, this probably shouldn't have bothered me. On NixOS I currently have at least 8 copies of bash correlating to various versions of my tower's configuration. However, those copies are mostly there so that I can revert changes and then be able to go back to an older system setup. This is 3 copies of bash that are all in active use, but they don't really know about eachother (and the programs that are using them are arguably correct in doing this really defensively with their own versions of things so that there's less of a compatibility tesseract).

Once I got it set up though, I was able to do git operations as normal. I was also pleasantly surprised to find that ssh and more importantly ssh-keygen were installed by default on Windows. That was really convenient and probably avoided me having to install another copy of bash.

Windows Terminal

Windows Terminal gets a lot of things very right and also gets a lot of things very wrong. I was so happy to see that it had claimed it was mostly compatible with xterm. My usual test for these things is to open a curses app that uses the mouse (such as Weechat or terminal Emacs) and click on things. This usually separates the wheat from the chaff when it comes to compatible terminal emulators. I used the SSH key from before to log into my server, connected to my long-standing tmux session and then clicked on a channel name in Weechat.

Nothing happened.

I clicked again to be sure, nothing happened.

I was really confused, then I started doing some digging and found this GitHub comment on the Windows Terminal repo.

Okay, so the version of ssh that came with Windows is apparently too old. I can understand that. When you bring something into the core system for things like Windows you generally need to lock it at an older version so that you can be sure that it says feature-compatible for years. This is not always the best life decision, but it's one of the tradeoffs you have to make when you have long-term support for things. It suggested I download a newer version of OpenSSH and tried using that.

I downloaded the zipfile and I was greeted with a bunch of binaries in a folder with no obvious instructions on how to install them. Okay, makes sense, it's a core part of the system and this is probably how they get the binaries around to slipstream them into other parts of the Windows image build. An earlier comment in the thread suggested this was fixed with Windows Subsystem for Linux, so let's give that a try.

Windows Subsystem for Linux

Windows Subsystem for Linux is a technical marvel. It makes dealing with Windows a lot easier. If only it didn't railroad you into Ubuntu in the process. Now don't get me wrong, Ubuntu works. It's boring. If you need to do something on a Linux system, nobody would get fired for suggesting Ubuntu. It just happens to not be the distro I want.

However, I can ssh into my server with the Ubuntu VM and then I can click around in Weechat to my heart's content. I can also do weird builds with Nix and it just works. Neat.

I should probably figure out how hard it would be to get a NixOS-like environment in WSL, but WSL can't run systemd so I've been kinda avoiding it. Excising systemd from NixOS really defeats most of the point in my book. I may end up installing Nix on Alpine or something. IDK.

PowerShell

They say you can learn a lot about the design of a command line interface by what commands are used to do things like change directory, list files in a directory and download files from the internet. In PowerShell these are Get-ChildItem, Set-Location and Invoke-WebRequest. However there are aliases for ls, dir, cd and wget (these aliases aren't always flag-compatible, so you may want to actually get used to doing things in the PowerShell way if you end up doing anything overly fancy).

Another annoying thing was that pressing Control-D on an empty prompt didn't end up closing the session. In order to do this you need to edit your shell profile file:

PS C:\Users\xena> code $profile

Then you add this to the .ps1 file:

Set-PSReadlineOption -EditMode Emacs

Save this file then close and re-open PowerShell.

If this was your first time editing your PowerShell config (like it was for me) you are going to have to mess with your execution policy to allow you to execute scrips on your local machine. I get the reason why they did this, PowerShell has a lot of...well...power over the system. Doing this must outright eliminate a lot of attack vectors without doing much on the admin's side. But this applies to your shell profile too. So you are going to need to make a choice as to what security level you want to have with PowerShell scripts. I personally went with RemoteSigned.

Themes

I use stuff cribbed from oh my fish for my fish prompt. I googled "oh my powershell" and hoped I would get lucky with finding some nice batteries-included tools. I got lucky.

After looking through the options I saw a theme named sorin that looks like this:

Project-local Dependencies

To get this I'd need to do everything in WSL and use Nix. VSCode even has some nice integration that makes this easy. I wish there was a more native option though.

Things Windows Gets Really Right

The big thing that Windows gets really right as a developer is backwards compatibility. For better or worse I can install just about any program from the last 30 years of released software targeting windows and it will Just Work.

All of the games that I play natively target windows, and I don't have to hack at Steam's linux setup to get things like Sonic Adventure 2 working. All of the VR stuff I want to do will Just Work. All of the games I download will Just Work. I don't have to do the Proton rain dance. I don't have to play with GPU driver paths. I don't have to disable my compositor to get Factorio to launch. And most of all when I report a problem it's likely to actually be taken seriously instead of moaned at because I run a distribution without /usr/lib.

Overall, I think I can at least tolerate this development experience. It's not really the most ideal setup, but it does work and I can get things done with it. It makes me miss NixOS though. NixOS really does ruin your expectations of what a desktop operating system should be. It leaves you with kind of impossible standards, and it can be a bit hard to unlearn them.

A lot of the software I use is closed source proprietary software. I've tried to fight that battle before. I've given up. When it works, Linux on the desktop is a fantastic experience. Everything works together there. The system is a lot more cohesive compared to the "download random programs and hope for the best" strategy that you end up taking with Windows systems. It's hard to do the "download random programs and hope for the best" strategy with Linux on the desktop because there really isn't one Linux platform to target. There's 20 or something. This is an advantage sometimes, but is a huge pain other times.

The conclusion here is that there is no conclusion.