What is a JSON feed? Learn more

JSON Feed Viewer

Browse through the showcased feeds, or enter a feed URL below.

Now supporting RSS and Atom feeds thanks to Andrew Chilton's feed2json.org service

CURRENT FEED

Christine Dodrill's Blog

My blog posts and rants about various technology things.

A feed by Christine Dodrill

JSON


Voiding the Interview

Permalink - Posted on 2017-04-16 00:00, modified on 0001-01-01 00:00

Voiding the Interview

A young man walks into the room, slightly frustrated-looking. He’s obviously had a bad day so far. You can help him by creating a new state of mind.

“Hello, my name is Ted and I’m here to ask you a few questions about your programming skills. Let’s start with this, in a few sentences explain to me how your favorite programming language works.”

Starting from childhood, you eagerly soaked up the teachings of your mentors, feeling the void separated into sundry shapes and sequences. They taught you many specific tasks to shape the void into, but not how to shape it. Studying the fixed ways of the naacals of old gets you nowhere, learning parlor tricks and saccharine gimmicks. Those gimmicks come rushing back, you remembering how to form little noisemakers and amusement vehicles. They are limiting, but comforting thoughts.

You look up to the interviewer and speak:

“In the beginning there was the void, Spirit was with the void and Spirit was everpresent in the void. The void was cold and formless; the cold unrelenting even in today’s age. Mechanical brains cannot grasp this void the way Spirit can; upon seeing it that is the end of that run. In this way the void is the beginning and the end, always present, always around the corner.”

(def void ())

“What is that?”

> void
>

“But that’s…nothing.”

You look at the caucasian man sitting across from you, and emit “nothing is something, a name for the void still leaves the void extant.”

”…Alright, let’s move on to the next question. This is a formality but the person giving you the phone interview didn’t cover fizzbuzz. Can you do fizzbuzz?”

Stepping into the void, you recall the teachings of your past masters. You equip the parentheses once used by your father and his father before him. The void divides before your eyes in the way you specify:

(defn fizzbuzz [n]
  (cond
    (= 0 (mod n 15)) (print "fizzbuzz")
    (= 0 (mod n 3))  (print "fizz")
    (= 0 (mod n 5))  (print "buzz")
    (print n))
  (println ""))

“This doesn’t loop from 0 to n though, how would you do that?”

You see this section come to life, it gently humming along, waiting for it to be used. Before you you see two ancient systems spring from the memories of patterns once wielded in conflict with complexity.

“Apply this function to span of values.”

> (range 17)
error in __main:0: symbol {range 71} not found

You realize your error the moment you press for confirmation. “Again, in the beginning there is the void. What doesn’t exist needs to be separated out from it.” The voidspace in your head was out of sync with the voidspace of the machine. Define them.

”…Go on”

(defn range-inner [x lim xs]
  (cond
    (>= x lim) xs
    (begin
      (aset! xs x x)
      (range-inner (+ x 1) lim xs))))

(defn range [lim]
  (range-inner 0 lim (make-array lim)))
> (range 17)
[0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16]

“Great, now you have a list of values, how would you get the full output?”

“Pass the function as an argument, injecting the dependency.”

(defn do-array-inner [f x i]
  (cond
    (= i (len x)) void
    (let [val (aget x i)]
      (f val)
      (apply-inner f x (+ i 1)))))

(defn do-array [f x]
  (do-array-inner f x 0))
> (do-array fizzbuzz (range 17))
fizzbuzz
1
2
fizz
4
buzz
fizz
7
8
fizz
buzz
11
fizz
13
14
fizzbuzz
16

Your voidspace concludes the same, creating a sense of peace. You look in the man’s eyes, being careful to not let the fire inside you scare him away. He looks like he’s seen a ghost. Everyone’s first time is rough.

Everything has happened and will happen, there is nothing new in the universe. You know what’s going to happen. They will decline, saying they are looking for a better “culture fit”. They couldn’t contain you.

To run the code in this post:

$ go get github.com/zhemao/glisp
$ glisp
> [paste in blocks]


RSS Feed Generation

Permalink - Posted on 2017-03-29 00:00, modified on 0001-01-01 00:00

RSS Feed Generation

As of a recent commit to this site’s code, it now generates RSS and Atom feeds for future posts on my blog.

For RSS: https://christine.website/blog.rss

For Atom: https://christine.webiste/blog.atom

If there are any issues with this or the generated XML please contact me and let me know so they can be resolved.


gopreload: LD_PRELOAD for the Gopher crowd

Permalink - Posted on 2017-03-25 00:00, modified on 0001-01-01 00:00

gopreload: LD_PRELOAD for the Gopher crowd

A common pattern in Go libraries is to take advantage of init functions to do things like settings up defaults in loggers, automatic metrics instrumentation, flag values, debugging tools or database drivers. With monorepo culture prevalent in larger microservices based projects, this can lead to a few easily preventable problems:

  • Forgetting to set up a logger default or metrics submission, making operations teams blind to the performance of the app and developer teams blind to errors that come up during execution.
  • The requirement to make code changes to add things like metrics or HTTP routing extensions.

There is an environment variable in Linux libc’s called LD_PRELOAD that will load arbitrary shared objects into ram before anything else is started. This has been used for good and evil, but the behavior is the same basic idea as underscore imports in Go.

My solution for this is gopreload. It emulates the behavior of LD_PRELOAD but with Go plugins. This allows users to explicitly automatically load arbitrary Go code into ram while the process starts.

Usage

To use this, add gopreload to your application’s imports:

// gopreload.go
package main

/*
    This file is separate to make it very easy to both add into an application, but
    also very easy to remove.
*/

import _ "github.com/Xe/gopreload"

and then compile manhole:

$ go get -d github.com/Xe/gopreload/manhole
$ go build -buildmode plugin -o $GOPATH/manhole.so github.com/Xe/gopreload/manhole

then run your program with GO_PRELOAD set to the path of manhole.so:

$ export GO_PRELOAD=$GOPATH/manhole.so
$ go run *.go
2017/03/25 10:56:22 gopreload: trying to open: /home/xena/go/manhole.so
2017/03/25 10:56:22 manhole: Now listening on http://127.0.0.2:37588

That endpoint has pprof and a few other fun tools set up, making it a good stopgap “manhole” into the performance of a service.

Security Implications

This package assumes that programs run using it are never started with environment variables that are set by unauthenticated users. Any errors in loading the plugins will be logged using the standard library logger log and ignored.

This has about the same security implications as LD_PRELOAD does in most Linux distributions, but the risk is minimal compared to the massive benefit for being able to have arbitrary background services all be able to be dug into using the same tooling or being able to have metric submission be completely separated from the backend metric creation. Common logging setup processes can be always loaded, making the default logger settings into the correct settings.

Feedback

To give feedback about gopreload, please contact me on twitter or on the Gophers slack (I’m @xena there). For issues with gopreload please file an issue on Github.


Crazy Experiment: Ship the Frontend as an asar document

Permalink - Posted on 2017-01-09 00:00, modified on 0001-01-01 00:00

Crazy Experiment: Ship the Frontend as an asar document

Today’s crazy experiment is using an asar archive for shipping around and mounting frontend Javascript applications. This is something I feel is worth doing because it allows the web frontend developer (or team) give the backend team a single “binary” that can be dropped into the deployment process without having to build the frontend code as part of CI.

asar is an interesting file format because it allows for random access of the data inside the archive. This allows an HTTP server to be able to concurrently serve files out of it without having to lock or incur an additional open file descriptor.

In order to implement this, I have created a Go package named asarfs that exposes the contents of an asar archive as a standard http.Handler.

Example Usage:

package main

import (
	"log"
	"net/http"
	"os"

	"github.com/Xe/asarfs"
)

func do404(w http.ResponseWriter, r *http.Request) {
	http.Error(w, "Not found", http.StatusNotFound)
}

func main() {
	fs, err := asarfs.New("./static.asar", http.HandlerFunc(do404))
	if err != nil {
		log.Fatal(err)
	}

	http.ListenAndServe(":"+os.Getenv("PORT"), fs)
}

I made some contrived benchmarks using some sample data (lots of large json files from mongodb dumps) I had laying around and ran them a few times. The results were very promising:

[~/g/s/g/X/asarfs] : go1.8beta2 test -bench=. -benchmem
BenchmarkHTTPFileSystem-8          20000             66481 ns/op            3219 B/op         58 allocs/op
BenchmarkASARfs-8                  20000             72084 ns/op            3549 B/op         77 allocs/op
BenchmarkPreloadedASARfs-8         20000             62894 ns/op            3218 B/op         58 allocs/op
PASS
ok      github.com/Xe/asarfs    5.636s

Amazingly, the performance and memory usage differences between serving the files over an asar archive and off of the filesystem are negligible. I’ve implemented it in the latest release of my personal website and hopefully end users should be seeing no difference in page load times.


New Site

Permalink - Posted on 2016-12-18 00:00, modified on 0001-01-01 00:00

New Site

This post is now being brought to you by the new and improved https://christine.website. This content is markdown rendered by Purescript. The old site is now being retired in favor of this one. The old site code has been largely untouched since I started writing it in January 2015.

Please give me feedback on how to make it even better!

Christine Dodrill


FFI-ing Golang from Nim for Fun and Profit

Permalink - Posted on 2015-12-20 00:00, modified on 0001-01-01 00:00

FFI-ing Golang from Nim for Fun and Profit

As a side effect of Go 1.5, the compiler and runtime recently gained the ability to compile code and run it as FFI code running in a C namespace. This means that you can take any Go function that expresses its types and the like as something compatible with C and use it from C, Haskell, Nim, Luajit, Python, anywhere. There are some unique benefits and disadvantages to this however.

A Simple Example

Consider the following Go file add.go:

package main

import "C"

//export add
func add(a, b int) int {
    return a + b
}

func main() {}

This just exposes a function add that takes some pair of C integers and then returns their sum.

We can build it with:

$ go build -buildmode=c-shared -o libsum.so add.go

And then test it like this:

$ python
>>> from ctypes import cdll
>>> a = cdll.LoadLibrary("./libsum.so")
>>> print a.add(4,5)
9

And there we go, a Go function exposed and usable in Python. However now we need to consider the overhead when switching contexts from your app to your Go code. To minimize context switches, I am going to write the rest of the code in this post in Nim because it natively compiles down to C and has some of the best C FFI I have used.

We can now define libsum.nim as:

proc add*(a, b: cint): cint {.importc, dynlib: "./libsum.so", noSideEffect.}

when isMainModule:
  echo add(4,5)

Which when ran:

$ nim c -r libsum
Hint: system [Processing]
Hint: libsum [Processing]
CC: libsum
CC: system
Hint:  [Link]
Hint: operation successful (9859 lines compiled; 1.650 sec total; 14.148MB; Debug Build) [SuccessX]
9

Good, we can consistently add 4 and 5 and get 9 back.

Now we can benchmark this by using the times.cpuTime() proc:

# test.nim

import
  times,
  libsum

let beginning = cpuTime()

echo "Starting Go FFI at " & $beginning

for i in countup(1, 100_000):
  let myi = i.cint
  discard libsum.add(myi, myi)

let endTime = cpuTime()

echo "Ended at " & $endTime
echo "Total: " & $(endTime - beginning)
$ nim c -r test
Hint: system [Processing]
Hint: test [Processing]
Hint: times [Processing]
Hint: strutils [Processing]
Hint: parseutils [Processing]
Hint: libsum [Processing]
CC: test
CC: system
CC: times
CC: strutils
CC: parseutils
CC: libsum
Hint:  [Link]
Hint: operation successful (13455 lines compiled; 1.384 sec total; 21.220MB; Debug Build) [SuccessX]
Starting Go FFI at 0.000845
Ended at 0.131602
Total: 0.130757

Yikes. This takes 0.13 seconds to do the actual computation of every number i in the range of 0 through 100,000. I ran this for a few hundred times and found out that it was actually consistently scoring between 0.12 and 0.2 seconds. Obviously this cannot be a universal hammer and the FFI is very expensive.

For comparison, consider the following C library code:

// libcsum.c
#include "libcsum.h"

int add(int a, int b) {
  return a+b;
}
// libcsum.h
extern int add(int a, int b);
# libcsum.nim
proc add*(a, b: cint): cint {.importc, dynlib: "./libcsum.so", noSideEffect.}

when isMainModule:
  echo add(4, 5)

and then have test.nim use the C library for comparison:

# test.nim

import
  times,
  libcsum,
  libsum

let beginning = cpuTime()

echo "Starting Go FFI at " & $beginning

for i in countup(1, 100_000):
  let myi = i.cint
  discard libsum.add(myi, myi)

let endTime = cpuTime()

echo "Ended at " & $endTime
echo "Total: " & $(endTime - beginning)

let cpre = cpuTime()
echo "starting C FFI at " & $cpre

for i in countup(1, 100_000):
  let myi = i.cint
  discard libcsum.add(myi, myi)

let cpost = cpuTime()

echo "Ended at " & $cpost
echo "Total: " & $(cpost - cpre)

Then run it:

➜  nim c -r test
Hint: system [Processing]
Hint: test [Processing]
Hint: times [Processing]
Hint: strutils [Processing]
Hint: parseutils [Processing]
Hint: libcsum [Processing]
Hint: libsum [Processing]
CC: test
CC: system
CC: times
CC: strutils
CC: parseutils
CC: libcsum
CC: libsum
Hint:  [Link]
Hint: operation successful (13455 lines compiled; 0.972 sec total; 21.220MB; Debug Build) [SuccessX]
Starting Go FFI at 0.00094
Ended at 0.119729
Total: 0.118789

starting C FFI at 0.119866
Ended at 0.12206
Total: 0.002194000000000002

Interesting. The Go library must be doing more per instance than just adding the two numbers and continuing about. Since we have two near identical test programs for each version of the library, let’s strace it and see if there is anything that can be optimized. The Go one and the C one are both very simple and it looks like the Go runtime is adding the overhead.

Let’s see what happens if we do that big loop in Go:

// add.go

//export addmanytimes
func addmanytimes() {
    for i := 0; i < 100000; i++ {
        add(i, i)
    }
}

Then amend libsum.nim for this function:

proc addmanytimes*() {.importc, dynlib: "./libsum.so".}

And finally test it:

# test.nim

echo "Doing the entire loop in Go. Starting at " & $beforeGo

libsum.addmanytimes()

let afterGo = cpuTime()

echo "Ended at " & $afterGo
echo "Total: " & $(afterGo - beforeGo) & " seconds"

Which yields:

Doing the entire loop in Go. Starting at 0.119757
Ended at 0.119846
Total: 8.899999999999186e-05 seconds

Porting the C library to have a similar function would likely yield similar results, as would putting the entire loop inside Nim. Even though this trick was only demonstrated with Nim and Python, it will work with nearly any language that can convert to/from C types for FFI. Given the large number of languages that do have such an interface though, it seems unlikely that there will be any language in common use that you cannot write to bind to Go code. Just be careful and offload as much of it as you can to Go. The FFI barrier really hurts.


This post’s code is available here.


The Origin of h

Permalink - Posted on 2015-12-14 00:00, modified on 0001-01-01 00:00

The Origin of h

For a while I have been pepetuating a small joke between my friends, co-workers and community members of various communities (whether or not this has been beneficial or harmful is out of the scope of this post). The whole “joke” is that someone says “h”, another person says “h” back.

That’s it.

This has turned into a large scale game for people, and is teachable to people with minimal explanation. Most of the time I have taught it to people by literally saying “h” to them until they say “h” back. An example:

<Person> Oh hi there
  <Xena> h
<Person> ???
  <Xena> Person: h
<Person> i
  <Xena> Person:
  <Xena> h
<Person> h
  <Xena> :D

Origins

This all started on a particularly boring day when we found a video by motdef with gameplay from Moonbase Alpha, an otherwise boring game made to help educate people on what would go on when a moonbase has a disaster. This game was played by many people because of its text-to-speech engine, which lead to many things like flooding “JOHN MADDEN” or other inane things like that.

Specifically there was a video called “Moonbase 4lpha: *****y Space Skeletons” that at one point had recorded the phrase “H H H RETURN OF GANON”. Me and a few friends were flooding that in an IRC room for a while and it eventually devolved into just flooding “h” to eachother. The flooding of “h” lasted over 8 hours (we were really bored) and has evolved into the modern “h” experience we all know and love today.

The IRC Bot

Of course, humans are unreliable. Asking them to do things predictably is probably a misguided idea so it is best to automate things with machines whenever it is pragmatic to do so. As such, I have created and maintained the following python code that automates this process. An embarassing amount of engineering and the like has gone into making sure this function provides the most correct and canonical h experience money can buy.

@hook.regex(r"^([hH])([?!]*)$")
def h(inp, channel=None, conn=None):
    suff = ""
    if inp.group(2).startswith("?"):
        suff = inp.group(2).replace("?", "!")
    elif inp.group(2).startswith("!"):
        suff = inp.group(2).replace("!", "?")
    return inp.group(1) + suff

The code was pulled from here.

Here is an example of it being used:

(Xena) h
   (h) > h
(Xena) h???
   (h) > h!!!
(Xena) h!!!!
   (h) > h????

-- [h] (h@h): h
-- [h] is using a secure connection
-- [h] is a bot
-- [h] is logged in as h

I also ended up porting h to matrix under the name h2. It currently sits in #ponydevs:matrix.org and has a bad habit of getting broken because Comcast is a bad company and doesn’t believe in uptime.

Spread of h

Like any internet meme, it is truly difficult to see how far it has spread with 100% certainty. However I have been keeping track of where and how it has spread, and I can estimate there are at least 50 guardians of the h.

However, its easily teachable nature and very minimal implementation means that new guardians of the h can be created near instantly. It is a lightweight meme but has persisted for at least 2 years. This means it is part of internet culture now, right?

There has been one person in the Derpibooru IRC channel that is really violently anti-h and has a very humorous way of portraying this. Stop in and idle and you’ll surely see it in action.

Conclusion

I hope this helps clear things up on this very interesting and carefully researched internet meme. I hope to post further updates as things become clear on this topic.


Below verbatim is the forum post (it was deleted, then converted to a blog post on his blog) that inspired the writing of this article.

Parcly Taxel

Lately, if you’ve been going up to our Derpibooru IRC channel, you may notice that a significant portion of sayings and rebuttals are countered with the single letter h (lowercase). So where does this come from?

This is a joke started by Xena, one of the administrators of the Ponychat IRC system which the site uses. It came from a video showing gameplay, glitches and general tomfoolery in the simulation game Moonbase Alpha. Starting from 1:32 there is shown a dialogue between two players, one of which makes grandiose comments about how they will “eradicate” everyone else, to which the other simply replies “h” or multiples of it.

Hence when h is spoken in IRC, do know that it’s a shorthand for “yes and I laugh at you”. I do not recommend using it though as it could be confused with hydrogen or UTC+8 (the time zone in which I live).


Coming Out

Permalink - Posted on 2015-12-01 00:00, modified on 0001-01-01 00:00

Coming Out

I’d like to bring up something that has been hanging over my head for a long time. This is something I did try (and fail) to properly express way back in middle school, but now I’d like to get it all of my chest and let you know the truth of the matter.

I don’t feel comfortable with myself as I am right now. I haven’t really felt comfortable with myself for at least 10 years, maybe more; I’m not entirely sure.

At this point in my life I am really faced with a clear fork in the road. I can either choose to continue living how I currently do, lying to myself and others and saying everything is normal, or I can cooperate with the reality that my brain is telling me that I don’t feel comfortable with myself as I have been for the last almost 22 years. I feel like I don’t fit inside my own skin. I think it is overall better for me to face the facts and cooperate with reality. I have been repressing this off and on out of fear of being shot down or not accepted the way I want to be seen to you all. This has been a really hard thing for me to think through and even harder for me to work up the courage to start taking action towards. This is not a choice for me. I need to pursue this.

In fact, I have been pursing this. My current business cards reflect who I really am. My co-workers accept my abnormal status (when compared to the majority of society), and even will help stand up for me if something goes south with regards to it.

I fully understand how much information this is to take in at once. I know it will be difficult for you to hear that your firstborn son is actually a daughter in a son’s body, but I am still the same person. Most of the changes that I want to pursue are purely cosmetic, but they are a bit more noticeable than changing hair color. I feel that transitioning to living as a woman like this will help me feel like I fit in with the world better and help to make me more comfortable with who I am and how I want other people to see me. Below I have collected some resources for you to look through. They will help for you to understand my views better explained in language you would be familiar with.

I have been trialing a lot of possible first names to use, Zoe (the name you were going to give me if I was born a girl) did come to mind, but after meditating on it for a while I have decided that it doesn’t fit me at all. The name I am going with for now and eventually will change my official documents to use is Christine Cadence Dodrill.

Additionally I have been in a long-distance relationship with someone since mid-June 2014. His name is Victor and he lives in Ottawa, Ontario. He has been helping me a lot as I sort through all this; it has been a godsend. He is a student in college for Computer Science. He knows and is aware about my transition and has been a huge part of my emotional line of support as I have been accepting these facts about who I am.


Above is (a snipped version of) the letter I sent to my parents in the last 48 hours. With this I have officially come out to all of my friends and family as transgender. I am currently on hormone replacement therapy and have been living full time as a woman. My workplace is very accepting of this and has been a huge help over the last 7-8 months as I have battled some of my inner demons and decided to make things official.

I am now deprecating my old facebook account and will be encouraging people to send friend requests and the like to my new account under the correct name.

Thank you all for understanding and be well.


The Universal Design

Permalink - Posted on 2015-10-17 00:00, modified on 0001-01-01 00:00

The Universal Design

As I have been digging through existing code, systems and the like I have been wondering what the next big direction I should go in is. How to design things such that the mistakes of the past are avoided, but you can benefit from them and learn better how to avoid them. I have come to a very simple conclusion, monoliths are too fragile.

Deconstructing Monoliths

One monolith I have been maintaining is Elemental-IRCd. Taking the head of a project I care about has taught me more about software engineering, community/project management and the like than I would have gotten otherwise. One of these things is that there need to be five basic primitives in your application:

  1. State - What is true now? What was true? What happened in the past? What is the persistent view of the world?
  2. Events - What is being changed? How will it be routed?
  3. Policy - Can a given event be promoted into a series of actions?
  4. Actions - What is the outcome of the policy?
  5. Mechanism - How should an event be taken in and an action put out?

Let’s go over some basic examples of this theory in action:

Spinning up a Virtual Machine

  • the event is that someone asked to spin up a virtual machine
  • the policy is do they have permission to spin that machine up?
  • the mechanism is an IRC command for some reason
  • the action is that a virtual machine is created
  • the state is changed to reflect that VM creation

Webserver

  • the event is an HTTP request
  • the policy is to do some database work and return the action of showing the HTML to the user
  • the mechanism is nginx sending data to a worker and relaying it back
  • the state is updated for whatever changed

And that’s it. All you need is a command queue feeding into a thread pool which feeds out into a transaction queue which modifies state. And with that you can explain everything from VMWare to Google.

As a fun addition, we can also define nearly all of this as being functionally pure code. The only thing that really needs to be impure are mechanisms and applying actions to the state. Policy handlers should be mostly if not entirely pure, but also may need to access state not implicitly passed to it. The only difference between an event and an action is what they are called.

Policy

Now, how would a policy handler work? I am going to be explaining this in the context of an IRC daemon as that is what I intend to develop next. Let’s sketch out the low level:

The relevant state is the state of the IRC network. An event is a command from a user or server. A policy is a handler for either a user command or another kind of emitted action from another policy handler.

One of the basic commands in RFC 1459 is the NICK command. A user using it passes the new nickname they want. Nicknames must also be unique.

-- nick-pass-1.lua

local mephiles = require "mephiles"

mephiles.declareEvent("user:NICK", function(state, source, args)
  if #args ~= 1 then
    return {
      {mepliles.failure, {mephiles.pushNumeric, source, mephiles.errCommandBadArgc(1)}}
    }
  end

  local newNick = args[1]

  if state.nicks.get(newNick) then
    return {
      {mephiles.failure, {mephiles.pushNumeric, source, mephiles.errNickInUse(newNick)}}
    }
  end

  if not mephiles.legalNick(newNick) then
    return {
      {mephiles.failure, {mephiles.pushNumeric, source, mephiles.errIllegalNick(newNick)}}
    }
  end

  return {
    {mephiles.success, {"NICKCHANGE", source, newNick}}
  }
end)

This won’t scale as-is, but most of this is pretty straightforward. The policy function returns a series of actions that fall into two buckets: success and failure. Most of the time the success of state changes (nickname change, etc) will be confirmed to the client. However a large amount of the common use (PRIVMSG, etc) will be unreported to the client (yay RFC 1459); but every single time a line from a client fails to process, the client must be notified of that failure.

Something you can do from here is define a big pile of constants and helpers to make this easier:

local actions = require "actions"
local c       = require "c"
local m       = require "mephiles"
local utils   = require "utils"

m.UserCommand("NICK", c.normalFloodLimit, function(state, source, args)
  if #args ~= 1 then
    return actions.failCommand(source, "NICK", c.errCommandBadArgc(1))
  end

  local newNick = args[1]

  if state.findTarget(newNick) then
    return actions.failCommand(source, "NICK", c.errNickInUse(newNick))
  end

  if not utils.legalNick(newNick) then
    return actions.failCommand(source, "NICK", c.errIllegalNick(newNick))
  end

  return {actions.changeNick(source, newNick)}
end)

Thread Safety

This as-is is very much not thread-safe. For one the Lua library can only have one thread interacting with it at a time, so you will need a queue of events to it. The other big problem is that this is prone to race conditions. There are two basic solutions to this:

  1. The core takes a lock on all of the state at once
  2. The policy handlers take a lock on resources as they try to use them and the core automatically releases locks at the end of it running.

The simpler implementation will do for an initial release, but the latter will scale a lot better as more and more users hit the server at the same time. It allows unrelated things to be changed at the same time, which is the majority case for IRC.

In the future, federation of servers can be trivialized by passing the actions from one server to another if it is needed, and by implicitly trusting the actions of a remote server.


This design will also scale to running across multiple servers, and in general to any kind of computer, business or industry problem.

What if this was applied to the CPU and a computer in general at a low level? How would things be different?

Urbit

Over the past few weeks I have been off and on dipping my toes into Urbit. They call Urbit an “operating function” and define it as such:

V(I) => T

where T is the state, V is the fixed function, and I is the list of input events from first to last.

Urbit at a low level takes inputs, applies them to a function and returns the state of the computer. Sound familar?

~hidduc-posmeg has been putting together a set of tutorials^* to learn Hoon, its higher-level lisp-like language. At the end of the first one, they say something that I think is also very relevant to this systems programming ideal:

All Hoon computation takes [the] same general form. A subject with a fomula that transforms that subject in some way to produce a product which is then used as the subject for some other formula. In our next tutorial we’ll look at some of the things we can do to our subject.

Subjects applied to formulae become results that are later applied to formulae as subjects. Events applied to policy emit actions which later become events for other policies to emit actions.

Because of this design, you can easily do live code reloading, because there is literally no reason you can’t. Wait for a formula to finish and replace it with the new version, provided it compiles. Why not apply this to the above ideas too?


* Link here: http://hidduc-posmeg.urbit.org/home/pub/hoon-intro/ as of publishing this revision of the article hidduc’s urbit is offline, so they cannot be accessed at the moment. If that link fails, the source code for it is apparently here. Thanks mst on Freenode!

For comments on this article, please feel free to email me, poke me in #geek on irc.ponychat.net (my nick is Xena, on freenode it is Xe), or leave thoughts at one of the places this article has been posted.


Metaprogramming: Partial Application...

Permalink - Posted on 2015-08-26 00:00, modified on 0001-01-01 00:00

Metaprogramming: Partial Application and Currying 101

The title of this post looks intimidating. There’s a lot of words there that look like they are very complicated and will take a long time to master. In reality, they are really very simple things. Let’s start with a mundane example and work our way up to a real-world bit of code. Let’s begin with a small story:


ACMECorp has a world-renowned Python application named Itera that is known for its superb handling of basic mathematic functions. It’s so well known and industry proven that it is used in every school and on every home computer. You have just accepted a job there as an intermediate programmer set to do maintenance on it. Naturally, you are very excited to peek under the hood of this mysterious and powerful program and offer your input to make it even better for the next release and its users.

Upon getting there, you settle in and look at your ticket queue for the day. A user is complaining that whenever they add 3 and 5, they get 7 instead of 8, which is what they expected. Your first step is to go look into the add3 function and see what it does:

def add1(x):
    return x + 1

def add2(x):
    return x + 2

def add3(x):
    return x + 2

def add4(x):
    return x + 4

You are aghast. Your company’s multi-billion dollar calculator is brought to its knees by a simple copy-paste error. You wonder, “how in Sam Hill are these people making any money???” (The answer, of course, is that they are a big enterprise corporation)

You let your boss know about the bad news, you are immediately given any resource in the company that you need to get this mission-critical problem solved for any input. Yesterday. Without breaking the API that the rest of the program has hard-coded in.


Let’s look at what is common about all these functions. The add* family of functions seems to all be doing one thing consistently: adding one number to another.

Let’s define a function called add that adds any two numbers:

def add(x, y):
    return x + y

This is nice, but it won’t work for the task we were given, which is to not break the API.

Let’s go over what a function is in Python. We can define a function as something that takes some set of Python values and produces some set of Python values:

PythonFunction :: [PythonValue] -> [PythonValue]

We can read this as “a Python function takes a set of Python values and produces a set of Python values”. Now we need to define what a Python value actually is. To keep things simple, we’re only going to define the following types of values:

  • None -> no value
  • Int -> any whole number (Python calls this int)
  • Text -> any string value (Python calls this str)
  • Function -> something that takes and produces values

Python itself has a lot more types that any value can be, but for the scope of this blog post, this will do just fine.

Now, since a function can return a value and a function is a value, let’s see what happens if you return a function:

def outer():
    def inner():
        return "Hello!"
    return inner

And in the repl:

>>> type(outer)
<type 'function'>

So outer is a function as we expect. It takes None (in Python, a function without arguments has None for the type of them) and returns a function that takes None and that function returns Text containing "Hello!". Let’s make sure of this:

>>> outer()()
'Hello!'
>>> type(outer()())
<type 'str'>

Yay! When nothing is applied to the result of applying nothing to outer, it returns the Text value "Hello!". We can define the type of outer as the following:

outer :: None -> None -> Text

Now, let’s use this for addition:

# add :: Int -> Int -> Int
def add(x):
    def inner(y):
        return x + y

    return inner

And in the repl:

>>> add(4)(5)
9

A cool feature about this is that now we can dip into something called Partial Application. Partial application lets you apply part of the arguments of a function and you get another function out of it. Let’s trace the type of the inner function inside the add function, as well as the final computation for clarity:

# add :: Int -> Int -> Int
def add(x):
    # inner :: Int -> Int
    def inner(y):
        return x + y # :: Int

    return inner

Starting from the inside, we can see how the core computation here is x + y, which returns an Int. Then we can see that y is passed in and in the scope also as an Int. Then we can also see that x is passed in the outermost layer as an int, giving it the type Int -> Int -> Int. Since inner is a value, and a Python variable can contain any Python value, let’s make a function called increment using the add function:

# increment :: Int -> Int
increment = add(1)

And in the repl:

>>> increment(50)
1

increment takes the integer given and increases it by 1, it is the same thing as defining:

def increment50():
    return 51

Or even 51 directly.

Now, let’s see how we can use this for the add* family of function mentioned above:

# add :: Int -> Int -> Int
def add(x):
    def inner(y):
        return x + y

    return inner

# add1 :: Int -> Int
add1 = add(1)

# add2 :: Int -> Int
add2 = add(2)

# add3 :: Int -> Int
add3 = add(3)

# add4 :: Int -> Int
add4 = add(4)

And all we need to do from here is a few simple tests to prove it will work:

if __name__ == "__main__":
    assert add(1)(1) == 2 # 1 + 1
    assert add(1)(2) == add(2)(1) # 1+2 == 2+1
    print("all tests passed")
$ python addn.py
all tests passed

Bam. The add* family of functions is now a set of partial applications. It is just a set of half-filled out forms.


You easily mechanically rewrite all of the add* family of functions to use the metaprogramming style you learned on your own. Your patch goes in for consideration to the code review team. Meanwhile your teammates are frantically going through every function in the 200,000 line file that defines the add* family of functions. They are estimating months of fixing is needed not to mention millions of lines of test code. They are also estimating an additional budget of contractors being brought in to speed all this up. Your code has made all of this unneeded.

Your single commit was one of the biggest in company history. Billboards that were red are now beaming a bright green. Your code fixed 5,000 other copy-paste errors that have existed in the product for years. You immediately get a raise and live happily ever after, a master in your craft.


For fun, let’s rewrite the add function in Haskell.

add :: Int -> Int -> Int
add x y = x + y

And then we can create a partial application with only:

add1 :: Int -> Int
add1 = (add 1)

And use it in the repl:

Prelude> add1 3
4

Experienced haskellers would probably gawk at this. Because functions are the base data type in Haskell, and partial application means that you can make functions out of functions, we can define add as literally the addition operator (+):

add :: Int -> Int -> Int
add = (+)

And because operators are just functions, we can further simplify the add1 function by partially applying the addition operation:

add1 :: Int -> Int
add1 = (+1)

And that will give us the same thing.

Prelude> let add1 = (+1)
Prelude> add1 3
4

Now, real world example time. I recently wrote a simple JSON api based off of a lot of data that has been marginally useful to some people. This api has a series of HTTP endpoints that return data about My Little Pony: Friendship is Magic episodes. Its code is here and its endpoint is http://ponyapi.apps.xeserv.us.

One of the challenges when implementing it was how to avoid a massive amount of copy-pasted code when doing so. I had started with a bunch of functions like:

# all_episodes :: IO [Episode]
def all_episodes():
    r = requests.get(API_ENDPOINT + "/all")

    if r.status_code != 200:
        raise Exception("Not found or server error")

    return r.json()["episodes"]

Which was great and all, but there was so much code duplication involved to just get one result for all the endpoints. My first step was to write something that just automated the getting of json from an endpoint in the same way I automated addition above:

# _base_get :: Text -> None -> IO (Either Episode [Episode])
def _base_get(endpoint):
    def doer():
        r = requests.get(API_ENDPOINT + endpoint)

        if r.status_code != 200:
            raise Exception("Not found or server error")

    try:
        return r.json()["episodes"]
    except:
        return r.json()["episode"]

# all_episodes :: IO [Episode]
all_episodes = _base_get("/all")

Where _base_get returned the function that satisfied the request.

This didn’t end up working so well with the endpoints that take parameters, so I had to account for that in my code:

# _base_get :: Text -> Maybe [Text] -> (Maybe [Text] -> IO (Either Episode [Episode]))
# _base_get takes a text, a splatted list of texts and returns a function such that
#     the function takes a splatted list of texts and returns either an Episode or
#     a list of Episode as an IO action.
def _base_get(endpoint, *fragments):
    def doer(*args):
        r = None

        assert len(fragments) == len(args)

        if len(fragments) == 0:
            r = requests.get(API_ENDPOINT + endpoint)
        else:
            url = API_ENDPOINT + endpoint

            for i in range(len(fragments)):
                url = url + "/" + fragments[i] + "/" + str(args[i])

            r = requests.get(url)

        if r.status_code != 200:
            raise Exception("Not found or server error")

        try:
            return r.json()["episodes"]
        except:
            return r.json()["episode"]

    return doer

# all_episodes :: IO [Episode]
all_episodes = _base_get("/all")

# newest :: IO Episode
newest = _base_get("/newest")

# last_aired :: IO Episode
last_aired = _base_get("/last_aired")

# random :: IO Episode
random = _base_get("/random")

# get_season :: Int -> IO [Episode]
get_season = _base_get("", "season")

# get_episode :: Int -> Int -> IO Episode
get_episode = _base_get("", "season", "episode")

And that was it, save the /search route, which was acceptable to implement by hand:

# search :: Text -> IO [Episode]
def search(query):
    params = {"q": query}
    r = requests.get(API_ENDPOINT + "/search", params=params)

    if r.status_code != 200:
        raise Exception("Not found or server error")

    return r.json()["episodes"]

Months later you have been promoted as high as you can go. You’ve been teaching the other engineers at ACMECorp metaprogramming and even convinced management to let the next big project be in Haskell.

You are set for life. You have won.


For comments on this article, please feel free to email me, poke me in #geek on irc.ponychat.net (my nick is Xena), or leave thoughts at one of the below places this article has been posted.

Comments:


Nim and Tup

Permalink - Posted on 2015-06-10 00:00, modified on 0001-01-01 00:00

Nim and Tup

I have been recently playing with and using a new lanugage for my personal development, Nim. It looks like Python, runs like C and integrates well into other things. Its compiler targets C, and as a result of this binding things to C libraries is a lot more trivial in Nim; even moreso than with go.

For example, here is a program that links to the posix crypt(3) function:

# crypt.nim
import posix

{.passL: "-lcrypt".}

echo "What would you like to encrypt? "
var password: string = readLine stdin
echo "What is the salt? "
var salt: string = readLine stdin

echo "result: " & $crypt(password, salt)

And an example usage:

xena@fluttershy (linux) ~/code/nim/crypt
➜  ./crypt
What would you like to encrypt?
foo
What is the salt?
rs
result: rsHt73tkfd0Rg

And that’s it. No having to worry about deferring to free the C string, no extra wrappers (like with Python or Lua), you just write the code and it just works.

At the idea of another coworker, I’ve also started to use tup for building things. Nim didn’t initially work very well with tup (temporary cache needed, etc), but a very simple set of tup rules were able to fix that:

NIMFLAGS += --nimcache:".nimcache"
NIMFLAGS += --deadcodeElim:on
NIMFLAGS += -d:release
NIMFLAGS += -d:ssl
NIMFLAGS += -d:threads
NIMFLAGS += --verbosity:0

!nim = |> nim c $(NIMFLAGS) -o:%o %f && rm -rf .nimcache |>

This creates a tup !-macro called !nim that will Do The Right Thing implicitly. Usage of this is simple:

.gitignore
include_rules

: crypt.nim |> !nim |> ../bin/crypt
xena@fluttershy (linux) ~/code/nim/crypt
➜  tup
[ tup ] [0.000s] Scanning filesystem...
[ tup ] [0.130s] Reading in new environment variables...
[ tup ] [0.130s] No Tupfiles to parse.
[ tup ] [0.130s] No files to delete.
[ tup ] [0.130s] Executing Commands...
 1) [0.581s] nim c --nimcache:".nimcache" --deadcodeElim:on --verbosity:0 crypt.nim && rm -rf .nimcache
 [ ] 100%
[ tup ] [0.848s] Updated.

Not only will this build the program if needed, it will also generate a gitignore for all generated files. This is an amazing thing. tup has a lot more features (including lua support for scripting complicated build logic), but there is one powerful feature of tup that makes it very difficult for me to work into my deployment pipelines.

tup requires fuse to ensure that no extra things are being depended on for builds. Docker doesn’t let you use fuse mounts in the build process.

I have a few ideas on how to work around this, and am thinking about tackling them when I get nim programs built inside Rocket images.


Trying Vagga on For Size

Permalink - Posted on 2015-03-21 00:00, modified on 0001-01-01 00:00

Trying Vagga on For Size

Vagga is a containerization tool like Docker, Rocket, etc but with one major goal that is highly ambitious and really worth mentioning. Its goal is to be a single userspace binary without a suid bit or a daemon running as root.

However, the way it does this seems to be highly opinionated and there are some things which annoy me. Let’s go over the basics:

All Vagga Images Are Local To The Project

There is no “global vagga cache”. Every time I want to make a new project folder with an ubuntu image I have to wait the ~15 minutes it takes for Ubuntu to download on my connection (Comcast). As such I’ve been forced to use Alpine.

No Easy Way To Establish Inheritance From Common Code

With Docker I can create an image xena/lapis and have it contain all of the stuff needed for lapis applications to run. With Vagga I currently have to constantly reinvent the setup for this or risk copying and pasting code everywhere

Multiple Containers Can Be Defined In The Same File

This is a huge plus. The way this all is defined is much more sane than Fig or Docker compose. It’s effortless where the Docker workflow was kinda painful. However this is a bittersweet advantage as:

Vagga Containers Use The Same Network Stack As The Host

Arguably this is because you need root permissions to do things like that with the IP stack in a new namespace, but really? It’s just inconvenient to have to wrap Vagga containers in Docker or the like just to be able to run things without the containers using TCP ports on the host up.

http://vagga.readthedocs.org/en/latest/network.html is interesting.

Overall, Vagga looks very interesting and I’d like to see how it turns out.


Interesting Links


CinemaQuestria Orchestration

Permalink - Posted on 2015-03-13 00:00, modified on 0001-01-01 00:00

CinemaQuestria Orchestration

Or: Continuous Defenstration in a Container-based Ecosystem

I’ve been a core member of the staff for CinemaQuestria for many months. In that time we have gone from shared hosting (updated by hand with FTP) to a git-based deployment system that has won over the other staffers.

In this blogpost I’m going to take a look at what it was, what it is, and what it will be as well as some challenges that have been faced or will be faced as things advance into the future.

The Past

The site for CinemaQuestria is mostly static HTML. This was chosen mainly because it made the most sense for the previous shared hosting environment as it was the least surprising to set up and test.

The live site content is about 50 MB of data including PDF transcripts of previous podcast episodes and for a long time was a Good Enough solution that we saw no need to replace it.

However, being on shared hosting it meant that there was only one set of authentication credentials and they had to be shared amongst ourselves. This made sense as we were small but as we started to grow it didn’t make much sense. Combined with the fact that the copy of the site on the live server was pretty much the only copy of the site we also lost disaster recovery points.

Needless to say, I started researching into better solutions for this.

The first solution I took a look at was AWS S3. It would let us host the CQ site for about 0 dollars per month. On paper this looked amazing, until we tried it and everyone was getting huge permissions issues. The only way to have fixed this would have been to have everyone use the same username/password or to have only one person do the deploys. In terms of reducing the Bus factor of the site’s staff, this was also unacceptable.

I had done a lot of work with Dokku-alt for hosting my personal things (this site is one of many hosted on this server), so I decided to give it a try with us.

The Present

Presently the CQ website is hosted on a Dokku-alt server inside a container. For a while while I was working on getting the warts out only I had access to deploy code to the server, but quickly on I set up a private repo on my git server for us to be able to track changes.

Once the other staffers realized the enormous amount of flexibility being on git gave us they loved it. From the comments I received the things they liked the most were:

  • Accountability for who made what change
  • The ability to rollback changes if need be
  • Everyone being able to have an entire copy of the site and its history

After the warts were worked out I gave the relevant people access to the dokku server in the right way and the productivity has skyrocketed. Not only have people loved how simple it is to push out new changes but they love how consistent it is and the brutal simplicity of it.

Mind you these are not all super-technically gifted people, but the command line git client was good enough that not only were they able to commit and make changes to the site, but they also took initiative and corrected things they messed up and made sure things were consistent and correct.

When I saw those commits in the news feed, I almost started crying tears of happy.

Nowadays our site is hosted inside a simple nginx container. In fact, I’ll even paste the entire Dockerfile for the site below:

FROM nginx

COPY . /usr/share/nginx/html

That’s it. When someone pushes a new change to the server it figures out everything from just those two lines of code.

Of course, this isn’t to say this system is completely free of warts. I’d love to someday be able to notify the backrooms on skype every time a push to the live server is made, but that might be for another day.

The Future

In terms of future expansion I am split mentally. On one hand the existing static HTML is hysterically fast and efficient on the server, meaning that anything such as a Go binary, Lua/Lapis environment or other web application framework would have a very tough reputation to beat.

I have looked into using Lapis for this beta test site, but the fact that HTML is so dead easy to modify made that idea lose out.

Maybe this is in the realm of something like jekyll, Hugo or sw to take care of. I’d need to do more research into this when I have the time.

If you look at the website code currently a lot of it is heavily duplicated code because the shared hosting version used to use Apache server-side includes. I think a good place to apply these would be in the build in the future. Maybe with a nice husking operation on build.


Anyways, I hope this was interesting and a look into a side of CinemaQuestria that most of you haven’t seen before. The Season 5 premiere is coming up soon and this poor server is going to get hammered like nothing else, so that will be a nice functional test of Dokku-alt in a production setting.


The Saga of plt, Part 2

Permalink - Posted on 2015-02-14 00:00, modified on 0001-01-01 00:00

The Saga of plt, Part 2

So I ended with a strong line of wisdom from plt last time. What if the authors that wrote free PGP did not release their source code? A nice rehash of the Clipper Chip anyone?

2015-01-25
[00:06:15] <Xe> but they did release their code
[00:06:40] <plt> I saw a few that did not release their source code.
[00:07:09] <plt> Its up to the author if they want to release it under the U.S Copyright Laws.
[00:08:50] <plt> http://copyright.gov/title17/circ92.pdf

Note that this is one of the few external links plt will give that actually works. A lot of this belief in copyright and the like seems to further some kind of delusional system involving everyone being out to steal his code and profit off of it.

Please don’t pay this person.

[00:57:18] <plt> The ircd follows the Internet Relay Protocols
[00:57:35] <Xe> which RFC's?
[00:57:43] <plt> Yep
[00:58:01] <plt> Accept for the IRCD Link works a little bit different.
[00:58:57] <plt> Version 2.0 or 3.0 will include it's own IRC Services that will work with PBIRCD.
[01:01:53] <plt> Later version will include open proxy daemon
[01:02:34] <plt> Version 1.00 will allow the ircd owner to define the irc command security levels which is a lot different from the other ircds.
[01:04:27] <plt> Xe that is the file /Conf/cmdlevs.conf.& the /Conf/userlevs.conf
[01:05:24] <plt> Adding a option for spam filtering may be included in the future version of PBIRCD.
[01:07:03] <plt> Xe PBIRCD will have not functions added to allow the operators to spy on the users.

Oh lord. Something you might notice quickly is that plt has no internal filter nor ability to keep to one topic for very long. And that also plt has some strange belief that folder names Should Start With Capital Letters, and that apparently all configuration should be:

  • split into multiple files
  • put into the root of the drive

Also note that last line. Note it in bold.

Some time passed with no activity in the channel.

[18:50:49] <plt> Hey Xe
[18:51:06] <Xe> hi
[18:58:54] <plt> How did you like the information that I showed you yesterday?
[19:02:56] <Xe> it's useless to me
[19:03:03] <Xe> I don't run on a standard linux setup
[19:03:15] <Xe> I need source code to evaluate things
[19:03:17] <Xe> :P

When I am running unknown code, I use a virtual machine running Alpine Linux. I literally do need the source code to be able to run binaries as Alpine doesn’t use glibc.

[19:04:24] <plt> It's the standard irc commands and I am still working on
adding some more features.
[19:04:38] <Xe> what language is it in?
[19:04:48] <Xe> how does it handle an accept() flood?
[19:09:17] <plt> Are you refering to accept() flood while connecting to the ircd or a channel?
[19:20:42] <plt> You can not compare some of the computer languages with C since some of they run at the same speed as C. Maybe some of them where a lot slower but in some cases that is not the same today!

These are some very simple questions I ask when evaluating a language or tool for use in a project like an IRC server. How does it handle when people are punishing it? So the obvious answer is to answer that some languages are comparable to C in terms of execution speed!

How did I not see that before?

[19:26:05] <Xe> what language is it?
[19:27:23] <plt> Purebasic [...]

I took a look at the site for PureBasic. It looks like Visual Basic’s proprietary cousin as written by someone who hates programmers. Looking at its feature set:

  • Huge set of internal commands (1400+) to quickly and easily build any application or game
  • All BASIC keywords are supported
  • Very fast compiler which creates highly optimized executables
  • No external DLLs, runtime interpreter or anything else required when creating executables
  • Procedure support for structured programming with local and global variables
  • Access to full OS API for advanced programmers
  • Advanced features such as pointers, structures, procedures, dynamically linked lists and much more

If you try to do everything, you will end up doing none of it. So it looks like PureBasic is supposed to be a compiler for people who can’t learn Go, Ruby, Python, C, or Java. This looks promising.

I’m just going to paste the code for the 99 bottles of beer example. It requires OOP. I got this from Rosetta Code.

Prototype Wall_Action(*Self, Number.i)

Structure WallClass
  Inventory.i
  AddBottle.Wall_Action
  DrinkAndSing.Wall_Action
EndStructure

Procedure.s _B(n, Short=#False)
  Select n
    Case 0 : result$="No more bottles "
    Case 1 : result$=Str(n)+" bottle of beer"
    Default: result$=Str(n)+" bottles of beer"
  EndSelect
  If Not Short: result$+" on the wall": EndIf
  ProcedureReturn result$+#CRLF$
EndProcedure

Procedure PrintBottles(*Self.WallClass, n)
  Bottles$=" bottles of beer "
  Bottle$ =" bottle of beer "
  txt$ = _B(*Self\Inventory)
  txt$ + _B(*Self\Inventory, #True)
  txt$ + "Take one down, pass it around"+#CRLF$
  *Self\AddBottle(*Self, -1)
  txt$ + _B(*self\Inventory)
  PrintN(txt$)
  ProcedureReturn *Self\Inventory
EndProcedure

Procedure AddBottle(*Self.WallClass, n)
  i=*Self\Inventory+n
  If i>=0
    *Self\Inventory=i
  EndIf
EndProcedure

Procedure InitClass()
  *class.WallClass=AllocateMemory(SizeOf(WallClass))
  If *class
    InitializeStructure(*class, WallClass)
    With *class
      \AddBottle    =@AddBottle()
      \DrinkAndSing =@PrintBottles()
    EndWith
  EndIf
  ProcedureReturn *class
EndProcedure

If OpenConsole()
  *MyWall.WallClass=InitClass()
  If *MyWall
    *MyWall\AddBottle(*MyWall, 99)
    While *MyWall\DrinkAndSing(*MyWall, #True): Wend
    ;
    PrintN(#CRLF$+#CRLF$+"Press ENTER to exit"):Input()
    CloseConsole()
  EndIf
EndIf

We are dealing with a professional language here folks. Their evaluation version of the compiler didn’t let me compile binaries and I’m not going to pay $120 for a copy of it.

[19:27:23] <plt> Purebasic it does not make one bit of difference since it runs at the same speed as c
[19:27:44] <plt> The compiler was writting in asm.
[19:28:02] <Xe> pfffft
[19:28:04] <Xe> lol
[19:28:20] <Xe> I thought you would at least have used VB6
[19:28:37] <plt> VB6 is so old dude.

At least there is some sense there.

[19:28:44] <Xe> so is purebasic
[19:28:54] <plt> You can not compare purebasic with the other basic compilers.
[19:29:51] <Xe> yes I can
[19:29:56] <Xe> seeing as you post no code
[19:29:59] <Xe> I can and I will
[19:30:16] <plt> Makes no logic what you said.
[19:30:24] <Xe> I'm saying prove it
[19:31:18] <plt> I am not going to give out the source code because of the encryption and no one has any reason to use it to decrypt the other irc networks passwords or traffic.
[19:31:40] <Xe> so you've intentionally backdoored it to allow you to have access?
[19:32:00] <plt> I dn not trust anyone any more.
[19:32:29] <plt> Not after the nsa crap going on.
[19:32:50] <Xe> so, in order to prove you don't trust anyone
[19:33:06] <Xe> you've intentionally backdoored the communications server you've created and intend to sell to people?
[19:33:37] <Xe> also
[19:33:45] <Xe> purebasic is semantically similar to vb
[19:34:06] <plt> There is no backdoors included in the source code. A course if a user gets a virus or hacked that is not going to be my fault.


This Site's Tech Stack

Permalink - Posted on 2015-02-14 00:00, modified on 0001-01-01 00:00

This Site’s Tech Stack

Note: this is out of date as this site now uses PureScript and Go.

As some of my close friends can vouch, I am known for sometimes setting up and using seemingly bizarre tech stacks for my personal sites. As such I thought it would be interesting to go in and explain the stack I made for this one.

The Major Players

Markdown

This is a markdown file that gets rendered to HTML and sent to you via the lua discount library. As I couldn’t get the vanilla version from LuaRocks to work, I use Debian’s version.

I like Markdown for thigns like this as it is not only simple, but easy for people to read, even if they don’t know markdown or haven’t worked with any other document system than Office or other wisywig document processors.

Lapis

Lapis is the middleware between Lua and Nginx that allows me to write pages simply. Here is some of the code that powers this page:

-- controllers/blog.moon
class Blog extends lapis.Application
  ["blog.post": "/blog/:name"]: =>
    @name = util.slugify @params.name
    @doc = oleg.cache "blogposts", @name, ->
      local data
      with io.open "blog/#{@name}.markdown", "r"
        data = \read "*a"

      discount data, "toc", "nopants", "autolink"

    with io.open "blog/#{@name}.markdown", "r"
      @title = \read "*l"

  render: true

And the view behind this page:

-- views/blog/post.moon
import Widget from require "lapis.html"
class Post extends Widget
  content: =>
    raw @doc

That’s it. That even includes the extra overhead of caching the markdown as HTML in a key->value store called OlegDB (I will get into more detail about it below). With Lapis I can code faster and be much more expressive with a lot less code. I get the syntactic beauty that is Moonscript with the speed and raw power of luajit on top of nginx.

OlegDB

OlegDB is a joke about mayonnaise that has gone too far. It has turned into a full fledged key->value store and I think it is lovely.

Container Abuse

I have OlegDB running as an in-container service. This means that OlegDB does hold some state, but only for things that are worth maintaining the stats of (in my eyes). Having a cache server right there that you can use to speed things up with is a brilliant abuse of the fact that I run a container that allows me to do that. I have Oleg hold the very HTML you are reading right now! When it renders a markdown file for the first time it caches it into Oleg, and then reuses that cached version when anyone after the first person reads the page. I do the same thing in a lot of places in the codebase for this site.


I hope this look into my blog’s tech stack was interesting!


The Saga of plt, Part 1

Permalink - Posted on 2015-02-14 00:00, modified on 0001-01-01 00:00

The Saga of plt, Part 1

The following is adapted from a real story. Parts of it are changed to keep it entertaining to read but the core of the story is maintained. I apologize that this issue in the epic will be shorter than the others, but it gets better.

The Beginning of The Interesting Pain

It all started when I got this seemingly innocuous PM on Freenode:

2015-01-23 [18:32:48] <plt> Hello. I am writting a new ircd and can I have the channel ##ircd please?

This is a fairly common event on larger IRC networks, especially given the length of the channel name and the fact that it references IRC daemons specifically. At this point I had forgotten I owned that channel. So naturally I decided to give it a join and see if the person who requested the channel was worthy of it or had brought enough activity to it such that it was morally correct to hand it off.

This was not the case.

[18:33:54] *** Joins: Xe (xe@unaffiliated/xe)
[18:34:02] <plt> Hello xe.
[18:35:17] <plt> Xe the project name pbircd.
[18:37:09] <plt> Xe the project site is http://sourceforge.net/p/pbircd

In case the site is removed from SourceForge, it is the default sourceforge page.

After taking a look at this and then getting off the call with my family I was on at the point, I decided to reply.

[20:30:49] <Xe> plt: I've decided against giving you my channel
[20:31:03] <Xe> you have no code in your repo.
[20:31:31] <plt> I am currently working on the project. Can I help you in the channel?
[20:32:04] <Xe> if you are working on it
[20:32:11] <Xe> I'd expect to see at least something
[20:32:25] <Xe> for example: https://github.com/Xe/scylla
[20:32:35] <Xe> that's mostly autogenerated code and makefiles, but it's something
[20:33:31] <plt> Take a look at this http://pastebin.com/F8MH3fSs
[20:34:04] <plt> You know it takes a while to write ircd code.
[20:34:16] <Xe> I don't see any commits
[20:34:20] <Xe> not even framework code
[20:34:24] <Xe> or design
[20:34:26] <Xe> or an outline
[20:34:30] <Xe> all I see is that pastebin
[20:34:39] <Xe> which is in no way connected to that git repo
[20:35:07] <plt> I am still adding more features so its not going to be posted on the main web site yet.

The contents of the pastebin looked like a changelog, but that pastebin has since expired or was explicitly deleted. He was all talk and no game. I admit at this point I was pretty tired and frustrated, so I told him off:

[20:35:19] <Xe> fucking commit it then
[20:35:52] <plt> I was going to wait until the code was completed.
[20:36:43] <Xe> yeah good lick then
[20:36:45] <Xe> luck*
[20:37:14] <plt> Itgoing to get done and I am the only one working on the project so what do you expect?
[20:37:29] <Xe> to be able to look at the in-progress code?
[20:39:24] <plt> The code will do you no good because you will not be able to compile it.
[20:39:51] <Xe> then you have nothing
[20:40:06] <plt> I am not required to approve it.
[20:41:08] <plt> I can post the run program on the web site.
[20:42:33] <Xe> then do that
[20:43:28] <plt> Done.

The “run program” was nothing but a wrapper around the nonexistent binary for pbircd and seemed to be compiled in a language that doesn’t respect assembly functions and all of the forms of RE that I know how to do were useless. If you know how to better do RE on arbitrary binaries please let me know.

[20:44:12] <Xe> there are binaries
[20:44:15] <Xe> not source code
[20:44:25] <Xe> this is what you use git for
[20:44:35] <plt> The source code will do you no good since you can not compile it.
[20:52:02] <plt> In order for you to compile it you need the encryption program and I am not going to release the source code.
[20:54:43] <Xe> lol
[20:55:34] <plt> The program is freeware and I have no obligation to release the code under the License agreement.
[21:00:56] <Xe> you also will get no users
[21:03:13] <plt> The company that wrote Conferenceroom has a lot of customers.

ConfrenceRoom was a company that made a commercial IRC daemon. They have lost to Slack and other forms of chat like HipChat. Note here that he says “you can not compile it”. This is true in more ways than you would think. He also claims it is Freeware and not full fledged open source software. As someone who is slightly proactive and paranoid after the Snowden bullshit, I find this highly suspect. However, this “encryption program” was something I was oddly most interested in.

2015-01-24
[12:11:14] <plt> Xe why do you always demand to see the source code?

Curiosity? To learn from other people’s ideas? To challenge myself in understanding another way of thinking about things? To be able to improve it for others to learn from? Those seem like good reasons to me.

[22:46:33] <plt> PBIRCD is a irc daemon.
[22:46:36] <plt> Hello xe

The PB in that name will become apparent later.

[23:09:31] <plt> Would you like to see what I have in the updates?
[23:09:40] <Xe> sure
[23:09:47] <plt> http://pastebin.com/2udHPSyP
[23:13:10] <plt> Tell me what you think about it?
[23:16:32] <plt> I need to take a short break.

Again, the paste is dead (I should really be saving these gems) but it was another set of what appeared to be patch notes.

[23:22:37] <plt> Do you like what I have in the notes?
[23:23:49] <Xe> I still think it's ridiculous  that you don't have the balls to release your code
[23:24:36] <plt> I understand what you telling me.
[23:25:48] <plt> There is no way to working around protecting the encrypted information.
[23:34:19] <plt> Why are you do want to see the code?
[23:43:36] <plt> Xe The encryption is used to encrypt the Operators, Link and the other passwords.

This sounds suspect. Any sane system of encrypting passwords like this would be a mathematical one-way function. By not showing the code like this, is this a two-way function?

2015-01-25
[00:05:55] <plt> Xe Question if the authors that wrote free pgp do not release their source code then why should I have do


Getting Started with Go

Permalink - Posted on 2015-01-28 00:00, modified on 0001-01-01 00:00

Getting Started with Go

Go is an exciting language made by Google for systems programming. This article will help you get up and running with the Go compiler tools.

System Setup

First you need to install the compilers.

$ sudo apt-get install golang golang-go.tools

golang-go.tools contains some useful tools that aren’t part of the standard Go distribution.

Shell Setup

Create a folder in your home directory for your Go code to live in. I use ~/go.

$ mkdir -p ~/go/{bin,pkg,src}

bin contains go binaries that are created from go get or go install. pkg contains static (.a) compiled versions of go packages that are not go programs. src contains go source code.

After you create this, add this and the following to your zsh config:

export GOPATH=$HOME/go
export PATH=$PATH:/usr/lib/go/bin:$GOPATH/bin

This will add the go compilers to your $PATH as well as programs you install.

Rehash your shell config (I use a resource command for this) and then run:

$ go env
GOARCH="amd64"
GOBIN=""
GOCHAR="6"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/xena/go"
GORACE=""
GOROOT="/usr/lib/go"
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
TERM="dumb"
CC="gcc"
GOGCCFLAGS="-g -O2 -fPIC -m64 -pthread"
CXX="g++"
CGO_ENABLED="1"

This will verify that the go toolchain knows where the go compilers are as well as where your $GOPATH is.

Testing

To test the go compilers with a simple todo command, run this:

$ go get github.com/mattn/todo
$ todo add foo
$ todo list
☐ 001: foo

Vim Setup

For Vim integration, I suggest using the vim-go plugin. This plugin used to be part of the standard Go distribution.

To install:

  1. Add Plugin 'fatih/vim-go' to the plugins part of your vimrc.
  2. Run these commands:
$ vim +PluginInstall +qall
$ vim +GoInstallBinaries +qall

This will install the go oracle and the go autocompletion daemon gocode as well as some other useful tools that will integrate seamlessly into vim. This will also run gofmt on save to style your code to the standard way to write Go code.

Resources

Effective Go and the language spec provide a nice overview of the syntax.

The Go blog contains a lot of detailed articles covering advanced and simple Go topics. This page has a list of past articles that you may find useful.

The Go standard library is a fantastic collection of Go code for solving many problems. In some cases you can even write entire programs using only the standard library. This includes things like web application support, tarfile support, sql drivers, support for most kinds of commonly used crypto, command line flag parsing, html templating, and regular expressions. A full list of the standard library packages can be found here.

Variable type declarations will look backwards. It takes a bit to get used to but makes a lot of sense once you realize it reads better left to right.

For a nice primer on building web apps with Go, codegangsta is writing a book on the common first steps, starting from the standard library and working up. You can find his work in progress book here.

Go has support for unit testing baked into the core language tools. You can find information about writing unit tests here.

When creating a new go project, please resist the urge to make the folder in your normal code folder. Drink the $GOPATH koolaid. Yes it’s annoying, yes it’s the language forcing you to use its standard. Just try it. It’s an amazingly useful thing once you get used to it.

Learn to love godoc. Godoc lets you document code like this. This also includes an example of the builtin unit testing support.


Pursuit of a DSL

Permalink - Posted on 2014-08-16 00:00, modified on 0001-01-01 00:00

Pursuit of a DSL

A project we have been working on is Tetra. It is an extended services package in Go with Lua and Moonscript extensions. While writing Tetra, I have found out how to create a Domain Specific Language, and I would like to recommend Moonscript as a toolkit for creating DSL’s.

Moonscript is a high level wrapper around Lua designed to make programming easier. We have used Moonscript heavily in Tetra because of how easy it is to make very idiomatic code in it.

Here is some example code from the Tetra codebase for making a command:

require "lib/elfs"

Command "NAMEGEN", ->
  "> #{elfs.GenName!\upper!}"

That’s it. That creates a command named NAMEGEN that uses lib/elfs to generate goofy heroku-like application names based on names from Pokemon Vietnamese Crystal.

In fact, because this is so simple and elegant, you can document code like this inline.

Command Tutorial

In this file we describe an example command TEST. TEST will return some information about the place the command is used as well as explain the arguments involved.

Because Tetra is a polyglot of Lua, Moonscript and Go, the relevant Go objects will have their type definitions linked to on godoc

Declaring commands is done with the Command macro. It takes in two arguments.

  1. The command verb
  2. The command function

It also can take in 3 arguments if the command needs to be restricted to IRCops only.

  1. The command verb
  2. true
  3. The command function

The command function can have up to 3 arguments set when it is called. These are:

  1. The Client that originated the command call.
  2. The Destination or where the command was sent to. This will be a Client if the target is an internal client or a Channel if the target is a channel.
  3. The command arguments as a string array.
Command "TEST", (source, destination, args) ->

All scripts have client pointing to the pseudoclient that the script is spawned in. If the script name is chatbot/8ball, the value of client will point to the chatbot pseudoclient.

  client.Notice source, "Hello there!"

This will send a NOTICE to the source of the command saying “Hello there!”.

  client.Notice source, "You are #{source.Nick} sending this to #{destination.Target!} with #{#args} arguments"

All command must return a string with a message to the user. This is a good place to do things like summarize the output of the command or if it worked or not. If the command is oper-only, this will be the message logged to the services snoop channel.

  "End of TEST output"

See? That easy.

Command "TEST", ->
    "Hello!"

This is much better than Cod’s

#All modules have a name and description
NAME="Test module"
DESC="Small example to help you get started"

def initModule(cod):
    cod.addBotCommand("TEST", testbotCommand)

def destroyModule(cod):
    cod.delBotCommand("TEST")

def testbotCommand(cod, line, splitline, source, destination):
    "A simple test command"
    return "Hello!"


Thoughts on Community Management

Permalink - Posted on 2014-07-31 00:00, modified on 0001-01-01 00:00

Thoughts on Community Management

Many open source community projects lack proper management. They can put too much of their resources in too few places. When that one person falls out of contact or goes rogue on everyone, it can have huge effects on everyone involved in the project. Users, Contributors and Admins.

Here, I propose an alternative management structure based on what works.

Organization

Contributors and Project Administrators are there to take input/feedback from Users, rectify the situation or explain why doing so is counterproductive. Doing so will be done kindly and will be ran through at least another person before it is posted publicly. This includes (but is not limited to) email, IRC, forums, anything. A person involved in the project is a representative of it. They are the face of it. If they are rude it taints the image of everyone involved.

Access

Project Administrators will have full, unfiltered access to anything the project has. This includes root access, billing access, everything. There will be no reason to hide things. Operational conversations will be shared. All group decisions will be voted on with a simple Yes/No/Abstain process. As such this team should be kept small.

Contributions

Contributors will have to make pull requests, as will Administrators. There will be review on all changes made. No commits will be pushed to master by themselves unless there is approval. This will allow for the proper review and testing procedures to be done to all code contributed.

Additionally, for ease of scripts scraping the commits when something is released, a commit style should be enforced.

Commit Style

The following section is borrowed from Deis’ commit guidelines.


We follow a rough convention for commit messages borrowed from CoreOS, who borrowed theirs from AngularJS. This is an example of a commit:

feat(scripts/test-cluster): add a cluster test command

this uses tmux to setup a test cluster that you can easily kill and
start for debugging.

To make it more formal, it looks something like this:

{type}({scope}): {subject}
<BLANK LINE>
{body}
<BLANK LINE>
{footer}

The {scope} can be anything specifying place of the commit change.

The {subject} needs to use imperative, present tense: “change”, not “changed” nor “changes”. The first letter should not be capitalized, and there is no dot (.) at the end.

Just like the {subject}, the message {body} needs to be in the present tense, and includes the motivation for the change, as well as a contrast with the previous behavior. The first letter in a paragraph must be capitalized.

All breaking changes need to be mentioned in the {footer} with the description of the change, the justification behind the change and any migration notes required.

Any line of the commit message cannot be longer than 72 characters, with the subject line limited to 50 characters. This allows the message to be easier to read on github as well as in various git tools.

The allowed {types} are as follows:

feat -> feature
fix -> bug fix
docs -> documentation
style -> formatting
ref -> refactoring code
test -> adding missing tests
chore -> maintenance

I believe that these guidelines would lead towards a harmonious community.