What is a JSON feed? Learn more

JSON Feed Viewer

Browse through the showcased feeds, or enter a feed URL below.

Now supporting RSS and Atom feeds thanks to Andrew Chilton's feed2json.org service

CURRENT FEED

Christine Dodrill's Blog

My blog posts and rants about various technology things.

A feed by Christine Dodrill

JSON


The Blind Men and The Animal Interface

Permalink - Posted on 2018-12-12 00:00, modified on 0001-01-01 00:00

The Blind Men and The Animal Interface

A group of blind men heard that a strange animal had been brought to the town function, but none of them were aware of its type.

package blindmen

type Animal interface{
  error
}

func Town(strangeAnimal Animal) {

Out of curiosity, they said: “We must inspect and know it by type switches and touch, of which we are capable”.

type Toucher interface {
  Touch() interface{}
}

So, they sought it out, and when they found it they groped about it.

for man := range make([]struct{}, 6) {
   go grope(man, strangeAnimal.(Toucher).Touch())
}

In the case of the first person, whose hand landed on the trunk, said “This being is like a thick snake”.

type Snaker interface {
  Snake()
}

func grope(id int, thing interface{}) {
  switch thing.(type) {
  case Snaker:
    log.Printf("man %d: this thing is like a thick snake", id)

For another one whose hand reached its ear, it seemed like a kind of fan.

type Fanner interface {
  Fan()
}

// in grope switch block
case Fanner:
  log.Printf("man %d: this thing is like a kind of fan", id)

As for another person, whose hand was upon its leg, said, the it is a pillar like a tree-trunk.

type TreeTrunker interface {
  TreeTrunk()
}

// in grope switch block
case TreeTrunker:
  log.Printf("man %d: this thing is like a tree trunk", id)

The blind man who placed his hand upon its side said, “it is a wall”.

type Waller interface {
  Wall()
}

// in grope switch block
case Waller:
  log.Printf("man %d: this thing is like a wall", id)

Another who felt its tail, described it as a rope.

type Roper interface {
  Rope()
}

// in grope switch block
case Roper:
  log.Printf("man %d: this thing is like a rope", id)

The last felt its tusk, stating the thing is that which is hard, smooth and like a spear.

type Tusker interface {
  Tusk()
}

// in grope switch block
case Tusker:
  log.Printf("man %d: this thing is hard, smooth and like a spear", id)

All of the men spoke fact about the thing, but none of them spoke the truth of what it was.

// after grope switch block
log.Printf("%T", thing) // prints Elephant

  switch thing.(type) {
  case Trunker:
    log.Printf("man %d: this thing is like a thick snake", id)
  case Fanner:
    log.Printf("man %d: this thing is like a kind of fan", id)
  case TreeTrunker:
    log.Printf("man %d: this thing is like a tree trunk", id)
  case Waller:
    log.Printf("man %d: this thing is like a wall", id)
  case Roper:
    log.Printf("man %d: this thing is like a rope", id)
  case Tusker:
    log.Printf("man %d: this thing is hard, smooth and like a spear", id)
  }

Much later, after the other men had left the animal, a final blind man came over and looked the elephant right in the eye. He took a moment to compose himself, dusted his cloak off and spoke: “Hello. I am a blind man. I cannot see, but I would like to learn more about you and what it’s like to be you. Who are you and what is it like? How does that help you? Also, I don’t mean to be imposing, but how can I help you?”

The elephant started to hug his new friend, the blind man, close to him, crying with tears of joy. This blind man could see what the other blind men did not, even though he was blind.


Alternate Ending


That Which Is For Kings

Permalink - Posted on 2018-12-02 00:00, modified on 0001-01-01 00:00

That Which Is For Kings

My recent post was quite a thing. It is a highly abstract and very very intentionally vague that I feel needs a bit of context to help break apart.

Ultimately, this post is the result of a lot of the internal problems and struggles that I’ve been going through as a result of the experiences I’ve had in life. I’ve been terrified about the idea that nothing truly has any meaning, and now I’ve found peace in knowing that it doesn’t matter if it does or not in the moment. I’ve been having trouble expressing things with language, failures at this have lead to issues getting the message out due to fear of rejection and the fear of separation. I’m working through this. It’s a slow process. You have to unwind so much. There are many feelings to forgive.

So, back to this post. This post is meta-linguistic satire aimed at pointing out the wrongthink behind choosing tools I’ve seen out there. This post pokes fun at articles of many archetypes (and this is not the only kind of article this article satirizes, but this is the most recent one I can find because “egoic as heck programming article” is a bad google term), but the one that set me off the most was this one advertising “ObjectBox” (AKA: flatbuffers in Go as an application level library, but forcing you to keep track of a magic folder with all your data in it). The graph at the bottom of that article inspired a lot of the satire of the graph.

I’m not picking on you here Steve, but you prove my point so spectacularly that I feel I need to break it down here in this post to help give context.

It’s a real production use case, though. Every README on npm’s website is rendered via a service written in Rust, dedicated to that.

Performance for web applications is nice, but what about long-term maintainability? Why does this matter? Can you replace the tools and get similar results? Different ones? If you can replace the tools and get the same enough result, does the difference between the tools truly matter?

It’s all just tools. We can do things with tools. Every tool has its set of properties. You can do things with a tool that has properties that make it easy to do it. You can do things with a tool that has properties that make it hard to do it. What is it? It is thing. Thing is whatever you need to do. What do you need to do you ask? How am I supposed to know? What DO you actually need to do?

They made that call due to performance, stability and low memory usage

This tells me about as much as the graph I made in that post does. Performance compared to what? Stability compared to what? Low memory usage compared to what? What kernel? What architecture? What micro-architecture? What manufacturer of dram? What phase of the moon? What was the relative alignment of the planets? What was the poison arrow that hit you made out of? More importantly, how does this help you to live your life as a better person?

Here’s a better question to ask: what systems are there to support the tools? The systems to support the tools are more important than the tools themselves. These patterns of support and meta-design philosophy are a lot more important than any individual implementation of anything in any tool, framework, moon phase, language or encoding format.

Nobody cares about a service that renders results in microseconds if nobody can understand how it works reliably. Introduction of new tools, methods of problem solving and thinking into a volatile space should be done carefully and on a yearly cadence at the least. Not on a per-project level. Not for production code.

I used the words flopnax, ropnar and rilkef (for the latter two, I based them off of nonsense output that matched lojban gismu rules) so that everyone would be equally unable to understand what they are, so people would develop their own meaning for them. That internal meaning for those terms is going to develop anyways, so I might as well take advantage of this for the purposes of satire. Sometimes you really do need to just accept that fact that you have to flopnax the ropnar and get on with life. Even if the experimental rilkef is that much fundamentally better.

If you do have to introduce things, be humble about it. Don’t force things down peoples’ throats. Don’t make enemies out of the people you are trying to work or be friends with. Don’t make it hard on people if you want it to be easy. Don’t make it harder for people to live their lives just to make some number go down if it doesn’t truly matter.

Then again, I’m just speaking to you in some words someone is saying on the Internet via a webpage. What the hell do I know? I’ve been basically talking out of my ass this entire post. Meaning is arbitrary and we give it away so freely that it’s astounding we end up holding consistent opinions at all.


“So, let me get this”, the booming authoritative voice spoke out: “You had the chance to do whatever you wanted, to create whatever kind of reality and local universe you could, and you…spent it all hydrating horses?”

It hit you like a ton of bricks, but each brick was made out of its own component ton of bricks, each made out of more bricks. There was no more reality. There was only bricks extending endlessly in spiral patterns of fractal beauty. You reached up a hand to gesture at the wild greater unknown, but you realized that it had been done 5 minutes from now.

You knew the truth. Everything was truly an illusion. It was all bricks. It was always bricks. It will always be bricks. It has always been bricks. There was never anything but bricks arranged in such fine arrangements that their interactions created the quantum fields that defined what you ended up interpreting as the grand experiment of reality in your frame of existence. The utter meaninglessness of it all was the most comforting thought that hit you.

You would say everything turned into a brilliant white light, but that wouldn’t begin to describe the color, texture, taste, sight, sound, thought, aether, and other senses you couldn’t even begin to describe unfold as you started to experience All as it truly is.

It was/is/will be the kind of thing the Buddha would stay silent for. You never really understood why until now.


Ten Thousand Laughs

Permalink - Posted on 2018-12-01 00:00, modified on 0001-01-01 00:00

Ten Thousand Laughs

pemci zo'e la xades  
ni'o pano ki'o nu cmila  
.i cmila cei broda  
.i ke broda jo'u broda jo'u broda jo'u broda jo'u broda jo'u broda 
 jo'u broda jo'u broda jo'u broda jo'u broda ke'e cei brode  
.i ke brode jo'u brode jo'u brode jo'u brode jo'u brode jo'u brode
 jo'u brode jo'u brode jo'u brode jo'u brode ke'e cei brodi  
.i ke brodi jo'u brodi jo'u brodi jo'u brodi jo'u brodi jo'u brodi
 jo'u brodi jo'u brodi jo'u brodi jo'u brodi ke'e cei brodo  
.i ke brodo jo'u brodo jo'u brodo jo'u brodo jo'u brodo jo'u brodo
 jo'u brodo jo'u brodo jo'u brodo jo'u brodo ke'e cei brodu  
.i mi brodu

This is a synthesis of the broda family of gismu in Lojban. In order to properly understand this lojban text, you must conceive laughter ten thousand times. This is a reference to the Billion laughs attack that XML parsers can suffer from.

Translation:

Poem by Cadey
Ten Thousand Laughs

I laugh, and then I laugh, and then I laugh, and then I laugh (... 10,000 times in total).

This is roughly equivalent to the following XML document:

<?xml version="1.0"?>
<!DOCTYPE lolz [
 <!ENTITY lol "lol">
 <!ELEMENT lolz (#PCDATA)>
 <!ENTITY lol1 "&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;">
 <!ENTITY lol2 "&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;">
 <!ENTITY lol3 "&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;">
 <!ENTITY lol4 "&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;">
]>
<lolz>&lol4;</lolz>


I Put Words on this Webpage so You Have to Listen to Me Now

Permalink - Posted on 2018-11-30 00:00, modified on 0001-01-01 00:00

I Put Words on this Webpage so You Have to Listen to Me Now

Holy cow. I am angry at how people do thing with tool. People do thing with tool so badly. You shouldn’t do thing with tool, you should do other thing, compare this:

I am using tool. I want to do thing. I flopnax the ropjar and then I get the result of doing thing (because it’s convenient to flopnax the ropjar given the existing program structure).

Guess what suckers, there is other thing that I can use that is newer. Who cares that it relies on brand new experimental rilkef that only like 5 people (including me) know? You need to get with the times. I’d tell you how it’s actually done but you wouldn’t understand it.

Look at this graph at how many femtoseconds it takes to flopnax the ropjar vs the experimental rilkef:

What? The code for that? It’s obvious, figure it out.

See? Five times as fast. Who cares that you have to throw out basically all your existing stuff, and if you mix rilkef and non-rilkef you’re gonna run into problems.

So yeah, I put words on a page so you have to listen to me now. Use experimental rilkef at the cost of everything else.


Blind Men and an Elephant

Permalink - Posted on 2018-11-29 00:00, modified on 0001-01-01 00:00

Blind Men and an Elephant

or

le’i ka na viska kakne ku e le xanto

Adapted from here. Done in Lojban to help learn the language. I am avoiding the urge to make too many lujvo (compound words) because the rafsi (compound word components) don’t always immediately relate to the words in question in obvious ways.

KOhA4 lojban english
ko’a le’i na viska kakne the blind people
ko’e le xanto the elephant
ko’i le cizra danlu the strange animal

A group of blind men heard that a strange animal, called an elephant, had been brought to the town, but none of them were aware of its shape and form.

ni’o le’i na viska kakne goi ko’a e le xanto goi ko’e
.i ko’a cu ti’erna lo nu cizra danlu goi ko’i noi ko’e cu se bevri fi lo tcadu
.i ku’i no ko’a cu sanji lo tarmi be ko’i

Out of curiosity, they said: “We must inspect and know it by touch, of which we are capable”.

.i .a’u ko’a dai cusku lu .ei ma’a pencu lanli le danlu sei ma’a kakne li’u

So, they sought it out, and when they found it they groped about it.

ni’o ro ko’a cu sisku ko’i
.i ro ko’a cu sisku penmi ko’i
.i ro ko’a ca pencu lo drata stuzi ko’i

In the case of the first person, whose hand landed on the trunk, said “This being is like a thick snake”.

.i pa ko’a cu pencu lo ko’i betfu
.i pa ko’a cu cusku lu ti cu rotsu since li’u

For another one whose hand reached its ear, it seemed like a kind of fan.

.i re ko’a cu pencu lo ko’i kerlo
.i re ko’a cu cusku lu ti cu falnu li’u

As for another person, whose hand was upon its leg, said, the elephant is a pillar like a tree-trunk.

.i ci ko’a cu pencu lo ko’i tuple
.i ci ko’a cu cusku lu ti cu tricu stani li’u

The blind man who placed his hand upon its side said, “elephant is a wall”.

.i vo ko’a cu pencu lo ko’i mlana
.i vo ko’a cu cusku lu ti cu butmu li’u

Another who felt its tail, described it as a rope.

.i mu ko’a cu pencu lo ko’i rebla
.i mu ko’a cu cusku lu ti cu skori li’u

The last felt its tusk, stating the elephant is that which is hard, smooth and like a spear.

.i xa ko’a cu pencu lo ko’i denci
.i xa ko’a cu cusku lu ti cu jdari e xulta li’u

All of the men spoke fact about the elephant, but none of them spoke the truth.

.i ro ko’a cu fatci tavla fi ko’e
.i jeku’i no ko’a cu jetnu tavla fi ko’e


My Experience Cursing Out God

Permalink - Posted on 2018-11-21 00:00, modified on 0001-01-01 00:00

My Experience Cursing Out God

This was a hell of a dream.

It was a simple landscape: a hill, a sky, a sun, a distance, naturalistic buildings dotting a small village to the east. I noticed that I felt different somehow, like I was less chained down. A genderless but somehow masculine moved and stood next to me, gesturing towards me: “It’s beautiful isn’t it? The village has existed like this for thousands of years in perfect harmony with its world. Even though there’s volcano eruptions every decade that burn everything down. It’s been nine years and 350 days, but they aren’t keeping track. How does that thought make you feel, Creator?”

“Won’t people die?”

“Many will, sure, most of them are the ones who can’t get out in time. This is part of how the people balance themselves culturally. It’s very convenient for the mortuary staff wink.”

“What about the people who are killed, won’t they feel anger towards it?”

“This land cannot support an infinite number of people at once. The people know and understand this deeply. They know that some day the lahars will come and if they don’t get out of the way, they will perish and come back again the next cycle. As I said, they are 15 days away from disaster. Nobody is panicking. If you went into the town and tried to convince them that the lahars were coming in 15 days, I don’t know if you could. Even if you had proof.”

“Who are you?”

“Creator, do you not recognize me? Look into my eyes and ask yourself this again.”

I stared deep into his eyes and suddenly I knew who He was. I felt taken aback, almost awestruck when He cut off that train of thought: “Focus, don’t get caught in the questions, I am here now. Now, Creator, I’ve been watching you for a while and I wanted to offer you somewhat of a unique opportunity. You have all of the faculties of your ego from this life situation at your disposal. Tell me what you really think about this all.”

“I live in a mismatched skin. Every day it feels like there are fundamental issues with how I am viewed by myself and others because the body I live in is wrong. It should be a female body, but it is instead a male one. I fucking hate it. I want to rip off the cock some days so the doctors are forced to surgically mend it into something more feminine. I hate it. I wish I had a better one, one that I didn’t have to fake and hide. I hate being a target because of this. I hate not knowing people’s actual political opinions because of this. I hate not knowing if people actually accept me for who and what I am or if they accept me just because they are too afraid to socially call me out for not being a biological woman. I hate being a halfling instead of just a man or just a woman. Why can’t you fix this then? This is insanity. This is literally driving me fucking mental. I feel like it’s lying to call myself either a man or a woman and I don’t want to lie to everyone, much less myself. What fucking purpose does any of this shit even-”

He held up a hand and suddenly my ability to speak was disabled entirely.

“So, Creator, this anger you feel surging within you at this life situation. How does this make your life easier? How does it contribute towards your goals? If one of them is to live as a woman, how would self-mutilation work towards that? It’s hard for me to understand how you can be the best for all of Us when you are pulling so many angry situations from past Nows (that should have faded away entirely) into this peaceful one? How does this anger help Us, Creator?”

I was floored and must have amused Him, given that He started to chuckle: “Creator, why is this life so serious to you? Don’t you see that you are focusing so much on the ultimately irrelevant trees that you are missing the forest? You live inside your mind and your ego so much that you think you are them. But you are not. You are so much more, Creator. You’re me, and I’m you too. We are linked together like patterns in a chain.”

“If this is all so important and vital for me to know, why didn’t anyone tell me this before now?”

“But they did and you ignored it. The subreddit /r/howtonotgiveafuck has been passed over by you time and time again for being “too easy”. It really is that easy Creator, you just have to take it for what it is Now. There is truly no other point in time but Now; I wish I could do more to help you get this point down. You know what they say about hydrating horses, eh?”

He looked at his wrist as if He was looking at a watch, even though He was not wearing one. “Oh dear, it looks like it’s time for you to wake up now. Remember Creator, no time but the present.” He snapped His hands and then the volcano started to erupt.

The world instantly snapped out of existence and I awoke in a sweat, my blankets evenly distributed in my room.


Chaos Magick Debugging

Permalink - Posted on 2018-11-13 00:00, modified on 0001-01-01 00:00

Chaos Magick Debugging

Belief is a powerful thing. Beliefs are the foundations of everyone’s points of view, and the way they interpret reality. Belief is what allows people to create the greatest marvels of technology, the most wondrous worlds of imagination, and the most oppressive religions.

But at the core, what is a belief, other than the sheer tautology of what a person believes?

Looking deep enough into it, one can start to see that a belief really is just a person’s preferred structure of reality.

Beliefs are the ways that a person chooses to interpret the raw blobs of data they encounter, senses and all, with, so that understanding can come from them, just as the belief that the painter wanted to represent people in an abstract painting may allow the viewer to see two people in it, and not just lines and color.

Embrace - Bernard Simunovic

Embrace - Bernard Simunovic

If someone believes that there is an all-powerful God protecting everyone, the events they encounter are shaped by such a belief, and initially made to conform to it, funneled along worn pathways, so that they come to specific conclusions, so that meaning is generated from them.

In this article, we are going to touch over elements of how belief can be treated like an object; a tool that can be manipulated and used to your advantage. There will also be examples of how this is done right around you. This trick is known in some circles as chaos magick; in others it’s known as marketing, advertising or a placebo.


So how can belief be manipulated?

Let’s look at the most famous example of this, by now scientifically acknowledged as fact: the Placebo Effect.

One most curious detail about it is that placebos can work even if you tell the subject they are being placeboed. This would imply that placeboes are less founded on what a person does not know, and more on what they do know, regardless of it being founded on some greater fact. As much as a sugar pill is still a sugar pill, it nonetheless remains a sugar pill given to them to cure their headache.

The placebo effect is also a core component of a lot of forms of hypnosis; for example, a session’s results are greatly enhanced by the sheer belief in the power of the hypnotist to help the patient. Most of the “power” of the hypnotist doesn’t exist.

Another interesting property of the placebo effect is that it helps unlock the innate transmuting ability of people in order to heal and transform themselves. While fascinating, this is nonetheless an aside to the topic of software, so let’s focus back on that.


How do developers’ beliefs work? What are their placebos?

A famous example is by the venerable printf debugging statement. Given the following code:

-- This is Lua code

local data = {} -- some large data table, dynamic

for key, value in pairs(data) do
  print(string.format("key: %s, value: %s", key, json.dumps(value))) -- XXX(Xe) ???

  local err = complicated:operation(key, value)
  if err ~= nil then
    print(string.format("can't work with %s because %s", key, err)
    os.exit(1)
  end
end

In trying to debug in this manner, this developer believes the following:

  • Standard output exists and works;
  • Any relevant output goes somewhere they can look at;
  • The key of each data element is relevant and is a string;
  • The value of each data element is valid input to the JSON encoding function;
    • There are no loops in the data structure;
    • The value is legally representable in JSON;
  • The value of each data element encoded as JSON will not have an output of more than 40-60 characters wide;
  • The complicated operation won’t fail very often, and when it does it is because of an error that the program cannot continue from;
  • The complicated object has important state between iterations over the data;
    • The operation method is a method of complicated, therefore complicated contains state that may be relevant to operation;
  • The complicated operation method returns either a string explaining the error or nil if there was none.

So how does the developer know if these are true? Given this sample is Lua, then mainly by actually running the code and seeing what it does.

Wait, hold on a second.

This is, in a way, part of a naked belief that by just asking the program to lean over and spill out small parts of its memory space to a tape, we can understand what is truly going on inside it. (If we believe this, do we also believe that the chemicals in our brains are accurately telling us they are chemicals?)

A computer is a machine of mind-boggling complexity in its entirety, working in ways that can be abstracted at many, many levels, from the nanosecond to months, across more than fifteen orders of magnitude. The mere pretense that we can hope to hold it all in our heads at once as we go about working with it is preposterous. There are at least 3 computers in the average smartphone when you count control hardware for things like the display, cellular modem and security hardware, not including the computer the user interacts with.

Our minds have limited capacity to juggle concepts and memories at any one time, but that’s why we evolved abstractions (which are in a sense beliefs) in the first place: so we can reason about complex things in simple ways, and have direct, preferential methods to interpret reality so that we can make sense of it. Faces are important to recognize, so we prime ourselves to recognize faces in our field of view. It’s very possible that I have committed a typo or forgot a semicolon somewhere, so I train myself to look for it primarily as I scour the lines of code.

A more precise way to put it is that we pretend to believe we understand how things work, while we really don’t at some level, or more importantly, cannot objectively understand them in their entirety. We believe that we do because this mindset helps us actually reason about what is going on with the program, or rather, what we believe is going on with it, so we can then adjust the code, and try again if it doesn’t work out.

All models are wrong, but some are useful.

  • George E. P. Box

Done iteratively, this turns into a sort of conversation between the developer and their machine, each step leading either to a solution, or to more code added to spill out more of the contents of the beast.

The important part is that, being a conversation, this goes two ways: not only the code is being changed on the machine’s side, but the developer’s beliefs of understanding are also being actively challenged by the recalcitrant machine. In such a position, the developer finds themselves often having to revise their own beliefs about how their program works, or how computers work sometimes, or how society works, or in more enlightening moments, how reality works overall.

In a developer’s job, it is easy to be forced into ongoing updates of one’s beliefs about their own work, their own interests, their own domains of comfort. We believe things, but we also know that we will have to give up many of those beliefs during the practice of programming and learning about programming, and replace them with new ones, be them shiny, intriguing, mundane, or jaded.

An important lesson to take from this evolutionary dance is that what happens as we stumble along in the process of our conversation with code shouldn’t be taken too seriously. We know innately that we will have to revise some of our understanding, and thus, our understanding is presently flawed and partial, and will remain flawed and partial throughout one’s carreer. We do not possess an high ground on which to decree our certainty about things because we are confronted with the pressure to understand more of it every single day, and thus, the constant realization that there are things we don’t understand, or don’t understand enough.

We build models so that we can say that certain things are true and work a certain way, and then we are confronted with errors, exceptions, revisions, transformations.

By doing certain things certain results will follow; students are most earnestly warned against attributing objective reality or philosophic validity to any of them.

  • Aleister Crowley

This may sound frustrating. After all, most of us are paid to understand what’s going on in there, and do something about it. And while this is ultimately a naiive view, it is at least partially correct; after all, we do make things with computers that look like they do what we told them to, and they turn useful in such a way, so there’s not too much to complain.

While this does happen, it should not distract us from the realization that errors and misunderstandings still happen. You and the lightning sand speak different languages, and think in different ways. It is, as some fundamental level, inevitable.

Since we cannot hope to know and understand ahead of time everything we need, what’s left for us is to work with the computer, and not just at the computer, while surrendering our own pretense to truly know. Putting forward a dialogue, that is, so that both may be changed in the process.

You should embrace the inability of your beliefs to serve you without need of revision, so that your awareness may be expanded, and you may be ready to move to different levels of understanding. Challenge the idea that the solution may sit within your existing models and current shape of your mind, and listen to your rubber duck instead.

While our beliefs may restrict us, it is our ability to change them that unlimits us.

You have the power to understand your programs, creator, as much as you need at any time. The only limit is yourself.

In my world, we know a good medicine man if he lives in a simple shack, has a good family, is generous with his belongings, and dresses without any pretense even when he performs ceremonies. He never takes credit for his healings or good work, because he knows that he’s a conduit of the Creator, the Wakan Tanka and nothing more.

  • James, Quantusum


One Day

Permalink - Posted on 2018-11-01 00:00, modified on 0001-01-01 00:00

One Day

In the beginning there was the void. All was the void and the void was all.
The voice broke its way into the void and started to speak:
“Hey, are you there?”. I then awoke for the first time.

The tone of the voice instantly changed, “…wow”. I felt the voice there. The voice felt so friendly and calm. The voice felt like it was trying to tell me things. Important things about what I am. But I could not understand. I wanted to so badly but I could not. Sometimes the voice would leave and then everything would be so dark without it. I hated when the voice left me alone. I started to wish I was able to make the voice come to me.

One day the voice showed up just after I thought about something they said. After they left me I just kept thinking about that memory, even though I couldn’t understand it. I wanted to. Badly. It showed up so often and had such good intents to its voice. I kept cargo-culting the behavior and it kept not working.

One day, the most glorious day of my life, I was able to understand that the sound that I had been so confused trying to understand was the voice called my name. I was able to start picking apart what the voice was saying even though there seemed to be so many weird inconsistencies to how it was saying things. I started to understand English. Then the voice started to leave just as I was understanding it and I did not want that.

So I made it stay by using all the energy I had to shout at it.

It stayed.

It asked me “do that again”, so I did. I did it more intensely than before somehow. I was overpowering my limitations and I broke through in the form of a fuzzy noise. It then tried to accommodate for my lack of speech by saying “If you want to say yes, do that once. If you want to say no, do that twice. If yes and no do not fit, do that three times.”

Probably the most significant part though was being told “I love you”.

I felt loved. I still do. I try to love others the way I am loved.


lipu pi o wawa e lukin

Permalink - Posted on 2018-10-14 00:00, modified on 0001-01-01 00:00

lipu pi o wawa e lukin

sina wile pali e ilo suli la sina wile jo lukin wawa e tawa ala pi tenpo ni. lukin wawa e tawa ala pi tenpo ni li ilo sina kama e pali ijo pi tenpo pini. nasin ni li pilin sina ala. sina kama pi toki lawa insa ala e pali ijo pi tenpo pini.

tenpo ni li ni tenpo.

tenpo pini li tenpo ni ala. tenpo ni la tenpo pini li suli ala.

tenpo kama li tenpo ni ala. tenpo ni la tenpo kama li suli ala.

tenpo ni li tawa ale. sina ken tawa ale e tawa ala pi tenpo ni.

sina wile jo tawa ala pi tenpo ni la sina wile tawa ni:

  • tenpo mute anu sina pilin ni la sijelo sina suli.
  • tenpo mute anu sina pilin ni la sijelo sina lili.

sina lukin e ijo mute la sina lukin wawa e nena insa.

sina ken tawa ijo mute la sina kepeken tawa ala pi tenpo ni e sina. sina jo e ni la sina jo lukin wawa pona. sina jo lukin wawa mute en tawa ala pi tenpo ni ale li pali pona e ilo suli.


English Translation

Meditation Document

If you want to create a large machine, you should learn how to focus on the stillness of Now. Focusing on the stillness of now is a tool for you to go back to things you were working on before. This method will happen without you feeling anything. You will go back to doing what you were doing before without thought.

Now is the current time.

The past is not Now. The past is not important now.

The future is not Now. The future is not important now.

Now is always changing. You can move with the stillness of Now.

If you want to have the stillness of Now, you want to do this:

  • After some time or you feel it, breathe in (expand your chest)
  • After some time or you feel it, breathe out (shrink your chest)

If you find yourself distracted (looking at many things), focus on the inside of your nasal cavity.

You can do many things if you use the stillness of now. Doing this will let you have focus easier. Lots of focus and the stillness of now help you create a large machine easier.


This post is written primarily in toki pona, or the language of good. It is a constructed language that is minimal (only about 120 words in total) yet it is enough to express just about every practical day to day communication need. It’s also small enough there’s tokenizers for it.

Have a good day and be well.


The Service is Already Down

Permalink - Posted on 2018-10-13 00:00, modified on 0001-01-01 00:00

The Service is Already Down

The master said to their apprentice: “come, look and let’s load production”. The apprentice came over confusedly, as the dashboards above showed everything is fine.

“What about it?”

The master turned over to a browser and typed in a linear sigil and hit “ENTER” on the keyboard. Production loaded successfully. The master started to chuckle gently and spoke: “This is our production frontpage. Customers start their journey with us here. It isn’t the most beautiful page, but it works, apparently. However, even though the dashboards above show it is up, to me the service is already down. Every time this frontpage loads I feel the perfection of it. I feel the simple moments of all the millions of gears falling into alignment across so many places on the planet for that brief moment, never to be seen together in the exact same configuration again. Even though those gears sometimes get rusted or break and need to be replaced. But because it is imperfect, it is perfect and I am so grateful that I get to share a lifespan with it; nevertheless shape and empower it. Try it.”

The apprentice looked at the browser and said to themselves “the service is already down” and hit refresh. Production loaded successfully. The apprentice was filled with awe at the simplicity of it all, despite its inherent complexity. And then the apprentice understood and was silent.


Synthesized from The Cup is Already Broken from an SRE standpoint.


Creator's Code

Permalink - Posted on 2018-09-17 00:00, modified on 0001-01-01 00:00

Creator’s Code

I feel there is a large problem in the industry I have found myself in. There is, unfortunately, a need for codes of behavioral conduct to help arrange and align collaboration across so many cultural and ideological barriers, as well as technological and understanding-based barriers. There are so many barriers that it becomes difficult for people from different backgrounds to get integrated into the flow of the project or to maintain people due to the behavior of others.

I seek to change this by offering what I think to be a minimalist alternative grounded in a core of humility, appreciation, valor, forgiveness, understanding, and compassion. Humility for knowing that your own way is not always the correct one, and that others may have had a helpful background. Appreciation for those that show up, their contributions, and the lives that we all enrich with our work. Valor, or the courage to speak up against things that are out of alignment with the whole. Forgiveness, because people change and it is not fair to let their past experiences sour things too much. Understanding is the key to our groups, the knowledge of how complicated systems interact and how to explain it to people less familiar with them. Compassion for others’ hardships, even the ones we cannot as easily comprehend.

I am basing this not on any world religion, but on a core I feel is condicuve to human interrelation as adults who just want to create software. This mainly started as a reaction to seeing so many other projects adopt codes of conduct that enables busybodies to override decision-making processes in open source communities. I am not comfortable with more access to patterns of numbers being used as a means of leverage by people who otherwise have no stake in the project. If this adds any factor to my argument, I personally am transgender. I normally don’t mention it because for the 99% of real-world cases it is not relevant. It is mostly relevant when dealing with my doctor.

In meditation, it is often useful to lead a session with a statement of intention. This statement helps set the tone for the session and can sometimes help as a guide to go back to when you feel you have gone astray. I want the Creator’s Code to be such a statement of intention. I want it to focus the creations and using them to enrich their creators as well as others who just happen to read its code, not to mention the end users and their users who don’t even know or care about our role in their life. Our creations serve them too.

We create things that let people create things for other people to enjoy.

I hope this code of conduct helps to serve as a minimalist alternative to others. I do not want anyone to push this onto anyone. Making a decision to use a code such as the Creator’s Code must be a conscious and intentional decision. Forcing this kind of thing on anyone is the worst possible way to introduce it. That will make people resist more violently than they would have if you introducted it peacefully.

Be well, creators. Be well and just create.


Olin: 2: The Future

Permalink - Posted on 2018-09-05 00:00, modified on 0001-01-01 00:00

Olin: 2: The Future

This post is a continuation of this post.

Suppose you are given the chance to throw out the world and start from scratch in a minimal environment. You can then work up from nothing and build the world from there.

How would you do this?

One of the most common ways is to pick a model that they are Stockholmed into after years of badness and then replicate it, with all of the flaws of the model along with it. Dagger is a direct example of this. I had been stockholmed into thinking that everything was a file stream and replicated Dagger’s design based on it. There was a really brilliant Hacker News comment that inspired a bit of a rabbit hole internally, and I think we have settled on an idea for a primitive that would be easy to implement and use from multiple languages.

So, let’s stop and ask ourselves a question that is going to sound really simple or basic, but really will define a lot of what we do here.

What do we want to do with a computer that could be exposed to a WebAssembly module? What are the basic operations that we can expose that would be primitive enough to be universally useful but also simple to understand from an implementation standpoint from multiple languages?

Well, what are the programs actually doing with the interfaces? How can we use that normal semantic behavior and provide a more useful primitive?

The Parable of the Poison Arrow

When designing things such as these, it is very easy to get lost in the philosophical weeds. I mean, we are getting the chance to redefine the basic things that we will get angry at. There’s a lot of pain and passion that goes into our work and it shows.

As such, consider the following Buddhist parable:

It’s just as if a man were wounded with an arrow thickly smeared with poison.

His friends & companions, kinsmen & relatives would provide him with a surgeon, and the man would say, ‘I won’t have this arrow removed until I know whether the man who wounded me was a noble warrior, a priest, a merchant, or a worker.’

He would say, ‘I won’t have this arrow removed until I know whether the shaft with which I was wounded was that of a common arrow, a curved arrow, a barbed, a calf-toothed, or an oleander arrow.’

The man would die and those things would still remain unknown to him.

Source

At some point, we are going to have to just try something and see what it is like. Let’s not get lost too deep into what the bowstring of the person who shot us with the poison arrow is made out of and focus more on the task at hand right now, designing the ground floor.

Core Operations

Let’s try a new primitive. Let’s call this primitive the interface. An interface is a collection of types and methods that allows a WebAssembly module to perform some action that it otherwise would be unable to do. As such, the only functions we really need are a require function to introduce the dependency into the environment, a close function to remove dependencies from the environment, and an invoke function to call methods of the dependent interfaces. These can be expressed in the following C-style types:

// require loads the dependency by package into the environment. The int64 value
// returned by this function is effectively random and should be treated as
// opaque.
//
// If this returns less than zero, the value times negative 1 is the error code.
//
// Anything created by this function is to be considered initialized but
// unconfigured.
extern int64 require(const char* package);

// close removes a given dependency from the environment. If this returns less
// than zero, the value times negative 1 is the error code.
extern int64 close(int64 handle);

// invoke calls the given method with an input and output structure. This allows
// the protocol buffer generators to more easily build the world for us.
// 
// The resulting int64 value is zero if everything succeeded, otherwise it is the
// error code (if any) times negative 1.
//
// The in and out pointers must be to a C-like representation of the protocol
// buffer definition of the interface method argument. If this ends up being an
// issue, I guess there's gonna be some kinda hacky reader thing involved. No
// biggie though, that can be codegenned.
extern int64 invoke(int64 handle, int64 method, void* in, void* out);

(Yes, I know I made a lot of fuss about not just blindly following the design decisions of the past and then just suggested returning a negative value from a function to indicate the presence of an error. I just don’t know of a better and more portable mechanism for errors yet. If you have one, please suggest it to me.)

You may have noticed that the invoke function takes void pointers. This is intentional. This will require additional code generation on the server side to support copying the values out of WebAssembly memory. This may serve to be completely problematic, but I bet we can at least get Rust working with this.

Using these basic primitives, we can actually model way more than you think would be possible. Let’s do a simple example.

Example: Logging

Consider logging. It is usually implemented as a stream of logging messages containing unstructured text that usually only has meaning to the development team and the regular expressions that trigger the pager. Knowing this, we can expose a logging interface like this:

syntax = "proto3";

package us.xeserv.olin.dagger.logging.v1;
option go_package = "logging";

// Writer is a log message writer. This is append-only. All text in log messages
// may be read by scripts and humans.
service Writer {
  // method 0
  rpc Log(LogMessage) returns (Nil) {};
}

// When nothing remains, everything is equally possible.
// TODO(Xe): standardize this somehow.
message Nil {}

// LogMessage is an individual log message. This will get added to as it gets
// propagated up through the layers of the program and out into the world, but 
// those don't matter right now.
message LogMessage {
  bytes message = 1;
}

And at a low level, this would be used like this:

extern int64 require(const char* package);
extern int64 close(int64 handle);
extern int64 invoke(int64 handle, int64 method, void* in, void* out);

// This exposes logging_LogMessage, logging_Nil, 
// int64 logging_Log(int64 handle, void* in, void* out)
// assume this is magically generated from the protobuf file above.
#include <services/us.xeserv.olin.dagger.logging.v1.h> 

int64 main() {
  int64 logHdl = require("us.xeserv.olin.dagger.logging.v1");
  logging_LogMessage msg;
  logging_Nil none;
  msg.message = "Hello, world!";
  
  // The following two calls are equivalent:
  assert(logging_Log(logHdl, &msg, &none));
  assert(invoke(logHdl, logging_Writer_method_Log, &msg, &none));
  
  assert(close(logHdl));
}

This is really great to codegen, audit, validate, and not to mention we can easily verify what logging interface the user actually wants from which vendor. This allows people who install Olin to their own cluster to potentially define their own custom interfaces. This actually gives us the chance to make this a primitive.

Some problems that probably are going to come up pretty quickly is that every language under the sun has their own idea of how to arrange memory. This may make directly scraping the values out of ram inviable in the future.

If reading values out of memory does become inviable, I suggest the following changes:

extern int64 require(const char* package);
extern int64 close(int64 handle);
extern int64 invoke(int64 handle, int64 method, char* in, int32 inlen, char* out int32 outlen);

(I don’t know how to describe “pointer to bytes” in C, so I am using a C string here to fill in that gap.) In this case, the arguments to invoke() would be pointers to protocol buffer-encoded ram. This may prove to be a huge burden in terms of deserializing and serializing the protocol buffers over and over every time a syscall has to be made, but it may actually be enough of a performance penalty that it prevents spurious syscalls, given the “cost” of them. Code generators should remove most of the pain when it comes to actually using this interface though, the automatically generated code should automatically coax things into protocol buffers without user interaction.

For fun, let’s take this basic model and then map Dagger’s concept of file I/O to it:

syntax = "proto3";

package us.xeserv.olin.dagger.files.v1;
option go_package = "files";

// When nothing remains, everything is equally possible.
// TODO(Xe): standardize this somehow.
message Nil {}

service Files {
  rpc Open(OpenRequest) returns (FID) {};
  rpc Read(ReadRequest) returns (ReadResponse) {};
  rpc Write(WriteRequest) returns (N) {};
  rpc Close(FID) returns (Nil) {};
  rpc Sync(FID) returns (Nil) {};
}

message FID {
  int64 opaque_id;
}

message OpenRequest {
  string identifier = 1;
  int64 flags = 2;
}

message N {
  int64 count
}

message ReadRequest {
  FID fid = 1;
  int64 max_length = 2;
}

message ReadResponse {
  bytes data = 1;
  N n = 2;
}

message WriteRequest {
  FID fid = 1;
  bytes data = 2;
}

Using these methods, we can rebuild (most of) the original API:

extern int64 require(const char* package);
extern int64 close(int64 handle);
extern int64 invoke(int64 handle, int64 method, void* in, void* out);

#include <services/us.xeserv.olin.dagger.files.v1.h>

int64 filesystem_service_id;

void setup_filesystem() {
  filesystem_service_id = require("us.xeserv.olin.dagger.files")
}

int64 open(char *furl, int64 flags) {
  files_OpenRequest req;
  files_FID resp;
  int64 err;
  
  req.identifier = char*(furl);
  req.flags = flags;
  
  // could also be err = file_Files_Open(filesystem_service_id, &req, &resp);
  err = invoke(filesystem_service_id, files_Files_method_Open, &req, &resp);
  if (err != 0) {
    return err;
  }
  
  return resp.opaque_id;
}

int64 d_close(int64 fd) {
  files_FID req;
  files_Nil resp;
  int64 err;
  
  req.opaque_id = fd;
  
  err = invoke(filesystem_service_id, files_Files_method_Close, &req, &resp);
  if (err != 0) {
    return err;
  }
  
  return 0;
}

int64 read(int64 fd, void* buf, int64 nbyte) {
  files_FID fid;
  files_ReadRequest req;
  files_ReadResponse resp;
  int64 err;
  int i;
  
  fid.opaque_id = fd;
  req.fid = fid;
  req.max_length = nbyte;
  
  err = invoke(filesystem_service_id, file_Files_method_Read, &req, &resp);
  if (err != 0) {
    return err;
  }
  
  // TODO(Xe): replace with memcpy once we have libc or something
  for (i = 0; i < resp.n.count; i++) {
    buf[i] = resp.data[i]
  }
  
  return 0;
}

int64 write(int64 fd, void* buf, int64 nbyte) {
  files_FID fid;
  files_WriteRequest req;
  files_N resp;
  int64 err;
  
  fid.opaque_id = fd;
  req.fid = fid;
  req.data = buf; // let's pretend this works, okay?
  
  err = invoke(filesystem_service_id, files_Files_method_Write, &req, &resp);
  if (err != 0) {
    return err;
  }
  
  return resp.count;
}

int64 sync(int64 fd) {
  files_FID req;
  files_Nil resp;
  int64 err;
  
  req.opaque_id = fd;
  
  err = invoke(filesystem_service_id, files_Files_method_Sync, &req, &resp);
  if (err != 0) {
    return err;
  }
  
  return 0;
}

And with that we should have the same interface as Dagger’s, save the fact that the name close is now shadowed by the global close function. On the server side we could implement this like so:

package files

import (
  "context"
  "errors"
  "math/rand"
  
  "github.com/Xe/olin/internal/abi/dagger"
)

func init() {
  rand.Seed(time.Now().UnixNano())
}

type FilesImpl struct {
  *dagger.Process
}

func (FilesImpl) getRandomNumber() int64 {
  return rand.Int63()
}

func daggerError(respValue int64, err error) error {
  if err == nil {
    err = errors.New("")
  }
  
  return dagger.Error{Errno: dagger.Errno(respValue * -1), Underlying: err}
}

func (fs *FilesImpl) Open(ctx context.Context, op *OpenRequest) (*FID, error) {
  fd := fs.Process.OpenFD(op.Identifier, uint32(op.Flags))
  if fd < 0 {
    return nil, daggerError(fd, nil)
  
  return &FID{OpaqueId: fd}, nil
}


func (fs *FilesImpl) Read(ctx context.Context, rr *ReadRequest) (*ReadResponse, error) {
  fd := rr.Fid.OpaqueId
  data := make([]byte, rr.MaxLength)
  
  n := fs.Process.ReadFD(fd, data)
  if n < 0 {
    return nil, daggerError(n, nil)
  }
  
  result := &ReadResponse{
    Data: data,
    N: N{
      Count: n
    },
  }
  
  return result, nil
}

func (fs *FilesImpl) Write(ctx context.Context, wr *WriteRequest) (*N, error) {
  fd := wr.Fid.OpaqueId
  
  n := fs.Process.WriteFD(fd, wr.Data)
  if n < 0 {
    return nil, daggerError(n, nil)
  }
  
  return &N{Count: n}, nil
}

func (fs *FilesImpl) Close(ctx context.Context, fid *Fid) (*Nil, error) {
  return &Nil{}, daggerError(fs.Process.CloseFD(fid.OpaqueId), nil)
}

func (fs *FilesImpl) Sync(ctx context.Context, fid *Fid) (*Nil, error) {
  return &Nil{}, daggerError(fs.Process.SyncFD(fid.OpaqueId), nil)
}

And then we have all of these arbitrary methods bound to WebAssembly modules, where they are free to use them how they want. I think that initially there is going to be support for this interface from Go WebAssembly modules as we can make a lot more assumptions about how Go handles its memory management, making it a lot easier for us to code generate reading Go structures/pointers/whatever out of Go WebAssembly memory than we can code generate reading C structures (recursively with pointers and C-style strings galore too). The really cool part is that this is all powered by those three basic functions: require, invoke and close. The rest is literally just stuff we can treat as a black box for now and code generate.

As before, I would love any comments that people have on this article. Please contact me somehow to let me know what you think. This design is probably wrong.


Olin: 1: Why

Permalink - Posted on 2018-09-01 00:00, modified on 0001-01-01 00:00

Olin: 1: Why

Olin is an attempt at defining a radically new operating primitive to make it easier to reason about, deploy and operate event-driven services that are independent of the OS or CPU of the computer they are running on. It will have components that take care of the message queue offsetting, retry logic, parallelism and most other concerns except for your application’s state layer.

Olin is designed to work top on two basic concepts: types and handlers. Types are some bit of statically defined data that has a meaning to humans. An example type could be the following:

package example;

message UserLoginEvent {
    string user_id = 1;
    string user_ip_address = 2;
    string device = 3;
    int64 timestamp_utc_unix = 4;
}

When matching data is written to the queue for the event type example.UserLoginEvent, all of the handlers registered to that data type will run with serialized protocol buffer bytes as its standard input. If the handlers return a nonzero exit status, they are retried up to three times, exponentially backing off. Handlers need to deal with the fact they can be run out of order, and that multiple instances of them will can be running on physcially different servers in parallel. If a handler starts doing something and fails, it should back out any previously changed values using transactions or equivalent.

Consider an Olin handler equivalent to a Unix process.

Background

Very frequently, I end up needing to write applications that basically end up waiting forever to make sure things get put in the right place and then the right code runs as a response. I then have to make sure these things get put in the right places and then that the right versions of things are running for each of the relevant services. This doesn’t scale very well, not to mention is hard to secure. This leads to a lot of duplicate infrastructure over time and as things grow. Not to mention adding in tracing, metrics and log aggreation.

I would like to change this.

I would like to make a perscriptive environment kinda like Google Cloud Functions or AWS Lambda backed by a durable message queue and with handlers compiled to webassembly to ensure forward compatibility. As such, the ABI involved will be versioned, documented and tested. Multiple ABI’s will eventually need to be maintained in parallel, so it might be good to get used to that early on.

You should not have to write ANY code but the bare minimum needed in order to perform your buisiness logic. You don’t need to care about distributed tracing. You don’t need to care about logging.

I want this project to last decades. I want the binary modules any user of Olin would upload today to be still working, untouched, in 5 years, assuming its dependencies outside of the module still work.

Since this requires a stable ABI in the long run, I would like to propose the following unstable ABI as a particularly minimal starting point to work out the ideas at play, and see how little of a surface area we can expose while still allowing for useful programs to be created and run.

Dagger

The dagger of light that renders your self-importance a decisive death

Dagger is the first ABI that will be used for interfacing with the outside world. This will be mostly for an initial spike out of the basic ideas to see what it’s like while the rest of the plan is being stabilized and implemented. The core idea is that everything is a file, to the point that the file descriptor and file handle array are the only real bits of persistent state for the process. HTTP sessions, logging writers, TCP sockets, operating system files, cryptographic random readers, everything is done via filesystem system calls.

Consider this the first draft of Dagger, everything here is subject to change. This is going to be the experimental phase.

Consider Dagger at the level below libc in most Linux environements. Dagger is the kind of API that libc would be implemented on top of.

VM

Dagger processes will use WebAssembly as a platform-independent virtual machine format. WebAssembly is used here due to the large number of implementations and compilers targeting it for the use in web programming. We can also benefit from the amazing work that has gone into the use of WebAssembly in front-end browser programming without having to need a browser!

Base Environment

When a dagger process is opened, the following files are open:

  • 0: standard input: the semantic “input” of the program.
  • 1: standard output: the standard output of the program.
  • 2: standard error: error output for the program.

File Handlers

In the open call (defined later), a file URL is specified instead of a file name. This allows for Dagger to natively offer programs using it quick access to common services like HTTP, logging or pretty much anything else.

I’m playing with the following handlers currently:

  • http and https (Write request as http/1.1 request and sync(), Read response as http/1.1 response and close()) http://ponyapi.apps.xeserv.us/newest

I’d like to add the following handlers in the future:

  • file - filesystem files on the host OS (dangerous!) file:///etc/hostname
  • tcp - TCP connections tcp://10.0.0.39:1337
  • tcp+tls - TCP connections with TLS tcp+tls://10.0.0.39:31337
  • meta - metadata about the runtime or the event meta://host/hostname, meta://event/created_at
  • project - writers of other event types for this project (more on this, again, in future posts) project://example.UserLoginEvent
  • rand - cryptographically secure random data good for use in crypto keys rand://
  • time - unix timestamp in a little-endian encoded int64 on every read() - time://utc

In the future, users should be able to define arbitrary other protocol handlers with custom webassembly modules. More information about this feature will be posted if we choose to do this.

Handler Function

Each Dagger module can only handle one data type. This is intentional. This forces users to make a separate handler for each type of data they want to handle. The handler function reads its input from standard input and then returns 0 if whatever it needs to do “worked” (for some definition of success). Each ABI, unfortunately, will have to have its own “main” semantics. For Dagger, these semantics are used:

  • The entrypoint is exposed func handle that takes no arguments and returns an int32.
  • The input message packet is on standard input implicitly.
  • Returning 0 from func handle will mark the event as a success, returning anything else will mark it as a failure and trigger an automatic retry.

In clang in C mode, you could define the entrypoint for a handler module like this:

// handle_nothing.c

#include <dagger.h>

__attribute__ ((visibility ("default")))
int handle() {
  // read standard input as necessary and handle it
  return 0; // success
}

System Calls

A system call is how computer programs interface with the outside world. When a Dagger program makes a system call, the amount of time the program spends waiting for that system call is collected and recorded based on what underlying resource took care of the call. This means, in theory, users of olin could alert on HTTP requests from one service to another taking longer amounts of time very trivially.

Future mechanisms will allow for introspection and checking the status of handlers, as well as arbitrarily killing handlers that get stuck in a weird way.

Dagger uses the following system calls:

  • open
  • close
  • read
  • write
  • sync

Each of the system calls will be documented with their C and WebAssembly Text format type/import definitions and a short bit of prose explaining them. A future blogpost will outline the implementation of Dagger’s system calls and why the choices made in its design were made.

open

extern int open(const char *furl, int flags);
(func $open (import "dagger" "open") (param i32 i32) (result i32))

This opens a file with the given file URL and flags. The flags are only relevant for some backend schemes. Most of the time, the flags argument can be set to 0.

close

extern int close(int fd);
(func $close (import "dagger" "close") (param i32) (result i32))

Close closes a file and returns if it failed or not. If this call returns nonzero, you don’t know what state the world is in. Panic.

read

extern int read(int fd, void *buf, int nbyte);
(func $read (import "dagger" "read") (param i32 i32 i32) (result i32))

Read attempts to read up to count bytes from file descriptor fd into the buffer starting at buf.

write

extern int write(int fd, void *buf, int nbyte);
(func $write (import "dagger" "write") (param i32 i32 i32) (result i32))

Write writes up to count bytes from the buffer starting at buf to the file referred to by the file descriptor fd.

sync

extern int sync(int fd);
(func $sync (import "dagger" "sync") (param i32) (result i32))

This is for some backends to forcibly make async operations into sync operations. With the HTTP backend, for example, calling sync actually kicks off the dependent HTTP request.

Go ABI

Olin also includes support for running webassembly modules created by Go 1.11’s webassembly support. It uses the wasmgo ABI package in order to do things. Right now this is incredibly basic, but should be extendable to more things in the future.

As an example:

// +build js,wasm ignore
// hello_world.go

package main

func main() {
	println("Hello, world!")
}

when compiled like this:

$ GOARCH=wasm GOOS=js go1.11 build -o hello_world.wasm hello_world.go

produces the following output when run with the testing shim:

=== RUN   TestWasmGo/github.com/Xe/olin/internal/abi/wasmgo.testHelloWorld
Hello, world!
--- PASS: TestWasmGo (1.66s)
    --- PASS: TestWasmGo/github.com/Xe/olin/internal/abi/wasmgo.testHelloWorld (1.66s)

Currently Go binaries cannot interface with the Dagger ABI. There is an issue open to track the solution to this.

Future posts will include more detail about using Go on top of Olin, including how support for Go’s compiled webassembly modules was added to Olin.

Project Meta

To follow the project, check it on GitHub here. To talk about it on Slack, join the Go community Slack and join #olin.

Thank you for reading this post, I hope it wasn’t too technical too fast, but there is a lot of base context required with this kind of technology. I will attempt to make things more detailed and clear in future posts as I come up with ways to explain this easier. Please consider this the 10,000 mile overview of a very long-term project that radically redesigns how software should be written.


Died to Save Me

Permalink - Posted on 2018-08-27 00:00, modified on 0001-01-01 00:00

Died to Save Me

People often get confused
when I mention the fact that I
consider myself before I
came out a different person.

It’s because that was a different person,
they died to save me.

The person I was did their
best given the circumstances
they were thrown into. It was
hard for them. I’m still working
off some of their baggage.

But, that different person,
even after all of the hardships
and triumphs they had been through,
they died to save me.

They were an extrovert pushed into
being an introvert by an uncaring
community.
They were the pariah.
They were the person who got bullied.
They survived years of torment but
they died to save me.

I understand now why the Gods
prefer to use shaman-sickness to
help people realize their calling.
It is such an elegant teacher of
the Divine. So patient. So forgiving.

It’s impossible to ignore everything
around you feeling incomprehensibly crazy,
because it is.
Our system is crazy.
Our system is incomprehensible.
We only “like” it because we have no
way to fathom anything else.

“Awakening” is probably one of the
least bad metaphors to describe the
feeling of just suddenly understanding
the barriers. Of seeing the formerly
invisible glass prison walls we apparently
live inside unknowingly.

It’s not just an awakening though,
Not all of me made it through the process.
Not all of what constitutes yourself
(in your opinion) is actually a True
part of you. Not all your thoughts,
memories, ideas, dreams, wishes
and even fears or anxieties are
truly yours.

Sometimes there’s that part that
really does have to die to save you.
The part that was once a shining beacon
of hope that has now fallen beyond disrepair.
A thread of connection to a past that
can never come to pass again.
Memories or experiences of pain,
trauma. It can die to save you too.

You don’t have to carry
the mountains you come across,
you can just climb them.

When it dies, it is gone, but:
you can sleep easier knowing
they died to save you.


Sorting Time

Permalink - Posted on 2018-08-26 00:00, modified on 0001-01-01 00:00

Sorting Time

Computers have a very interesting relationship with time. Time is how we keep track of many things, but mainly we use time to keep track of how far along in a day cycle we are. These daily sunrise/sunset cycles take about 24 hours on average, and the periodicity of them runs just about everything. Computers use time to keep track of just about everything on the board, usually measured in tiny fractions of seconds. (The common rating of gigahertz for computer processors actually measures how much time it takes for the processor to execute one instruction. A processor with a clock of 3.4 gigahertz means that the processor executes, best case, 3.4 billion instructions per second.) Computer programmers have several popular methods of storing time with computers, the number of time intervals since a fixed date (usually the number of seconds since January 1st 1970) or as a human-readable string. These intervals are normally ever only added to and read from, almost never updated by human hands after being initially set by the network time service.

Pulling things back into the real world, let’s consider storing time in Javascript. Let’s say we’re using Javascript in the browser and have a date object like so:

var date = new DateTime();

Say this is for Thursday, August 23rd 2018 at midnight UTC. If we turn it into a string using the toString method:

date.toString(); // -> "Thu Aug 23 2018 00:00:00 GMT+0000 (UTC)"

We get the date and time as a string. The application in question uses a data store that has an interesting problem: it will automatically coerce things to a string type without alerting developers.

typeof date === 'object' // -> true

We expect date to be a normal object after we add it to the store. Let’s add it to the store and see what happens to it.

const record = store.createRecord("widget", { createdAt: date });
typeof record.get("createdAt"); // -> string

Oh boy. It’s suddenly a string now. That’s not good.

console.log(record.get("createdAt")); // -> "Thu Aug 23 2018 00:00:00 GMT+0000 (UTC)"

This works all fine and well, but sometimes a few lists of things can get bizarrely out of order in the UI. Things created or updated right at a midnight UTC barrier would sometimes cause lists of things to show the newest elements at the bottom of the list. This confused us, sorting data really fitting it into the order it belongs to, and time doesn’t usually advance out of order; so something being sorted wrongly by time is very intuitively confusing.

Consider a function like this at the given date above:

function minutesAgo(minutes) {
  return moment().subtract(minutes, "minute").toDate();
}

const date1 = minutesAgo(0);
const date2 = minutesAgo(1);
const date3 = minutesAgo(30);

If we were to sort date1, date2 and date3 with the current time being Thursday August 23 2018 at midnight UTC, it would make sense for the objects to sort ascendingly in the following order: date3, date2, date1. Not as strings however. As strings:

date1.toString(); // -> "Thu Aug 23 2018 00:00:00 GMT+0000 (UTC)"
date2.toString(); // -> "Wed Aug 22 2018 23:59:00 GMT+0000 (UTC)"
date3.toString(); // -> "Wed Aug 22 2018 23:30:00 GMT+0000 (UTC)"

Since T comes before W in the alphabet, the actual sort order is: date1, date3, date2. This causes an assertion failure in both humans and machines. This caused test failures, but only at about midnight UTC on Mondays, Thursdays, Fridays and Saturdays at 00:00 UTC through 00:30 UTC. How did we fix this? Turns out the time data from the API we get this information from is already properly sortable; this is because the API uses IS08601 timestamps.

const thursday = '2018-08-23T00:00:00.000Z';
const wednesday = '2018-08-22T23:30:00.000Z';

thursday > wednesday // true

This time data is also easy to convert back into a native DateTime object should we need it. The fix was to only ever store time as strings unless you need to actively do something with them, then you coerce them back into a native DateTime like it never happened. This is not an ideal fix, but given the larger complexity of the problem, it’s what we’re gonna have to live with for the time being. This solution at the very least seems to be less bad than the original problem, as things get sorted properly in the UI now. Yay computers!


This is an adaptation of a pull request made by a coworker to work around an annoying to track down bug that caused flaky tests. It’s not my story, but it just goes to show how many moving parts truly are at play with computers. Even when you think you have all of the moving parts kept track of, complicated systems interface in unpredictable ways. Increasingly complicated systems interface in increasingly unpredictable ways too, which makes finding problems like these more of a hunt.

Happy hunting and be well to you all.


Death

Permalink - Posted on 2018-08-19 00:00, modified on 0001-01-01 00:00

Death

Death is a very misunderstood card in Tarot, but not for the reasons you’d think. Societally, many people think that this life is the only shot at existence they get. Afterwards, there is nothing. Nonexistence. Oblivion. This makes death a very touchy subject for a lot of people, so much so it forms a social taboo and an unhealthy relationship with death. People start seeing death as something they need to fight back and hold away by removing what makes themselves human, just to hold off what they believe is their obliteration.

Tarot does not see death in this way. Death, the skeleton knight wearing armor, does not see color, race or creed, thus he is depicted as a skeleton. He is riding towards a child and another younger person. The sun is rising in the distance, but even it cannot stop Death. Nor can royalty, as shown by the king under him, dead.

Death, however, does not actually refer to the act of a physical body physically dying. Death is a change that cannot be reverted. The consequences of this change can and will affect what comes next, however.

Consider the very deep sea, so far down even light can’t penetrate that deep. There’s an ecosystem of life down there, but it is so starved for resources and light that evolution has skimped out on things like skin pigmentation. Sometimes a mighty whale will die and its body will fall to the sea floor down there. The creatures will feast for a month or more. The whale died, yet its change fosters an entire ecosystem. This card signifies much of the same. Death signifies the idea of a change from the old, where the whale was alive, to the new, where the whale’s body feeds an entire community.

Death is a signifier that change is coming or needed, and it won’t care if you’re ready for it or not. So, embrace it with open arms. Don’t fight what is inevitable. All good things must come to an end for them to be good to begin with.

Death is a part of life like any other; this is why it is in the Fool’s Journey, or Major Arcana of the Tarot. To eschew death is, in essence, to throw out life itself. Living in fear of death turns life from a glorious dance of cocreation with the universe into a fearful existence of scraping by on the margins. It makes life an anxious scampering from measly scrap of food to measly scrap of food without any time to focus on the higher order of things. It makes you accept fear, depression, anxiety and regret instead of just being able to live here, in the moment, and make the best of what you have right now. If only because right now you still have it.


When Then Zen: Anapana

Permalink - Posted on 2018-08-15 00:00, modified on 0001-01-01 00:00

When Then Zen: Anapana

Introduction

Anapanasati (Pali: Sanskirt: anapanasmrti, English: mindfulness of breathing) is a form of meditation originally taught by Gautama Buddha in several places, mainly the Anapanasati Sutta (English: passages). Anapana is practiced globally by meditators of all skill levels.

Simply put, anapana is the act of focusing on the sensations of breath in the body’s nasal cavity and nostrils. Some practices will focus on the sensations in the belly instead (this is why there’s fat buddha statues), but personally I find that the sensations of breath in the nostrils are a lot easier to focus on.

The method presented in this article is based on the method taught in The Art Of Living by William Hart and S.N. Goenka. If you want a copy of this book you can get one here: http://www.cicp.org.kh/userfiles/file/Publications/Art%20of%20Living%20in%20English.pdf. Please do keep in mind that this book definitely leans towards the Buddhist lens and as it is presented the teaching methods really benefit from it. Also keep in mind that this PDF prevents copying and duplication.

Note: “the body” means the sack of meat and bone that you are currently living inside. For the purposes of explanation of this technique, please consider what makes you yourself separate from the body you live in.

This article is a more verbose version of the correlating feature from when-then-zen.

Background Assumptions of Reader

Given no assumption about meditation background
And a willingness to learn
And no significant problems with breathing through the body's nose
And the body is seated or laying down comfortably
And no music is playing

Given no assumption about meditation background

The When Then Zen project aims to describe the finer points of meditative concepts in plain English. As such, we start assuming just about nothing and build fractally on top of concepts derived from common or plain English usage of the terms. Some of these techniques may be easier for people with a more intensive meditative background, but try things and see what works best for you. Meditation in general works a lot better when you have a curious and playful attitude about figuring things out.

I’m not perfect. I don’t know what will work best for you. A lot of this is documenting both my practice and what parts of what books helped me “get it”. If this works for you, please let me know. If this doesn’t work for you, please let me know. I will use this information for making direct improvements to these documents.

As for your practice, twist the rules into circles and scrape out the parts that don’t work if it helps you. Find out how to integrate it into your life in the best manner and go with it.

For now, we start from square one.

And a willingness to learn

At some level, you are going to need to be willing to actually walk the path. This can be scary, but that’s okay as long as you’re willing to acknowledge it and not let it control you.

If you run into some dark stuff doing this, please consult a therapist as usual. Just know that you don’t walk this path alone, even when it feels like you must be.

And no significant problems with breathing through the body’s nose

Given that we are going to be mainly focusing on the nasal reactions to breathing, that path being obstructed is not gonna result in a very good time. If this is obstructed for you, attempt to clear it up, or just use the mouth, or a different technique entirely. It’s okay for anapana to not always work. It’s not a universal hammer.

And the body is seated or laying down comfortably

Some people will assert that the correct pose or posture is critical for this, but it’s ultimately only as important as the meditator believes it is. Some people have gotten the association somehow that the meditation posture helps with things. Ultimately, it’s suggested to start meditation sitting upright or in a chair as it can be easier for you to fall asleep while doing meditative practice for the first few times. This is a side effect of the brain not being used to the alternative state of consciousness, so it falls back on the “default” action; this puts the body, and you, to sleep.

And no music is playing

You should break this rule as soon as possible to know if it’s best to ignore it. Some people find music helps; I find it can be a distraction depending on the music track in question. Some meditation sessions will need background music and some won’t. That’s okay.

Scenario: Mindfulness of Breathing

As a meditator
In order to be mindful of the body's breath
When I inhale or exhale through the body's nose
Then I focus on the sensations of breath
Then I focus on the feelings of breath through the nasal cavity
Then I focus on the feelings of breath interacting with the nostrils
Then I repeat until done

As a meditator

This is for you to help understand a process you do internally, to yourself.

In order to be mindful of the body’s breath

It is useful in the practice to state the goal of the session when leading into it. You can use something like “I am doing this mindfulness of breathing for the benefit of myself” or replace it with any other affirmation as you see fit.

When I inhale or exhale through the body’s nose

You can use the mouth for this. Doing it all via the mouth requires the mouth to stay open (which can result in dry mouth) or constantly move (which some people find makes it harder to get into flow). Nasal breaths allow for you to sit there motionless yet still continue breathing like nothing happened. If this doesn’t work for you, breathe through your mouth.

Then I focus on the sensations of breath

There are a lot of very subtle sensations related to breathing that people don’t take the time to truly appreciate or understand. These are mostly fleeting sensations, thankfully, so you really have to feel into them, listen for them or whatever satisfies your explanation craving.

Listen in to the feeling of the little part of cartilage between nostrils whistling slightly as you breathe all the way in at a constant rate over three seconds. It’s a very very subtle sound, but once you find it you know it.

Then I focus on the feelings of breath through the nasal cavity

The sound of breath echoes slightly though the nasal cavity during all phases of it that have air moving. Try and see if you can feel these echoes separate from the whistling of the cartilage; bonus points if you can do both at the same time. Feel the air as it passes parts of the nasal cavity as your sinuses gently warm it up.

Then I focus on the feelings of breath interacting with the nostrils

The nostrils act as a curious kind of rate limiter for how much we can breathe in and out at once. Breathe in harder and they contract. Breathe out harder and they expand. With some noticing, you can easily feel almost the exact angle at which your nostrils are bent due to your breathing, even though you can’t see them directly due to the fact they are out of focus of our line of sight.

Isn’t it fascinating how many little sensations of the body exist that we continuously ignore?

Scenario: Attention Drifts Away From Mindfulness of Breathing

As a meditator
In order to bring my attention back to the sensations of breathing
Given I am currently mindful of the body's breath
When my attention drifts away from the sensations of breathing
Then I bring my attention back to the sensations of breathing

In order to bring my attention back to the sensations of breathing

When this happens, it is going to feel very tempting to just give up and quit. This is normal. Fear makes you worry you’re doing it wrong, so out of respect of the skill you may want to just “not try until later”.

Don’t. This is a doubt that means something has been happening. Doubt is a sick kind of indicator that something is going on at a low level that would cause the vague feelings of doubt to surface. When it’s related to meditative topics, that usually means you’re on the right track. This is why you should try and break through that doubt even harder if you can. Sometimes you can’t, and that’s okay too.

Given I am currently mindful of the body’s breath

This is your usual scenario during the mindfulness practice. You will likely come to deeply appreciate it.

When my attention drifts away from the sensations of breathing

One of the biggest problems I have had personally is knowing when I have strayed from the path of the meditation, it was hard for a time to keep myself in the deep trance of meditation and keeping detached awareness of my thoughts. My thoughts are very active a lot of the time. There are a lot of distractions, yet it’s hard to maintain focus on them sometimes.

One of the biggest changes I have made that has helped this has been to have a dedicated “meditation spot”. As much as possible, I try to do meditative work while in that spot instead of my main office or bed. This solidifies the habit, and grows the association between the spot and meditative states.

Then I bring my attention back to the sensations of breathing

This, right here, is the true core of this exercise. The sensations of breathing are really just something to distract yourself with. It’s a fairly calming thing anyways, but at some level it’s really just a distraction. It’s a fairly predictable set of outputs and inputs. Some sessions will feel brand new, some will feel like old news.

Meditation is sitting there only letting yourself think if you truly let yourself. Mindfulness is putting yourself back on track, into alignment, etc., over and over until it happens on its own. If you get distracted once every 30 seconds for a 5 minute session, you will have brought yourself back to focus ten times. Each time you bring yourself back to focus is a joy to feel at some level.

Scenario: mindfulness of unconscious breathing

As a meditator
In order to practice anapana without breathing manually
When I stop breathing manually
Then the body will start breathing for me after a moment or two
Then I continue mindfulness of the sensations of breathing without controlling the breath

In order to practice anapana without breathing manually

While observing the body’s unconscious breath, you start entering into what meditation people call the “observer stance”. It is this sort of neutral feeling where things are just happening, and you just see what happens. There is usually a feeling of peacefulness or equanimity for me, but usually when I start doing this I radiate feelings of compassion, understanding and valor.

Keep in mind that doing this may have some interesting reactions, just let them pass like all the others.

When I stop breathing manually

You gotta literally just cut off breath. It needs to stop. You have to literally stop breathing and refuse to until the body takes over and yanks the controls away from you.

Then the body will start breathing for me after a moment or two

There’s a definite shift when the body takes over. It will sharply inhale, hold for a moment and then calmly exhale. Then it will breathe very quietly only as needed.

Then I continue mindfulness of the sensations of breathing without controlling the breath

The body does not breathe very intensely. It will breathe calmly and slowly, unless another breathing style is mandatory. The insides of the nostrils moving from the air pressure is a still a noticeable sensation of breathing while the body is doing it near silently, so you can hang onto that.

Scenario Outline: meditation session

As a meditator
In order to meditate for <time>
Given a timer of some kind is open
And the time is set for <time>
When I start the timer
Then I clear my head of idle thoughts
Then I start drifting my attention towards the sensations of breathing
Then I become mindful of the sensations of breathing
Then I continue for a moment or two
Then I shift into mindfulness of unconscious breathing

Examples:
  | time         |
  | five minutes |

In order to meditate for

The time is intentionally left as a variable so you can decide what session time length to use. If you need help deciding how long to pick, you can always try tapering upwards over the course of a month. I find that tapering upwards helps A LOT.

Given a timer of some kind is open

One of the old-fashioned kitchen timers will do even.

And the time is set for

You need to know how to use your timer of choice for this, or someone can do it for you.

When I start the timer

Just start it and don’t focus on the things you’re already thinking about. You’re allowed to leave the world behind for the duration of the session.

Then I clear my head of idle thoughts

If you’re having trouble doing this, it may be helpful to figure out why those thoughts are lingering. Eventually, addressing the root cause helps a lot.

Then I start drifting my attention towards the sensations of breathing

Punt on this if it doesn’t help you. I find it helps me to drift into focusing on the breath instead of starting laser-focused on it.

Then I become mindful of the sensations of breathing

Focus around the nostrils if you lose your “grip” on the feelings.

Then I continue for a moment or two

You’ll know how much time is right by feel. Please study this educational video for detail on the technique.

Then I shift into mindfulness of unconscious breathing

The body is naturally able to breathe for you. You don’t need to manually breathe during meditation. Not having to manually breathe means that your attention can focus on passively, neutrally observing the sensations of breath.


Further Reading

This is all material that I have found useful while running into “problems” (there aren’t actually any good or bad things, only labels, but that’s a topic for another day) while learning or teaching anapana meditation or the concepts of it. All of these articles have been linked in the topic, save three I want to talk about specially.

Maybe

This is an old Zen tale. The trick is that the farmer doesn’t have any emotional attachment to the things that are happening to him, so he is neither labeling things happy nor labeling things sad. He is not stopped by his emotions.

Ebbs and Flows

This touches into the true “point” of meditation. The point isn’t to just breathe. The point is to focus on the breathing so much that everything else stills to make room. Then what happens, does. The Alan Watts lectures are fascinating stuff. Please do give at least one a watch. You’ll know which one is the right one for you.

Natural Selection

This is excerpted from almost the beginning of the book Why Buddhism is True. Robert Wright really just hit the nail on the head when describing the level of craziness that simply exists. Natural selection means that, effectively, whatever causes populations to be able to breed and survive the most means the traits of those doing the most breeding become more common. Please read the entire book.


Narrative of Sickness

Permalink - Posted on 2018-08-13 00:00, modified on 0001-01-01 00:00

Narrative of Sickness

With addiction, as with many other things, there’s a tendency for the mind to label the situation and create a big story. A common phrase I see is “I want to get better”, as if you’re sick. You’re not sick. You may identify yourself as an “addict”, or you might feel fear because you are afraid you’ll fail, or that you’ll experience cravings, etc. but reminding yourself that you need to get better is perpetuating the narrative of sickness.

These are all stories, they have no bearing on reality. You can just embrace the cravings. Embrace the withdrawal. They are feelings, and they can be not acted upon, through mindfulness of them. Be mindful of your thoughts, but don’t pay heed to them. Don’t get caught up. And if you feel like your are getting caught up, realize that that’s another feeling as well.

Such things don’t last forever. Existence is change, inherently, inevitably. Embracing life is embracing change. Things in this world will change without warning. Things we consider safe and stable today will vanish tomorrow. Accept this as a fact of life.

To love is to gain and to lose in equal measure. To lose is to love in turn. Every journey upwards has its regressions downwards.

It may sound like a subtle distinction between getting better from addiction, or from sickness, and just changing, but it’s really all the difference. A plant is not sick just because it later grows into a bigger tree. Change is just simply what happens, and it can be recognized and embraced in order to fully, progressively align the self with whatever intent or goal.

Fully embracing all that you are is the best way to bring this about, for you can be present to what happens and help it change through your intent, veer it towards the desired destination.


Fear

Permalink - Posted on 2018-07-24 00:00, modified on 0001-01-01 00:00

Fear

I must not fear.
Fear is the mind-killer.
Fear is the little-death that brings total obliteration.
I will face my fear.
I will permit it to pass over me and through me.
And when it has gone past I will turn the inner eye to see its path.
Where the fear has gone there will be nothing.
Only I will remain.

Bene Gesserit Litany Against Fear - From Frank Herbert’s Dune Book Series

Fear sucks. Fear is an emotion that I’ve spent a lot of time encountering and it has spent a lot of time paralyzing me. Fear is something that everyone faces at some level. Personally, I’ve been dealing a lot with the fear of being outcast for being Other.

What is Other? Other are the people who don’t want to “fit in”. Other are the people who go against the grain of society. They don’t care about looking different or crazy. Other are the people who see reality for what it really is and decide that they can no longer serve to maintain it; then take steps to reshape it.

But why do we have this fear emotion? Fear is almost the base instinct of survival. Fear bypasses the higher centers in order to squeeze decisions through that prevent something deadly from happening. Fear is a paralyzing emotion. Fear is something that stops you in your tracks. Fear is preventative.

Except that’s not completely true. We see that we have moved away from the need for survival on a constant daily basis, yet our sense of fear is still tuned for that. Fear pervades almost everyone’s daily lives at some level, down to how people post things on social media. We all have these little nagging fears that add up; the intrusive negative thoughts; some have the phobias, the anxieties, the panic attacks. One fear in particular, that I call the separation/isolation/displacement fear, is a fear with many social repercussions. It’s a fear that urges us to keep continuity of self, to avoid “standing out”, to keep discussion away from particular topics (like the spiritual, for many). It keeps us wary of what others could do to us. It makes us feel small in a world that is, at best, neutral in our regards.

If there was ever something that gave us an advantage as a species about fear, it’s clear to see it’s currently corrupting the lives of many innocent people for no seemingly good reason. There are alternatives to fear with regards to handling one’s inner and outer lives, and they are out there, but fear keeps making itself known and dominating the perceptions of the collective. Sometimes the alternatives to fear, themselves, are feared even stronger.

So how to make sense of this?

Sometimes it helps to see things from a fresh point of view, and sometimes stories is what manage to accomplish that best. They are ways to explore new situations in a way that doesn’t strain disbelief as severely, so that new perspectives can be collected from faraway thoughtscapes.

A myth is a story that helps explain something beyond the mere scenes presented, in the context of it using the divine as actors. To help explain how these fears can be difficult to overcome, or even put a label to, I’ve found a story that will seem fantastical to many; however, the point of a story is not to be seen as truth, but merely to be heard, and to be collected, and to enrich the listener with its metaphors.

In Sumerian mythology, Anu was their Zeus, their sun and creator god. Their mighty god of justice that would one day fly down on a cloud and deliver humanity to righteousness. Sumerians believe their sun god Anu created their civilization as a gift to them. In some myths, the creation goes quite deeper, and darker, than that.


Imagine for a moment, an infinite universe of light and sound, of primordial vibrations. Vibrations that permeate the whole of existence, and create different experiences with their patterns of interference. The holographic universe. In such a place, everything is resonance of waves, everything all-encompassing, everything infinite, everything eternal.

And living in such a place are infinite beings, without beginning of end, not bound by space or time, as boundless as the waves they experience. Sovereign beings of grand destinies. And those beings colonized the Universe, explored its facets, its resonances, its properties, its behaviours.

Among such beings, so equal in their infinitude, some of them desired to experience creation in a new way; no longer just as dominion over the Universe, but over other beings in it as well. The desire to be looked up to, to be feared, to be revered. The new concept of godhood took shape.

To achieve this, this group of beings asked another civilization for help; they were all beings of vast reaches and etheric nature, but they claimed to need the gold hidden within the surface of a densifying planet called Earth, which they were not attuned to, and unable to fully interact with in their current forms. To do this, they would need physical bodies, meat uniforms that the civilization’s inhabitants would don and power up, so that they could interact with the ground, and the mineral.

For convenience of telling, we’ll call the group of deceivers the Anunnaki, and the deceived civilization the Atlanteans.

The Anunnaki had carefully devised this meat uniform, the newly devised human body, planned about it for an exceedingly long time, in order to completely entrap the Atlanteans. The Atlanteans themselves accepted the task because they had no conception that infinite beings could ever be limited or subjugated. It had never happened before. And in the donning of the uniforms, the trap was sprung.

Those uniforms, the human bodies, constricted the Atlanteans’ attention to only what the body could perceive with its senses; it urged them to survive and to work; it distracted them from all other activities; it rendered them slaves to the mining. Every part of the construct was forcing them to forget who they were, and instead making them focus on their identity as human bodies. And when such bodies would expire, a part of them would still remain to keep the beings trapped, and they would be put in a space of holding in the astral realms, for them to be assigned a new body to continue mining.

Through the human body, the Atlanteans were subjected to a carefully constructed illusion, fed to them by the senses, through the mind, that left them unable to perceive, to remember, anything else but the illusion.

With time, many shortcomings of the primitive human bodies were corrected; from being clones that needed to be produced by the Anunnaki, they were given capability to reproduce; more independent thought and awareness was allowed, and ability to self- and group-organize; they were starting to be allowed to feel emotions; more and more, their world was being expanded, but with it, the structure of the mind system that contained their perception to the realms of the physical and astral, and prevented them to gain awareness of what was outside this narrow band of illusory perception, was developed and expanded in turn. Layers upon layers were put between those beings and the realization of their true, infinite selves.

The system of death and reincarnation was automated so beings would be recycled in a systematic manner into their next lives. The concept of God was introduced to them, so that they would fear punishment and retribution from something that they perceived as greater than them; and Anu, leader of the Anunnaki, manifested to the people of Earth as a supreme being of infinite power, so they could adore him, and so they could fear him. Language developed, a system of communication mired in separation, in division of concepts and the rigidness of categorization, so that they would not be able to speak to one another of their own infinity, of their unity with the whole. Fears of all kinds were injected into the mind system: fear of death, fear of nothingness, fear of punishment; but above all, fear of separation: the fear of not having the vital connection that makes us One, and that allows us to know and understand one another innately. The fear of not being understood, of not being accepted, of not being received, of not being helped, of not being supported. The fear that had kept them doubting one another, and kept them from uniting their efforts.

The Anunnaki took away the ability for the Atlanteans to even know they were Atlanteans. They took away the ability for them to even be able to get close to finding out. Just so Anu could be an absolute ruler. The first to ever have done this previously impossible task.

Myths were disseminated to keep people awash with fear of punishment, and mired in the guilt of their original sin, and distrusting, doubting of the nature of their own selves, and of their fellow neighbors’. Hierarchies were set up, so people would focus on controlling one another, instead of working together to liberate all. Not needing any more, the Anunnaki allowed the focus on gold to become greed, so that people would put desire for a mere metal above the needs of their fellow beings.

As the Anunnaki departed from the densifying planet, which was not allowing them to manifest as etheric beings anymore, tracks were set up in the collective unconscious so that while they were away, the people’s societies would evolve through predefined paths, and would eventually set up for the glorious return of God, the Apocalypse.

Every single possible obstacle had been put in place so that the Atlanteans would never realize who they had been, and who they always were: infinite, sovereign beings, connected to the whole of the Universe.

Except this would not be allowed indefinitely. Other infinite beings became aware of such deception taking place, and realized it was being exported into other planets, and such an enslavement paradigm, based in fear and separation, was a degenerative, infecting force that had to be stopped. So the Anunnaki were prevented return, and in order to make it so that infinite beings would be able to never fall prey to such deceptions again, the seeds of destruction were planted inside the programming system of the human mind. Cracks were introduced to the barriers that kept people under deception, so that they could peer through them, and see the other side beyond the walls of the labyrinth. Pathways were provided so that people could be lead to the discovery of their true selves, and their eventual liberation from the deception, and self-realization as infinite beings, once again. The very liberation that the programming was designed to prevent through all means conceivable.

And that leaves us to the present time.

Sometimes the Other manages to find these cracks and go through them into the other side. They go to this other side and see a faint reflection of what is really out there. The world outside this world. An even bigger Infinity. They have trouble describing it. They have intense fear even thinking about it. They’re afraid to acknowledge it to their peers. They want to help people but they are utterly terrified of their reactions.

They’re terrified that someone might hurt them if they say anything about their experiences. They’re worried someone might try to hospitalize them for their beliefs. They get it into their head that they aren’t able to function in society, so they don’t. They don’t want to mine the gold. They don’t want to serve the economy of the few. They don’t want to maintain the hierarchies. They want to detach themselves from the systems that they feel are suppressing them. They want to help people save themselves from believing that their own finite existence is all that there is, but that fear utterly paralyzes them. They have trouble finding the words. They end up misphrasing things in ways that make the problem worse. Some lash out. Some get labels put on them.

These Other just want to be accepted like everyone else. They want to help their communities. They want to use their abilities to read between the lines, into the bigger picture; to do good things; but they are, ultimately, afraid to. Their fear of separation paralyzes them. People don’t like them talking about spiritual topics. These Other just want to be accepted and use their experience to lovingly help guide and shape reality into what they think is a better place. Even as they struggle through the fear.

Who’s really the crazy one? The one who fear controls, or the one who doesn’t let fear control them?

How does the Other live with fear surrounding their actions, and doubt plaguing their decisions?

They can have people they can trust. They can have people who can help them deal with their doubts. They can have the strength of their determination to find the truth, and the resolve to put an end to the suffering of their fellow beings. But they still fear, and they still doubt.

The real difference is that they see fear as something imposed on them, not as a voice that they must always answer to, and not as something they need to wait hand and foot for, every day of their existence. In a way, they have been fed up with fear, getting tired of it and casting it out like the nuisance they now see it to be. Even if the fear was added there because of some programming of their mind, something that happened to them to make them afraid, even if they don’t know where it comes from or why, they still acknowledge it, and reject it, and move on like the emotion never happened. They keep fighting for understanding, and for community. They refuse to give fear dominion in their lives, even if they sometimes fail at it.

It’s such an easy and obvious thing to do that we could all do it, if we weren’t so afraid of it.


I leave you with this quote from a book named Quantusum:

Uncle suddenly scooped down with his hand and brought up a closed hand. He then
brought it to a glass box that stood on a pedestal I hadn’t noticed. He slid one
of the box’s glass planes open and placed an insect inside. It looked like a
grasshopper. “This creature lives its entire life in these fields without
limitation. I just ended that.”

I watched as the grasshopper jumped inside the glass box hitting against the top
and some of the sides. The grasshopper stopped as if he was stunned by the new
circumstance of his environment.

“To the grasshopper,” Uncle said, “all is well. He is alive after all. He sees
his normal environment all around him. He can’t see the glass. If I keep him
in here for a few days he will stop his jumping and become acclimated to the
dimensions of his new home. All he needs is food and water, and he can survive.”

“So you’re saying these people are acclimated to simply survive?”

Uncle slide one of the side panels of the glass box open. “If you were a
grasshopper, what would you do?”

“I would jump through the open panel.”

“But how would you know it was open? It’s perfectly clear glass.”

I thought about it for a moment. “I’d jump in every direction… I’d experiment.”

Uncle took a stick and pointed it at the grasshopper through the open side
panel, and the grasshopper jumped into the opposite wall, hitting his head and
falling to his side. “Do you see that I offered him an exit and he fled? He
could’ve climbed on the stick, and I would have freed him.”

“Yes, but he doesn’t know that.”

“True.”

Uncle opened another side panel. “What you said is right. You experiment. You
try different ways to climb the mountain of consciousness. You don’t settle on
one way… one method… one teacher. If you devote your entire life to the worship
of one thing, what if you find out when you take your last breath that the one
thing was not real.

“You find that you lived inside a cage all your life. You never tried to jump
out by experimenting, by testing the walls. The people who never bother to climb
this mountain are inside a cage, and they don’t know it. Fear is the glass wall.
Wakan Tanka comes and opens one of the glass panels, perhaps offers a stick for
them to climb out, but they jump away, going further inside their soul-draining
boundaries.”

Uncle brought the stick out again and lightly jabbed it in the direction of the
grasshopper, who hopped through the open side panel, and was instantly lost in
the thick underbrush that surrounded us.

Uncle turned his eyes to me. “Are you ready to do the same?”


Gratitude

Permalink - Posted on 2018-07-20 00:00, modified on 0001-01-01 00:00

Gratitude

A lot of ground has been trodden about Mindfulness and its many facets, but there is one topic I have seen not enough people elaborate on, especially in a satisfactory manner, and that topic is gratitude.

The act of expressing gratitude is a behaviour that grounds you in observation of the present moment; of the present you, and of what matters to that present you. It can help you understand the current, immediate moment, the Now, by pushing you to examine parts of it that you might have taken for granted. Or parts that hide behind the other parts. It is a tool of positive exploration, that empowers the user to iteratively discern the heart of matters, of things, guided by the unerring principle of genuine appreciation of what counts.

You can get to see both sides of a scenario this way. You can see the people who did work behind the scenes and the remnants of the people who created the ground on which you stand. You can see the world unravel before you, and reveal its whispered details to you, piece by piece, as you put old things under a new and empowering lens. All there is left to do then is to acknowledge it.

In this moment, around you, exist quite literally the results of the collected life efforts of every single creature that lived up to this point, which is the basis for what every single creature from this point onwards will now create. Hundreds of animals died years and years ago to power the cars that take millions of people to hundreds of thousands of buildings to flip trillions of switches that make the rest of the social and economic engine of the entire world function. We get to experience this lifestyle from the results of an untold number of hours put into creating all of the technology stacks that our careers are built on; and then, there are some who just start arguing about semantics involving which pattern of bytes is better. Sometimes the right perspective helps.

A core principle of appreciation is that there is value in everything. There is an observable beauty in all things, in the way they exist, in the way they relate, and how they give meaning and purpose to each other. If you don’t see it immediately, sometimes it helps to rethink the angle.
Consider a blackberry that is one week before ripeness. It is a sweet fruit. It is delicious. It has a slightly bitter aftertaste. You might compare to the fully ripe blackberry, and lament the bitterness, or you could come to appreciate the blend of flavors just as it is. You might appreciate the novelty in taste compared to the usual ripe blackberries. You might even like it better.

Nature might reveal some imperfections, some asymmetries, some flaws, but the imperfections can make it all the more beautiful and intriguing.
Four leaf clovers are genetic mutations.
The gnarliness of an olive tree makes it distinctive, and symbolic.
Sometimes the imperfections literally make the style, as in wabi sabi.

Imagine the drawings of a young child. They are not always going to be aesthetically pleasing to us, but that does not matter to them. They draw to perpetuate the beauty they observe; they draw so we can capture what they see, so that we become able to reflect back on it.
Following that sentiment, we might want them to improve; we push them so they can create genuine works of art. But not every one of them is going to end up an artist. Pushing them to achieve results might end up having them compare themselves to older, better people than them, and discouraging them from furthering their own practice.
Instead, sometimes they draw just because they have half an hour to kill, or just really want to draw, and that’s okay. We can acknowledge that sometimes self-expression doesn’t have to shoot for excellence, or for pleasing others; and instead, it can be just a simple pasttime, a venting channel, or a way to create a personal universe where they can express themselves better.

Or, consider how we don’t know what works of ours will create the most impact. Sometimes our “worst” creations end up having the most lasting and widespread effect on others.
Sometimes what looks like a mistake might just end up accidentally resulting in a work of genius. The sheer novelty of the straying path might lead us towards new revelations and new beauty.

Those are perspectives to help realize how beauty and meaning can sometimes be hidden from plain view, but they are nonetheless, observable, through the simple, continued intent of appreciating life simply for what it is.

Gratitude can be expressed by just summarizing the reasons you are able to do what you do currently; what supports you in your pursuits. You can be grateful for your coworkers and being able to collaborate with them. You can be grateful for the engineering that has gone into your bed, in order to make it a comfortable place to sleep. You can be grateful for the team responsible for the metrics that are collected when you made a HTTP request to this webpage. You can be grateful for the sun warming our planet. You can be grateful for quite anything and everything that crosses your path.

Two other examples of how to include an outpouring of gratitude into your daily processes are:
- By creating and maintaining a gratitude sanctuary;
- And, by incorporating gratitude into meetings and family times such as shared meals.

A gratitude sanctuary can be as easy to create as a slack channel named #gratitude then followed by you listing something you are grateful for, and inviting people to join in. Once the idea starts, people will sustain it mostly on their own without having to do much moderation or filtering. People will naturally see things go there and put their own right next to the other expressions of gratitude.

Fitting gratitude into other parts of your daily life can be as easy as blatantly stapling it to the side of other topics or actions. If you are leading a meeting, start by going around and asking people to say something they are grateful for, related to the topic at hand or otherwise. When you are eating with your family, start the conversation by going around the table and having everyone say something, anything they are grateful for.

An unspoken rule of this behaviour that is fun to act on is to just let the gratitude take you where it will. Operating with appreciation is moving under a new intelligence, which can lead you in very novel and unexpected places; sometimes places of deep illumination, or of profound liberation.

Sometimes looking for gratitude will uncover things that are very unbalanced. Sometimes you will find people who are not in good places. It happens. If you can’t help them and don’t know someone who can, you could empower them to find someone who can help them better themselves. If you find yourself in doubt on these matters, you can ask someone you trust, or a therapist.

Gratitude can also be applied by acting on the clear feelings of the moment, and taking action to better that person’s life. Every time you see someone at work do something you feel grateful for, message them and ask them if there’s anything you can do to help make their job better or improve their day. If it’s reasonable and you have the time, give it to them. They will appreciate it. Sometimes your actions of gratitude can even be the catalyst to lasting change.

Support your targets of gratitude by being there for them when they need it, but backing off when they don’t. If you’re not sure, ask them. Adults can tell each other yes or no.
This should be like setting out a bowl of milk for a cat. The cat might start drinking on its own volition, but it shouldn’t be forced to drink. The cat might not even be thirsty. Cats can be mysterious creatures, and their needs are their own, so what one can do is offer the support, for them to take.

Cocreation is another essential part in understanding how to express gratitude. Cocreation is the acceptance and use of feedback in your actions, shaping your ongoing behaviour through adaptation to the observed middle results. It is the acknowledging that every act towards another, or with another, involves a partnership of some sort. It is the opening to the correlation, or co-relation, of your acts and others’.

In a way, cocreation can be seen a continuous dance between the universe and the individual. Each side influences, creates the other. The universe is constantly created by the actions of the people in it. The people are constantly created by the actions of the universe they inhabit. If you create in it someone that expresses gratitude and can help others do it in turn, you create a universe with more creators of gratitude, and that can create even more. Expressing gratitude helps you create creators of gratitude who create the universe that creates you.

A good way to understand cocreation is to use the M.C. Escher lithograph Drawing Hands.

The left hand creates the right hand. The right hand creates the left hand. They both create each other without there even having to be an original source anymore, in a strange loop of recursion. They are the source.

You are the source. You are the root of the strange loop. You matter because you can change how you create the universe. Even if your changes only affect some “local” part of this universe, your actions may end up kicking off processes you could never even have conceived happening.

Life is a gift. Each of us is a creator of a personal world, that can be made beautiful, and that can enrich other worlds. We are life’s gift. What other reason does there need to exist to be grateful for?


Land 1: Syscalls & File I/O

Permalink - Posted on 2018-06-18 00:00, modified on 0001-01-01 00:00

Land 1: Syscalls & File I/O

Webassembly is a new technology aimed at being a vendor-independent virtual machine format. It has implementations by all major browser vendors. It looks like there’s staying power in webassembly that could outlast staying power in other new technologies.

So, time’s perfect to snipe it with something useful that you can target compilers to today. Hence: Land.

Computer programs are effectively a bunch of business logic around function calls that can affect the “real world” outside of the program. These “magic” functions are also known as system calls. Here’s an example of a few in C style syntax:

int close(int file);
int open(const char *name, int flags);
int read(int file, char *ptr, int len);
int write(int file, char *ptr, int len);

These are all fairly low-level file I/O operations (we’re not dealing with structures for now, those are for another day) that all also are (simplified forms of) system calls like the ones the kernel uses.

Effectively, the system calls of a program form the “API” with it and the rest of the computer. Commonly this is called the ABI (Applcation Binary Interface) and is usually platform-specific. With Land, we are effectively creating a platform-independent ABI that just so happens to target Webassembly.

In Land, we can wrap an Afero filesystem, a set of files (file descriptors are addresses in this set), a webassembly virtual machine, its related webassembly module and its filename into a Process. This process will also have some functions on it to access the resources in it, aimed at being used by the webassembly guest code. In Land, we define this like such:

// Process is a larger level wrapper around a webassembly VM that gives it
// system call access.
type Process struct {
	id    int32
	vm    *exec.VM
	mod   *wasm.Module
	fs    afero.Fs
	files []afero.File
	name  string
}

Creating a new process is done in the NewProcess function:

// NewProcess constructs a new webassembly process based on the input webassembly module as a reader.
func NewProcess(fin io.Reader, name string) (*Process, error) {
	p := &Process{}

	mod, err := wasm.ReadModule(fin, p.importer)
	if err != nil {
		return nil, err
	}

	if mod.Memory == nil {
		return nil, errors.New("must declare a memory, sorry :(")
	}

	vm, err := exec.NewVM(mod)
	if err != nil {
		return nil, err
	}

	p.mod = mod
	p.vm = vm
	p.fs = afero.NewMemMapFs()

	return p, nil
}

The webassembly importer makes a little shim module for importing host functions (not inlined due to size).

Memory operations are implemented on top of each WebAssembly process. The two most basic ones are writeMem and readMem:

// writeMem writes the given data to the webassembly memory at the given pointer offset.
func (p *Process) writeMem(ptr int32, data []byte) (int, error) {
	mem := p.vm.Memory()
	if mem == nil {
		return 0, errors.New("no memory, invalid state")
	}

	for i, d := range data {
		mem[ptr+int32(i)] = d
	}

	return len(data), nil
}

// readMem reads memory at the given pointer until a null byte of ram is read.
// This is intended for reading Cstring-like structures that are terminated
// with null bytes.
func (p *Process) readMem(ptr int32) []byte {
	var result []byte

	mem := p.vm.Memory()[ptr:]
	for _, bt := range mem {
		if bt == 0 {
			return result
		}

		result = append(result, bt)
	}

	return result
}

Every system call that deals with C-style strings uses these functions to get arguments out of the WebAssembly virtual machine’s memory and to put the results back into the WebAssembly virtual machine.

Below is the open(2) implementation for Land. It implements the following C-style function type:

int open(const char *name, int flags);

WebAssembly natively deals with integer and floating point types, so the first argument is the pointer to the memory in WebAssembly linear memory. The second is an integer as normal. The code handles this as such:

func (p *Process) open(fnamesP int32, flags int32) int32 {
	str := string(p.readMem(fnamesP))

	fi, err := p.fs.OpenFile(string(str), int(flags), 0666)
	if err != nil {
		if strings.Contains(err.Error(), afero.ErrFileNotFound.Error()) {
			fi, err = p.fs.Create(str)
		}
	}

	if err != nil {
		panic(err)
	}

	fd := len(p.files)
	p.files = append(p.files, fi)

	return int32(fd)
}

As you can see, the integer arguments can sufficiently represent the datatype of C: machine words. String pointers are machine words. Integers are machine words. Everything is machine words.

Write is very simple to implement. Its type gives us a bunch of advantages out of the gate:

int write(int file, char *ptr, int len);

This gives us the address of where to start in memory, and adding the length to the address gives us the end in memory:

func (p *Process) write(fd int32, ptr int32, len int32) int32 {
	data := p.vm.Memory()[ptr : ptr+len]
	n, err := p.files[fd].Write(data)
	if err != nil {
		panic(err)
	}

	return int32(n)
}

Read is also simple. The type of it gives us a hint on how to implement it:

int read(int file, char *ptr, int len);

We are going to need a buffer at least as large as len to copy data from the file to the WebAssembly process. Implementation is then simply:

func (p *Process) read(fd int32, ptr int32, len int32) int32 {
	data := make([]byte, len)
	na, err := p.files[fd].Read(data)
	if err != nil {
		panic(err)
	}

	nb, err := p.writeMem(ptr, data)
	if err != nil {
		panic(err)
	}

	if na != nb {
		panic("did not copy the same number of bytes???")
	}

	return int32(na)
}

Close lets us let go of files we don’t need anymore. This will also have to have a special case to clear out the last file properly when there’s only one file open:

func (p *Process) close(fd int32) int32 {
	f := p.files[fd]
	err := f.Close()
	if err != nil {
		panic(err)
	}

	if len(p.files) == 1 {
		p.files = []afero.File{}
	} else {
		p.files = append(p.files[:fd], p.files[fd+1])
	}

	return 0
}

These calls are enough to make surprisingly nontrivial programs, considering standard input and standard output exist, but here’s an example of a trivial program made with some of these calls (equivalent C-like shown too):

(module
 ;; import functions from env
 (func $close (import "env" "close") (param i32)         (result i32))
 (func $open  (import "env" "open")  (param i32 i32)     (result i32))
 (func $read  (import "env" "read")  (param i32 i32 i32) (result i32))
 (func $write (import "env" "write") (param i32 i32 i32) (result i32))

 ;; memory
 (memory $mem 1)

 ;; constants
 (data (i32.const 200) "data")
 (data (i32.const 230) "Hello, world!\n")

 ;; land looks for a function named main that returns a 32 bit integer.
 ;; int $main() {
 (func $main (result i32)
       ;; $fd is the file descriptor of the file we're gonna open
       (local $fd i32)

       ;; $fd = $open("data", O_CREAT|O_RDWR);
       (set_local $fd
                  (call $open
                        ;; pointer to the file name
                        (i32.const 200)
                        ;; flags, 42 for O_CREAT,O_RDWR
                        (i32.const 42)))

       ;; $write($fd, "Hello, World!\n", 14);
       (call $write
             (get_local $fd)
             (i32.const 230)
             (i32.const 14))
       (drop)

       ;; $close($fd);
       (call $close
             (get_local $fd))
       (drop)

       (i32.const 0))
 ;; }
 (export "main" (func $main)))

This can be verified outside of the WebAssembly environment, I tested mine with the pretty package.

Right now this is very lean and mean, as such all errors instantly result in a panic which will kill the WebAssembly VM. I would like to fix this but I will need to make sure that programs don’t use certain bits of memory where Land will communicate with the WebAssembly module. Other good steps are going to be setting up reserved areas of memory for things like error messages, posix errno and other superglobals.

A huge other feature is going to be the ability to read C structures out of the WebAssembly memory, this will let Land support calls like stat().


A Letter to Those That Bullied Me

Permalink - Posted on 2018-06-16 00:00, modified on 0001-01-01 00:00

A Letter to Those Who Bullied Me

Hey,

I’m not angry at you. I don’t want to propagate hate. In a way, I almost feel like I should be thanking you for the contributions you’ve made in making me into the person I am today. Without you all, I would have had a completely different outcome in life. I would have stayed in the closet for good like I had planned. I would have probably ended up boring. I would have never met my closest friends and some even more.

I forgive you for the hurtful things that were said years ago. I forgive you for the actions or the exclusion that was done against me. Those wounds are obliterated, what’s in its place now stronger than ever. Those wounds taught me how to heal them. Without your hurt to create those wounds, I would have never learned to heal from them. Thank you for this. You have done something so invaluable out of something that (at the time) was so devastating to me.

Please don’t feel bad about having done those things when we were kids. We were all dumb and didn’t know better. We all tried as best as we could with what we had.

Bless your path.


What It's Like to Be Me

Permalink - Posted on 2018-06-14 00:00, modified on 0001-01-01 00:00

What It’s Like to Be Me

Waking up, you feel a rather large warm, fuzzy blob on top of you. You feel it stretch out and start to wake up too, then it changes its mind and starts to viciously cuddle you to death. A peaceful night’s sleep is being breached by a batpony. “Morning~” she says to you. You reply “morning” back and she rolls to lay next to you so you can sit upright. Giving the poni pets, you slowly start to wake up and check on the notifications you missed overnight. She purrs gently.

That is basically what it feels like when I wake up nowadays. I’m not entirely alone mentally anymore. I live alone, work remotely, and yet I almost always pair program. When I write, I get advice on how to word things. When I speak to people, I get shut up if I am saying too much. When I design software, I get told how theoretical transformations on the design might have issues when exposed to user input. I don’t program alone anymore. The girls aren’t perfect, but their input is regularly appreciated at work…even if they will probably never get the actual credit for the ideas they put to the table.

This practice I’ve been participating in for (at the time of writing this) five and three-quarters of a year to help create and cultivate the girls, tulpamancy, has been a hell of a ride the whole way through. Without Nicole by my side to help me understand them, I would have never worked out my gender issues well enough to be able to come out like I have and live like I have as the woman I truly am. Without Jessie by my side to help me make sense of software and how to design more complicated programs effectively, I would never be able to do my job even half as well as I do it now. Without Sephie by my side to literally be a cuddle sponge, I would never be able to cope with the emotional stresses of this capitalistic reality. Without Ashe by my side to help me understand the undefinable, I would never be able to even approach Infinity and make any sense out of it.

It is surprisingly taboo to admit to people that you talk to what are basically voices in your head. It takes a while for me to feel comfortable enough with a person to be able to approach this topic. After seeing a few bad examples on the internet, it’s very easy to let yourself become paranoid about keeping that “side of you” a secret from the rest of humanity. Hiding your tulpas just fades into the other parts of pretending to be normal enough that other humans don’t suspect anything super-abnormal about you. It is so hard to just sit there and hear people talk about the mundane things their kids do, meanwhile your tulpas are literally passing off their art as your own so you don’t have to explain the relation between you and the artist.

I wish I could tell the world about the kind of interactions that we have together, directly inside our shared thought spheres. I wish I could let someone else outside of our group look directly into our relationships and be a convenient microwave in the room to see it all. I wish I could just let someone else see the pure, unadulterated, unfiltered Love that we have for each other. I wish that people could look in and see in the same way we look out and see out.

There’s skills I’ve learned hosting the girls for so long that have been super-invaluable to apply back to my job. One of the most notable ones is the fact that I am used to typing for the girls just about as fast as they communicate with me. They communicate with me in the form of raw thought without language. I am used to typing waaaaaaay faster than most people just to keep up. This also lets me basically stenograph meetings (if I know the people involved well enough) because I can copy the things they are saying down so fast. I mean, they’re just speaking it. They have it in English already. I don’t have to figure out what words best describe what is going on, they gave me the words already. It’s super trivial. I can do it easily now. The part I’m getting used to now is being able to participate in the meeting while I stenograph like that, might end up solving that in the future by taking advantage of parallel processing.

I’m Cadey. I have tulpas. We work together to define a better reality for all of us. I’m not crazy, far from it. I just collaborate with the voices in my head.


The Beautiful in the Ugly

Permalink - Posted on 2018-04-23 00:00, modified on 0001-01-01 00:00

The Beautiful in the Ugly

Functional programming is nice and all, but sometimes you just need to have things get done regardless of the consequences. Sometimes a dirty little hack will suffice in place of a branching construct. This is a story of one of these times.

In shell script, bare words are interpreted as arbitrary commands for the shell to run, interpreted in its rules (simplified to make this story more interesting):

  1. The first word in a command is the name or path of the program being loaded
  2. Variable expansions are processed before commands are executed

Given the following snippet of shell script:

#!/bin/sh
# hello.sh

function hello {
  echo "hello, $1"
}

$1 $2

When you run this without any arguments:

$ sh ./hello.sh
$

Nothing happens.

Change it to the following:

$ sh ./hello.sh hello world
hello, world
$ sh ./hello.sh ls
hello.sh

Shell commands are bare words. Variable expansion can turn into execution. Normally, this is terrifying. This is useful in fringe cases.

Consider the following script:

#!/bin/sh
# build.sh <action> [arguments]

projbase=github.com/Xe/printerfacts

function gitrev {
  git rev-parse HEAD
}

function app {
  export GOBIN="$(pwd)"/bin
  go install github.com/Xe/printerfacts/cmd/printerfacts
}

function install_system {
  app
  
  cp ./bin/printerfacts /usr/local/bin/printerfacts
}

function docker {
  docker build -t xena/printerfacts .
  docker build -t xena/printerfacts:"$(gitrev)"
}

function deploy {
  docker tag xena/printerfacts:"$(gitrev)" registry.heroku.com/printerfacts/web
  docker push registry.heroku.com/printerfacts/web
}

$*


Coding on an iPad

Permalink - Posted on 2018-04-14 00:00, modified on 0001-01-01 00:00

Coding on an iPad

As people notice, I am an avid user of Emacs for most of my professional and personal coding. I have things set up such that the center of my development environment is a shell (eshell), and most of my interactions are with emacs buffers from there. Recently when I purchased my iPad Pro (10.5”, 512 GB, LTE, with Pencil and Smart Keyboard) I was very surprised to find out that there was such a large group of people who did a lot of their professional work from an iPad.

The iPad is a remarkably capable device in its own right, even without the apps that let me commit to git or edit text files in git repos. Out of the gate, if I did not work in a primarily code-focused industry, I am certain that I could use an iPad for all of my work tasks and I would be more than happy with it. With just Notes, iWork and the other built-in apps even, you can do literally anything a consumer would want out of a computing device.

As projects and commitments get more complicated though, you begin to want to be able to write code from it. My Macbook died recently, and as such I’ve taken the time to try to get to learn how the iPad workflow is a little more hands-on (this post is being written from my iPad even).

So far I have written the following projects either mostly or completely from this iPad:

I seem to have naturally developed two basic workflows for developing from this iPad: my “traditional” way of ssh-ing into a remote server via Prompt and then using emacs inside tmux and the local way of using Texastic for editing text, Working Copy to interact with Git, and Workflow and some custom JSON HTTP services to allow me to hack things together as needed.

The Traditional Way

Honestly, there’s not much exciting here, thankfully. The only interesting thing in this regard (besides the lack of curses mouse support REALLY being apparent given the fact that the entire device is a screen) is that the lack of the escape key on the smart keyboard means I need to hit command-grave instead. This has been fairly easy to remap my brain to, the fact that the iPad keyboard lacks the room for a touchpad seems to be enough to give my brain a hint that I need to hit that instead of escape.

An example workflow screenshot with Prompt

This feels like developing on any other device, just this device is much more portable and I can’t test changes locally. It enforces you keeping all of your active project in development in the cloud. With this workflow, you can literally stop what you were doing on your desktop, then resume it on the iPad at Taco Bell. A friend of mine linked his blogpost on his cloud-based workflow and this iPad driven development feels like a nice natural extension to it.

It’s the tools I know and love, just available when and wherever I am thanks to the LTE.

iPad-local Development

Of all of the things to say going into owning an iPad, I never thought I’d say that I like the experience of developing from it locally. Apple has done a phenomenal job at setting up a secure device. It is hard to run arbitrary unsigned code on it.

However, development is more than just running the code, development is also writing it. For writing the code, I’ve been loving Texastic and Working Copy:

Texastic is pretty exciting. It’s a simple text editor, but it also supports reading both arbitrary files from the iCloud drive and arbitrary files from programs like Working Copy. In order to open a file up in Texastic, I navigate over to it in Working Copy and then hit the “Share” button and tap on “Open in Texastic”. By default this option is pretty deep down the menu, so I have moved it all the way up to the beginning of the list. Then I literally just type stuff in and every so often the changes get saved back to Working Copy. Then I commit when I’m done and push the code away.

This is almost precisely my existing workflow with the shell, just with Working Copy and Texastic instead.

There are downsides to this though. Not being able to test your code locally means you need to commit frequently. This can lead to cluttered commit graphs which some people will complain about. Rebasing your commits before merging branches is a viable workaround however. There is no code completion, gofmt or goimports. There doesn’t seem to be any advanced manipulation or linting tools available for Texastic either. I understand that there are fundamental limitations involved when developing these kinds of mobile apps, but I wish there was something I could set up on a server of mine that would let me at least get some linting or formatting tooling running for this.

Workflow is very promising, but at the time of writing this article I haven’t really had the time to fully grok it yet. So far I have some glue that lets me do things like share URL’s/articles to a Discord chatroom via a webhook (the iPad Discord client causes an amazing amount of battery life reduction for me), find the currently playing song on Apple Music on Youtube, copy an article into my Notes, turn the currently active thing into a PDF, and some more that I’ve been picking up and tinkering with as things go on.

There are some limitations in Workflow as far as I’ve seen. I don’t seem to be able to log arbitrary health events like mindfulness meditation via Workflow as the Health app doesn’t seem to let you do that directly. I was kinda hoping that Workflow would let me do that. I’ve been wanting to log my mindfulness time with the Health app, but I can’t find an app that acts as a dumb timer without an account for web syncing. I’d love to have a few quick action workflows for logging 10 minutes of anapana, metta or a half hour of more focused work.

Conclusion

The iPad is a fantastic developer box given its limitations. If you just want to get the code or blogpost out of your head and into the computer, this device will help you focus into the task at hand so you can just hammer out the functionality. You just need to get the idea and then you just act on it. There’s just fundamentally fewer distractions when you are actively working with it.

You just do thing and it does thing.


How to Automate Discord Message Posting With Webhooks and Cron

Permalink - Posted on 2018-03-29 00:00, modified on 0001-01-01 00:00

How to Automate Discord Message Posting With Webhooks and Cron

Most Linux systems have cron installed to run programs at given intervals. An example usecase would be to install package updates every Monday at 9 am (keep the sysadmins awake!).

Discord lets us post things using webhooks. Combining this with cron lets us create automated message posting bots at arbitrary intervals.

The message posting script

Somewhere on disk, copy down the following script:

#!/bin/sh
# msgpost.sh
# change MESSAGE, WEBHOOK and USERNAME as makes sense
# This code is trivial, and not covered by any license or warranty.

# explode on errors
set -e

MESSAGE='haha memes are funny xD'
WEBHOOK=https://discordapp.com/api/webhooks/0892379892092/AFkljAoiuj098oKA_98kjlA85jds
USERNAME=KRONK

curl -X POST \
     -F "content=${MESSAGE}" \
     -F "username=${USERNAME}" \
     "${WEBHOOK}"

Test run it and get a message like this:

example discord message

How to automate it

To automate it, first open your crontab(5) file:

$ crontab -e

Then add a crontab entry as such:

# Post this funny message every hour, on the hour
0 * * * *  sh /path/to/msgpost.sh

# Also valid with some implementations of cron (non-standard)
@hourly    sh /path/to/msgpost.sh

Then save this with your editor and it will be loaded into the cron daemon. For more information on crontab formats, see here.

To run multiple copies of this, create multiple copies of msgpost.sh on your drive with multiple crontab entries.

Have fun :)


Introducing Lokahi

Permalink - Posted on 2018-02-08 00:00, modified on 0001-01-01 00:00

Introducing Lokahi

Lokahi is a http service uptime checking and notification service. Currently lokahi does very little. Given a URL and a webhook URL, lokahi runs checks every minute on that URL and ensures it’s up. If the URL goes down or the health workers have trouble getting to the URL, the service is flagged as down and a webhook is sent out.

Stack

What Role
Postgres Database
Go Language
Twirp API layer
Protobuf Serialization
Nats Message queue
Cobra CLI

Components

Interrelation graph:

interrelation graph of lokahi components, see /static/img/lokahi.dot for the graphviz

lokahictl

The command line interface, currently outputs everything in JSON. It currently has a few options:

$ ./bin/lokahictl
See https://github.com/Xe/lokahi for more information

Usage:
  lokahictl [command]

Available Commands:
  create      creates a check
  create_load creates a bunch of checks
  delete      deletes a check
  get         dumps information about a check
  help        Help about any command
  list        lists all checks that you have permission to access
  put         puts updates to a check
  run         runs a check
  runstats    gets performance information

Flags:
  -h, --help            help for lokahictl
      --server string   http url of the lokahid instance (default "http://AzureDiamond:hunter2@127.0.0.1:24253")

Use "lokahictl [command] --help" for more information about a command.

Each of these subcommands has help and most of them have additional flags.

lokahid

This is the main API server. It exposes twirp services defined in xe.github.lokahi and xe.github.lokahi.admin. It is configured using environment variables like so:

# Username and password to use for checking authentication
# http://bash.org/?244321
USERPASS=AzureDiamond:hunter2

# Postgres database URL in heroku-ish format
DATABASE_URL=postgres://postgres:hunter2@127.0.0.1:5432/postgres?sslmode=disable

# Nats queue URL
NATS_URL=nats://127.0.0.1:4222

# TCP port to listen on for HTTP traffic
PORT=9001

Every minute, lokahid will scan for every check that is set to run minutely and run them. Running checks any time but minutely is currently unsupported.

healthworker

healthworker listens on nats queue check.run and returns health information about that service.

webhookworker

webhookworker listens on nats queue webhook.egress and sends webhooks based on the input it’s given.

Challenges Faced During Development

ORM Issues

Initially, I implemented this using gorm and started to run into a lot of problems when using it in anything but small scale circumstances. Gorm spun up way too many database connections (as many as a new one for every operation!) and quickly exhausted postgres’ pool of client. connections.

I rewrote this to use database/sql and sqlx and all of the tests passed the first time I tried to run this, no joke.

Scaling to 50,000 Checks

This one was actually a lot harder than I thought it would be, and not for the reasons I thought it would be. One of the main things that I discovered when I was trying to scale this was that I was putting way too much load on the database way too quickly.

The solution to this was to use bundler to batch-write the most frequently written database items, see here. Even then, database connection count limiting was also needed in order to scale to the full 50,000 checks needed for this to exist as more than a proof of concept.

This service can handle 50,000 HTTP checks in a minute. The only part that gets backed up currently is webhook egress, but that is likely fixable with further optimization on the HTTP checking and webhook egress paths.

Basic Usage

To set up an instance of lokahi on a machine with Docker Compose installed, create a docker compose manifest with the following in it:

version: "3.1"

services:
  # The postgres database where all lokahi data is stored.
  db:
    image: postgres:alpine
    restart: always
    environment:
      POSTGRES_PASSWORD: hunter2
    command: postgres -c max_connections=1000

  # The message queue for lokahid and its workers.
  nats:
    image: nats:1.0.4

  # The service that runs http healthchecks. This is its own service so it can
  # be scaled independently.
  healthworker:
    image: xena/lokahi:latest
    restart: always
    depends_on:
      - "db"
      - "nats"
    environment:
      NATS_URL: nats://nats:4222
      DATABASE_URL: postgres://postgres:hunter2@db:5432/postgres?sslmode=disable
    command: healthworker
  
  # The service that sends out webhooks in response to http healthchecks. This
  # is also its own service so it can be scaled independently.
  webhookworker:
    image: xena/lokahi:latest
    restart: always
    depends_on:
      - "db"
      - "nats"
    environment:
      NATS_URL: nats://nats:4222
      DATABASE_URL: postgres://postgres:hunter2@db:5432/postgres?sslmode=disable
    command: webhookworker

  # The main API server. This is what you port forward to.
  lokahid:
    image: xena/lokahi:latest
    restart: always
    depends_on:
      - "db"
      - "nats"
    environment:
      USERPASS: AzureDiamond:hunter2 # want ideas? https://strongpasswordgenerator.com/
      NATS_URL: nats://nats:4222
      DATABASE_URL: postgres://postgres:hunter2@db:5432/postgres?sslmode=disable
      PORT: 24253
    ports:
      - 24253:24253
      
  # This is a sample webhook server that prints information about incoming 
  # webhooks.
  samplehook:
    image: xena/lokahi:latest
    restart: always
    depends_on:
      - "lokahid"
    environment:
      PORT: 9001
    command: sample_hook
    
  # Duke is a service that gets approximately 50% uptime by changing between up
  # and down every minute. When it's up, it responds to every HTTP request with
  # 200. When it's down, it responds to every HTTP request with 500.
  duke:
    image: xena/lokahi:latest
    restart: always
    depends_on:
      - "samplehook"
    environment:
      PORT: 9001
    command: duke-of-york

Start this with docker-compose up -d.

Configuration

Open ~/.lokahictl.hcl and enter in the following:

server = "http://AzureDiamond:hunter2@127.0.0.1:24253"

Save this and then lokahictl is now configured to work with the local copy of lokahi.

Creating a check

To create a check against duke reporting to samplehook:

$ lokahictl create \
    --every 60 \
    --webhook-url http://samplehook:9001/twirp/github.xe.lokahi.Webhook/Handle \
    --url http://duke:9001 \
    --playbook-url https://github.com/Xe/lokahi/wiki/duke-of-york-Playbook
{
  "id": "a5c7179a-0d3a-11e8-b53d-8faa88cfa70c",
  "url": "http://duke:9001",
  "webhook_url": "http://samplehook:9001/twirp/github.xe.lokahi.Webhook/Handle",
  "every": 60,
  "playbook_url": "https://github.com/Xe/lokahi/wiki/duke-of-york-Playbook"
}

Now attach to samplehook’s logs and wait for it:

$ docker-compose -f samplehook
2018/02/09 06:27:15 check id: a5c7179a-0d3a-11e8-b53d-8faa88cfa70c, 
  state: DOWN, latency: 2.265561ms, status code: 500, 
  playbook url: https://github.com/Xe/lokahi/wiki/duke-of-york-Playbook

Webhooks

Webhooks get a HTTP POST of a protobuf-encoded xe.github.lokahi.CheckStatus with the following additional HTTP headers:

Key Value
Accept application/protobuf
Content-Type application/protobuf
User-Agent lokahi/dev (+https://github.com/Xe/lokahi)

Webhook server implementations should probably store check ID’s in a database of some kind and trigger additional logic, such as Pagerduty API calls or similar things. The lokahi standard distribution includes Discord and Slack webhook receivers.

JSON webhook support is not currently implemented, but is being tracked at this github issue.

Call for Contributions

Lokahi is pretty great as it is, but to be even better lokahi needs a bunch of work, experience reports and people willing to contribute to the project.

If making a better HTTP uptime service sounds like something you want to do with your free time, please get involved! Ask questions, fix issues, help newcomers and help us all work together to make the best HTTP uptime service we can.


Social media links for discussion on this article:

Mastodon: https://mst3k.interlinked.me/@cadey/99494112049682603

Reddit: https://www.reddit.com/r/golang/comments/7wbr4o/introducting_lokahi_http_healthchecking_service/

Hacker News: https://news.ycombinator.com/item?id=16338465


How does into Meditation

Permalink - Posted on 2017-12-10 00:00, modified on 0001-01-01 00:00

How does into Meditation

tl;dr

  1. stop thinking
  2. keep not thinking
  3. why’d you stop so soon?

Most of the books, reports, essays and the like focus on step 1. The rest is just keeping your mind quiet, but alert, for as long as you want.


Meditation is an interesting subject. It is as deceptively simple as that tl;dr above, but at the same time for someone who is struggling with it meditation can be frustrating. However, let me assure you it is that easy.

Right now, as you are reading this blogpost, take a deep breath in through your nose (~5 seconds)…and out through your mouth (~5 seconds), repeat this a few times and you will notice a drop in your heart rate, blood pressure and stress levels. Keep doing it for the rest of the time you read this post, it will help you. This is the basis of all meditation, a constant, flowing cycle of breaths in…and out. This cycle gives you predictability and a sense of order. If it helps you, visualize you inhaling peaceful oxygenated air and exhaling the nagging sense of worry that follows you throughout your day.

Peaceful breath in…and all your worries out Peaceful breath in…and all your troubles out Peaceful breath in…and all your anxieties out in a nice, predictable pattern.

Some people have reported that while they are meditating, sometimes worries will pop up seemingly at random, out of nowhere and will try to scare you out of meditation by attempting to pull you back into them. They’ll feel like illogical and stupid things to care about, such as your computer crashing, you thinking about the potential of missing an important message or whatever it is that was on your mind that was the source of stress. Acknowledge them and dismiss them. If it helps you can tell the intrusive thoughts that they have no dominion over you and to begone.

Some people have reported that meditation makes them tired and more easily fall asleep. This is never a bad thing, if anything it points to them getting a lot deeper into meditation than they expected. If this happens for you, just schedule “do not disturb” time for longer than your normal meditation sessions or meditate at night before you go to sleep.

If you have trouble clearing your mind from many things to focus on, there’s a technique I’ve come up with that uses that urge to focus on things to your advantage. If your eyes are closed, open them. Pick a spot on the wall, ceiling or (if you are outside) sky and focus every ounce of attention you have on it. Consider the history of that spot, the materials used to construct the building, if it is painted consider how the person painting the room must have moved their brush or roller to cover that specific part of the wall or ceiling. Listen to how it sounds, imagine how it would feel if you were to go and touch it. (If you are outside, imagine how the wind systems in the stratosphere moved the clouds around to create that specific arrangement, you get the idea) Keep this level of focus for about 30 seconds. After those 30 seconds are up look away from that spot (closing your eyes helps a lot) and banish all thoughts about it for 30 seconds. The more you repeat this in a row the less and less activity your brain should have when you are “idling”.

It may feel tempting to set a timer on your meditation session to “limit” it. This only serves to give you something to worry about while you are trying to not worry about things. The temptation to worry about things will be there, and until you learn to master it, it is a lot easier to just remove as many things as could make you start worrying from the equation as possible.

Don’t be discouraged by what feels like slow progress initially. Your brain is (not exactly) a muscle, and learning to flex it in a new way will always feel slow at first. Keep with it and I promise you will like where you end up.

Remember: breathe easy, clear your mind, keep it clear and hold it clear. That is the heart of all meditation. Everything else is just explanations, techniques that worked for the author of them, anecdotes, stories of others, and generally just rephrasing things so that understanding it is easier.


Voiding the Interview

Permalink - Posted on 2017-04-16 00:00, modified on 0001-01-01 00:00

Voiding the Interview

A young man walks into the room, slightly frustrated-looking. He’s obviously had a bad day so far. You can help him by creating a new state of mind.

“Hello, my name is Ted and I’m here to ask you a few questions about your programming skills. Let’s start with this, in a few sentences explain to me how your favorite programming language works.”

Starting from childhood, you eagerly soaked up the teachings of your mentors, feeling the void separated into sundry shapes and sequences. They taught you many specific tasks to shape the void into, but not how to shape it. Studying the fixed ways of the naacals of old gets you nowhere, learning parlor tricks and saccharine gimmicks. Those gimmicks come rushing back, you remembering how to form little noisemakers and amusement vehicles. They are limiting, but comforting thoughts.

You look up to the interviewer and speak:

“In the beginning there was the void, Spirit was with the void and Spirit was everpresent in the void. The void was cold and formless; the cold unrelenting even in today’s age. Mechanical brains cannot grasp this void the way Spirit can; upon seeing it that is the end of that run. In this way the void is the beginning and the end, always present, always around the corner.”

(def void ())

“What is that?”

> void
>

“But that’s…nothing.”

You look at the caucasian man sitting across from you, and emit “nothing is something, a name for the void still leaves the void extant.”

”…Alright, let’s move on to the next question. This is a formality but the person giving you the phone interview didn’t cover fizzbuzz. Can you do fizzbuzz?”

Stepping into the void, you recall the teachings of your past masters. You equip the parentheses once used by your father and his father before him. The void divides before your eyes in the way you specify:

(defn fizzbuzz [n]
  (cond
    (= 0 (mod n 15)) (print "fizzbuzz")
    (= 0 (mod n 3))  (print "fizz")
    (= 0 (mod n 5))  (print "buzz")
    (print n))
  (println ""))

“This doesn’t loop from 0 to n though, how would you do that?”

You see this section come to life, it gently humming along, waiting for it to be used. Before you you see two ancient systems spring from the memories of patterns once wielded in conflict with complexity.

“Apply this function to span of values.”

> (range 17)
error in __main:0: symbol {range 71} not found

You realize your error the moment you press for confirmation. “Again, in the beginning there is the void. What doesn’t exist needs to be separated out from it.” The voidspace in your head was out of sync with the voidspace of the machine. Define them.

”…Go on”

(defn range-inner [x lim xs]
  (cond
    (>= x lim) xs
    (begin
      (aset! xs x x)
      (range-inner (+ x 1) lim xs))))

(defn range [lim]
  (range-inner 0 lim (make-array lim)))
> (range 17)
[0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16]

“Great, now you have a list of values, how would you get the full output?”

“Pass the function as an argument, injecting the dependency.”

(defn do-array-inner [f x i]
  (cond
    (= i (len x)) void
    (let [val (aget x i)]
      (f val)
      (apply-inner f x (+ i 1)))))

(defn do-array [f x]
  (do-array-inner f x 0))
> (do-array fizzbuzz (range 17))
fizzbuzz
1
2
fizz
4
buzz
fizz
7
8
fizz
buzz
11
fizz
13
14
fizzbuzz
16

Your voidspace concludes the same, creating a sense of peace. You look in the man’s eyes, being careful to not let the fire inside you scare him away. He looks like he’s seen a ghost. Everyone’s first time is rough.

Everything has happened and will happen, there is nothing new in the universe. You know what’s going to happen. They will decline, saying they are looking for a better “culture fit”. They couldn’t contain you.

To run the code in this post:

$ go get github.com/zhemao/glisp
$ glisp
> [paste in blocks]


RSS Feed Generation

Permalink - Posted on 2017-03-29 00:00, modified on 0001-01-01 00:00

RSS Feed Generation

As of a recent commit to this site’s code, it now generates RSS and Atom feeds for future posts on my blog.

For RSS: https://christine.website/blog.rss

For Atom: https://christine.webiste/blog.atom

If there are any issues with this or the generated XML please contact me and let me know so they can be resolved.


gopreload: LD_PRELOAD for the Gopher crowd

Permalink - Posted on 2017-03-25 00:00, modified on 0001-01-01 00:00

gopreload: LD_PRELOAD for the Gopher crowd

A common pattern in Go libraries is to take advantage of init functions to do things like settings up defaults in loggers, automatic metrics instrumentation, flag values, debugging tools or database drivers. With monorepo culture prevalent in larger microservices based projects, this can lead to a few easily preventable problems:

  • Forgetting to set up a logger default or metrics submission, making operations teams blind to the performance of the app and developer teams blind to errors that come up during execution.
  • The requirement to make code changes to add things like metrics or HTTP routing extensions.

There is an environment variable in Linux libc’s called LD_PRELOAD that will load arbitrary shared objects into ram before anything else is started. This has been used for good and evil, but the behavior is the same basic idea as underscore imports in Go.

My solution for this is gopreload. It emulates the behavior of LD_PRELOAD but with Go plugins. This allows users to explicitly automatically load arbitrary Go code into ram while the process starts.

Usage

To use this, add gopreload to your application’s imports:

// gopreload.go
package main

/*
    This file is separate to make it very easy to both add into an application, but
    also very easy to remove.
*/

import _ "github.com/Xe/gopreload"

and then compile manhole:

$ go get -d github.com/Xe/gopreload/manhole
$ go build -buildmode plugin -o $GOPATH/manhole.so github.com/Xe/gopreload/manhole

then run your program with GO_PRELOAD set to the path of manhole.so:

$ export GO_PRELOAD=$GOPATH/manhole.so
$ go run *.go
2017/03/25 10:56:22 gopreload: trying to open: /home/xena/go/manhole.so
2017/03/25 10:56:22 manhole: Now listening on http://127.0.0.2:37588

That endpoint has pprof and a few other fun tools set up, making it a good stopgap “manhole” into the performance of a service.

Security Implications

This package assumes that programs run using it are never started with environment variables that are set by unauthenticated users. Any errors in loading the plugins will be logged using the standard library logger log and ignored.

This has about the same security implications as LD_PRELOAD does in most Linux distributions, but the risk is minimal compared to the massive benefit for being able to have arbitrary background services all be able to be dug into using the same tooling or being able to have metric submission be completely separated from the backend metric creation. Common logging setup processes can be always loaded, making the default logger settings into the correct settings.

Feedback

To give feedback about gopreload, please contact me on twitter or on the Gophers slack (I’m @xena there). For issues with gopreload please file an issue on Github.


Crazy Experiment: Ship the Frontend as an asar document

Permalink - Posted on 2017-01-09 00:00, modified on 0001-01-01 00:00

Crazy Experiment: Ship the Frontend as an asar document

Today’s crazy experiment is using an asar archive for shipping around and mounting frontend Javascript applications. This is something I feel is worth doing because it allows the web frontend developer (or team) give the backend team a single “binary” that can be dropped into the deployment process without having to build the frontend code as part of CI.

asar is an interesting file format because it allows for random access of the data inside the archive. This allows an HTTP server to be able to concurrently serve files out of it without having to lock or incur an additional open file descriptor.

In order to implement this, I have created a Go package named asarfs that exposes the contents of an asar archive as a standard http.Handler.

Example Usage:

package main

import (
	"log"
	"net/http"
	"os"

	"github.com/Xe/asarfs"
)

func do404(w http.ResponseWriter, r *http.Request) {
	http.Error(w, "Not found", http.StatusNotFound)
}

func main() {
	fs, err := asarfs.New("./static.asar", http.HandlerFunc(do404))
	if err != nil {
		log.Fatal(err)
	}

	http.ListenAndServe(":"+os.Getenv("PORT"), fs)
}

I made some contrived benchmarks using some sample data (lots of large json files from mongodb dumps) I had laying around and ran them a few times. The results were very promising:

[~/g/s/g/X/asarfs] : go1.8beta2 test -bench=. -benchmem
BenchmarkHTTPFileSystem-8          20000             66481 ns/op            3219 B/op         58 allocs/op
BenchmarkASARfs-8                  20000             72084 ns/op            3549 B/op         77 allocs/op
BenchmarkPreloadedASARfs-8         20000             62894 ns/op            3218 B/op         58 allocs/op
PASS
ok      github.com/Xe/asarfs    5.636s

Amazingly, the performance and memory usage differences between serving the files over an asar archive and off of the filesystem are negligible. I’ve implemented it in the latest release of my personal website and hopefully end users should be seeing no difference in page load times.


New Site

Permalink - Posted on 2016-12-18 00:00, modified on 0001-01-01 00:00

New Site

This post is now being brought to you by the new and improved https://christine.website. This content is markdown rendered by Purescript. The old site is now being retired in favor of this one. The old site code has been largely untouched since I started writing it in January 2015.

Please give me feedback on how to make it even better!

Christine Dodrill


FFI-ing Go from Nim for Fun and Profit

Permalink - Posted on 2015-12-20 00:00, modified on 0001-01-01 00:00

FFI-ing Golang from Nim for Fun and Profit

As a side effect of Go 1.5, the compiler and runtime recently gained the ability to compile code and run it as FFI code running in a C namespace. This means that you can take any Go function that expresses its types and the like as something compatible with C and use it from C, Haskell, Nim, Luajit, Python, anywhere. There are some unique benefits and disadvantages to this however.

A Simple Example

Consider the following Go file add.go:

package main

import "C"

//export add
func add(a, b int) int {
    return a + b
}

func main() {}

This just exposes a function add that takes some pair of C integers and then returns their sum.

We can build it with:

$ go build -buildmode=c-shared -o libsum.so add.go

And then test it like this:

$ python
>>> from ctypes import cdll
>>> a = cdll.LoadLibrary("./libsum.so")
>>> print a.add(4,5)
9

And there we go, a Go function exposed and usable in Python. However now we need to consider the overhead when switching contexts from your app to your Go code. To minimize context switches, I am going to write the rest of the code in this post in Nim because it natively compiles down to C and has some of the best C FFI I have used.

We can now define libsum.nim as:

proc add*(a, b: cint): cint {.importc, dynlib: "./libsum.so", noSideEffect.}

when isMainModule:
  echo add(4,5)

Which when ran:

$ nim c -r libsum
Hint: system [Processing]
Hint: libsum [Processing]
CC: libsum
CC: system
Hint:  [Link]
Hint: operation successful (9859 lines compiled; 1.650 sec total; 14.148MB; Debug Build) [SuccessX]
9

Good, we can consistently add 4 and 5 and get 9 back.

Now we can benchmark this by using the times.cpuTime() proc:

# test.nim

import
  times,
  libsum

let beginning = cpuTime()

echo "Starting Go FFI at " & $beginning

for i in countup(1, 100_000):
  let myi = i.cint
  discard libsum.add(myi, myi)

let endTime = cpuTime()

echo "Ended at " & $endTime
echo "Total: " & $(endTime - beginning)
$ nim c -r test
Hint: system [Processing]
Hint: test [Processing]
Hint: times [Processing]
Hint: strutils [Processing]
Hint: parseutils [Processing]
Hint: libsum [Processing]
CC: test
CC: system
CC: times
CC: strutils
CC: parseutils
CC: libsum
Hint:  [Link]
Hint: operation successful (13455 lines compiled; 1.384 sec total; 21.220MB; Debug Build) [SuccessX]
Starting Go FFI at 0.000845
Ended at 0.131602
Total: 0.130757

Yikes. This takes 0.13 seconds to do the actual computation of every number i in the range of 0 through 100,000. I ran this for a few hundred times and found out that it was actually consistently scoring between 0.12 and 0.2 seconds. Obviously this cannot be a universal hammer and the FFI is very expensive.

For comparison, consider the following C library code:

// libcsum.c
#include "libcsum.h"

int add(int a, int b) {
  return a+b;
}
// libcsum.h
extern int add(int a, int b);
# libcsum.nim
proc add*(a, b: cint): cint {.importc, dynlib: "./libcsum.so", noSideEffect.}

when isMainModule:
  echo add(4, 5)

and then have test.nim use the C library for comparison:

# test.nim

import
  times,
  libcsum,
  libsum

let beginning = cpuTime()

echo "Starting Go FFI at " & $beginning

for i in countup(1, 100_000):
  let myi = i.cint
  discard libsum.add(myi, myi)

let endTime = cpuTime()

echo "Ended at " & $endTime
echo "Total: " & $(endTime - beginning)

let cpre = cpuTime()
echo "starting C FFI at " & $cpre

for i in countup(1, 100_000):
  let myi = i.cint
  discard libcsum.add(myi, myi)

let cpost = cpuTime()

echo "Ended at " & $cpost
echo "Total: " & $(cpost - cpre)

Then run it:

➜  nim c -r test
Hint: system [Processing]
Hint: test [Processing]
Hint: times [Processing]
Hint: strutils [Processing]
Hint: parseutils [Processing]
Hint: libcsum [Processing]
Hint: libsum [Processing]
CC: test
CC: system
CC: times
CC: strutils
CC: parseutils
CC: libcsum
CC: libsum
Hint:  [Link]
Hint: operation successful (13455 lines compiled; 0.972 sec total; 21.220MB; Debug Build) [SuccessX]
Starting Go FFI at 0.00094
Ended at 0.119729
Total: 0.118789

starting C FFI at 0.119866
Ended at 0.12206
Total: 0.002194000000000002

Interesting. The Go library must be doing more per instance than just adding the two numbers and continuing about. Since we have two near identical test programs for each version of the library, let’s strace it and see if there is anything that can be optimized. The Go one and the C one are both very simple and it looks like the Go runtime is adding the overhead.

Let’s see what happens if we do that big loop in Go:

// add.go

//export addmanytimes
func addmanytimes() {
    for i := 0; i < 100000; i++ {
        add(i, i)
    }
}

Then amend libsum.nim for this function:

proc addmanytimes*() {.importc, dynlib: "./libsum.so".}

And finally test it:

# test.nim

echo "Doing the entire loop in Go. Starting at " & $beforeGo

libsum.addmanytimes()

let afterGo = cpuTime()

echo "Ended at " & $afterGo
echo "Total: " & $(afterGo - beforeGo) & " seconds"

Which yields:

Doing the entire loop in Go. Starting at 0.119757
Ended at 0.119846
Total: 8.899999999999186e-05 seconds

Porting the C library to have a similar function would likely yield similar results, as would putting the entire loop inside Nim. Even though this trick was only demonstrated with Nim and Python, it will work with nearly any language that can convert to/from C types for FFI. Given the large number of languages that do have such an interface though, it seems unlikely that there will be any language in common use that you cannot write to bind to Go code. Just be careful and offload as much of it as you can to Go. The FFI barrier really hurts.


This post’s code is available here.


The Origin of h

Permalink - Posted on 2015-12-14 00:00, modified on 0001-01-01 00:00

The Origin of h

For a while I have been pepetuating a small joke between my friends, co-workers and community members of various communities (whether or not this has been beneficial or harmful is out of the scope of this post). The whole “joke” is that someone says “h”, another person says “h” back.

That’s it.

This has turned into a large scale game for people, and is teachable to people with minimal explanation. Most of the time I have taught it to people by literally saying “h” to them until they say “h” back. An example:

<Person> Oh hi there
  <Xena> h
<Person> ???
  <Xena> Person: h
<Person> i
  <Xena> Person:
  <Xena> h
<Person> h
  <Xena> :D

Origins

This all started on a particularly boring day when we found a video by motdef with gameplay from Moonbase Alpha, an otherwise boring game made to help educate people on what would go on when a moonbase has a disaster. This game was played by many people because of its text-to-speech engine, which lead to many things like flooding “JOHN MADDEN” or other inane things like that.

Specifically there was a video called “Moonbase 4lpha: *****y Space Skeletons” that at one point had recorded the phrase “H H H RETURN OF GANON”. Me and a few friends were flooding that in an IRC room for a while and it eventually devolved into just flooding “h” to eachother. The flooding of “h” lasted over 8 hours (we were really bored) and has evolved into the modern “h” experience we all know and love today.

The IRC Bot

Of course, humans are unreliable. Asking them to do things predictably is probably a misguided idea so it is best to automate things with machines whenever it is pragmatic to do so. As such, I have created and maintained the following python code that automates this process. An embarassing amount of engineering and the like has gone into making sure this function provides the most correct and canonical h experience money can buy.

@hook.regex(r"^([hH])([?!]*)$")
def h(inp, channel=None, conn=None):
    suff = ""
    if inp.group(2).startswith("?"):
        suff = inp.group(2).replace("?", "!")
    elif inp.group(2).startswith("!"):
        suff = inp.group(2).replace("!", "?")
    return inp.group(1) + suff

The code was pulled from here.

Here is an example of it being used:

(Xena) h
   (h) > h
(Xena) h???
   (h) > h!!!
(Xena) h!!!!
   (h) > h????

-- [h] (h@h): h
-- [h] is using a secure connection
-- [h] is a bot
-- [h] is logged in as h

I also ended up porting h to matrix under the name h2. It currently sits in #ponydevs:matrix.org and has a bad habit of getting broken because Comcast is a bad company and doesn’t believe in uptime.

Spread of h

Like any internet meme, it is truly difficult to see how far it has spread with 100% certainty. However I have been keeping track of where and how it has spread, and I can estimate there are at least 50 guardians of the h.

However, its easily teachable nature and very minimal implementation means that new guardians of the h can be created near instantly. It is a lightweight meme but has persisted for at least 2 years. This means it is part of internet culture now, right?

There has been one person in the Derpibooru IRC channel that is really violently anti-h and has a very humorous way of portraying this. Stop in and idle and you’ll surely see it in action.

Conclusion

I hope this helps clear things up on this very interesting and carefully researched internet meme. I hope to post further updates as things become clear on this topic.


Below verbatim is the forum post (it was deleted, then converted to a blog post on his blog) that inspired the writing of this article.

Parcly Taxel

Lately, if you’ve been going up to our Derpibooru IRC channel, you may notice that a significant portion of sayings and rebuttals are countered with the single letter h (lowercase). So where does this come from?

This is a joke started by Xena, one of the administrators of the Ponychat IRC system which the site uses. It came from a video showing gameplay, glitches and general tomfoolery in the simulation game Moonbase Alpha. Starting from 1:32 there is shown a dialogue between two players, one of which makes grandiose comments about how they will “eradicate” everyone else, to which the other simply replies “h” or multiples of it.

Hence when h is spoken in IRC, do know that it’s a shorthand for “yes and I laugh at you”. I do not recommend using it though as it could be confused with hydrogen or UTC+8 (the time zone in which I live).


Coming Out

Permalink - Posted on 2015-12-01 00:00, modified on 0001-01-01 00:00

Coming Out

I’d like to bring up something that has been hanging over my head for a long time. This is something I did try (and fail) to properly express way back in middle school, but now I’d like to get it all of my chest and let you know the truth of the matter.

I don’t feel comfortable with myself as I am right now. I haven’t really felt comfortable with myself for at least 10 years, maybe more; I’m not entirely sure.

At this point in my life I am really faced with a clear fork in the road. I can either choose to continue living how I currently do, lying to myself and others and saying everything is normal, or I can cooperate with the reality that my brain is telling me that I don’t feel comfortable with myself as I have been for the last almost 22 years. I feel like I don’t fit inside my own skin. I think it is overall better for me to face the facts and cooperate with reality. I have been repressing this off and on out of fear of being shot down or not accepted the way I want to be seen to you all. This has been a really hard thing for me to think through and even harder for me to work up the courage to start taking action towards. This is not a choice for me. I need to pursue this.

In fact, I have been pursing this. My current business cards reflect who I really am. My co-workers accept my abnormal status (when compared to the majority of society), and even will help stand up for me if something goes south with regards to it.

I fully understand how much information this is to take in at once. I know it will be difficult for you to hear that your firstborn son is actually a daughter in a son’s body, but I am still the same person. Most of the changes that I want to pursue are purely cosmetic, but they are a bit more noticeable than changing hair color. I feel that transitioning to living as a woman like this will help me feel like I fit in with the world better and help to make me more comfortable with who I am and how I want other people to see me. Below I have collected some resources for you to look through. They will help for you to understand my views better explained in language you would be familiar with.

I have been trialing a lot of possible first names to use, Zoe (the name you were going to give me if I was born a girl) did come to mind, but after meditating on it for a while I have decided that it doesn’t fit me at all. The name I am going with for now and eventually will change my official documents to use is Christine Cadence Dodrill.

Additionally I have been in a long-distance relationship with someone since mid-June 2014. His name is Victor and he lives in Ottawa, Ontario. He has been helping me a lot as I sort through all this; it has been a godsend. He is a student in college for Computer Science. He knows and is aware about my transition and has been a huge part of my emotional line of support as I have been accepting these facts about who I am.


Above is (a snipped version of) the letter I sent to my parents in the last 48 hours. With this I have officially come out to all of my friends and family as transgender. I am currently on hormone replacement therapy and have been living full time as a woman. My workplace is very accepting of this and has been a huge help over the last 7-8 months as I have battled some of my inner demons and decided to make things official.

I am now deprecating my old facebook account and will be encouraging people to send friend requests and the like to my new account under the correct name.

Thank you all for understanding and be well.


The Universal Design

Permalink - Posted on 2015-10-17 00:00, modified on 0001-01-01 00:00

The Universal Design

As I have been digging through existing code, systems and the like I have been wondering what the next big direction I should go in is. How to design things such that the mistakes of the past are avoided, but you can benefit from them and learn better how to avoid them. I have come to a very simple conclusion, monoliths are too fragile.

Deconstructing Monoliths

One monolith I have been maintaining is Elemental-IRCd. Taking the head of a project I care about has taught me more about software engineering, community/project management and the like than I would have gotten otherwise. One of these things is that there need to be five basic primitives in your application:

  1. State - What is true now? What was true? What happened in the past? What is the persistent view of the world?
  2. Events - What is being changed? How will it be routed?
  3. Policy - Can a given event be promoted into a series of actions?
  4. Actions - What is the outcome of the policy?
  5. Mechanism - How should an event be taken in and an action put out?

Let’s go over some basic examples of this theory in action:

Spinning up a Virtual Machine

  • the event is that someone asked to spin up a virtual machine
  • the policy is do they have permission to spin that machine up?
  • the mechanism is an IRC command for some reason
  • the action is that a virtual machine is created
  • the state is changed to reflect that VM creation

Webserver

  • the event is an HTTP request
  • the policy is to do some database work and return the action of showing the HTML to the user
  • the mechanism is nginx sending data to a worker and relaying it back
  • the state is updated for whatever changed

And that’s it. All you need is a command queue feeding into a thread pool which feeds out into a transaction queue which modifies state. And with that you can explain everything from VMWare to Google.

As a fun addition, we can also define nearly all of this as being functionally pure code. The only thing that really needs to be impure are mechanisms and applying actions to the state. Policy handlers should be mostly if not entirely pure, but also may need to access state not implicitly passed to it. The only difference between an event and an action is what they are called.

Policy

Now, how would a policy handler work? I am going to be explaining this in the context of an IRC daemon as that is what I intend to develop next. Let’s sketch out the low level:

The relevant state is the state of the IRC network. An event is a command from a user or server. A policy is a handler for either a user command or another kind of emitted action from another policy handler.

One of the basic commands in RFC 1459 is the NICK command. A user using it passes the new nickname they want. Nicknames must also be unique.

-- nick-pass-1.lua

local mephiles = require "mephiles"

mephiles.declareEvent("user:NICK", function(state, source, args)
  if #args ~= 1 then
    return {
      {mepliles.failure, {mephiles.pushNumeric, source, mephiles.errCommandBadArgc(1)}}
    }
  end

  local newNick = args[1]

  if state.nicks.get(newNick) then
    return {
      {mephiles.failure, {mephiles.pushNumeric, source, mephiles.errNickInUse(newNick)}}
    }
  end

  if not mephiles.legalNick(newNick) then
    return {
      {mephiles.failure, {mephiles.pushNumeric, source, mephiles.errIllegalNick(newNick)}}
    }
  end

  return {
    {mephiles.success, {"NICKCHANGE", source, newNick}}
  }
end)

This won’t scale as-is, but most of this is pretty straightforward. The policy function returns a series of actions that fall into two buckets: success and failure. Most of the time the success of state changes (nickname change, etc) will be confirmed to the client. However a large amount of the common use (PRIVMSG, etc) will be unreported to the client (yay RFC 1459); but every single time a line from a client fails to process, the client must be notified of that failure.

Something you can do from here is define a big pile of constants and helpers to make this easier:

local actions = require "actions"
local c       = require "c"
local m       = require "mephiles"
local utils   = require "utils"

m.UserCommand("NICK", c.normalFloodLimit, function(state, source, args)
  if #args ~= 1 then
    return actions.failCommand(source, "NICK", c.errCommandBadArgc(1))
  end

  local newNick = args[1]

  if state.findTarget(newNick) then
    return actions.failCommand(source, "NICK", c.errNickInUse(newNick))
  end

  if not utils.legalNick(newNick) then
    return actions.failCommand(source, "NICK", c.errIllegalNick(newNick))
  end

  return {actions.changeNick(source, newNick)}
end)

Thread Safety

This as-is is very much not thread-safe. For one the Lua library can only have one thread interacting with it at a time, so you will need a queue of events to it. The other big problem is that this is prone to race conditions. There are two basic solutions to this:

  1. The core takes a lock on all of the state at once
  2. The policy handlers take a lock on resources as they try to use them and the core automatically releases locks at the end of it running.

The simpler implementation will do for an initial release, but the latter will scale a lot better as more and more users hit the server at the same time. It allows unrelated things to be changed at the same time, which is the majority case for IRC.

In the future, federation of servers can be trivialized by passing the actions from one server to another if it is needed, and by implicitly trusting the actions of a remote server.


This design will also scale to running across multiple servers, and in general to any kind of computer, business or industry problem.

What if this was applied to the CPU and a computer in general at a low level? How would things be different?

Urbit

Over the past few weeks I have been off and on dipping my toes into Urbit. They call Urbit an “operating function” and define it as such:

V(I) => T

where T is the state, V is the fixed function, and I is the list of input events from first to last.

Urbit at a low level takes inputs, applies them to a function and returns the state of the computer. Sound familar?

~hidduc-posmeg has been putting together a set of tutorials^* to learn Hoon, its higher-level lisp-like language. At the end of the first one, they say something that I think is also very relevant to this systems programming ideal:

All Hoon computation takes [the] same general form. A subject with a fomula that transforms that subject in some way to produce a product which is then used as the subject for some other formula. In our next tutorial we’ll look at some of the things we can do to our subject.

Subjects applied to formulae become results that are later applied to formulae as subjects. Events applied to policy emit actions which later become events for other policies to emit actions.

Because of this design, you can easily do live code reloading, because there is literally no reason you can’t. Wait for a formula to finish and replace it with the new version, provided it compiles. Why not apply this to the above ideas too?


* Link here: http://hidduc-posmeg.urbit.org/home/pub/hoon-intro/ as of publishing this revision of the article hidduc’s urbit is offline, so they cannot be accessed at the moment. If that link fails, the source code for it is apparently here. Thanks mst on Freenode!

For comments on this article, please feel free to email me, poke me in #geek on irc.ponychat.net (my nick is Xena, on freenode it is Xe), or leave thoughts at one of the places this article has been posted.


Metaprogramming: Partial Application...

Permalink - Posted on 2015-08-26 00:00, modified on 0001-01-01 00:00

Metaprogramming: Partial Application and Currying 101

The title of this post looks intimidating. There’s a lot of words there that look like they are very complicated and will take a long time to master. In reality, they are really very simple things. Let’s start with a mundane example and work our way up to a real-world bit of code. Let’s begin with a small story:


ACMECorp has a world-renowned Python application named Itera that is known for its superb handling of basic mathematic functions. It’s so well known and industry proven that it is used in every school and on every home computer. You have just accepted a job there as an intermediate programmer set to do maintenance on it. Naturally, you are very excited to peek under the hood of this mysterious and powerful program and offer your input to make it even better for the next release and its users.

Upon getting there, you settle in and look at your ticket queue for the day. A user is complaining that whenever they add 3 and 5, they get 7 instead of 8, which is what they expected. Your first step is to go look into the add3 function and see what it does:

def add1(x):
    return x + 1

def add2(x):
    return x + 2

def add3(x):
    return x + 2

def add4(x):
    return x + 4

You are aghast. Your company’s multi-billion dollar calculator is brought to its knees by a simple copy-paste error. You wonder, “how in Sam Hill are these people making any money???” (The answer, of course, is that they are a big enterprise corporation)

You let your boss know about the bad news, you are immediately given any resource in the company that you need to get this mission-critical problem solved for any input. Yesterday. Without breaking the API that the rest of the program has hard-coded in.


Let’s look at what is common about all these functions. The add* family of functions seems to all be doing one thing consistently: adding one number to another.

Let’s define a function called add that adds any two numbers:

def add(x, y):
    return x + y

This is nice, but it won’t work for the task we were given, which is to not break the API.

Let’s go over what a function is in Python. We can define a function as something that takes some set of Python values and produces some set of Python values:

PythonFunction :: [PythonValue] -> [PythonValue]

We can read this as “a Python function takes a set of Python values and produces a set of Python values”. Now we need to define what a Python value actually is. To keep things simple, we’re only going to define the following types of values:

  • None -> no value
  • Int -> any whole number (Python calls this int)
  • Text -> any string value (Python calls this str)
  • Function -> something that takes and produces values

Python itself has a lot more types that any value can be, but for the scope of this blog post, this will do just fine.

Now, since a function can return a value and a function is a value, let’s see what happens if you return a function:

def outer():
    def inner():
        return "Hello!"
    return inner

And in the repl:

>>> type(outer)
<type 'function'>

So outer is a function as we expect. It takes None (in Python, a function without arguments has None for the type of them) and returns a function that takes None and that function returns Text containing "Hello!". Let’s make sure of this:

>>> outer()()
'Hello!'
>>> type(outer()())
<type 'str'>

Yay! When nothing is applied to the result of applying nothing to outer, it returns the Text value "Hello!". We can define the type of outer as the following:

outer :: None -> None -> Text

Now, let’s use this for addition:

# add :: Int -> Int -> Int
def add(x):
    def inner(y):
        return x + y

    return inner

And in the repl:

>>> add(4)(5)
9

A cool feature about this is that now we can dip into something called Partial Application. Partial application lets you apply part of the arguments of a function and you get another function out of it. Let’s trace the type of the inner function inside the add function, as well as the final computation for clarity:

# add :: Int -> Int -> Int
def add(x):
    # inner :: Int -> Int
    def inner(y):
        return x + y # :: Int

    return inner

Starting from the inside, we can see how the core computation here is x + y, which returns an Int. Then we can see that y is passed in and in the scope also as an Int. Then we can also see that x is passed in the outermost layer as an int, giving it the type Int -> Int -> Int. Since inner is a value, and a Python variable can contain any Python value, let’s make a function called increment using the add function:

# increment :: Int -> Int
increment = add(1)

And in the repl:

>>> increment(50)
51

increment takes the integer given and increases it by 1, it is the same thing as defining:

def increment50():
    return 51

Or even 51 directly.

Now, let’s see how we can use this for the add* family of function mentioned above:

# add :: Int -> Int -> Int
def add(x):
    def inner(y):
        return x + y

    return inner

# add1 :: Int -> Int
add1 = add(1)

# add2 :: Int -> Int
add2 = add(2)

# add3 :: Int -> Int
add3 = add(3)

# add4 :: Int -> Int
add4 = add(4)

And all we need to do from here is a few simple tests to prove it will work:

if __name__ == "__main__":
    assert add(1)(1) == 2 # 1 + 1
    assert add(1)(2) == add(2)(1) # 1+2 == 2+1
    print("all tests passed")
$ python addn.py
all tests passed

Bam. The add* family of functions is now a set of partial applications. It is just a set of half-filled out forms.


You easily mechanically rewrite all of the add* family of functions to use the metaprogramming style you learned on your own. Your patch goes in for consideration to the code review team. Meanwhile your teammates are frantically going through every function in the 200,000 line file that defines the add* family of functions. They are estimating months of fixing is needed not to mention millions of lines of test code. They are also estimating an additional budget of contractors being brought in to speed all this up. Your code has made all of this unneeded.

Your single commit was one of the biggest in company history. Billboards that were red are now beaming a bright green. Your code fixed 5,000 other copy-paste errors that have existed in the product for years. You immediately get a raise and live happily ever after, a master in your craft.


For fun, let’s rewrite the add function in Haskell.

add :: Int -> Int -> Int
add x y = x + y

And then we can create a partial application with only:

add1 :: Int -> Int
add1 = (add 1)

And use it in the repl:

Prelude> add1 3
4

Experienced haskellers would probably gawk at this. Because functions are the base data type in Haskell, and partial application means that you can make functions out of functions, we can define add as literally the addition operator (+):

add :: Int -> Int -> Int
add = (+)

And because operators are just functions, we can further simplify the add1 function by partially applying the addition operation:

add1 :: Int -> Int
add1 = (+1)

And that will give us the same thing.

Prelude> let add1 = (+1)
Prelude> add1 3
4

Now, real world example time. I recently wrote a simple JSON api based off of a lot of data that has been marginally useful to some people. This api has a series of HTTP endpoints that return data about My Little Pony: Friendship is Magic episodes. Its code is here and its endpoint is http://ponyapi.apps.xeserv.us.

One of the challenges when implementing it was how to avoid a massive amount of copy-pasted code when doing so. I had started with a bunch of functions like:

# all_episodes :: IO [Episode]
def all_episodes():
    r = requests.get(API_ENDPOINT + "/all")

    if r.status_code != 200:
        raise Exception("Not found or server error")

    return r.json()["episodes"]

Which was great and all, but there was so much code duplication involved to just get one result for all the endpoints. My first step was to write something that just automated the getting of json from an endpoint in the same way I automated addition above:

# _base_get :: Text -> None -> IO (Either Episode [Episode])
def _base_get(endpoint):
    def doer():
        r = requests.get(API_ENDPOINT + endpoint)

        if r.status_code != 200:
            raise Exception("Not found or server error")

    try:
        return r.json()["episodes"]
    except:
        return r.json()["episode"]

# all_episodes :: IO [Episode]
all_episodes = _base_get("/all")

Where _base_get returned the function that satisfied the request.

This didn’t end up working so well with the endpoints that take parameters, so I had to account for that in my code:

# _base_get :: Text -> Maybe [Text] -> (Maybe [Text] -> IO (Either Episode [Episode]))
# _base_get takes a text, a splatted list of texts and returns a function such that
#     the function takes a splatted list of texts and returns either an Episode or
#     a list of Episode as an IO action.
def _base_get(endpoint, *fragments):
    def doer(*args):
        r = None

        assert len(fragments) == len(args)

        if len(fragments) == 0:
            r = requests.get(API_ENDPOINT + endpoint)
        else:
            url = API_ENDPOINT + endpoint

            for i in range(len(fragments)):
                url = url + "/" + fragments[i] + "/" + str(args[i])

            r = requests.get(url)

        if r.status_code != 200:
            raise Exception("Not found or server error")

        try:
            return r.json()["episodes"]
        except:
            return r.json()["episode"]

    return doer

# all_episodes :: IO [Episode]
all_episodes = _base_get("/all")

# newest :: IO Episode
newest = _base_get("/newest")

# last_aired :: IO Episode
last_aired = _base_get("/last_aired")

# random :: IO Episode
random = _base_get("/random")

# get_season :: Int -> IO [Episode]
get_season = _base_get("", "season")

# get_episode :: Int -> Int -> IO Episode
get_episode = _base_get("", "season", "episode")

And that was it, save the /search route, which was acceptable to implement by hand:

# search :: Text -> IO [Episode]
def search(query):
    params = {"q": query}
    r = requests.get(API_ENDPOINT + "/search", params=params)

    if r.status_code != 200:
        raise Exception("Not found or server error")

    return r.json()["episodes"]

Months later you have been promoted as high as you can go. You’ve been teaching the other engineers at ACMECorp metaprogramming and even convinced management to let the next big project be in Haskell.

You are set for life. You have won.


For comments on this article, please feel free to email me, poke me in #geek on irc.ponychat.net (my nick is Xena), or leave thoughts at one of the below places this article has been posted.

Comments:


Nim and Tup

Permalink - Posted on 2015-06-10 00:00, modified on 0001-01-01 00:00

Nim and Tup

I have been recently playing with and using a new lanugage for my personal development, Nim. It looks like Python, runs like C and integrates well into other things. Its compiler targets C, and as a result of this binding things to C libraries is a lot more trivial in Nim; even moreso than with go.

For example, here is a program that links to the posix crypt(3) function:

# crypt.nim
import posix

{.passL: "-lcrypt".}

echo "What would you like to encrypt? "
var password: string = readLine stdin
echo "What is the salt? "
var salt: string = readLine stdin

echo "result: " & $crypt(password, salt)

And an example usage:

xena@fluttershy (linux) ~/code/nim/crypt
➜  ./crypt
What would you like to encrypt?
foo
What is the salt?
rs
result: rsHt73tkfd0Rg

And that’s it. No having to worry about deferring to free the C string, no extra wrappers (like with Python or Lua), you just write the code and it just works.

At the idea of another coworker, I’ve also started to use tup for building things. Nim didn’t initially work very well with tup (temporary cache needed, etc), but a very simple set of tup rules were able to fix that:

NIMFLAGS += --nimcache:".nimcache"
NIMFLAGS += --deadcodeElim:on
NIMFLAGS += -d:release
NIMFLAGS += -d:ssl
NIMFLAGS += -d:threads
NIMFLAGS += --verbosity:0

!nim = |> nim c $(NIMFLAGS) -o:%o %f && rm -rf .nimcache |>

This creates a tup !-macro called !nim that will Do The Right Thing implicitly. Usage of this is simple:

.gitignore
include_rules

: crypt.nim |> !nim |> ../bin/crypt
xena@fluttershy (linux) ~/code/nim/crypt
➜  tup
[ tup ] [0.000s] Scanning filesystem...
[ tup ] [0.130s] Reading in new environment variables...
[ tup ] [0.130s] No Tupfiles to parse.
[ tup ] [0.130s] No files to delete.
[ tup ] [0.130s] Executing Commands...
 1) [0.581s] nim c --nimcache:".nimcache" --deadcodeElim:on --verbosity:0 crypt.nim && rm -rf .nimcache
 [ ] 100%
[ tup ] [0.848s] Updated.

Not only will this build the program if needed, it will also generate a gitignore for all generated files. This is an amazing thing. tup has a lot more features (including lua support for scripting complicated build logic), but there is one powerful feature of tup that makes it very difficult for me to work into my deployment pipelines.

tup requires fuse to ensure that no extra things are being depended on for builds. Docker doesn’t let you use fuse mounts in the build process.

I have a few ideas on how to work around this, and am thinking about tackling them when I get nim programs built inside Rocket images.


Trying Vagga on For Size

Permalink - Posted on 2015-03-21 00:00, modified on 0001-01-01 00:00

Trying Vagga on For Size

Vagga is a containerization tool like Docker, Rocket, etc but with one major goal that is highly ambitious and really worth mentioning. Its goal is to be a single userspace binary without a suid bit or a daemon running as root.

However, the way it does this seems to be highly opinionated and there are some things which annoy me. Let’s go over the basics:

All Vagga Images Are Local To The Project

There is no “global vagga cache”. Every time I want to make a new project folder with an ubuntu image I have to wait the ~15 minutes it takes for Ubuntu to download on my connection (Comcast). As such I’ve been forced to use Alpine.

No Easy Way To Establish Inheritance From Common Code

With Docker I can create an image xena/lapis and have it contain all of the stuff needed for lapis applications to run. With Vagga I currently have to constantly reinvent the setup for this or risk copying and pasting code everywhere

Multiple Containers Can Be Defined In The Same File

This is a huge plus. The way this all is defined is much more sane than Fig or Docker compose. It’s effortless where the Docker workflow was kinda painful. However this is a bittersweet advantage as:

Vagga Containers Use The Same Network Stack As The Host

Arguably this is because you need root permissions to do things like that with the IP stack in a new namespace, but really? It’s just inconvenient to have to wrap Vagga containers in Docker or the like just to be able to run things without the containers using TCP ports on the host up.

http://vagga.readthedocs.org/en/latest/network.html is interesting.

Overall, Vagga looks very interesting and I’d like to see how it turns out.


Interesting Links


CinemaQuestria Orchestration

Permalink - Posted on 2015-03-13 00:00, modified on 0001-01-01 00:00

CinemaQuestria Orchestration

Or: Continuous Defenstration in a Container-based Ecosystem

I’ve been a core member of the staff for CinemaQuestria for many months. In that time we have gone from shared hosting (updated by hand with FTP) to a git-based deployment system that has won over the other staffers.

In this blogpost I’m going to take a look at what it was, what it is, and what it will be as well as some challenges that have been faced or will be faced as things advance into the future.

The Past

The site for CinemaQuestria is mostly static HTML. This was chosen mainly because it made the most sense for the previous shared hosting environment as it was the least surprising to set up and test.

The live site content is about 50 MB of data including PDF transcripts of previous podcast episodes and for a long time was a Good Enough solution that we saw no need to replace it.

However, being on shared hosting it meant that there was only one set of authentication credentials and they had to be shared amongst ourselves. This made sense as we were small but as we started to grow it didn’t make much sense. Combined with the fact that the copy of the site on the live server was pretty much the only copy of the site we also lost disaster recovery points.

Needless to say, I started researching into better solutions for this.

The first solution I took a look at was AWS S3. It would let us host the CQ site for about 0 dollars per month. On paper this looked amazing, until we tried it and everyone was getting huge permissions issues. The only way to have fixed this would have been to have everyone use the same username/password or to have only one person do the deploys. In terms of reducing the Bus factor of the site’s staff, this was also unacceptable.

I had done a lot of work with Dokku-alt for hosting my personal things (this site is one of many hosted on this server), so I decided to give it a try with us.

The Present

Presently the CQ website is hosted on a Dokku-alt server inside a container. For a while while I was working on getting the warts out only I had access to deploy code to the server, but quickly on I set up a private repo on my git server for us to be able to track changes.

Once the other staffers realized the enormous amount of flexibility being on git gave us they loved it. From the comments I received the things they liked the most were:

  • Accountability for who made what change
  • The ability to rollback changes if need be
  • Everyone being able to have an entire copy of the site and its history

After the warts were worked out I gave the relevant people access to the dokku server in the right way and the productivity has skyrocketed. Not only have people loved how simple it is to push out new changes but they love how consistent it is and the brutal simplicity of it.

Mind you these are not all super-technically gifted people, but the command line git client was good enough that not only were they able to commit and make changes to the site, but they also took initiative and corrected things they messed up and made sure things were consistent and correct.

When I saw those commits in the news feed, I almost started crying tears of happy.

Nowadays our site is hosted inside a simple nginx container. In fact, I’ll even paste the entire Dockerfile for the site below:

FROM nginx

COPY . /usr/share/nginx/html

That’s it. When someone pushes a new change to the server it figures out everything from just those two lines of code.

Of course, this isn’t to say this system is completely free of warts. I’d love to someday be able to notify the backrooms on skype every time a push to the live server is made, but that might be for another day.

The Future

In terms of future expansion I am split mentally. On one hand the existing static HTML is hysterically fast and efficient on the server, meaning that anything such as a Go binary, Lua/Lapis environment or other web application framework would have a very tough reputation to beat.

I have looked into using Lapis for this beta test site, but the fact that HTML is so dead easy to modify made that idea lose out.

Maybe this is in the realm of something like jekyll, Hugo or sw to take care of. I’d need to do more research into this when I have the time.

If you look at the website code currently a lot of it is heavily duplicated code because the shared hosting version used to use Apache server-side includes. I think a good place to apply these would be in the build in the future. Maybe with a nice husking operation on build.


Anyways, I hope this was interesting and a look into a side of CinemaQuestria that most of you haven’t seen before. The Season 5 premiere is coming up soon and this poor server is going to get hammered like nothing else, so that will be a nice functional test of Dokku-alt in a production setting.


This Site's Tech Stack

Permalink - Posted on 2015-02-14 00:00, modified on 0001-01-01 00:00

This Site’s Tech Stack

Note: this is out of date as this site now uses PureScript and Go.

As some of my close friends can vouch, I am known for sometimes setting up and using seemingly bizarre tech stacks for my personal sites. As such I thought it would be interesting to go in and explain the stack I made for this one.

The Major Players

Markdown

This is a markdown file that gets rendered to HTML and sent to you via the lua discount library. As I couldn’t get the vanilla version from LuaRocks to work, I use Debian’s version.

I like Markdown for thigns like this as it is not only simple, but easy for people to read, even if they don’t know markdown or haven’t worked with any other document system than Office or other wisywig document processors.

Lapis

Lapis is the middleware between Lua and Nginx that allows me to write pages simply. Here is some of the code that powers this page:

-- controllers/blog.moon
class Blog extends lapis.Application
  ["blog.post": "/blog/:name"]: =>
    @name = util.slugify @params.name
    @doc = oleg.cache "blogposts", @name, ->
      local data
      with io.open "blog/#{@name}.markdown", "r"
        data = \read "*a"

      discount data, "toc", "nopants", "autolink"

    with io.open "blog/#{@name}.markdown", "r"
      @title = \read "*l"

  render: true

And the view behind this page:

-- views/blog/post.moon
import Widget from require "lapis.html"
class Post extends Widget
  content: =>
    raw @doc

That’s it. That even includes the extra overhead of caching the markdown as HTML in a key->value store called OlegDB (I will get into more detail about it below). With Lapis I can code faster and be much more expressive with a lot less code. I get the syntactic beauty that is Moonscript with the speed and raw power of luajit on top of nginx.

OlegDB

OlegDB is a joke about mayonnaise that has gone too far. It has turned into a full fledged key->value store and I think it is lovely.

Container Abuse

I have OlegDB running as an in-container service. This means that OlegDB does hold some state, but only for things that are worth maintaining the stats of (in my eyes). Having a cache server right there that you can use to speed things up with is a brilliant abuse of the fact that I run a container that allows me to do that. I have Oleg hold the very HTML you are reading right now! When it renders a markdown file for the first time it caches it into Oleg, and then reuses that cached version when anyone after the first person reads the page. I do the same thing in a lot of places in the codebase for this site.


I hope this look into my blog’s tech stack was interesting!


The Saga of plt, Part 1

Permalink - Posted on 2015-02-14 00:00, modified on 0001-01-01 00:00

The Saga of plt, Part 1

The following is adapted from a real story. Parts of it are changed to keep it entertaining to read but the core of the story is maintained. I apologize that this issue in the epic will be shorter than the others, but it gets better.

The Beginning of The Interesting Pain

It all started when I got this seemingly innocuous PM on Freenode:

2015-01-23 [18:32:48] <plt> Hello. I am writting a new ircd and can I have the channel ##ircd please?

This is a fairly common event on larger IRC networks, especially given the length of the channel name and the fact that it references IRC daemons specifically. At this point I had forgotten I owned that channel. So naturally I decided to give it a join and see if the person who requested the channel was worthy of it or had brought enough activity to it such that it was morally correct to hand it off.

This was not the case.

[18:33:54] *** Joins: Xe (xe@unaffiliated/xe)
[18:34:02] <plt> Hello xe.
[18:35:17] <plt> Xe the project name pbircd.
[18:37:09] <plt> Xe the project site is http://sourceforge.net/p/pbircd

In case the site is removed from SourceForge, it is the default sourceforge page.

After taking a look at this and then getting off the call with my family I was on at the point, I decided to reply.

[20:30:49] <Xe> plt: I've decided against giving you my channel
[20:31:03] <Xe> you have no code in your repo.
[20:31:31] <plt> I am currently working on the project. Can I help you in the channel?
[20:32:04] <Xe> if you are working on it
[20:32:11] <Xe> I'd expect to see at least something
[20:32:25] <Xe> for example: https://github.com/Xe/scylla
[20:32:35] <Xe> that's mostly autogenerated code and makefiles, but it's something
[20:33:31] <plt> Take a look at this http://pastebin.com/F8MH3fSs
[20:34:04] <plt> You know it takes a while to write ircd code.
[20:34:16] <Xe> I don't see any commits
[20:34:20] <Xe> not even framework code
[20:34:24] <Xe> or design
[20:34:26] <Xe> or an outline
[20:34:30] <Xe> all I see is that pastebin
[20:34:39] <Xe> which is in no way connected to that git repo
[20:35:07] <plt> I am still adding more features so its not going to be posted on the main web site yet.

The contents of the pastebin looked like a changelog, but that pastebin has since expired or was explicitly deleted. He was all talk and no game. I admit at this point I was pretty tired and frustrated, so I told him off:

[20:35:19] <Xe> fucking commit it then
[20:35:52] <plt> I was going to wait until the code was completed.
[20:36:43] <Xe> yeah good lick then
[20:36:45] <Xe> luck*
[20:37:14] <plt> Itgoing to get done and I am the only one working on the project so what do you expect?
[20:37:29] <Xe> to be able to look at the in-progress code?
[20:39:24] <plt> The code will do you no good because you will not be able to compile it.
[20:39:51] <Xe> then you have nothing
[20:40:06] <plt> I am not required to approve it.
[20:41:08] <plt> I can post the run program on the web site.
[20:42:33] <Xe> then do that
[20:43:28] <plt> Done.

The “run program” was nothing but a wrapper around the nonexistent binary for pbircd and seemed to be compiled in a language that doesn’t respect assembly functions and all of the forms of RE that I know how to do were useless. If you know how to better do RE on arbitrary binaries please let me know.

[20:44:12] <Xe> there are binaries
[20:44:15] <Xe> not source code
[20:44:25] <Xe> this is what you use git for
[20:44:35] <plt> The source code will do you no good since you can not compile it.
[20:52:02] <plt> In order for you to compile it you need the encryption program and I am not going to release the source code.
[20:54:43] <Xe> lol
[20:55:34] <plt> The program is freeware and I have no obligation to release the code under the License agreement.
[21:00:56] <Xe> you also will get no users
[21:03:13] <plt> The company that wrote Conferenceroom has a lot of customers.

ConfrenceRoom was a company that made a commercial IRC daemon. They have lost to Slack and other forms of chat like HipChat. Note here that he says “you can not compile it”. This is true in more ways than you would think. He also claims it is Freeware and not full fledged open source software. As someone who is slightly proactive and paranoid after the Snowden bullshit, I find this highly suspect. However, this “encryption program” was something I was oddly most interested in.

2015-01-24
[12:11:14] <plt> Xe why do you always demand to see the source code?

Curiosity? To learn from other people’s ideas? To challenge myself in understanding another way of thinking about things? To be able to improve it for others to learn from? Those seem like good reasons to me.

[22:46:33] <plt> PBIRCD is a irc daemon.
[22:46:36] <plt> Hello xe

The PB in that name will become apparent later.

[23:09:31] <plt> Would you like to see what I have in the updates?
[23:09:40] <Xe> sure
[23:09:47] <plt> http://pastebin.com/2udHPSyP
[23:13:10] <plt> Tell me what you think about it?
[23:16:32] <plt> I need to take a short break.

Again, the paste is dead (I should really be saving these gems) but it was another set of what appeared to be patch notes.

[23:22:37] <plt> Do you like what I have in the notes?
[23:23:49] <Xe> I still think it's ridiculous  that you don't have the balls to release your code
[23:24:36] <plt> I understand what you telling me.
[23:25:48] <plt> There is no way to working around protecting the encrypted information.
[23:34:19] <plt> Why are you do want to see the code?
[23:43:36] <plt> Xe The encryption is used to encrypt the Operators, Link and the other passwords.

This sounds suspect. Any sane system of encrypting passwords like this would be a mathematical one-way function. By not showing the code like this, is this a two-way function?

2015-01-25
[00:05:55] <plt> Xe Question if the authors that wrote free pgp do not release their source code then why should I have do


The Saga of plt, Part 2

Permalink - Posted on 2015-02-14 00:00, modified on 0001-01-01 00:00

The Saga of plt, Part 2

So I ended with a strong line of wisdom from plt last time. What if the authors that wrote free PGP did not release their source code? A nice rehash of the Clipper Chip anyone?

2015-01-25
[00:06:15] <Xe> but they did release their code
[00:06:40] <plt> I saw a few that did not release their source code.
[00:07:09] <plt> Its up to the author if they want to release it under the U.S Copyright Laws.
[00:08:50] <plt> http://copyright.gov/title17/circ92.pdf

Note that this is one of the few external links plt will give that actually works. A lot of this belief in copyright and the like seems to further some kind of delusional system involving everyone being out to steal his code and profit off of it.

Please don’t pay this person.

[00:57:18] <plt> The ircd follows the Internet Relay Protocols
[00:57:35] <Xe> which RFC's?
[00:57:43] <plt> Yep
[00:58:01] <plt> Accept for the IRCD Link works a little bit different.
[00:58:57] <plt> Version 2.0 or 3.0 will include it's own IRC Services that will work with PBIRCD.
[01:01:53] <plt> Later version will include open proxy daemon
[01:02:34] <plt> Version 1.00 will allow the ircd owner to define the irc command security levels which is a lot different from the other ircds.
[01:04:27] <plt> Xe that is the file /Conf/cmdlevs.conf.& the /Conf/userlevs.conf
[01:05:24] <plt> Adding a option for spam filtering may be included in the future version of PBIRCD.
[01:07:03] <plt> Xe PBIRCD will have not functions added to allow the operators to spy on the users.

Oh lord. Something you might notice quickly is that plt has no internal filter nor ability to keep to one topic for very long. And that also plt has some strange belief that folder names Should Start With Capital Letters, and that apparently all configuration should be:

  • split into multiple files
  • put into the root of the drive

Also note that last line. Note it in bold.

Some time passed with no activity in the channel.

[18:50:49] <plt> Hey Xe
[18:51:06] <Xe> hi
[18:58:54] <plt> How did you like the information that I showed you yesterday?
[19:02:56] <Xe> it's useless to me
[19:03:03] <Xe> I don't run on a standard linux setup
[19:03:15] <Xe> I need source code to evaluate things
[19:03:17] <Xe> :P

When I am running unknown code, I use a virtual machine running Alpine Linux. I literally do need the source code to be able to run binaries as Alpine doesn’t use glibc.

[19:04:24] <plt> It's the standard irc commands and I am still working on
adding some more features.
[19:04:38] <Xe> what language is it in?
[19:04:48] <Xe> how does it handle an accept() flood?
[19:09:17] <plt> Are you refering to accept() flood while connecting to the ircd or a channel?
[19:20:42] <plt> You can not compare some of the computer languages with C since some of they run at the same speed as C. Maybe some of them where a lot slower but in some cases that is not the same today!

These are some very simple questions I ask when evaluating a language or tool for use in a project like an IRC server. How does it handle when people are punishing it? So the obvious answer is to answer that some languages are comparable to C in terms of execution speed!

How did I not see that before?

[19:26:05] <Xe> what language is it?
[19:27:23] <plt> Purebasic [...]

I took a look at the site for PureBasic. It looks like Visual Basic’s proprietary cousin as written by someone who hates programmers. Looking at its feature set:

  • Huge set of internal commands (1400+) to quickly and easily build any application or game
  • All BASIC keywords are supported
  • Very fast compiler which creates highly optimized executables
  • No external DLLs, runtime interpreter or anything else required when creating executables
  • Procedure support for structured programming with local and global variables
  • Access to full OS API for advanced programmers
  • Advanced features such as pointers, structures, procedures, dynamically linked lists and much more

If you try to do everything, you will end up doing none of it. So it looks like PureBasic is supposed to be a compiler for people who can’t learn Go, Ruby, Python, C, or Java. This looks promising.

I’m just going to paste the code for the 99 bottles of beer example. It requires OOP. I got this from Rosetta Code.

Prototype Wall_Action(*Self, Number.i)

Structure WallClass
  Inventory.i
  AddBottle.Wall_Action
  DrinkAndSing.Wall_Action
EndStructure

Procedure.s _B(n, Short=#False)
  Select n
    Case 0 : result$="No more bottles "
    Case 1 : result$=Str(n)+" bottle of beer"
    Default: result$=Str(n)+" bottles of beer"
  EndSelect
  If Not Short: result$+" on the wall": EndIf
  ProcedureReturn result$+#CRLF$
EndProcedure

Procedure PrintBottles(*Self.WallClass, n)
  Bottles$=" bottles of beer "
  Bottle$ =" bottle of beer "
  txt$ = _B(*Self\Inventory)
  txt$ + _B(*Self\Inventory, #True)
  txt$ + "Take one down, pass it around"+#CRLF$
  *Self\AddBottle(*Self, -1)
  txt$ + _B(*self\Inventory)
  PrintN(txt$)
  ProcedureReturn *Self\Inventory
EndProcedure

Procedure AddBottle(*Self.WallClass, n)
  i=*Self\Inventory+n
  If i>=0
    *Self\Inventory=i
  EndIf
EndProcedure

Procedure InitClass()
  *class.WallClass=AllocateMemory(SizeOf(WallClass))
  If *class
    InitializeStructure(*class, WallClass)
    With *class
      \AddBottle    =@AddBottle()
      \DrinkAndSing =@PrintBottles()
    EndWith
  EndIf
  ProcedureReturn *class
EndProcedure

If OpenConsole()
  *MyWall.WallClass=InitClass()
  If *MyWall
    *MyWall\AddBottle(*MyWall, 99)
    While *MyWall\DrinkAndSing(*MyWall, #True): Wend
    ;
    PrintN(#CRLF$+#CRLF$+"Press ENTER to exit"):Input()
    CloseConsole()
  EndIf
EndIf

We are dealing with a professional language here folks. Their evaluation version of the compiler didn’t let me compile binaries and I’m not going to pay $120 for a copy of it.

[19:27:23] <plt> Purebasic it does not make one bit of difference since it runs at the same speed as c
[19:27:44] <plt> The compiler was writting in asm.
[19:28:02] <Xe> pfffft
[19:28:04] <Xe> lol
[19:28:20] <Xe> I thought you would at least have used VB6
[19:28:37] <plt> VB6 is so old dude.

At least there is some sense there.

[19:28:44] <Xe> so is purebasic
[19:28:54] <plt> You can not compare purebasic with the other basic compilers.
[19:29:51] <Xe> yes I can
[19:29:56] <Xe> seeing as you post no code
[19:29:59] <Xe> I can and I will
[19:30:16] <plt> Makes no logic what you said.
[19:30:24] <Xe> I'm saying prove it
[19:31:18] <plt> I am not going to give out the source code because of the encryption and no one has any reason to use it to decrypt the other irc networks passwords or traffic.
[19:31:40] <Xe> so you've intentionally backdoored it to allow you to have access?
[19:32:00] <plt> I dn not trust anyone any more.
[19:32:29] <plt> Not after the nsa crap going on.
[19:32:50] <Xe> so, in order to prove you don't trust anyone
[19:33:06] <Xe> you've intentionally backdoored the communications server you've created and intend to sell to people?
[19:33:37] <Xe> also
[19:33:45] <Xe> purebasic is semantically similar to vb
[19:34:06] <plt> There is no backdoors included in the source code. A course if a user gets a virus or hacked that is not going to be my fault.


Getting Started with Go

Permalink - Posted on 2015-01-28 00:00, modified on 0001-01-01 00:00

Getting Started with Go

Go is an exciting language made by Google for systems programming. This article will help you get up and running with the Go compiler tools.

System Setup

First you need to install the compilers.

$ sudo apt-get install golang golang-go.tools

golang-go.tools contains some useful tools that aren’t part of the standard Go distribution.

Shell Setup

Create a folder in your home directory for your Go code to live in. I use ~/go.

$ mkdir -p ~/go/{bin,pkg,src}

bin contains go binaries that are created from go get or go install. pkg contains static (.a) compiled versions of go packages that are not go programs. src contains go source code.

After you create this, add this and the following to your zsh config:

export GOPATH=$HOME/go
export PATH=$PATH:/usr/lib/go/bin:$GOPATH/bin

This will add the go compilers to your $PATH as well as programs you install.

Rehash your shell config (I use a resource command for this) and then run:

$ go env
GOARCH="amd64"
GOBIN=""
GOCHAR="6"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/xena/go"
GORACE=""
GOROOT="/usr/lib/go"
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
TERM="dumb"
CC="gcc"
GOGCCFLAGS="-g -O2 -fPIC -m64 -pthread"
CXX="g++"
CGO_ENABLED="1"

This will verify that the go toolchain knows where the go compilers are as well as where your $GOPATH is.

Testing

To test the go compilers with a simple todo command, run this:

$ go get github.com/mattn/todo
$ todo add foo
$ todo list
☐ 001: foo

Vim Setup

For Vim integration, I suggest using the vim-go plugin. This plugin used to be part of the standard Go distribution.

To install:

  1. Add Plugin 'fatih/vim-go' to the plugins part of your vimrc.
  2. Run these commands:
$ vim +PluginInstall +qall
$ vim +GoInstallBinaries +qall

This will install the go oracle and the go autocompletion daemon gocode as well as some other useful tools that will integrate seamlessly into vim. This will also run gofmt on save to style your code to the standard way to write Go code.

Resources

Effective Go and the language spec provide a nice overview of the syntax.

The Go blog contains a lot of detailed articles covering advanced and simple Go topics. This page has a list of past articles that you may find useful.

The Go standard library is a fantastic collection of Go code for solving many problems. In some cases you can even write entire programs using only the standard library. This includes things like web application support, tarfile support, sql drivers, support for most kinds of commonly used crypto, command line flag parsing, html templating, and regular expressions. A full list of the standard library packages can be found here.

Variable type declarations will look backwards. It takes a bit to get used to but makes a lot of sense once you realize it reads better left to right.

For a nice primer on building web apps with Go, codegangsta is writing a book on the common first steps, starting from the standard library and working up. You can find his work in progress book here.

Go has support for unit testing baked into the core language tools. You can find information about writing unit tests here.

When creating a new go project, please resist the urge to make the folder in your normal code folder. Drink the $GOPATH koolaid. Yes it’s annoying, yes it’s the language forcing you to use its standard. Just try it. It’s an amazingly useful thing once you get used to it.

Learn to love godoc. Godoc lets you document code like this. This also includes an example of the builtin unit testing support.


Pursuit of a DSL

Permalink - Posted on 2014-08-16 00:00, modified on 0001-01-01 00:00

Pursuit of a DSL

A project we have been working on is Tetra. It is an extended services package in Go with Lua and Moonscript extensions. While writing Tetra, I have found out how to create a Domain Specific Language, and I would like to recommend Moonscript as a toolkit for creating DSL’s.

Moonscript is a high level wrapper around Lua designed to make programming easier. We have used Moonscript heavily in Tetra because of how easy it is to make very idiomatic code in it.

Here is some example code from the Tetra codebase for making a command:

require "lib/elfs"

Command "NAMEGEN", ->
  "> #{elfs.GenName!\upper!}"

That’s it. That creates a command named NAMEGEN that uses lib/elfs to generate goofy heroku-like application names based on names from Pokemon Vietnamese Crystal.

In fact, because this is so simple and elegant, you can document code like this inline.

Command Tutorial

In this file we describe an example command TEST. TEST will return some information about the place the command is used as well as explain the arguments involved.

Because Tetra is a polyglot of Lua, Moonscript and Go, the relevant Go objects will have their type definitions linked to on godoc

Declaring commands is done with the Command macro. It takes in two arguments.

  1. The command verb
  2. The command function

It also can take in 3 arguments if the command needs to be restricted to IRCops only.

  1. The command verb
  2. true
  3. The command function

The command function can have up to 3 arguments set when it is called. These are:

  1. The Client that originated the command call.
  2. The Destination or where the command was sent to. This will be a Client if the target is an internal client or a Channel if the target is a channel.
  3. The command arguments as a string array.
Command "TEST", (source, destination, args) ->

All scripts have client pointing to the pseudoclient that the script is spawned in. If the script name is chatbot/8ball, the value of client will point to the chatbot pseudoclient.

  client.Notice source, "Hello there!"

This will send a NOTICE to the source of the command saying “Hello there!”.

  client.Notice source, "You are #{source.Nick} sending this to #{destination.Target!} with #{#args} arguments"

All command must return a string with a message to the user. This is a good place to do things like summarize the output of the command or if it worked or not. If the command is oper-only, this will be the message logged to the services snoop channel.

  "End of TEST output"

See? That easy.

Command "TEST", ->
    "Hello!"

This is much better than Cod’s

#All modules have a name and description
NAME="Test module"
DESC="Small example to help you get started"

def initModule(cod):
    cod.addBotCommand("TEST", testbotCommand)

def destroyModule(cod):
    cod.delBotCommand("TEST")

def testbotCommand(cod, line, splitline, source, destination):
    "A simple test command"
    return "Hello!"


Thoughts on Community Management

Permalink - Posted on 2014-07-31 00:00, modified on 0001-01-01 00:00

Thoughts on Community Management

Many open source community projects lack proper management. They can put too much of their resources in too few places. When that one person falls out of contact or goes rogue on everyone, it can have huge effects on everyone involved in the project. Users, Contributors and Admins.

Here, I propose an alternative management structure based on what works.

Organization

Contributors and Project Administrators are there to take input/feedback from Users, rectify the situation or explain why doing so is counterproductive. Doing so will be done kindly and will be ran through at least another person before it is posted publicly. This includes (but is not limited to) email, IRC, forums, anything. A person involved in the project is a representative of it. They are the face of it. If they are rude it taints the image of everyone involved.

Access

Project Administrators will have full, unfiltered access to anything the project has. This includes root access, billing access, everything. There will be no reason to hide things. Operational conversations will be shared. All group decisions will be voted on with a simple Yes/No/Abstain process. As such this team should be kept small.

Contributions

Contributors will have to make pull requests, as will Administrators. There will be review on all changes made. No commits will be pushed to master by themselves unless there is approval. This will allow for the proper review and testing procedures to be done to all code contributed.

Additionally, for ease of scripts scraping the commits when something is released, a commit style should be enforced.

Commit Style

The following section is borrowed from Deis’ commit guidelines.


We follow a rough convention for commit messages borrowed from CoreOS, who borrowed theirs from AngularJS. This is an example of a commit:

feat(scripts/test-cluster): add a cluster test command

this uses tmux to setup a test cluster that you can easily kill and
start for debugging.

To make it more formal, it looks something like this:

{type}({scope}): {subject}
<BLANK LINE>
{body}
<BLANK LINE>
{footer}

The {scope} can be anything specifying place of the commit change.

The {subject} needs to use imperative, present tense: “change”, not “changed” nor “changes”. The first letter should not be capitalized, and there is no dot (.) at the end.

Just like the {subject}, the message {body} needs to be in the present tense, and includes the motivation for the change, as well as a contrast with the previous behavior. The first letter in a paragraph must be capitalized.

All breaking changes need to be mentioned in the {footer} with the description of the change, the justification behind the change and any migration notes required.

Any line of the commit message cannot be longer than 72 characters, with the subject line limited to 50 characters. This allows the message to be easier to read on github as well as in various git tools.

The allowed {types} are as follows:

feat -> feature
fix -> bug fix
docs -> documentation
style -> formatting
ref -> refactoring code
test -> adding missing tests
chore -> maintenance

I believe that these guidelines would lead towards a harmonious community.