What is a JSON feed? Learn more

JSON Feed Viewer

Browse through the showcased feeds, or enter a feed URL below.

Now supporting RSS and Atom feeds thanks to Andrew Chilton's feed2json.org service

CURRENT FEED

Henri Bergius

Hacker and an occasional adventurer. Author of Create.js and NoFlo, founder of Flowhub UG. Decoupling software, one piece at a time. This blog tells the story of that.

JSON


My blog, the 2017 edition

Permalink - Posted on 2017-12-14 00:00

I guess every five years is a good cadence for blog redesigns. This year’s edition started as a rewrite of the technical implementation, but I ended up also updating the visuals. Here I’ll go through the design goals, and how I met them.

More robust and secure delivery

This year the web has been strongly turning towards encryption. While my site doesn’t contain any interactive elements, using HTTPS still makes it harder for malicious parties to track and modify the contents people read.

For the past five years, my blog has been hosted on GitHub Pages. While otherwise that has been a pretty robust solution, they sadly don’t support SSL for custom domains. A common workaround would be to utilize Cloudflare as a HTTPS proxy, but that only works if you let them manage your domain. Since bergie.iki.fi is a subdomain, that was off the cards.

Instead, what I did was turn towards Amazon Web Services. I used Amazon Certificate Manager with my iki subdomain to get an SSL certificate, and utilized Travis CI to build the Jekyll site and upload to S3.

From there the site updates are served using Amazon CloudFront CDN, routed using Route53.

With this, I only need to push new changes to this site’s GitHub repository, and robots will take care of the rest, from producing the HTML pages to distributing them via a global content delivery network.

And, I get the friendly green lock icon.

SSL certificate for bergie.iki.fi

Easier image rescaling

I moved the site from Midgard CMS to the Jekyll static site generator in 2012. At that point, images were stored in the same GitHub repository alongside the textual contents.

However, the sheer volume of pictures accumulated on this site over the years made the repository quite unwieldy, and so I moved them to Amazon S3 couple of years ago.

This made working with different sizes of images a bit more unwieldy, as I’d have to produce the different variants locally and upload them separately.

Now, with the new redesign I built an Amazon Lambda function to resize images on-demand. My solution is implemented in NoFlo, roughly following the ideas from this tutorial but utilizing the excellent noflo-sharp library.

This is a topic I should write about in more detail, but it turns out NoFlo works really well with Amazon Lambda. You can use any Node.js NoFlo graph there by simply wrapping it using the asCallback embedding API.

The end result is that I only need to upload original size images to S3 using some tool (NoFlo, s3cmd, AWS console, or the nice DropShare app), and I can get different sizes by tweaking the URL.

I could have gone with ImgFlo, but right now I need only rescaling, and running the whole GIMP engine felt like an overkill.

New visuals

After the technical side of the blog revamp was done, I turned towards the design aspects. I wanted more color, and also to benefit from the features of the modern web. This meant that performance-hindering things like Bootstrap, jQuery, and Google Fonts were out, since nowadays you can do pretty nice sites with pure CSS alone.

In addition to the better CDN setup, the redesign improved the site’s PageSpeed score. And I think it looks pretty good.

Here’s the front page:

2017 edition of bergie.iki.fi

For reference, here is how the 2012 edition looked like:

2012 edition of bergie.iki.fi

I also spent a bit of time to make sure the site looks nice on both smartphones and tablets, since those are the devices most people use to browse the web these days.

Here is how the site looks like on different devices, courtesy of Am I Responsive

2017 front page

2017 article page

Better content discoverability

This site has over 1000 articles, and it is easy to lost in those volumes. To make it easier to discover content, I implemented a related posts feature.

I originally wanted to use Jekyll’s Latent Semantic Indexing feature, but with this amount of content that simply blows up.

Instead, I ended up building my own hacky implementation based on categorization and similar keywords in posts using Liquid templates. This makes full site builds a bit slow, but the results seem quite good:

Related posts to the NoFlo 1.0 announcement

Staying up to date

While most people probably discover content now via Twitter or Facebook (both of which I occasionally share my things in, in addition to places like Reddit or Hacker News as needed), RSS is still the underpinning of receiving blog updates.

For this, the site is available as both:

Feel free to add one of them to the news aggregator of your choice!

I also supply /now page for current activities, inspired by the NowNowNow movement. Here is how Derek Sivers described the idea:

People often ask me what I’m doing now.

Each time I would type out a reply, describing where I’m at, what I’m focused on, and what I’m not.

So earlier this year I added a /now page to my site: https://sivers.org/now

A simple link. Easy to remember. Easy to type.

It’s a nice reminder for myself, when I’m feeling unfocused. A public declaration of priorities.

Previous redesigns

I’ve been running this site since 1997. Here is what I’ve written about some of the previous redesigns:

I hope you enjoy the new design! Let me know what you think.


Get ready for NoFlo 1.0

Permalink - Posted on 2017-11-02 00:00

After six years of work, and bunch of different projects done with NoFlo, we’re finally ready for the big 1.0. The two primary pull requests for the 1.0.0 cycle landed today, and so it is time to talk about how to prepare for it.

tl;dr If your project runs with NoFlo 0.8 without deprecation warnings, you should be ready for NoFlo 1.0

ES6 first

The primary difference between NoFlo 0.8 and 1.0 is that now we’re shipping it as ES6 code utilizing features like classes and arrow functions.

Now that all modern browsers support ES6 out of the box, and Node.js 8 is the long-term supported release, it should be generally safe to use ES6 as-is.

If you need to support older browsers, Node.js versions, or maybe PhantomJS, it is of course possible to compile the NoFlo codebase into ES5 using Babel.

We recommend new components to be written in ES6 instead of CoffeeScript.

Easier webpack builds

It has been possible to build NoFlo projects for browsers since 2013. Last year we switched to webpack as the module bundler.

However, at that stage there was still quite a lot of configuration magic happening inside grunt-noflo-browser. This turned out to be sub-optimal since it made integrating NoFlo into existing project build setups difficult.

Last week we extracted the difficult parts out of the Grunt plugin, and released the noflo-component-loader webpack loader. With this, you can generate a configured NoFlo component loader in any webpack build. See this example.

In addition to generating the component loader, your NoFlo browser project may also need two other loaders, depending how your NoFlo graphs are built: json-loader for JSON graphs, and fbp-loader for graphs defined in the .fbp DSL.

Removed APIs

There were several old NoFlo APIs that we marked as deprecated in NoFlo 0.8. In that series, usage of those APIs logged warnings. Now in 1.0 the deprecated APIs are completely removed, giving us a lighter, smaller codebase to maintain.

Here is a list of the primary API removals and the suggested migration strategy:

  • noflo.AsyncComponent class: use WirePattern or Process API instead
  • noflo.ArrayPort class: use InPort/OutPort with addressable: true instead
  • noflo.Port class: use InPort/OutPort instead
  • noflo.helpers.MapComponent function: use WirePattern or Process API instead
  • noflo.helpers.WirePattern legacy mode: now WirePattern always uses Process API internally
  • noflo.helpers.WirePattern synchronous mode: use async: true and callback
  • noflo.helpers.MultiError function: send errors via callback or error port
  • noflo.InPort process callback: use Process API
  • noflo.InPort handle callback: use Process API
  • noflo.InPort receive method: use Process API getX methods
  • noflo.InPort contains method: use Process API hasX methods
  • Subgraph EXPORTS mechanism: disambiguate with INPORT/OUTPORT

The easiest way to verify whether your project is compatible is to run it with NoFlo 0.8.

You can also make usage of deprecated APIs throw errors instead of just logging them by setting the NOFLO_FATAL_DEPRECATED environment variable. In browser applications you can set the same flag to window.

Scopes

Scopes are a flow isolation mechanism that was introduced in NoFlo 0.8. With scopes, you can run multiple simultaneous flows through a NoFlo network without a risk of data leaking from one scope to another.

The primary use case for scope isolation is building things like web API servers, where you want to isolate the processing of each HTTP request from each other safely, while reusing a single NoFlo graph.

Scope isolation is handled automatically for you when using Process API or WirePattern. If you want to manipulate scopes, the noflo-packets library provides components for this.

NoFlo in/outports can also be set as scoped: false to support getting out of scopes.

asCallback and async/await

noflo.asCallback provides an easy way to expose NoFlo graphs to normal JavaScript consumers. The produced function uses the standard Node.js callback mechanism, meaning that you can easily make it return promises with Node.js util.promisify or Bluebird. After this your NoFlo graph can be run via normal async/await.

Component libraries

There are hundreds of ready-made NoFlo components available on NPM. By now, most of these have been adapted to work with NoFlo 0.8.

Once 1.0 ships, we’ll try to be as quick as possible to update all of them to run with it. In the meanwhile, it is possible to use npm shrinkwrap to force them to depend on NoFlo 1.0.

If you’re relying on a library that uses deprecated APIs, or hasn’t otherwise been updated yet, please file an issue in the GitHub repo of that library.

This pull request for noflo-gravatar is a great example of how to implement all the modernization recommendations below in an existing component library.

Recommendations for new projects

This post has mostly covered how to adapt existing NoFlo projects for 1.0. How about new projects? Here are some recommendations:

  • While NoFlo projects have traditionally been written in CoffeeScript, for new projects we recommend using ES6. In particular, follow the AirBnB ES6 guidelines
  • Use fbp-spec for test automation
  • Use NPM scripts instead of Grunt for building and testing
  • Make browser builds with webpack utilizing noflo-component-loader
  • Use Process API when writing components
  • If you expose any library functionality, provide an index file using noflo.asCallback for non-NoFlo consumers

The BIG IoT Node.js bridge is a recent project that follows these guidelines if you want to see an example in action.

There is also a project tutorial available on the NoFlo website.


Building an IoT dashboard with NASA Open MCT

Permalink - Posted on 2017-10-05 00:00

One important aspect of any Internet of Things setup is being able to collect and visualize data for analysis. Seeing trends in sensor readings over time can be useful for identifying problems, and for coming up with new ways to use the data.

We wanted an easy solution for this for the c-base IoT setup. Since the c-base backstory is that of a crashed space station, using space technology for this made sense.

OpenMCT view on c-base

NASA Open MCT is a framework for building web-based mission control tools and dashboards that they’ve released as open source. It is intended for bringing together tools and both historical and real-time data, as can be seen in their Mars Science Laboratory dashboard demo.

c-beam telemetry server

As a dashboard framework, Open MCT doesn’t really come with batteries included. You get a bunch of widgets and library functionality, but out of the box there is no integration with data sources.

However, they do provide a tutorial project for integrating data sources. We started with that, and built the cbeam-telemetry-server project which gives a very easy way to integrate Open MCT with an existing IoT setup.

With the c-beam telemetry server we combine Open MCT with the InfluxDB timeseries database and the MQTT messaging bus. This gives a “turnkey” setup for persisting and visualizing IoT information.

Getting started

The first step is to install the c-beam telemetry server. If you want to do a manual setup, first install a MQTT broker, InfluxDB and Node.js. Optionally you can also install CouchDB for sharing custom dashboard layouts between users.

Then just clone the c-beam telemetry server repo:

$ git clone https://github.com/c-base/cbeam-telemetry-server.git

Install the dependencies and build Open MCT with:

$ npm install

Now you should be able to start the service with:

$ npm start

Running with Docker

There is also an easier way to get going: we provide pre-built Docker images of the c-beam telemetry server for both x86 and ARM.

There are also docker-compose configuration files for both environments. To install and start the whole service with all its dependencies, grab the docker-compose.yml file (or the Raspberry Pi 3 version) and start with:

$ docker-compose up -d

We’re building these images as part of our continuous integration pipeline (ARM build with this recipe), so they should always be reasonably up-to-date.

Configuring your data

The next step is to create a JavaScript configuration file for your Open MCT. This is where you need to provide a “dictionary” listing all data you want your dashboard to track.

Data sets are configured like the following (configuring a temperature reading tracked for the 2nd floor):

var floor2 = new app.Dictionary('2nd floor', 'floor2');
floor2.addMeasurement('temperature', 'floor2_temperature', [
  {
    units: 'degrees',
    format: 'float'
  }
], {
  topic: 'bitraf/temperature/1'
});

You can have multiple dictionaries in the same Open MCT installation, allowing you to group related data sets. Each measurement needs to have a name and a unit.

Getting data in

In the example above we also supply a MQTT topic to read the measurement from. Now sending data to the dashboard is as easy as writing numbers to that MQTT topic. On command-line that would be done with:

$ mosquitto_pub -t bitraf/temperature/1 -m 27.3

If you were running the telemetry server when you sent that message, you should’ve seen it appear in the appropriate dashboard.

Bitraf temperature graph with Open MCT

There are MQTT libraries available for most programming languages, making it easy to connect existing systems with this dashboard.

The telemetry server is also compatible with our MsgFlo framework, meaning that you can also configure the connections between your data sources and Open MCT visually in Flowhub.

This makes it possible to utilize the existing MsgFlo libraries for implementing data sources. For example, with msgflo-arduino you can transmit sensor data from Tiva-C or NodeMcu microcontrollers to the dashboard.

Status and how you can help

The c-beam telemetry server is currently in production use in a couple of hackerspaces, and seems to run quite happily.

We’d love to get feedback from other deployments!

If you’d like to help with the project, here are couple of areas that would be great:

  • Adding tests to the project
  • Implementing downsampling of historical data
  • Figuring out ways to control IoT devices via the dashboard (so, to write to MQTT instead of just reading)

Please file issues or make pull requests to the repository.


Flowhub IoT hack weekend at c-base: buttons, sensors, the Big Switch

Permalink - Posted on 2017-07-11 00:00

Last weekend we held the c-base IoT hack weekend, focused on the Flowhub IoT platform. This was continuation from the workshop we organized at the Bitraf makerspace a week earlier. Same tools and technologies, but slightly different focus areas.

c-base is one of the world’s oldest hackerspaces and a crashed space station under Berlin. It is also one of the earliest users of MsgFlo with quite a lot of devices connected via MQTT.

Hack weekend debriefing

Hack weekend

Just like at Bitraf, the workshop aimed to add new IoT capabilities to c-base, as well as to increase the number of members who know how to make the station’s setup do new things. For this, we used three primary tools:

Internet of Things

The workshop started in Friday evening after a lecture on nuclear pulse propulsion ended in the main hall. We continued all the way to late Sunday evening with some sleep breaks in between. There is something about c-base that makes you want to work there at night.

Testing a humidity sensor

By Sunday evening, we had built and deployed 15 connected IoT devices, with five additional ones pretty far in development. You can find the source code in the c-flo repository.

Idea wall

Sensor boxes

Quite a lot of c-base was already instrumented when we started the workshop. We had details on electricity consumption, internet traffic, and more. But one thing we didn’t have was information on the physical environment at the station. To solve this, we decided to build a set of sensor boxes that we could deploy in different areas of the hackerspace.

Building sensors

The capabilities shared by all the sensor boxes we deployed were:

  • Temperature
  • Humidity
  • Motion (via passive infrared)

For some areas of interest we provided some additional sensors:

  • Sound level (for the workshop)
  • Light level (for c-lab)
  • Carbon dioxide
  • Door open/closed
  • Gravity

Workshop sensor on a breadboard

We found a set of nice little electrical boxes that provided a convenient housing for these sensor boxes. This way we were able to mount them in proper places quickly. This should also protect them from dust and other elements to some degree.

Installed weltenbaulab sensor

The Big Switch

The lights of the c-base main hall are controllable via MsgFlo, and we have a system called farbgeber to produce pleasing color schemes for any given time.

However, when there are events we need to enable manual control of all lights and sound. To make this “MsgFlo vs. IP lounge” control question clearer, we built a Big Switch to decide which controls the lights:

Big Switch in action

The switch is an old electric mains switch from an office building. It makes a satisfying sound when you turn it, and is big enough that you can see which way the setting is from across the room.

To complement the Big Switch we also added a “c-boom” button to trigger the disco mode in the main hall:

c-boom button

Info screens

One part of the IoT setup was to make statistics and announcements about c-base visible in different areas of the station. We did this by rolling out a set of displays with Raspberry Pi 3s connected to the MsgFlo MQTT environment.

Info screens ready for installing

The announcements shown on the screens range from mission critical information like station power consumption or whether the bar is open, to more fictional ones like the NoFlo-powered space station announcements.

Air lock

We also built an Android version of the info display software, which enabled deploying screens using some old donated tablets.

Info screen tablet

Conclusions

This was another successful workshop. Participants got to do new things, and we got lots of new IoT infrastructure installed around c-base. The Flowhub graph is definitely starting to look populated:

c-base is a graph

We also deployed NASA OpenMCT so that we get a nice overview on the station status. Our telemetry server provides MsgFlo participants that receive data via MQTT, store it in InfluxDB, and then visualize it on the dashboard:

OpenMCT view on c-base

All the c-base IoT software is available on Github:


If you’d like to have a similar IoT workshop at your company, we’re happy to organize one. Get in touch!


Flowhub IoT workshop at Bitraf: sensors, access control, and more

Permalink - Posted on 2017-07-04 00:00

I just got back to Berlin from the Bitraf IoT hackathon we organized in Oslo, Norway. This hackathon was the first of two IoT workshops around MsgFlo and Flowhub IoT. The second will be held at c-base in Berlin this coming weekend.

Bitraf and the existing IoT setup

Bitraf is a large non-profit makerspace in the center of Oslo. It provides co-working facilities, as well as labs and a large selection of computer controlled tools for building things. Members have 24/7 access to the space, and are provided with everything needed for CNC milling, laser cutting, 3D-printing and more.

The space uses the Flowhub IoT stack of MsgFlo and Mosquitto for business-critical things like the door locks that members can open with their smartphone.

Bitraf lock system

In addition to access control, they also had various environmental sensors available on the MQTT network.

With the workshop, our aim was to utilize these existing things more, as well as to add new IoT capabilities. And of course to increase the number of Bitraf members with the knowledge to work with the MsgFlo IoT setup.

Preparations

Being a makerspace, Bitraf already had everything needed for the physical side of the workshop — tons of sensors, WiFi-enabled microcontrollers, tools for building cases and mounting solutions. So the workshop preparations mostly focused on the software side of things.

The primary tools for the workshop were:

To help visualize the data coming from the sensors people were building, I integrated the NASA OpenMCT dashboard with MsgFlo and InfluxDB time series database. This setup is available at the cbeam-telemetry-server project.

OpenMCT at Bitraf

This gave us a way to send data from any interesting sensors in the IoT network to a dashboard and visualize it. Down the line the persisted data can also be interesting for further analysis or machine learning.

Kick-off session

We started the workshop with a quick intro session about Flowhub, MsgFlo, and MQTT development. There is unfortunately no video, but the slides are available:

After the intro, we did a round of all attendees to see what skills people already had, and what they were interested in learning. Then we started collecting ideas what to work on.

Bitraf IoT ideas

People picked their ideas, and the project work started.

Idea session at Bitraf IoT

I’d like to highlight couple of the projects.

New sensors for the makerspace

Teams at work

Building new sensors was a major part of the workshop. There were several projects, all built on top of msgflo-arduino and the ESP8266 microcontroller:

Working on a motion sensor

There was also a project to automatically open and close windows, but this one didn’t get completed over the weekend. You can follow the progress in the altF4 GitHub repo.

Tool locking

All hackerspaces have the problem that people borrow tools and then don’t return them when finished. This means that the next person needing the tool will have to spend time searching for it.

To solve this, the team designed a system that enabled tools to be locked to a wall, with a web interface where members can “check out” a tool they want to use. This way the system constantly knows what tools are in their right places, and which tools are in use, and by who.

You can see the tool lock system in action in this demo video:

Source code and schematics: https://github.com/einsmein/bitraf-thelock.

After the hackathon

Before my flight out, we sat down with Jon to review how things went. In general, I think it is clear the event was a success — people got to learn and try new things, and all projects except one were completed during the two days.

Our unofficial goal was to double the number of nodes in the Bitraf Flowhub graph, and I think we succeeded in this:

Bitraf as a graph

Here are couple of comments from the attendees:

Really fun and informative. The development pipeline also seems complete. Made it a lot easier for beginner to get started.

this was a very fantastic hackathon! Lots of interesting things to learn, very enthusiastic participants, great stewardship and we actually got quite a few projects finished. Well done everbody.

In general the development tools we provided worked well. Everybody was able to run the full Flowhub IoT environment on their own machines using the Docker setup we provided. And apart from a couple of corner cases, msgflo-arduino was easy to get going on the NodeMCUs.

With these two, everybody could easily wire up some sensors and see their data in both Flowhub and the OpenMCT dashboard. From the local setup going to production was just a matter of switching the MQTT broker configuration.


If you’d like to have a similar IoT workshop at your company, we’re happy to organize one. Get in touch!


Two hackathons in a week: thoughts on NoFlo and MsgFlo

Permalink - Posted on 2017-06-19 00:00

Last week I participated in two hackathons, events where a group of strangers would form a team for two or three days and build a product prototype. In the end all teams pitch their prototypes, and the best ones would be given some prizes.

Hackathons are typically organized to get feedback from developers on some new API or platform. Sometimes they’re also organized as a recruitment opportunity.

Apart from the free beer and camaraderie, I like going to hackathons since they’re a great way to battle test the developer tools I build. The time from idea to having to have a running prototype is short, people are used to different ways of working and different toolkits.

If our tools and flow-based programming work as intended, they should be ideal for these kind of situations.

Minds + Machines hackathon and Electrocute

Minds + Machines hackathon was held on a boat and focused on decarbonizing power and manufacturing industries. The main platform to work with was Predix, GE’s PaaS service.

Team Electrocute

Our project was Electrocute, a machine learning system for forecasting power consumption in a changing climate.

1.5°C is the global warming target set by the Paris Agreement. How will this affect energy consumption? What kind of generator assets should utilities deploy to meet these targets? When and how much renevable energy can be utilized?

The changing climate poses many questions to utilities. With Electrocute’s forecasting suite power companies can have accurate answers, on-demand.

Electrocute forecasts

The system was built with a NoFlo web API server talking over MsgFlo with a Python machine learning backend. We also built a frontend where users could see the energy usage forecasts on a heatmap.

NoFlo-Xpress in action

Unfortunately we didn’t win this one.

Recoding Aviation and Skillport

Recoding Aviation was held at hub:raum and focused on improving the air travel experience through usage of open APIs offered by the various participating airports.

Team Skillport

Skillport was our project to make long layovers more bearable by connecting people who’re stuck at the airport at the same time.

Long layovers suck. But there is ONE thing amazing about them: You are surrounded by highly skilled people with interesting stories from all over the world. It sometimes happens that you meet someone randomly - we all have a story like that. But usually we are too shy and lazy to communicate and see how we could create a valuable interaction. You never know if the other person feels the same.

We built a mobile app that turns airports into a networking, cultural exchange and knowledge sharing hub. Users tell each other through the app that they are available to meet and what value they can bring to an interaction.

The app connected with a J2EE API service that then communicated over MsgFlo with NoFlo microservices doing all the interactions with social and airport APIs. We also did some data enrichment in NoFlo to make smart recommendations on meeting venues.

MsgFlo in action

This time our project went well with the judges and we were selected as the winner of the Life in between airports challenge. I’m looking forward to the helicopter ride over Berlin!

Category winners

Skillport also won a space at hub:raum, so this might not be the last you’ll hear of the project…

Lessons learned

Benefits of a message queue architecture

I’ve written before on why to use message queues for microservices, but that post focused more on the benefits for real-life production usage.

The problems and tasks for a system architecture in a hackathon are different. Since the time is short, you want to enable people to work in parallel as much as possible without stepping on each other’s toes. Since people in the team come from different backgrounds, you want to enable a heterogeneous, polyglot architecture where each developer can use the tools they’re most productive with.

MsgFlo is by its nature very suitable for this. Components can be written in any language that supports the message queue used, and we have convenience libraries for many of them. The discovery mechanism makes new microservices appear on the Flowhub graph as soon as they start, enabling services to be wired together quickly.

Mock early, mock often

Mocks are a useful way to provide a microservice to the other team members even before the real implementation is ready.

For example in the GE Predix hackathon, we knew the machine learning team would need quite a bit of time to build their model. Until that point we ran their microservice with a simple msgflo-python component that just gave random() as the forecast.

This way everybody else was able to work with the real interface from the get-go. When the learning model was ready we just replaced that Python service, and everything was live.

Mocks can be useful also in situations where you have a misbehaving third-party API.

Don’t forget tests

While shooting for a full test coverage is probably not realistic within the time constraints of a hackathon, it still makes sense to have at least some “happy path” tests. When you’re working with multiple developers each building a different parts of the service, interface tests serve a dual purpose:

  • They show the other team members how to use your service
  • They verify that your service actually does what it is supposed to

And if you’re using a continuous integration tool like Travis, the tests will help you catch any breakages quickly, and also ensure the services work on a clean installation.

For a message queue architecture, fbp-spec is a great tool for writing and running these interface tests.

Talk with the API providers

The reason API and platform providers organize these events is to get feedback. As a developer that works with tons of different APIs, this is a great opportunity to make sure your ideas for improvement are heard.

On the flip side, this usually also means the APIs are in a pretty early stage, and you may be the first one using them in a real-world project. When the inevitable bugs arise, it is a good to have a channel of communications open with the API provider on site so you can get them resolved or worked around quickly.

Room for improvement

The downside of the NoFlo and MsgFlo stack is that there is still quite a bit of a learning curve. NoFlo documentation is now in a reasonable place, but with Flowhub and MsgFlo we have tons of work ahead on improving the onboarding experience.

Right now it is easy to work with if somebody sets it up properly first, but getting there is a bit tricky. Fixing this will be crucial for enabling others to benefit from these tools as well.


Reprogramming the hackerspace: MsgFlo IoT workshops at c-base and Bitraf

Permalink - Posted on 2017-06-07 00:00

This July we’re organizing two hack weekends around MsgFlo and Flowhub:

Both of these will focus on reprogramming the Internet of Things setup of the hackerspace. The aim is to connect more devices at the spaces to the MsgFlo environment, and come up with new connections between systems, and ways to visualize them.

If you’re interested, feel welcome to join! Bring your own laptop. For the Bitraf event, please register on the Meetup page. For c-base, add yourself to the Facebook event.

c-base disco mode

Focus areas

  • MsgFlo IoT and new functionality — connecting internet services with MsgFlo, adding new smarts to the hackerspace IoT setup. Skills: Python, Node.js, Rust
  • Hardware hacking — connecting more devices with MsgFlo. Skills: microcontroller programming, electronics
  • Information displays — new infoscreen designs, data visualizations. Skills: web design, React.js, Django
  • Mobile app — bringing the hackerspace IoT functionality to mobile. Skills: Android
  • Woodworking — new cases, mounts, decorations for various systems. Skills: woodworking, painting

You don’t have to an expert to participate. We’ll be there to help you get up to speed!

More information

c-flo station at c-base


If you’d like to have a similar IoT workshop at your company, we’re happy to organize one. Get in touch!


NoFlo: six years of JavaScript dataflow

Permalink - Posted on 2017-06-05 00:00

Quite a bit of time has passed since my two years of NoFlo post, and it is time to take another look at the state of the NoFlo ecosystem. To start with the basics, NoFlo is a JavaScript implementation of Flow-Based Programming:

In computer programming, flow-based programming (FBP) is a programming paradigm that defines applications as networks of “black box” processes, which exchange data across predefined connections by message passing, where the connections are specified externally to the processes. These black box processes can be reconnected endlessly to form different applications without having to be changed internally. FBP is thus naturally component-oriented.

With NoFlo software is built by creating graphs that contain reusable components and define the program logic by determining how these components talk to each other.

I started the NoFlo open source project six years ago in Mountain View, California. My aim was to improve the JavaScript programming experience by bringing the FBP paradigm to the ecosystem. At the time the focus was largely on web API servers and extract, transform, load (ETL) programs, but the scope has since expanded quite a bit:

NoFlo is not a web framework or a UI toolkit. It is a way to coordinate and reorganize data flow in any JavaScript application. As such, it can be used for whatever purpose JavaScript can be used for. We know of NoFlo being used for anything from building web servers and build tools, to coordinating events inside GUI applications, driving robots, or building Internet-connected art installations.

Flowhub

Four years ago I wrote how UI was the missing part of NoFlo. Later the same year we launched a Kickstarter campaign to fix this.

Our promise was to design a new way to build software & manage complexity - a visual development environment for all.

This was wildly successful, being at the time the 5th highest funded software crowdfunding campaign. The result — Flowhub — was released to the public in 2014. Big thanks to all of our backers!

Here is how Fast Company wrote about NoFlo:

If NoFlo succeeds, it could herald a new paradigm of web programming. Imagine a world where anyone can understand and build web applications, and developers can focus on programming efficient components to be put to work by this new class of application architects. In a way, this is the same promise as the “learn to code” movement, which wants to teach everyone to be a programmer. Just without the programming.

With Flowhub you can manage full NoFlo projects in your browser. This includes writing components in JavaScript or CoffeeScript, editing graphs and subgraphs, running and introspecting the software, and creating unit tests. You can keep your project in sync with the GitHub integration.

Live programming NoFlo in Flowhub

Celebrating six years of NoFlo

Earlier this year we incorporated Flowhub in Germany. Now, to celebrate six years of NoFlo we’re offering a perpetual 30% discount on Flowhub plans. To lock in the discount, subscribe to a Flowhub plan before June 12th 2017 using the code noflo6.

Ecosystem

While NoFlo itself has by no means taken over the world yet, the overall ecosystem it is part of is looking very healthy. Sure, JavaScript fatigue is real, but at the same time it has gone through a pretty dramatic expansion.

JavaScript

As I wrote around the time I started NoFlo, JavaScript has indeed become a universal runtime. It is used on web browsers, server-side, as well as for building mobile and desktop applications. And with NoFlo you can target all those platforms with a single programming model and toolchain.

The de-facto standard for sharing JavaScript libraries — NPM has become the most popular software repository for open source modules. Apart from the hundreds of thousands of other packages, you can also get prebuild NoFlo components from NPM to cover almost any use case.

Dataflow

After a long period of semi-obscurity, our Kickstarter campaign greatly increased the awareness in FBP and dataflow programming. Several open source projects expanded the reach of FBP to other platforms, like MicroFlo to microcontroller programming, or PhpFlo to data conversion pipelines in PHP. To support more of these with common tooling, we standardized the FBP protocol that allows IDEs like Flowhub manage flow-based programs across different runtimes.

Dataflow also saw uptake in the bigger industry. Facebook’s Flux architecture brought flow-based programming to reactive web applications. Google’s TensorFlow made dataflow the way to build machine learning applications. And Google’s Cloud Dataflow uses these techniques for stream processing.

Tooling for flow-based programming

One big area of focus for us has been improving the tooling around NoFlo, as well as the other FBP systems. The FBP protocol has been a big enabler for both building better tools, and for collaboration between different FBP and dataflow systems.

FBP protocol

Here are some of the tools currently available for NoFlo developers:

  • Flowhub — browser-based visual programming IDE for NoFlo and other flow-based systems
  • noflo-nodejs — command-line interface for running NoFlo programs on Node.js
  • noflo-browser-app — template for building browser applications in NoFLo
  • MsgFlo — for running NoFlo and other FBP runtimes as a distributed system
  • fbp-specdata-driven tests for NoFlo and other FBP environments
  • flowtrace — tool for retroactive debugging of NoFlo programs. Supports visual replay with Flowhub

NoFlo 0.8

NoFlo 0.8, released in March this year is probably our most important release so far. It introduced a new component API and greatly clarified the component and network lifecycle.

NoFlo 0.8 program lifecycle

With this release, it is easier than ever to build well-behaved NoFlo components and to deal with the mixture of asynchronous and synchronous data processing. It also brings NoFlo a lot closer to the classical FBP concepts.

As part of the release process, we also fully overhauled the NoFlo documentation and wrote a new data transformations tutorial project.

To find out more about NoFlo 0.8, watch my recent NoFlo talk from Berlin Node.js meetup:

Road to 1.0

In addition to providing lots of new APIs and functionality, NoFlo 0.8 also acts as the transitional release as we head towards the big 1.0 release. In this version we marked many old APIs as deprecated.

NoFlo 1.0 will essentially be 0.8, but with all the deprecated APIs removed. If you haven’t done so already, now is a good time to upgrade your NoFlo projects to 0.8 and make sure everything runs without deprecation warnings.

We intend to release NoFlo 1.0 later this summer once more of our open source component libraries have been updated to utilize the new features.


Edit as project and Flowhub live mode

Permalink - Posted on 2017-05-08 00:00

In Flowhub you can create and edit full flow-based programming projects. The live mode enables introspecting running FBP systems. This weekend we rolled out Flowhub 0.19 which makes it easy to move between these modes.

Live mode

Flowhub’s live mode is designed for making software introspectable — flow-based systems provide access to the graph and network state information, and tools like Flowhub then visualize it.

Opening a system in live mode can be done with a URL that contains the connection details to the runtime. This means runtimes can make their live mode discoverable in many ways, from printing it at system start-up to a NFC tag or QR Code on a deployed device:

In live mode, users can see the graph structure of the running software and the packets flowing through it. And with the right permissions, you can also navigate the possible sub-graphs and component source code.

Edit as project

To make changes to the running software, hit the Edit as project button. Flowhub will download the currently running software from the runtime and turn it into a project.

MsgFlo in live mode

In nutshell:

  • Live mode provides a read-only view to a running system
  • Project mode allows live programming a full project including its graphs and components
  • Edit as project can be used to turn a live mode session into an editable project
  • Projects can be synchronized with GitHub

Edit as project also works with the example programs that ship with Flowhub.


Atreus: Building a custom ergonomic keyboard

Permalink - Posted on 2017-04-20 00:00

As mentioned in my Working on Android post, I’ve been using a mechanical keyboard for a couple of years now. Now that I work on Flowhub from home, it was a good time to re-evaluate the whole work setup. As far as regular keyboards go, the MiniLa was nice, but I wanted something more compact and ergonomic.

The Atreus keyboard

My new Atreus

Atreus is a 40% ergonomic mechanical keyboard designed by Phil Hagelberg. It is an open hardware design, but he also sells kits for easier construction. From the kit introduction:

The Atreus is a small mechanical keyboard that is based around the shape of the human hand. It combines the comfort of a split ergonomic keyboard with the crisp key action of mechanical switches, all while fitting into a tiny profile.

My use case was also quite travel-oriented. I wanted a small keyboard that would enable me to work with it also on the road. There are many other small-ish DIY keyboard designs like Planck and Gherkin available, but Atreus had the advantage of better ergonomics. I really liked the design of the Ergodox keyboard, and Atreus essentially is that made mobile:

I found the split halves and relatively large size (which are fantastic for stationary use at a desk) make me reluctant to use it on the lap, at a coffee shop, or on the couch, so that’s the primary use case I’ve targeted with the Atreus. It still has most of the other characteristics that make the Ergodox stand out, like mechanical Cherry switches, staggered columns instead of rows, heavy usage of the thumbs, and a hackable microcontroller with flexible firmware, but it’s dramatically smaller and lighter

I had the opportunity to try a kit-built Atreus in the Berlin Mechanical Keyboard meetup, and it felt nice. It was time to start the project.

Sourcing the parts

When building an Atreus the first decision is whether to go with the kit or hand-wire it yourself. Building from a kit is certainly easier, but since I’m a member of a hackerspace, doing a hand-wired build seemed like the way to go.

To build a custom keyboard, you need:

  • Switches: in my case 37 Cherry MX blues and 5 Cherry MX blacks
  • Diodes: one 1N4148 per switch
  • Microcontroller: a Arduino Pro Micro on my keyboard
  • Keycaps: started with recycled ones and later upgraded to DSA blanks
  • Case: got a set of laset-cut steel plates

Even though Cherry — the maker of the most common mechanical key switches — is a German company, it is quite difficult to get switches in retail here. Luckily a fellow hackerspace member had just dismantled some old mechanical keyboards, and so I was able to get the switches I needed via barter.

Keyswitches

The Cherry MX blues are tactile clicky switches that feel super-nice to type on, but are quite loud. For modifiers I went with Cherry MX blacks that are linear. This way there is quite a clear difference in feel between keys you typically hold down compared to the ones you just press.

The diodes and the microcontroller I ordered from Amazon for about 20€ total.

Arduino Pro Micro

At first I used a set of old keycaps that I got with the switches, but once the keyboard was up and running I upgraded to a very nice set of blank DSA-profile keycaps that I ordered from AliExpress for 30€. That set came with enough keycaps that I’ll have myself covered if I ever build a second Atreus.

All put together, I think the parts ended up costing me around 100€ total.

Preparations

When I received all the parts, there were some preparation steps to be made. Since the key switches were 2nd hand, I had to start by dismantling them and removing old diodes that had been left inside some of them.

Opening the key switches

The keycaps I had gotten with the switches were super grimy, and so I ended up sending them to the washing machine. After that you could see that they were not new, but at least they were clean.

With the steel mounting plate there had been a slight misunderstading, and the plates I received were a few millimeters thicker than needed, so the switches wouldn’t “click” in place. While this could’ve been worked around with hot glue, we ended up filing the mounting holes down to the right thickness.

Filing the plate

Little bit of help

Wiring the keyboard

Once the mounting plate was in the right shape, I clicked the switches in and it was time to solder.

All switches in place

Hand-wiring keyboards is not that tricky. You have to attach a diode to each keyswitch, and then connect each row together via the diodes.

Connecting diodes

First row ready

The two thumb keys are wired to be on the same column, but different rows.

All rows ready diodes

Then each column is connected together via the other pin on the switches.

Soldering columns

This is how the matrix looks like:

Completed matrix

After these are done, connect a wire from each column, and each row to a I/O pin on the microcontroller.

Adding column wires

If you haven’t done it earlier, this is a good stage to test all connections with a multimeter!

Connecting the microcontroller

Firmware

After finishing the wiring, I downloaded the QMK firmware, changed the PIN mapping for how my Atreus is wired up, switched the layout to Colemak, and the keyboard was ready to go.

Atreus in use

Don’t mind the key labels in the picture above. These are the second-hand keycaps I started with. Since then I’ve switched to blank ones.

USB-C

The default Atreus design has the USB cable connected directly to the microcontroller, meaning that you’ll have to open the case to change the cable. To mitigate that I wanted to add a USB breakout board to the project, and this being 2017, it felt right to go with USB-C.

USB-C breakouts

I found some cheap USB-C breakout boards from AliExpress. Once they arrived, it was time to figure out how the spec works. Since USB-C is quite new, there are very few resources available on how to use it with microcontrollers. These tutorials were quite helpful:

Here is how we ended up wiring the breakout board. After these you only have four wires to connect to the microcontroller: ground, power, and the positive and negative data pins.

USB-C breakout with wiring

This Atreus build log was useful for figuring out where to connect the USB wires on the Pro Micro. Once all was done, I had a custom, USB-C keyboard!

USB-C keyboard

Next steps

Now I have the Atreus working nicely on my new standing desk. Learning Colemak is a bit painful, but the keyboard itself feels super nice!

New standing desk

However, I’d still like to CNC mill a proper wooden case for the keyboard. I may update this post once that happens.

I’m also considering to order an Atreus kit so I’d have a second, always packed for travel keyboard. The kit comes with a PCB, which might work better at airport security checks than the hand-wired build.

Another thing that is quite tempting is to make a custom firmware with MicroFlo. I have no complaints on how QMK works, but it’d be super-cool to use our visual programming tool to tweak the keyboard live.

Big thanks to Technomancy for the Atreus design, and to XenGi for all the help during the build!