Browse through the showcased feeds, or enter a feed URL below.
Hacker and an occasional adventurer. Author of Create.js and NoFlo, founder of Flowhub UG and GuvScale. Decoupling software, one piece at a time. This blog tells the story of that.
Permalink - Posted on 2017-07-11 00:00
Last weekend we held the c-base IoT hack weekend, focused on the Flowhub IoT platform. This was continuation from the workshop we organized at the Bitraf makerspace a week earlier. Same tools and technologies, but slightly different focus areas.
Just like at Bitraf, the workshop aimed to add new IoT capabilities to c-base, as well as to increase the number of members who know how to make the station’s setup do new things. For this, we used three primary tools:
The workshop started in Friday evening after a lecture on nuclear pulse propulsion ended in the main hall. We continued all the way to late Sunday evening with some sleep breaks in between. There is something about c-base that makes you want to work there at night.
By Sunday evening, we had built and deployed 15 connected IoT devices, with five additional ones pretty far in development. You can find the source code in the c-flo repository.
Quite a lot of c-base was already instrumented when we started the workshop. We had details on electricity consumption, internet traffic, and more. But one thing we didn’t have was information on the physical environment at the station. To solve this, we decided to build a set of sensor boxes that we could deploy in different areas of the hackerspace.
The capabilities shared by all the sensor boxes we deployed were:
For some areas of interest we provided some additional sensors:
We found a set of nice little electrical boxes that provided a convenient housing for these sensor boxes. This way we were able to mount them in proper places quickly. This should also protect them from dust and other elements to some degree.
The lights of the c-base main hall are controllable via MsgFlo, and we have a system called farbgeber to produce pleasing color schemes for any given time.
However, when there are events we need to enable manual control of all lights and sound. To make this “MsgFlo vs. IP lounge” control question clearer, we built a Big Switch to decide which controls the lights:
The switch is an old electric mains switch from an office building. It makes a satisfying sound when you turn it, and is big enough that you can see which way the setting is from across the room.
To complement the Big Switch we also added a “c-boom” button to trigger the disco mode in the main hall:
One part of the IoT setup was to make statistics and announcements about c-base visible in different areas of the station. We did this by rolling out a set of displays with Raspberry Pi 3s connected to the MsgFlo MQTT environment.
The announcements shown on the screens range from mission critical information like station power consumption or whether the bar is open, to more fictional ones like the NoFlo-powered space station announcements.
We also built an Android version of the info display software, which enabled deploying screens using some old donated tablets.
This was another successful workshop. Participants got to do new things, and we got lots of new IoT infrastructure installed around c-base. The Flowhub graph is definitely starting to look populated:
We also deployed NASA OpenMCT so that we get a nice overview on the station status. Our telemetry server provides MsgFlo participants that receive data via MQTT, store it in InfluxDB, and then visualize it on the dashboard:
All the c-base IoT software is available on Github:
If you’d like to have a similar IoT workshop at your company, we’re happy to organize one. Get in touch!
Permalink - Posted on 2017-07-04 00:00
I just got back to Berlin from the Bitraf IoT hackathon we organized in Oslo, Norway. This hackathon was the first of two IoT workshops around MsgFlo and Flowhub IoT. The second will be held at c-base in Berlin this coming weekend.
Bitraf is a large non-profit makerspace in the center of Oslo. It provides co-working facilities, as well as labs and a large selection of computer controlled tools for building things. Members have 24/7 access to the space, and are provided with everything needed for CNC milling, laser cutting, 3D-printing and more.
In addition to access control, they also had various environmental sensors available on the MQTT network.
With the workshop, our aim was to utilize these existing things more, as well as to add new IoT capabilities. And of course to increase the number of Bitraf members with the knowledge to work with the MsgFlo IoT setup.
Being a makerspace, Bitraf already had everything needed for the physical side of the workshop — tons of sensors, WiFi-enabled microcontrollers, tools for building cases and mounting solutions. So the workshop preparations mostly focused on the software side of things.
The primary tools for the workshop were:
To help visualize the data coming from the sensors people were building, I integrated the NASA OpenMCT dashboard with MsgFlo and InfluxDB time series database. This setup is available at the cbeam-telemetry-server project.
This gave us a way to send data from any interesting sensors in the IoT network to a dashboard and visualize it. Down the line the persisted data can also be interesting for further analysis or machine learning.
We started the workshop with a quick intro session about Flowhub, MsgFlo, and MQTT development. There is unfortunately no video, but the slides are available:
After the intro, we did a round of all attendees to see what skills people already had, and what they were interested in learning. Then we started collecting ideas what to work on.
People picked their ideas, and the project work started.
I’d like to highlight couple of the projects.
Building new sensors was a major part of the workshop. There were several projects, all built on top of msgflo-arduino and the ESP8266 microcontroller:
There was also a project to automatically open and close windows, but this one didn’t get completed over the weekend. You can follow the progress in the altF4 GitHub repo.
All hackerspaces have the problem that people borrow tools and then don’t return them when finished. This means that the next person needing the tool will have to spend time searching for it.
To solve this, the team designed a system that enabled tools to be locked to a wall, with a web interface where members can “check out” a tool they want to use. This way the system constantly knows what tools are in their right places, and which tools are in use, and by who.
You can see the tool lock system in action in this demo video:
Source code and schematics: https://github.com/einsmein/bitraf-thelock.
Before my flight out, we sat down with Jon to review how things went. In general, I think it is clear the event was a success — people got to learn and try new things, and all projects except one were completed during the two days.
Our unofficial goal was to double the number of nodes in the Bitraf Flowhub graph, and I think we succeeded in this:
Here are couple of comments from the attendees:
Really fun and informative. The development pipeline also seems complete. Made it a lot easier for beginner to get started.
this was a very fantastic hackathon! Lots of interesting things to learn, very enthusiastic participants, great stewardship and we actually got quite a few projects finished. Well done everbody.
In general the development tools we provided worked well. Everybody was able to run the full Flowhub IoT environment on their own machines using the Docker setup we provided. And apart from a couple of corner cases, msgflo-arduino was easy to get going on the NodeMCUs.
With these two, everybody could easily wire up some sensors and see their data in both Flowhub and the OpenMCT dashboard. From the local setup going to production was just a matter of switching the MQTT broker configuration.
If you’d like to have a similar IoT workshop at your company, we’re happy to organize one. Get in touch!
Permalink - Posted on 2017-06-19 00:00
Last week I participated in two hackathons, events where a group of strangers would form a team for two or three days and build a product prototype. In the end all teams pitch their prototypes, and the best ones would be given some prizes.
Hackathons are typically organized to get feedback from developers on some new API or platform. Sometimes they’re also organized as a recruitment opportunity.
Apart from the free beer and camaraderie, I like going to hackathons since they’re a great way to battle test the developer tools I build. The time from idea to having to have a running prototype is short, people are used to different ways of working and different toolkits.
If our tools and flow-based programming work as intended, they should be ideal for these kind of situations.
Our project was Electrocute, a machine learning system for forecasting power consumption in a changing climate.
1.5°C is the global warming target set by the Paris Agreement. How will this affect energy consumption? What kind of generator assets should utilities deploy to meet these targets? When and how much renevable energy can be utilized?
The changing climate poses many questions to utilities. With Electrocute’s forecasting suite power companies can have accurate answers, on-demand.
Unfortunately we didn’t win this one.
Skillport was our project to make long layovers more bearable by connecting people who’re stuck at the airport at the same time.
Long layovers suck. But there is ONE thing amazing about them: You are surrounded by highly skilled people with interesting stories from all over the world. It sometimes happens that you meet someone randomly - we all have a story like that. But usually we are too shy and lazy to communicate and see how we could create a valuable interaction. You never know if the other person feels the same.
We built a mobile app that turns airports into a networking, cultural exchange and knowledge sharing hub. Users tell each other through the app that they are available to meet and what value they can bring to an interaction.
The app connected with a J2EE API service that then communicated over MsgFlo with NoFlo microservices doing all the interactions with social and airport APIs. We also did some data enrichment in NoFlo to make smart recommendations on meeting venues.
This time our project went well with the judges and we were selected as the winner of the Life in between airports challenge. I’m looking forward to the helicopter ride over Berlin!
Skillport also won a space at hub:raum, so this might not be the last you’ll hear of the project…
I’ve written before on why to use message queues for microservices, but that post focused more on the benefits for real-life production usage.
The problems and tasks for a system architecture in a hackathon are different. Since the time is short, you want to enable people to work in parallel as much as possible without stepping on each other’s toes. Since people in the team come from different backgrounds, you want to enable a heterogeneous, polyglot architecture where each developer can use the tools they’re most productive with.
MsgFlo is by its nature very suitable for this. Components can be written in any language that supports the message queue used, and we have convenience libraries for many of them. The discovery mechanism makes new microservices appear on the Flowhub graph as soon as they start, enabling services to be wired together quickly.
Mocks are a useful way to provide a microservice to the other team members even before the real implementation is ready.
For example in the GE Predix hackathon, we knew the machine learning team would need quite a bit of time to build their model. Until that point we ran their microservice with a simple msgflo-python component that just gave
random() as the forecast.
This way everybody else was able to work with the real interface from the get-go. When the learning model was ready we just replaced that Python service, and everything was live.
Mocks can be useful also in situations where you have a misbehaving third-party API.
While shooting for a full test coverage is probably not realistic within the time constraints of a hackathon, it still makes sense to have at least some “happy path” tests. When you’re working with multiple developers each building a different parts of the service, interface tests serve a dual purpose:
And if you’re using a continuous integration tool like Travis, the tests will help you catch any breakages quickly, and also ensure the services work on a clean installation.
For a message queue architecture, fbp-spec is a great tool for writing and running these interface tests.
The reason API and platform providers organize these events is to get feedback. As a developer that works with tons of different APIs, this is a great opportunity to make sure your ideas for improvement are heard.
On the flip side, this usually also means the APIs are in a pretty early stage, and you may be the first one using them in a real-world project. When the inevitable bugs arise, it is a good to have a channel of communications open with the API provider on site so you can get them resolved or worked around quickly.
The downside of the NoFlo and MsgFlo stack is that there is still quite a bit of a learning curve. NoFlo documentation is now in a reasonable place, but with Flowhub and MsgFlo we have tons of work ahead on improving the onboarding experience.
Right now it is easy to work with if somebody sets it up properly first, but getting there is a bit tricky. Fixing this will be crucial for enabling others to benefit from these tools as well.
Permalink - Posted on 2017-06-07 00:00
Both of these will focus on reprogramming the Internet of Things setup of the hackerspace. The aim is to connect more devices at the spaces to the MsgFlo environment, and come up with new connections between systems, and ways to visualize them.
You don’t have to an expert to participate. We’ll be there to help you get up to speed!
If you’d like to have a similar IoT workshop at your company, we’re happy to organize one. Get in touch!
Permalink - Posted on 2017-06-05 00:00
In computer programming, flow-based programming (FBP) is a programming paradigm that defines applications as networks of “black box” processes, which exchange data across predefined connections by message passing, where the connections are specified externally to the processes. These black box processes can be reconnected endlessly to form different applications without having to be changed internally. FBP is thus naturally component-oriented.
With NoFlo software is built by creating graphs that contain reusable components and define the program logic by determining how these components talk to each other.
Our promise was to design a new way to build software & manage complexity - a visual development environment for all.
Here is how Fast Company wrote about NoFlo:
If NoFlo succeeds, it could herald a new paradigm of web programming. Imagine a world where anyone can understand and build web applications, and developers can focus on programming efficient components to be put to work by this new class of application architects. In a way, this is the same promise as the “learn to code” movement, which wants to teach everyone to be a programmer. Just without the programming.
Earlier this year we incorporated Flowhub in Germany. Now, to celebrate six years of NoFlo we’re offering a perpetual 30% discount on Flowhub plans. To lock in the discount, subscribe to a Flowhub plan before June 12th 2017 using the code
After a long period of semi-obscurity, our Kickstarter campaign greatly increased the awareness in FBP and dataflow programming. Several open source projects expanded the reach of FBP to other platforms, like MicroFlo to microcontroller programming, or PhpFlo to data conversion pipelines in PHP. To support more of these with common tooling, we standardized the FBP protocol that allows IDEs like Flowhub manage flow-based programs across different runtimes.
Dataflow also saw uptake in the bigger industry. Facebook’s Flux architecture brought flow-based programming to reactive web applications. Google’s TensorFlow made dataflow the way to build machine learning applications. And Google’s Cloud Dataflow uses these techniques for stream processing.
One big area of focus for us has been improving the tooling around NoFlo, as well as the other FBP systems. The FBP protocol has been a big enabler for both building better tools, and for collaboration between different FBP and dataflow systems.
Here are some of the tools currently available for NoFlo developers:
With this release, it is easier than ever to build well-behaved NoFlo components and to deal with the mixture of asynchronous and synchronous data processing. It also brings NoFlo a lot closer to the classical FBP concepts.
To find out more about NoFlo 0.8, watch my recent NoFlo talk from Berlin Node.js meetup:
In addition to providing lots of new APIs and functionality, NoFlo 0.8 also acts as the transitional release as we head towards the big 1.0 release. In this version we marked many old APIs as deprecated.
NoFlo 1.0 will essentially be 0.8, but with all the deprecated APIs removed. If you haven’t done so already, now is a good time to upgrade your NoFlo projects to 0.8 and make sure everything runs without deprecation warnings.
We intend to release NoFlo 1.0 later this summer once more of our open source component libraries have been updated to utilize the new features.
Permalink - Posted on 2017-05-08 00:00
In Flowhub you can create and edit full flow-based programming projects. The live mode enables introspecting running FBP systems. This weekend we rolled out Flowhub 0.19 which makes it easy to move between these modes.
Flowhub’s live mode is designed for making software introspectable — flow-based systems provide access to the graph and network state information, and tools like Flowhub then visualize it.
Opening a system in live mode can be done with a URL that contains the connection details to the runtime. This means runtimes can make their live mode discoverable in many ways, from printing it at system start-up to a NFC tag or QR Code on a deployed device:
In live mode, users can see the graph structure of the running software and the packets flowing through it. And with the right permissions, you can also navigate the possible sub-graphs and component source code.
To make changes to the running software, hit the Edit as project button. Flowhub will download the currently running software from the runtime and turn it into a project.
Edit as project also works with the example programs that ship with Flowhub.
Permalink - Posted on 2017-04-20 00:00
As mentioned in my Working on Android post, I’ve been using a mechanical keyboard for a couple of years now. Now that I work on Flowhub from home, it was a good time to re-evaluate the whole work setup. As far as regular keyboards go, the MiniLa was nice, but I wanted something more compact and ergonomic.
The Atreus is a small mechanical keyboard that is based around the shape of the human hand. It combines the comfort of a split ergonomic keyboard with the crisp key action of mechanical switches, all while fitting into a tiny profile.
My use case was also quite travel-oriented. I wanted a small keyboard that would enable me to work with it also on the road. There are many other small-ish DIY keyboard designs like Planck and Gherkin available, but Atreus had the advantage of better ergonomics. I really liked the design of the Ergodox keyboard, and Atreus essentially is that made mobile:
I found the split halves and relatively large size (which are fantastic for stationary use at a desk) make me reluctant to use it on the lap, at a coffee shop, or on the couch, so that’s the primary use case I’ve targeted with the Atreus. It still has most of the other characteristics that make the Ergodox stand out, like mechanical Cherry switches, staggered columns instead of rows, heavy usage of the thumbs, and a hackable microcontroller with flexible firmware, but it’s dramatically smaller and lighter
I had the opportunity to try a kit-built Atreus in the Berlin Mechanical Keyboard meetup, and it felt nice. It was time to start the project.
When building an Atreus the first decision is whether to go with the kit or hand-wire it yourself. Building from a kit is certainly easier, but since I’m a member of a hackerspace, doing a hand-wired build seemed like the way to go.
To build a custom keyboard, you need:
Even though Cherry — the maker of the most common mechanical key switches — is a German company, it is quite difficult to get switches in retail here. Luckily a fellow hackerspace member had just dismantled some old mechanical keyboards, and so I was able to get the switches I needed via barter.
The Cherry MX blues are tactile clicky switches that feel super-nice to type on, but are quite loud. For modifiers I went with Cherry MX blacks that are linear. This way there is quite a clear difference in feel between keys you typically hold down compared to the ones you just press.
The diodes and the microcontroller I ordered from Amazon for about 20€ total.
At first I used a set of old keycaps that I got with the switches, but once the keyboard was up and running I upgraded to a very nice set of blank DSA-profile keycaps that I ordered from AliExpress for 30€. That set came with enough keycaps that I’ll have myself covered if I ever build a second Atreus.
All put together, I think the parts ended up costing me around 100€ total.
When I received all the parts, there were some preparation steps to be made. Since the key switches were 2nd hand, I had to start by dismantling them and removing old diodes that had been left inside some of them.
The keycaps I had gotten with the switches were super grimy, and so I ended up sending them to the washing machine. After that you could see that they were not new, but at least they were clean.
With the steel mounting plate there had been a slight misunderstading, and the plates I received were a few millimeters thicker than needed, so the switches wouldn’t “click” in place. While this could’ve been worked around with hot glue, we ended up filing the mounting holes down to the right thickness.
Once the mounting plate was in the right shape, I clicked the switches in and it was time to solder.
Hand-wiring keyboards is not that tricky. You have to attach a diode to each keyswitch, and then connect each row together via the diodes.
The two thumb keys are wired to be on the same column, but different rows.
Then each column is connected together via the other pin on the switches.
This is how the matrix looks like:
After these are done, connect a wire from each column, and each row to a I/O pin on the microcontroller.
If you haven’t done it earlier, this is a good stage to test all connections with a multimeter!
Don’t mind the key labels in the picture above. These are the second-hand keycaps I started with. Since then I’ve switched to blank ones.
The default Atreus design has the USB cable connected directly to the microcontroller, meaning that you’ll have to open the case to change the cable. To mitigate that I wanted to add a USB breakout board to the project, and this being 2017, it felt right to go with USB-C.
I found some cheap USB-C breakout boards from AliExpress. Once they arrived, it was time to figure out how the spec works. Since USB-C is quite new, there are very few resources available on how to use it with microcontrollers. These tutorials were quite helpful:
Here is how we ended up wiring the breakout board. After these you only have four wires to connect to the microcontroller: ground, power, and the positive and negative data pins.
This Atreus build log was useful for figuring out where to connect the USB wires on the Pro Micro. Once all was done, I had a custom, USB-C keyboard!
Now I have the Atreus working nicely on my new standing desk. Learning Colemak is a bit painful, but the keyboard itself feels super nice!
However, I’d still like to CNC mill a proper wooden case for the keyboard. I may update this post once that happens.
I’m also considering to order an Atreus kit so I’d have a second, always packed for travel keyboard. The kit comes with a PCB, which might work better at airport security checks than the hand-wired build.
Another thing that is quite tempting is to make a custom firmware with MicroFlo. I have no complaints on how QMK works, but it’d be super-cool to use our visual programming tool to tweak the keyboard live.
Big thanks to Technomancy for the Atreus design, and to XenGi for all the help during the build!
Permalink - Posted on 2017-04-13 00:00
In an ideal world, microservices are a realization of the Unix philosophy as applied to building internet services: writing programs that do one thing, and do it well; writing programs that work together.
However, when most people think about microservices, they think systems that communicate with each other using HTTP APIs. I think this is quite limited, and something that makes microservices a lot more fragile than they could be. Message queues provide a much better solution. This is mentioned in Martin Fowler’s Microservices article:
The second approach in common use is messaging over a lightweight message bus. The infrastructure chosen is typically dumb (dumb as in acts as a message router only) - simple implementations such as RabbitMQ or ZeroMQ don’t do much more than provide a reliable asynchronous fabric - the smarts still live in the end points that are producing and consuming messages; in the services.
When we were building The Grid, we went with an architecture heavily reliant on microservices communicating using message queues. This gave us several useful capabilities:
HTTP is something all backend developers are familiar with. With message queues you have to deal with some new concepts, and hence new tools are also needed.
Client libraries for talking with common message queues exist for pretty much every language. However, this still means that you’ll have to handle things like message pre-fetch limits, acknowledging handled messages, and setting up queues yourself. And of course keeping track of what service talks to what can become a burden.
We developed the MsgFlo tool to solve these problems. MsgFlo provides open source client libraries for bunch of different programming languages, providing a simple model to handle message-related workflow.
To give you an overview of the dataflow between services, MsgFlo also provides a way to define the whole system as a flow-based programming graph. This means you’ll see the whole system visually, and can change connections between services directly from graphical tools like Flowhub.
As mentioned above, queue-based microservices make autoscaling quite easy. If you’re on Heroku, then our GuvScale can be used to automate scaling operations for all of your background dynos.
I wrote more about GuvScale in a recent blog post.
If you’d like to explore using message queues for your services a bit more, here are couple of good articles:
There are also some MsgFlo example applications available:
Permalink - Posted on 2017-03-31 00:00
As I’m preparing for a NoFlo talk in Bulgaria Web Summit next week, I went through some older videos of my conference talks. Sadly a lot of the older ones are not online, but the ones I found I compiled in playlists:
Decoupling Content Management with Create and PHPCR from SymfonyLive Paris 2012:
If you find any I’ve missed, please let me know!
Permalink - Posted on 2017-03-22 00:00
Last fall we bought the Flowhub and other flow-based programming assets from The Grid, and now after some paperwork we’re up and running as Flowhub UG, a company registered in Berlin, Germany.
We’re also now selling the new Pro and Supporter plans, and can also provide a dedicated on-premise version of Flowhub. Please check out our Plans for more information:
Read more in the latest Kickstarter update.