What is a JSON feed? Learn more

JSON Feed Viewer

Browse through the showcased feeds, or enter a feed URL below.

Now supporting RSS and Atom feeds thanks to Andrew Chilton's feed2json.org service

CURRENT FEED

InfluxData

JSON


In the News: Manufacturing Tomorrow | Managing the Metrics

Permalink - Posted on 2019-01-25 16:00, modified at 17:03

Publication: Manufacturing Tomorrow
Title: Managing the Metrics

Abstract: This Manufacturing Tomorrow Q&A with Head of Product Marketing at InfluxData Navdeep Sidhu discusses how manufacturing companies are creating real business value from their data. First, Sidhu provides an overview of InfluxData. Then he covers some of the pressing data-related issues that manufacturing companies face today. He explains why measurement, monitoring and access to real-time data greatly matter for manufacturing companies and also discusses barriers that the manufacturing industry faces in adopting data analysis. Finally, he covers InfluxData expansion plans in the global manufacturing arena.

 

Read full coverage

The post In the News: Manufacturing Tomorrow | Managing the Metrics appeared first on InfluxData.


Release Announcement: InfluxDB Enterprise 1.7.3

Permalink - Posted on 2019-01-25 01:33

A new feature-bearing release for InfluxDB Enterprise is now available.

This the first official release of InfluxDB Enterprise on the 1.7 line.

Key highlights for the Enterprise release include:

  • Technical Preview of Flux is available. If you wish to explore this new query language, you will need to modify the configuration to activate it. This is not recommended for production environments. The same authentication and authorization applies to Flux queries, however fine-grained authorization support has not been implemented.
  • For customers using LDAP as the backing security store, a new LDAP configuration setting, security, allows you to configure InfluxDB to use StartTLS, per the LDAPv3 standard, to upgrade an unencrypted connection to TLS.
  • Additional TLS configuration options are available now allowing for TLS to be enabled between the data nodes, and TLS support can also be activated on port 8089 for the meta nodes.
  • The Anti-Entropy (AE) service is configured to be ‘off’ by default. Customers should explore and understand the performance overhead and impact of turning this feature on, prior to activating it within a production environment.

Features released with InfluxDB 1.7 OSS and all subsequent defect fixes from versions 1.7.1, 1.7.2 and 1.7.3 are available with InfluxDB Enterprise 1.7.3.

For our InfluxDB Enterprise customers, log in to the InfluxDB Enterprise portal and download the binaries from there.

 

The post Release Announcement: InfluxDB Enterprise 1.7.3 appeared first on InfluxData.


In the News: The New Stack | How I Created a Telegraf Plugin to Monitor Solar Panels

Permalink - Posted on 2019-01-24 17:37, modified at 18:02

Publication: The New Stack
Title: How I Created a Telegraf Plugin to Monitor Solar Panels
Author: Julius Marozas

Abstract: In this article, Julius Marozas explains how he created the http_listener_v2 Telegraf input plugin. After installing solar panels at his house, he realized that he wanted more access to how metrics were displayed, such as how much power and energy the panels were able to generate from the sun. He also wanted to correlate this data with temperature, cloudiness and sun intensity. To retrieve metrics from the solar panels, he was using Fronius solar inverter, which has a push service that can be configured to periodically send HTTP POST or FTP requests with the data encoded in JSON to the provided endpoint. Since he was also already using InfluxDB for storing other metrics from IoT devices, he thought of persisting the inverter-produced data there. At the time, the InfluxDB metrics collection agent Telegraf didn’t support parsing JSON (and other data formats) data sent via HTTP POST/PUT requests. So he decided to  create a http_listener_v2 plugin for supporting that purpose.

 

Read full coverage

The post In the News: The New Stack | How I Created a Telegraf Plugin to Monitor Solar Panels appeared first on InfluxData.


Introducing the Next-Generation InfluxDB 2.0 Platform

Permalink - Posted on 2019-01-24 00:16, modified at 00:35

Today, we are proud to give you alpha access to the next generation of InfluxDB. We’ve taken our original mantra of Fastest Time to Awesome™ to a whole new level, and we can’t wait to share it with you. You can find links to download the latest build on our website. I’m going to walk you through what you can do in the Alpha. For a broader vision of InfluxDB 2.0 Platform, check out this post from our CTO, Paul Dix.

Everything you need in one binary

The first thing you’ll notice about InfluxDB 2.0 is that there’s only one thing to download and install. Users who have installed the TICK Stack in the past might be wondering how to get data into the system, visualize that data, and process it.

InfluxDB 2.0 contains everything you need in a monitoring platform in a single binary. This allows us to provide a simplified setup experience while maintaining the power and flexibility of individual components.

Users, organizations, and buckets

When you start up InfluxDB 2.0 for the first time, you’ll be asked to set up a few things. Our updated onboarding experience walks you through the process in one easy step. Everything you do in InfluxDB 2.0 requires you to authenticate as a user. This helps ensure a single unified access control across all components of the platform including querying data and building dashboards.

You’ll also need an organization, which acts as a workspace for you and your colleagues to share things like dashboards and queries, as well as a bucket, which is where you store your time series data with a retention policy.

Your data, fully loaded

InfluxDB 2.0 comes with a few ways to quickly load data into the platform. First, we’ve added native Prometheus metric scraping directly into the platform, so you can pull data from anywhere without additional software. Simply add a new Prometheus scrape target through the browser and watch as your metrics start flowing into the system. The “Quick Start” path from onboarding automatically configures the InfluxDB 2.0 scrape target and populates your instance with metrics about itself. If you are just looking to explore the Alpha, this is the quickest path to ingest data and get started.

For those of you who prefer to push your metrics into InfluxDB, you can continue to use the Telegraf collection agent. We’ve made it easier than ever to create Telegraf configurations and distribute the agents wherever they are needed. We’ve added a Telegraf agent configuration option in the InfluxDB 2.0 user interface. This allows you to quickly build a Telegraf configuration file and best of all, InfluxDB 2.0 stores the configuration so Telegraf 1.9 (or above) can pull it remotely. It’s the fastest and easiest way to build a Telegraf agent configuration.

And finally, for users that already have data in Line Protocol, you can upload those files directly via the browser as well. Just select the Line Protocol option when you add data to your bucket, and you’ll be able to upload raw line protocol files from your local machine into the designated bucket.

Never stop exploring (your data)

InfluxDB 2.0 has a whole new Data Explorer that makes it easier than ever to quickly visualize your data. The new explorer allows you to harness the flexibility of Flux by allowing you to select any measurement, tag or field you’d like to filter with and going from there. Would you like to see what tags and fields are being collected from a specific host? Simply filter by that host first, then continue refining using additional tags and fields specific only to that host.

For power users, we’ve made it easy to switch to edit the raw Flux query powering your visualization behind the scenes. You can dive in and start customizing to your heart’s content. We’ve also updated the documentation for all the functions available in Flux today.

Finally, once you’ve got your query and graph absolutely perfect, use the Save As feature to quickly save it to a dashboard or create a new task for scheduling it to run.

Tasks: schedulable Flux queries for aggregations and more

Tasks are the brand new way to execute queries on a schedule and store the results. You could use tasks to build advanced aggregation scripts for analyzing time series data over a longer period of time, similar to how you might use continuous queries within InfluxDB 1.x. Soon we’ll have integrations in Flux to send data and alerts to third party systems, making Tasks useful for monitoring and alerting in addition to aggregation. This is similar to the anomaly detection and alerting functionality from Kapacitor via TICKscripts, but instead of a separate language for queries and tasks, we’ve unified these concepts with Flux.

Tasks can be completely managed through the Tasks page in the UI, but the easiest way to create a task is to start from the Data Explorer, verify the shape of the data you’d like to save, then use the Save As option to create a task from that query. You’ll be able to set the bucket you want to store the results in when you create the Task. Of course, you can also create new Tasks from scratch using the Add Task option.

The power of the command line

I’ve been highlighting the features and capabilities of the browser, but we also have a fully featured command line interface for interacting with InfluxDB 2.0. From the command line, you can do everything you can in the browser. The command line also contains an interactive shell for writing Flux queries, and the ability to script queries easily. Give it a spin and let us know what additional capabilities or enhancements you’d like to see.

Unified API

Everything you see in the browser and the command line is backed by a unified API that’s documented in this Swagger file. This allows us (and community members) to quickly create their own applications on top of this new platform. We are going to start with libraries in Go and Javascript and then move onto other languages (Java, Ruby, C#, Python, Kotlin, PHP, C are all planned) as we see what our users are building with. With the API you should be able to automate and manage everything from the creation of users and access tokens to dashboards to tasks all through your own code.

This is only the beginning

This is our first release of the new vision for InfluxDB, and we plan to keep iterating on it until we are satisfied that it meets our — and more importantly — our community’s standards for software. But we can’t do it alone.

We need your help. We are asking for your feedback on all aspects of the alpha to make it amazing for developers to use. If you find yourself having trouble figuring out how to do something, or see behavior that doesn’t quite make sense, we want to hear about it. Please open issues for anything you would like to see improved, so we can take a look. Anticipate running into bugs as with any new software. In weekly updates, we plan to tackle the most serious ones and continue to iterate based on the feedback we receive.

Thank you to all the developers and community members who help make this software. Great open source communities make the best open source software. Together, let’s build the next-generation time series platform!

The post Introducing the Next-Generation InfluxDB 2.0 Platform appeared first on InfluxData.


InfluxDB 2.0 Alpha Release and the Road Ahead

Permalink - Posted on 2019-01-23 22:08, modified on 2019-01-24 00:15

Today we’re releasing the first alpha build of InfluxDB 2.0. Our vision for 2.0 is to collapse the TICK Stack into one cohesive and consistent whole which combines the time series database, the UI and dashboarding tool, and the background processing and monitoring agent behind a single API. The move from the 1.x line to 2.0 represents the biggest evolution of our product since we started InfluxDB in 2013. In this post I’ll cover what we’re trying to accomplish with the 2.x line of InfluxDB and Flux, why we’re building it out the way we are, and what the development process through alpha, then beta, and then general availability of the 2.0 release will be.

If you want to skip the history and higher level goal and just get to playing around, have a look at our getting started guide for installation and setup instructions in addition to documentation on Flux. We’d love to hear your feedback on our community site under the influxdb2 category. Or if you’re looking for more of a product walkthrough have a look at Russ Savage’s blog post on the release.

Flux, a new language

One of the biggest parts of InfluxDB 2.0 is Flux, our new data scripting and query language. We decided to build a new language after reviewing years of feature requests, community questions, and issues with InfluxQL, our current query language, and TICKscript, our language for working with Kapacitor. From a design goals perspective, Flux should:

  • Support language services that drive graphical user experiences so that users can accomplish more without learning the actual language. Flux’s design should make this easier than we could achieve with other language choices.
  • Designed to integrate across disparate data sources. These could be other databases, third-party APIs, filesystems, or anywhere data lives. Integration with other systems is a core language design feature.
  • Able to cross-compile. Working with time series is possible in many different languages like InfluxQL, PromQL, Flux and others. We wanted this to work on on a single optimizer that could plan against many disparate sources. Reasonable developers can disagree about language choices, and we want to embrace a broader ecosystem for working with data.
  • In addition to supporting multiple syntaxes for querying, Flux must integrate seamlessly with other analytics tools and environments. This includes things like integration with Jupyter and using Apache Arrow as an underlying data interchange format so that it can integrate with other big data analytics systems.
  • Flux is a fourth generation programming language designed for data scripting, ETL, monitoring and alerting. It’s more than a query language, but it’s not just a programming language. It includes a planner and optimizer. It should be seamless between querying and programming, combining them into a single consistent whole.

All of this leads to a few things. First, Flux will be Turing complete and useful for querying and working with not only time series data, but data in general. In the coming months we’ll detail how to create these connectors and pair them with the query planner and optimizer to convert Flux queries to use each data store’s unique capabilities. In the meantime you can hop over to the Fluxlang section of our community site to tell us which connectors you’d like to see first.

For this release we’ve focused on the query abilities against the embedded InfluxDB 2.0 data store. Long standing feature requests like math across measurements, top N, time shift, group by field, and sorting by values are all possible in this initial alpha release. Creating a new language and a new query engine are a daunting and significant task, but that effort is starting to pay dividends in the amount of new functionality we’re able to add. In fact, adding new functions to the language couldn’t be easier. For example, you can see how count is implemented. A primary design goal of the Flux codebase is to make it easy for external contributors to add new functions, just like we do with input functions in Telegraf.

We’ve kept Flux as a separate repository from InfluxDB because the language will have a life of its own as an open source project that is useful whether you’re using InfluxDB or not. To that end we’ve picked the liberal MIT license and we’ll accept pull requests to integrate Flux with all sorts of third party systems, even if they’re competitive with InfluxDB. This mirrors our philosophy around contributions to Telegraf, which is widely in use not only by InfluxDB users, but also by customers of other companies like Microsoft, DataDog, SignalFX, Wavefront, and many others. The reason for accepting connectors which invite competition is pretty simple: we recognize that the value of Telegraf is driven by its input plugins and if everyone can use it as their collector then it will attract a larger community contributing plugins which also benefits InfluxDB users. We’ve watched this play out over the last 3 and 1/2 years with Telegraf now having over 200 plugins mostly contributed by community (including competitors) and a level of community contribution and use that eclipses InfluxDB itself. It’s our most popular piece of open source software based on usage and community contribution.

TICK is combining into one cohesive whole

The TICK stack is what we call the set of components that make up the InfluxData platform. In order it represents four components for solving problems with time series data: Telegraf, our data collector, InfluxDB, our time series database, Chronograf our UI for visualization, and Kapacitor, our service for processing and monitoring. The current incarnations of those components are an artifact of how the company was built over time.

I hatched the idea for the TICK components in 2014 while raising the series A round for Influx. The pitch was pretty simple: time series is a base level abstraction that is useful for solving many problems and we’re building a platform for working with time series data. Said another way: many data problems are actually time series data problems and we’re going to help people solve those. We had a prototype database and now we needed to hire developers and designers to finish that work and build out the other three components. Over the course of 2015 and 2016 we hired the teams to start building these components with the initial Telegraf release in June of 2015, Kapacitor in November of 2015, InfluxDB 1.0 in September of 2016, and a rebooted Chronograf in November of 2016.

Over that time we’ve learned more about how people are using the platform and what types of problems they frequently encounter. With InfluxQL, there were a large number of feature requests that we simply couldn’t figure out an elegant way to add the language. Kapacitor is currently used by about 13% of InfluxDB users. We’ve frequently heard users mention that TICKscript was difficult to learn and difficult to debug, but for the users that made the journey, they frequently think of it as a killer app. Chronograf is used by about 20% of InfluxDB users. In the early days of InfluxDB we had the UI accessible as part of the database. The move to put all UI into Chronograf was contentious at the time and we now realize a better user experience could be presented if we simply made that UI available as part of the database.

Many of these things led to the creation of Flux, a new language to deliver the longstanding query feature requests we couldn’t address in InfluxQL. Flux as a language looks much more familiar than TICKscript, which is a deliberate language design choice. Having Flux as the language for both interactive ad-hoc queries and background processing will make it easier to develop tooling to help users debug and see what’s going on in their monitoring, alerting, and ETL tasks.

The other big push was to create an API that is unified for the entire platform. It has concepts built in that are designed to make InfluxDB a multi-tenanted, time series service. Because of all these components, InfluxDB is actually much more than just a database. It’s a monitoring system, it a dashboarding engine, it’s an analytics service, it’s a event and metrics processor. As the capabilities of Flux expand, so will InfluxDB’s capabilities and scope.

Telegraf continues to have a life of its own outside of InfluxDB, but we will be adding APIs in InfluxDB 2.0 for Telegraf to integrate with it. One example which is available in the alpha is that we now have a user interface for creating Telegraf configurations, which Telegraf can pull directly from InfluxDB.

Push and pull in one place

Another big part of InfluxDB 2.0 is our first class support for the Prometheus exposition format. InfluxDB 2.0 can act as a metrics scraper and the initial onboarding experience has an option to set up scraping of the local InfluxDB server’s metrics. With InfluxDB 2.0 we support both push and pull models out of the box. Eventually we’ll also support querying via PromQL out of the box.

Future work

We wanted to get this early release out for users in the community to play with to give us feedback. There are some core features missing that need to get done before we’re ready to move into the beta phase. We still have to add compatibility (being able to query InfluxDB 2.0 via InfluxQL and the 1.x API), and being able write to InfluxDB 2.0 as though it were a 1.x server. We also need to add features for backup & restore, bulk data import and export, and data deletes.

Most of these features represent lower level API work. As we’re getting that done, the UI team will be able to iterate quickly on user feedback to add more user facing features that can be built on top of the core 2.0 API. We plan to cut weekly versioned alpha builds starting in the first week of February. Once we get to the minimum set of features we’ll move into the beta phase of the project. It is during the beta period that we’ll be doing additional bug fix, performance testing and other operational readiness activities. The alpha builds are not meant for testing performance, but more for giving feedback on the UI and API.

The entire InfluxDB 2.0 API is documented and implemented based on this Swagger file. Once we’ve moved into the beta phase of the project, the API should be stable with only additive changes. We still need to add all the capabilities of TICKscript into Flux, which will make Tasks useful for monitoring and alerting workloads. We also have to create a TICKscript to Flux converter to help Kapacitor users migration to InfluxDB 2.0.

This is definitely an early release and there’s much more we’re planning on doing over the course of the 2.x line of InfluxDB releases. This will set us up for years of continued innovation with Flux as a core building block.

We’re also building out our new InfluxData Cloud offering, which will be a fully managed and hosted version of the InfluxDB 2.x API. It will have usage based pricing similar to what you see in other fully hosted multi-tenant services. We’ll have more news on this soon.

We need your feedback!

This is only the beginning of a journey of continued development and iteration out in the open. Please have a look at our getting started guide for installation and setup instructions in addition to documentation on Flux. We’d love to hear your feedback on our community site under the influxdb2 category. For more of a product walkthrough have a look at Russ Savage’s blog post on the release.

I’d love to hear from you so don’t hesitate to get at me on Twitter or submit to HN and I’ll comment there.

The post InfluxDB 2.0 Alpha Release and the Road Ahead appeared first on InfluxData.


In the News: Manufacturing Tomorrow | Using Open Software in Manufacturing

Permalink - Posted on 2019-01-23 17:45, modified at 18:21

Publication: Manufacturing Tomorrow
Title: Using Open Software in Manufacturing

Abstract: This Manufacturing Tomorrow article features a Q&A with Frederik Van Leeckwyck, Business Development Manager at Factry, a Belgian software company that brings the power of open software to the world of manufacturing, and a Consulting Partner of InfluxData. Van Leeckwyck covers what Factry does, prominent trends in Industrial IT, the importance of open source for manufacturing companies, the reasons for adopting open source, and what makes historians ripe for disruption. In his answers, he mentions InfluxDB: “Historians exist for decades already, and from a purely functional standpoint their basic functionality will not change that much. However, it’s because open software and protocols such as InfluxDB and OPC-UA are available that historians become increasingly available to smaller companies. We’ve helped large companies make the change from incumbent historian vendors to Factry Historian, based on InfluxDB.”

 

Read full coverage

The post In the News: Manufacturing Tomorrow | Using Open Software in Manufacturing appeared first on InfluxData.


In the News: VMblog.com | Monitoring Kubernetes in 2019 Will Get A Whole Lot Easier

Permalink - Posted on 2019-01-23 16:57, modified at 17:24

Publication: VMblog.com
Title: Monitoring Kubernetes in 2019 Will Get A Whole Lot Easier
Author: Navdeep Sidhu

Abstract: InfluxData Head of Product Marketing Navdeep Sidhu, in this VMblog article, notes that the shift to container architectures is demonstrating to DevOps teams that legacy monitoring tools are unsuitable for this new deployment architecture. He points out InfluxData’s prediction for 2019 that the move to containerization and Kubernetes will gain further ground, resulting in a number of expected trends. Among them he outlines five key trends: the evolution of metrics and events observability; growth in CX Monitoring beyond monitoring infrastructure and DevOps toolchains; traction for time series platforms; increased need for purpose-built platforms; and additional gained ground for Kubernetes but also the continued presence of workloads operating on non-K8s infrastructure.

 

Read full coverage

The post In the News: VMblog.com | Monitoring Kubernetes in 2019 Will Get A Whole Lot Easier appeared first on InfluxData.


Release Announcement: Telegraf 1.9.3

Permalink - Posted on 2019-01-23 01:37

A new maintenance release for Telegraf is available now.

This maintenance release of Telegraf 1.9.3 includes improvements to the following plugins:

  1. Agent
    • Updated to use a last-in, first-out ordering for the internal metrics buffer.
    • Fixed internal_write buffer_size not being reset on timed writes.
  2. AMQP Consumer Input (amqp_consumer)
    • This plugin no longer stops consuming when it receives an unparsable message.
  3. Couchbase Input (couchbase)
    • Removed userinfo from the cluster tag to prevent credential leaking.
  4. Prometheus Input (prometheus)
    • Fixed an issue where this plugin was not detecting added and removed pods.
  5. SQL Server Input (sqlserver)
    • Fixed an issue where when using the money type, large values could cause arithmetic overflow.

The binaries for the latest open source release can be found on our downloads page.

The post Release Announcement: Telegraf 1.9.3 appeared first on InfluxData.


In the News: GeekWire | Tech Moves: InfluxData hires former Salesforce exec; Alder adds to C-suite (again); Aiqudo brings on Alexa vet; and more

Permalink - Posted on 2019-01-22 16:00, modified at 12:06

Publication: GeekWire
Title: Tech Moves: InfluxData hires former Salesforce exec; Alder adds to C-suite (again); Aiqudo brings on Alexa vet; and more
Author: James Thorne

Abstract: GeekWire covered InfluxData’s recent announcement about its new hire for Senior Vice President of Engineering, Jim Walsh, who held a similar role at Salesforce. The article mentions that prior to Salesforce, Walsh co-founded cybersecurity firm Versive, which was acquired by eSentire in 2018, and spent over two decades at Microsoft, where he co-founded the Bing engineering team and started the Cosmos data platform. The article quotes what Walsh told GeekWire in an email: “As technology embeds itself into everything, I’m super excited to be joining the world leader in time series platforms as we drive the ability to derive value and insight from the constantly growing trends of increasing sensorization and instrumentation of both virtual and physical worlds.”

 

 

Read full coverage

The post In the News: GeekWire | Tech Moves: InfluxData hires former Salesforce exec; Alder adds to C-suite (again); Aiqudo brings on Alexa vet; and more appeared first on InfluxData.


Using GraphQL with InfluxDB and Flux

Permalink - Posted on 2019-01-17 19:48, modified at 19:49

In a previous post, we showed you how to make Flux queries in a Rails app. In this post, we’ll extend that knowledge to use GraphQL in a Rails app which leverages the power of Flux to query InfluxDB. This allows the client to fine-tune the results of an underlying Flux query. You’ll start to understand the powerful capabilities you can access when you pair GraphQL with Flux.

Before we begin, let’s talk a bit about GraphQL and why you would use it over a traditional REST API. I highly recommend doing the tutorials and looking through the documentation at How to GraphQL to better understand what GraphQL is and how to implement it in your apps. I did a lot of my own learning there.

What is GraphQL?

GraphQL is a query language that allows the client to specify exactly what data it wants in a single request rather than having to make multiple requests to fetch the same data or deal with over-fetching or under-fetching of data. It is database agnostic in that it does not itself directly query the database. Backend logic still handles that, as usual (in our case, a Flux query will be doing the heavy lifting). GraphQL uses resolvers to handle incoming queries and then implements the backend logic to fetch data. It allows the client to have more power over fetching data than it does with a traditional REST API that returns pre-determined data structures.

What are We Building?

We’re going to build a backend that triggers Flux to query our InfluxDB database when a client initiates a GraphQL query. In this case, we are a train company that uses sensors to capture data about our trains along with other pertinent information, such as outdoor temperature. We want our client to be able to choose exactly which data it wants to retrieve, such as speed and/or track temperature, rather than over-fetching or under-fetching data. We will therefore give our client a few GraphQL queries that it can customize to suit its needs.

Trains, Planes, and Data Points

In the last post, I used a measurement called cpu_load_short with various tags and fields. In this post I’ve seeded my InfluxDB database, called trains, with train data. The measurement is train_speed. My tags are driverlocation, and train. My fields are outdoor_tempspeed, and track_temp. Below you can see a sample of data for three different drivers.

Below is a sample of data for one driver.

The method to format the annotated CSV response to our Flux query, which you saw in the previous post, now looks like this after some refactoring. It performs the same function as before—except I’m now generating a train object rather than a cpu load object:

def parse_trains(response)
 header, *rows = response.body.split("\r\n").drop(3)
 column_names = parse_row(header)
 rows.map do |row|
   values = parse_row(row)
   train_data = column_names.zip(values).to_h
   Train.new(train_data)
 end
end

def parse_row(raw_row)
 raw_row.split(",").drop(3)
end

I also made a new PORO (plain old ruby object) for my train object:

class Train
   attr_reader :time, :measurement, :driver, :location, :train, :outdoor_temp, :speed, :track_temp

   def initialize(train)
      @time = train["_time"]
      @measurement = train["_measurement"]
      @driver = train["driver"]
      @location = train["location"]
      @train = train["train"]
      @outdoor_temp = train["outdoor_temp"]
      @speed = train["speed"]
      @track_temp = train["track_temp"]
   end

end

This PORO, generated at the end of my parse_trains method, comes in handy because the GraphQL gem simply fetches the respective instance variable value when resolving the incoming GraphQL query.

Adding GraphQL to a Rails Project

I found the How to GraphQL Rails tutorial to be really useful in getting started, and I recommend following the steps outlined there to add the graphql gem and the GraphiQL tool to your project.

The rest of that tutorial uses Active Record queries to fetch data from a traditional relational database using GraphQL resolvers. It’s helpful to understand how that works, but what if we want to fetch data from an InfluxDB instance? We can’t use Active Record for that, but as we learned in the last post, we can use a Flux query inside an HTTP call to fetch that data (and when the InfluxDB v2.0 client libraries are ready, we can use those).

For my new train data points, my HTTP request to the query endpoint of my InfluxDB database now looks like this:

def query_influxdb(time, filter_param = nil)
 uri = URI.parse("http://localhost:8086/api/v2/query")
 request = Net::HTTP::Post.new(uri)
 request.content_type = "application/vnd.flux"
 request["Accept"] = "application/csv"
 request.body = flux_query(time, filter_param)

 req_options = {
 use_ssl: uri.scheme == "https",
 }
 response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
   http.request(request)
 end
end

This is a generic HTTP request that is invoked for every GraphQL query. A Flux query is then dynamically generated and passed into its request.body.

Depending on the GraphQL query that is being resolved, the Flux query is constructed as follows:

def flux_query(time, filter_param = nil)
 %Q[
   #{flux_from}
   |> #{flux_range(time)}
   |> #{flux_filter(filter_param)}
   |> #{flux_pivot}
   |> #{flux_yield}
 ]
end

def flux_from
 "from(bucket:\"trains/autogen\")"
End

# There are two possible options in the range method because for most queries I just
# want a default range of the last 60 days, but in order to search by
# timestamp, I need to set my range to that exact timestamp. Flux
# allows me to do that by choosing an inclusive start time, which is my timestamp,
# and an exclusive stop time, which is one second later.
def flux_range(time)
 if time == TIME
   "range(start: #{time})"
 else
   start_time = Time.parse(time)
   stop_time = start_time + 1.second
   "range(start: #{start_time.rfc3339}, stop: #{stop_time.rfc3339})"
 end
end

def flux_filter(filter_param = nil)
 if filter_param
   "filter(fn: (r) => r.driver == \"#{filter_param}\" and
                      r._measurement == \"train_speed\")"
 else
   "filter(fn: (r) => r._measurement == \"train_speed\")"
 end
end

def flux_pivot
 "pivot(
       rowKey:[\"_time\"],
       columnKey: [\"_field\"],
       valueColumn: \"_value\"
     )"
end

def flux_yield
 "yield()"
end

Defining GraphQL Types

After following the initial setup instructions outlined in the How to GraphQL Rails tutorial, I created a new file, app/graphql/types/train_type.rb, into which I placed the following code:

# defines a new GraphQL type

Types::TrainType = GraphQL::ObjectType.define do
   # this type is named 'Train'
   name 'Train'

   # it has the following fields
   field :time, types.String
   field :measurement, types.String
   field :driver, types.String
   field :location, types.String
   field :train, types.String
   field :outdoor_temp, types.String
   field :speed, types.String
   field :track_temp, types.String

end

So what’s going on here, exactly? I’m defining an object type. I know I’ve got an InfluxDB database populated with train data points, with my measurement being train_speed, tags of driverlocation, and train, and my fields being speedoutdoor_temp, and track_temp. I need the shape of my GraphQL object type to mimic the shape of my data points in order to render each of the fields. Each of the GraphQL fields thus corresponds to a column of my InfluxDB data points. Note that the Flux pivot function is what allows me to have my InfluxDB fields appear as their own separate columns, which makes the data more readable.

In addition to object types, there are also root types in GraphQL. Root types include a query type and a mutation type. Since we will be querying data, we need to establish our query type in which we will place the code for all of our queries. When we ran rails generate graphql:install in the initial setup, we generated a template file called app/graphql/types/query_type.rb. We now place into this file our various queries, which are just defined as fields, similar to the object type. My entire file looks like this (for easier reading I placed the aforementioned HTTP call, parsing method, and Flux query generator methods at the bottom as private methods):

require 'net/http'
require 'uri'

TIME = "-60d"

Types::QueryType = GraphQL::ObjectType.define do
 name "Query"

   field :allTrains, !types[Types::TrainType] do
     description "Return all train data"
     resolve -> (obj, args, ctx) {
       parse_trains(query_influxdb(TIME))
     }
   end

   field :trainByDriver, !types[Types::TrainType] do
     argument :driver, !types.String
     description "Find train stats by driver"
     resolve -> (obj, args, ctx) {
       parse_trains(query_influxdb(TIME, args[:driver]))
     }
   end

   field :trainByTime, !types[Types::TrainType] do
     argument :time, !types.String
     description "Find train stats by time"
     resolve -> (obj, args, ctx) {
       parse_trains(query_influxdb(args[:time]))
     }
   end

end

private

def parse_trains(response)
 header, *rows = response.body.split("\r\n").drop(3)
 column_names = parse_row(header)
 rows.map do |row|
   values = parse_row(row)
   train_data = column_names.zip(values).to_h
   Train.new(train_data)
 end
end

def parse_row(raw_row)
 raw_row.split(",").drop(3)
end

def query_influxdb(time, filter_param = nil)
 uri = URI.parse("http://localhost:8086/api/v2/query")
 request = Net::HTTP::Post.new(uri)
 request.content_type = "application/vnd.flux"
 request["Accept"] = "application/csv"
 request.body = flux_query(time, filter_param)

 req_options = {
 use_ssl: uri.scheme == "https",
 }
 response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
   http.request(request)
 end
end

def flux_query(time, filter_param = nil)
 %Q[
   #{flux_from}
   |> #{flux_range(time)}
   |> #{flux_filter(filter_param)}
   |> #{flux_pivot}
   |> #{flux_yield}
 ]
end

def flux_from
 "from(bucket:\"trains/autogen\")"
end

def flux_range(time)
 if time == TIME
   "range(start: #{time})"
 else
   start_time = Time.parse(time)
   stop_time = start_time + 1.second
   "range(start: #{start_time.rfc3339}, stop: #{stop_time.rfc3339})"
 end
end

def flux_filter(filter_param = nil)
 if filter_param
   "filter(fn: (r) => r.driver == \"#{filter_param}\" and
                      r._measurement == \"train_speed\")"
 else
   "filter(fn: (r) => r._measurement == \"train_speed\")"
 end
end

def flux_pivot
 "pivot(
       rowKey:[\"_time\"],
       columnKey: [\"_field\"],
       valueColumn: \"_value\"
     )"
end

def flux_yield
 "yield()"
end

The allTrains query returns, as expected, all trains. My resolver is just returning that entire data set, and behind the scenes, the grapqhl gem is simply using dot syntax to call the methods to access each of the instance variables I defined in my Train PORO.

If we completed the initial setup properly, we can use the GraphiQL tool to simulate client queries. Let’s head over to http://localhost:3000/graphiql to test out our queries. I can return all of the fields with the allTrains query, as you see below:

I can also return a selection of only the fields I want:

This is why GraphQL is so useful. The client gets to decide what fields it wants returned rather than having a pre-determined data set returned by a conventional REST API endpoint. On the backend, I can provide my client with a number of options to fetch the desired data, and these options are highly customizable.

For example, I can allow my client to pass in an argument to return all trains for a single driver using the trainByDriver query. In this query, the transformation is passed down to Flux to do the work of returning the train data associated with only the requested driver while GraphQL handles the field selection. The same functionality of only returning the requested fields exists in this query as well.

Since this is time series data we’re working with, it makes sense to allow our client to search by a single timestamp using the trainByTime query. As you can see in the Flux query generator code we discussed earlier, we again push the work of transforming the data into a Flux query to retrieve the data associated with that timestamp, and then have GraphQL return only the fields requested by the client.

You can probably come up with a number of other queries that would be useful to a client, all of which will have the benefit of providing only the information the client specifically requests.

Summary

GraphQL makes a powerful companion to your InfluxDB database. It allows the client to specify exactly what information it needs without over-fetching or under-fetching data. Combined with the results of a Flux query, you now have two loci of control: the range and filter functions of your Flux query, and the types of GraphQL queries you open up to the client. The client can simplify its code and avoid superfluous code, particularly when you push the work of transformation down to the database query rather than requiring the client to make that transformation. Furthermore, you no longer require your client to change the endpoint it hits as requirements change because it is using GraphQL to query the API instead of directly hitting your endpoints. This creates a lot of flexibility in the backend while empowering the client to make data decisions on its own.

One thing that’s worth noting is that you could expose a Flux endpoint to the front end developer and they could write Flux rather than GraphQL to get the same result. However, the advantages of GraphQL are that you have client libraries and the other tooling (like the explorer) that are built on top of it. Also, if you end up hooking into other databases, those results can be joined together in a single API call (i.e. InfluxDB + Postgres).

The post Using GraphQL with InfluxDB and Flux appeared first on InfluxData.