This article is an in-depth write-up on Grafana – An open-source tool for running analytics & monitoring our systems online. It contains answers to all our questions about it such as What is it? Why use it? Can I deploy it on-prem? Are there any companies using it already? How popular is it?
Also, I’ll share a bit of my industry experience with the tool.
So, without any further ado.
Let’s get started.
Affiliate Disclaimer: A few of the resources stated in this article contain affiliate links. That means if you find these resources helpful, worthy of spending your money on and you buy a subscription, a course or a book; I get a small cut without you paying anything extra.
I recommend these resources to you because I think the content they offer is pretty good & these will assist you big time in upskilling yourself enabling you to soar in your career.
1. What is Grafana & What Is It Used For?
Grafana is an open source solution for running data analytics, pulling up metrics that make sense of the massive amount of data & to monitor our apps with the help of cool customizable dashboards.
Grafana connects with every possible data source, commonly referred to as databases such as Graphite, Prometheus, Influx DB, ElasticSearch, MySQL, PostgreSQL etc.
Grafana being an open source solution also enables us to write plugins from scratch for integration with several different data sources.
The tool helps us study, analyse & monitor data over a period of time, technically called time series analytics.
It helps us track the user behaviour, application behaviour, frequency of errors popping up in production or a pre-prod environment, type of errors popping up & the contextual scenarios by providing relative data.
A big upside of the project is it can be deployed on-prem by organizations which do not want their data to be streamed over to a vendor cloud for security & other reasons.
Over time this framework has gained a lot of popularity in the industry & is deployed by big guns such as PayPal, eBay, Intel & many more. I’ll talk about the industry use cases up ahead in the article.
Besides the core open source solution there are other two services offered by the Grafana team for businesses known as the Grafana Cloud & the Enterprise. What are they? More on that up ahead in the article.
Before that, let’s dig a little deeper into the functionality & the architectural flow of the tool with having an understanding of what dashboards are? & How do they work?
If you are a developer and find it hard to cope with constant changes in technology. You are sick and tired of it. You are looking for ways to jump off that endless upskilling treadmill staying relevant and hireable.
You might want to check out my ebook, DEVELOPER’S ROADMAP TO EXCELLENCE AND BUILDING YOUR OWN THING, where I share with you the roadmap and techniques that I follow to keep my sanity in this ever-changing world of software development without killing myself. In it, you’ll find actionable advice and critical points that will enable you to make informed career decisions and accelerate your career at MACH speed.
2. What Is a Grafana Dashboard?
Here is a snapshot of a Grafana dashboard, monitoring stuff.
The dashboards pull data from the plugged-in data sources such as Graphite, Prometheus, Influx DB, ElasticSearch, MySQL, PostgreSQL etc. These are a few of many data sources which Grafana supports by default.
The dashboards contain a gamut of visualization options such as geo maps, heat maps, histograms, all the variety of charts & graphs which a business typically requires to study data.
A dashboard contains several different individual panels on the grid. Each panel has different functionalities.
2.1 My Industry Experience with Grafana
In my former organization where I worked as a backend dev, we used Grafana for monitoring purposes, well naturally.
First on the pre-production environment to study the errors popping up & the server uptime.
The app instances were deployed as Docker containers managed by a docker swarm. There were times when the instances were down or a critical issue caused the system to crash. All of these scenarios were tracked on the Grafana dashboard. Which made our lives a lot easier.
The data was pulled from Prometheus which was plugged-in to the Grafana dashboard as a data source. Queries were fired from the dashboard with different expressions such as min, avg etc.
And Prometheus pulled data from cAdvisor. Have a look at the architectural flow.
After the initial testing & monitoring was complete.
The tool was used to monitor the stuff happening when the app was live, in the production environment. Several pre-meditated checks were put in place & alarms were configured when they occurred. This helped us massively, in gaining an in-depth understanding of the system’s behaviour.
We also used past data, which we could track on the dashboard by filtering down by time range for planning out the future operations.
We also used Kibana for monitoring but that was primarily for tracking & studying logs.
To educate yourself on software architecture from the right resources, to master the art of designing large scale distributed systems that would scale to millions of users, to understand what tech companies are really looking for in a candidate during their system design interviews. Read my blog post on master system design for your interviews or web startup.
3. What Are the Features Offered by Grafana?
Let’s go through the features offered by the open-source analytics dashboard framework.
This open-source framework takes care of all the analytics of our app. We can easily query, visualize, set up alerts, understand the data with the help of metrics.
The dashboard is pretty equipped, & continually evolving, to make sense of complex data. From displaying graphs to heatmaps, histograms, Geo maps. The tool has a plethora of visualization options to understand data as per our business requirements.
Alerts are set up & triggered like trip wires whenever an anticipated scenario occurs. These happenings can be notified on Slack or whatever communication platform the monitoring team uses.
Grafana has native support for approx. a dozen databases. And with many more, facilitated by respective plugins.
Either host it on-prem or any cloud platform of your choice.
It has built-in support for Graphite & expressions like add, filter, avg, min, max functions etc. to custom fetch data. What is Graphite? I’ll come to that.
It also has a built-in Influx DB, Prometheus, ElasticSearch, CloudWatch support. I’ll talk about all up ahead.
Become a Pro In DevOps Starting Right From The Basics 🙂
Educative.io’s – DevOps For Developers Learning Track gets you up to speed on everything you need to know to master Docker and Kubernetes. By the time you’re done, you’ll be able to incorporate Docker and Kubernetes into your work projects with confidence. Educative.io helps software developers level up on in-demand technologies & prepare for their interviews via interactive text-based courses with embedded coding environments. They have over 300,000 learners on their platform.
4. What is Grafana Cloud?
Grafana Cloud is a cloud-native, highly available, fast fully managed open SaaS Software as a Service metrics platform. Pretty helpful for those who do not want to take the load of hosting the solution on-prem & want to stay worry free of managing the entire deployment infrastructure.
It runs on Kubernetes clusters. The backend is Prometheus & Graphite compatible. So, we can either chose Grafana cloud instance or both. For more information on navigate here.
5. What is Grafana Enterprise?
The Enterprise service comes with all the Grafana Cloud features plus premium plugins, data sources, premium support from the core team. We get response SLAs, trainings & a lot more. For more information visit
6. What Are Some of the Real-World Industry Use Cases of Grafana?
Grafana dashboards are deployed all over the industry be it Gaming, IoT, FinTech or E-Comm.
StackOverflow used the tool to enable their developers & site reliability teams create tailored dashboards to visualize data & optimize their server performance.
Digital Ocean uses Grafana to share visualization data between their teams & have in place a common visual data sharing platform.
These are some of the instances, why we love this tool. For further reading on the industry use cases here you go
By now I am pretty sure you have an idea what Grafana is & why use it?
Now let’s find out what are Graphite & Prometheus?
7. What Is Prometheus Grafana?
Prometheus is an open source data monitoring tool. The combination of Prometheus & Grafana is the de-facto tool combination in the industry for deploying a data visualization setup. Grafana dashboard is used for visualizing the data whereas the backend is powered by Prometheus.
Though Prometheus too has data visualization features & stuff. But still, Grafana is preferred for visualizing data. Queries are fired from the dashboard & the data is fetched from Prometheus.
It acts as a perfect open source data model for storing time series data.
8. What Is Graphite Grafana?
Grafana has a pretty advanced Graphite query editor which enables us to interact with the data with the help of expressions & functions.
Should you wish to delve into the details. How Grafana, Graphite & Prometheus are different in terms of features they offer? This is a nice read.
9. Grafana Vs Kibana?
As I stated earlier. In my previous project, Kibana was primarily used for analyzing & monitoring logs. Kibana is the K in the ELK stack. The whole intention of writing Kibana by the ElasticSearch team was to have an effective tool to monitor logs. Just click around & track the context of exceptions occurring in prod instead of running Linux commands in the console to find them. It was a pain, believe me.
Well, on the other hand, Grafana is written as a generic monitoring solution for running monitoring & analytics on pretty much anything. Well, this is a very bird’s eye view of the difference between both the tools.
Read this, if you wish to get a deeper insight.
Recommended Read: Master System Design For Your Interviews Or Your Web Startup
Handpicked Resources to Learn Software Architecture and Large Scale Distributed Systems Design
I’ve put together a list of resources (online courses + books) that I believe are super helpful in building a solid foundation in software architecture and designing large-scale distributed systems like Facebook, YouTube, Gmail, Uber, and so on. Check it out.
Subscribe to the newsletter to stay notified of the new posts.
If you liked the article, share it on the web. You can follow scaleyourapp.com on social media, links below, to stay notified of the new content published. I am Shivang, you can read about me here!
> Spotify Engineering: From Live to Recording
> Ingesting LIVE video streams at a global scale at Twitch
> $64,944 spent on AWS, to support 25,000 customers, in August by ConvertKit.
> Read how Storytel engineering computes customer consumption of books transitioning from batch processing to streaming bookmarks data with Apache Beam and Google Cloud.
> How Pokemon Go scales to millions of requests per second?
> Insight into how Grab built a high-performance ad server.
SUBSCRIBE TO MY NEWSLETTER to be notified of new additions to the list. Fortnight/monthly emails.
Looking for developer, software architect jobs? Try Jooble. Jooble is a job search engine created for a single purpose: To help you find the job of your dreams!!
- Web Application Architecture Explained With Designing a Real-World Service
- Wide-column, Column-oriented and Column Family Databases – A Deep Dive with Bigtable and Cassandra
- Design For Scale and High Availability – What Does 100 Million Users On A Google Service Mean?
- How Razorpay handled significant transaction bursts during events like IPL
- Facebook’s Photo Storage Architecture