Monitoring Solution using a TimeSeries Database exporting Infra Metrics (Part 3)

 

What is InfluxDB?

InfluxDB is an open-source schema-less Time-series database that specializes in the storage of time-series data for operations monitoring, application metrics, IoT sensors, and real-time analytics.

InfluxQL is an SQL-like query language for interacting with data in InfluxDB. It is a push-based database which means it can’t collect your data and you need to push to it.

Telegraf : It is a plugin-driven server agent for collecting and sending metrics and events from databases, systems, and IoT sensors.

Its purpose is to collect data from the given data sources and push it to InfluxDB. It has a lot of plugins which helps in collecting data from multiple sources and write to multiple targets.

InfluxDB2 Installation

Manual Download (Ubuntu): (Method 1)

Step 1: Prerequisites

a) You should have a running Ubuntu 20.04 LTS Server.

b) You should have sudo or root access to run privileged commands.

c) You should have apt or apt-get utility available in your System.

Step 2: Add GPG Key

1.wget -qO- https://repos.influxdata.com/influxdb.key | gpg — dearmor | sudo tee /etc/apt/trusted.gpg.d/influxdb.gpg > /dev/null

Step 3: Setup Repo

1.export DISTRIB_ID=$(lsb_release -si); export DISTRIB_CODENAME=$(lsb_release -sc)

2.echo “deb [signed-by=/etc/apt/trusted.gpg.d/influxdb.gpg] https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable” | sudo tee /etc/apt/sources.list.d/influxdb.list > /dev/null

Step 4: Update Your Server

sudo apt-get update

Step 5: Install InfluxDB2

sudo apt-get install influxdb2

Step 6: Verify Installation

dpkg -L influxdb2

Step 7: Check InfluxDB Version

influx version

Step 8: Start InfluxDB Service

sudo systemctl start influxdb

sudo systemctl status influxdb

Docker Compose (Method 2):

Fluent-bit is pushing the system metrics data to Apache Kafka topic “monitor-topic” and Telgraf is fetching the data from Apache Kafka and pushing to InfluxDB bucket “monitordb”

DOCKER_INFLUXDB_INIT_USERNAME: Sets the username for InfluxDB

DOCKER_INFLUXDB_INIT_PASSWORD: Sets the password for InfluxDB

DOCKER_INFLUXDB_INIT_ORG: Sets the organization name

DOCKER_INFLUXDB_INIT_BUCKET: Bucket name where you want to push your data in Influxdb

DOCKER_INFLUXDB_INIT_ADMIN_TOKEN: It helps to push data to InfluxDB.

In the volumes section of Telegraf, we are replacing our updated Telegraf config file with the default one.

Telegraf Configuration File

In the OUTPUT section Data is send to the InfluxDB2 URL and it is writing the data into the given organization and bucket with the admin token which are all provided by the environment variables.

In the INPUT section, we are giving Kafka’s URL, Kafka topic to listen to, and a consumer group name and data format field which means Telegraf will expect data in JSON format from Apache Kafka .

$ docker-compose -f influxdb.yaml up -d

Go to localhost:8086 and put your username and password

Amount of Memory Used Data in Graph :

Amount of CPU Usage in Graph and Histogram :

Consumption of Messages by Telegraf (Confluent Platform) — Consumer:

For Part 4 : Intergration of InfluxDB2 with Grafana : Integration of InfluxDB2 with Grafana

To get the code for the above section :  Monitoring-Solution


Comments

Post a Comment

Popular posts from this blog

Overview of Ansible and Ansible Playbooks

Image
In this article, you will get a brief idea about ansible, ansible playbooks, roles and how to use include module in your playbooks and we will be showing how to automate the process of creating a small application using database – mysql and webserver – flask by using ansible playbooks.   Why Do We Need Ansible? Ansible is an open source IT Configuration Management, Deployment & Orchestration tool. It aims to provide large productivity gains to a wide variety of automation challenges. This tool is very simple to use yet powerful enough to automate complex multi-tier IT application environments. It models your IT infrastructure by describing how all of your systems inter-relate, rather than just managing one system at a time.   For an example : Assume that if you want to install some software in a large number of servers , you will not install those softwares one by one into the server right ….. It will take a hell lot of time !! .     Advantages of using Ansible ...
Read more

Getting Started with ArgoCD

Image
      In this article, we will learn how to use ArgoCD as continuous delivery tool in our workflow. It facilitates the adoption of GitOps, where teams leverage declarative configuration and infrastructure definitions from Git as the single source of truth. What is ArgoCD? ArgoCD is a Kubernetes-native continuous deployment (CD) tool. Unlike external CD tools that only enable push-based deployments, Argo CD can pull updated code from Git repositories and deploy it directly to Kubernetes resources. It enables developers to manage both infrastructure configuration and application updates in one system. There are many features of ArgoCD : 1. Whole Kubernetes configuration defined as code in GIT Repository. 2. Web user interface and command-line interface (CLI). 3. It has easy rollback mechanism through which we can rollback our changes to any previous state. 4. It has cluster disaster recovery feature which helps to create the exact cluster if cluster dies because all co...
Read more

How to Start with Kustomize — it’s Features

Image
  In this article, you will learn about why we use kustomize, what is kustomize and how it is linked with Kubernetes and different features of kustomize like Overlays, Base, how to change namespace, namePrefix, nameSuffix, images, ConfigMap and Secrets in your yaml files. Why Kustomize? Deploying applications to Kubernetes can sometimes feel unmanageable. Organizations having complex applications manage it by using several resources like deployments, services, configMap, secrets, replicas, service discovery and networking through a single API Interface. Many organizations have multiple environments of their applications so that they develop and then test their applications before deploying into production environment. So that’s when Kustomize comes into picture. So using Kustomize you will be able to customize Kubernetes configuration files such deployments, services, ingresses, etc., to a specific environment. For example, replica-count or resource limit for pods , In dev and test...
Read more