Monitoring Virtual Machines: Gnocchi and OpenStack Aodh

Owned by Chris Green
Last updated: Jul 10, 2023
24 min read

This component is currently disabled and not available.

This document is a brief introduction to Aodh and Gnocchi.

 

We can monitor virtual machines by using alarms which fire when a threshold for a given measurement (e.g memory usage) has been met, when a VM has unexpectedly powered off, or when a VM has entered an error state.

In OpenStack, Aodh provides the ability to create, update and assign alarms to virtual machines.

To use alarms to monitor specific metrics from a virtual machine, Aodh uses metrics provided by Gnocchi.

Command Line Clients

The command line clients for Aodh and Gnocchi can be installed using pip:

pip install aodhclient pip install gnocchiclient

Aodh

Aodh is the telemetry alarming service which triggers alarm when the collected data or metrics have met or broken defined rules.

Alarms are governed by defined rules and have three states that they can enter:

  • ok - the rule has been evaluated as false

  • alarm - the rule has been evaluated as true

  • insufficient_data - there is not enough data available to meaningfully determine the alarm state.

There are three types of alarms which can be created using Aodh:

  • Threshold Alarms: these alarms move to an alarm state when a given threshold has been met or exceeded. An example of a threshold alarm would be a memory usage alarm, which uses the memory usage metric of the specific VM to determine the alarm’s state. These alarms will move back to an ok state when the threshold is no longer being exceeded.

  • Event Alarms: these alarms move to an alarm state when a given event has occurred to a resource e.g a VM has unexpectedly powered off. These alarms only move to the alarm state once and will need their status updated manually to ‘reset’ the alarm.

    Note: Event alarms will always start in an insufficient_data state once created and will only change state when the event has occurred.

  • Composite Alarms: these alarms use a combination of rules to determine the state of an alarm. These rules are combined with logical operators.

Please see the Create Aodh Alarms documentation for more information on how to create Aodh Alarms in the command line.

Gnocchi

Gnocchi is a time series database which process data from the telemetry service Ceilometer. It correlates measurements from Ceilometer to resources and metrics.

Metrics are measurable quantities (e.g CPU time, memory usage etc) which are sampled from objects called resources. These resources could be instances (VMs), storage volume etc.

The gnocchi CLI enables us to use openstack commands to access and work with some of the metrics being stored.

openstack metric <command> #The complete list of metric commands can be viewed using openstack metric --help

Metrics

A metric is a measurable resource property. This property could be memory usage, the amount of CPU time used, the number of vCPUs etc.

Metrics consist of:

  • Name: a unique name for the metric

  • Unit: e.g MB, ns, B/s

  • Resource ID: the resource where the measurement is from

  • Unique User ID (UIID): a unique identifier for the metric

  • Archive Policy: The policy for aggregating and storing the measurement for the metric.

    In OpenStack, Ceilometer automatically populates Gnocchi with metrics and resources.

The list of measurements from Ceilometer that Gnocchi can aggregate and store data from can be found here:

https://docs.openstack.org/ceilometer/train/admin/telemetry-measurements.html

Note: Since the Stein Release of OpenStack, the CPU utilization meter has been deprecated and the meter cpu_util cannot be used. However, gnocchi does provide a way to get the CPU utilization of a resource manually.

For example, metrics which are collected include:

+----------------------------+-------------------+ | Metric | Unit | +============================+===================+ | network.incoming.bytes | B | +----------------------------+-------------------+ | network.incoming.packets | packet | +----------------------------+-------------------+ | network.outgoing.bytes | B | +----------------------------+-------------------+ | network.outgoing.packets | packet | +----------------------------+-------------------+ | disk.device.read.bytes | B | +----------------------------+-------------------+ | disk.device.read.requests | request | +----------------------------+-------------------+ | disk.device.write.bytes | B | +----------------------------+-------------------+ | disk.device.write.requests | request | +----------------------------+-------------------+ | cpu | ns | +----------------------------+-------------------+ | disk.ephemeral.size | GB | +----------------------------+-------------------+ | disk.root.size | GB | +----------------------------+-------------------+ | memory.usage | MB | +----------------------------+-------------------+ | memory | MB | +----------------------------+-------------------+ | vcpus | vcpu | +----------------------------+-------------------+

 

CPU Utilization

Although the cpu_util meter has been deprecated since the OpenStack Stein release, we can use gnocchi to calculate the CPU utilization of a VM manually.

We can use the command gnocchi agggregates <options> to do this.

To calculate the CPU utilization of a VM, we can use the following command:

This will return a table similar to the following:

 

Archive Policies

Archive policies are linked to every metric for each resource. These policies determines how many data points to collect over a given time period and the method for aggregating this data.

To view the list of archive policies, we can use the command:

This will return a table listing each archive policy and how the policies are defined. The definitions given to each policy determines how raw datapoints for metrics from OpenStack Ceilometer are collected and aggregated.

We can view details for an archive policy using the command:

This will return a table with information about the named archive policy. For example:

This table shows each archive-policy and how the raw datapoints for each metric is stored. As an example, let’s view the details for one of the archive policies.

 

In Gnocchi, granularity refers to the time interval between each aggregated data point. We can see from this table that for metrics collected using the archive policy ceilometer-high:

  • The mean is stored for each interval.

  • Stores one hour of data in one second intervals. (3600 data points)

  • Stores one day of data in one minute intervals. (1440 data points)

  • Stores one year of data in one hour intervals. (8760 data points)

    Note: When creating threshold alarms which monitors metrics, it is important to check which archive policy they are using to collect the data. This is because each archive policy will have a different value for granularity. If a threshold alarm time interval is shorter than the granularity for that specific metric, the alarm will remain in an insufficient_data state.

Metric Commands

We can list the metrics in our project using the command:

To view more information about the metric, we can use:

We can view the metric resource list as well. The following command will return a table containing every single resource (instance, instance disk, etc) which metrics are attached to.

To view the measurements of a metric, we can use the command:

To view the resource which has a metric linked to it, we can use the command:

To view the measurements of a metric, we can use the command:

To view the list of resource types:

References

Gnocchi Documentation: Gnocchi – Metric as a Service — Gnocchi 4.5.1 documentation

Telemetry Measurements for Train: https://docs.openstack.org/ceilometer/train/admin/telemetry-measurements.html

Gnocchi Aggregation: How I Learned to Stop Worrying and Love Gnocchi aggregation

Gnocchi Glossary: Glossary — Gnocchi 4.5.1 documentation

Aodh Alarms: https://docs.openstack.org/aodh/train/admin/telemetry-alarms.html