Mastering PromQL: A Comprehensive Guide to Prometheus Query Language

This article is an excerpt from the book, " 
         Figure 5.1 – A simplified view of metric data stored in the TSDB 

• The labels or dimensions of a metric: Prometheus labels provide metadata to identify data of interest. These labels create metrics, time series, and samples: 

* Each unique __name__ value creates a metric. In the preceding figure, the metric is app_ frontend_requests. 

* Each unique set of labels creates a time series. In the preceding figure, the set of all labels is the time series. 

* A time series will contain multiple samples, each with a unique timestamp. The preceding figure shows a single sample, but over time, multiple samples will be collected for each  

time series. 

* The number of unique values for a metric label is referred to as the cardinality of the l abel. Highly cardinal labels should be avoided, as they signifi cantly increase the storage costs of the metric. 

The following diagram shows a single metric containing two time series and five samples: 

 
                            Figure 5.5 – Range vectors 

• Scalars are simple numeric values, with no labels or timestamps involved. 

Selecting data 

PromQL offers several tools for you to select data to show in a dashboard or a list, or just to understand a system’s state. Some of these are described in the following table: 

Table 5.2 – The comparison operators available in PromQL 

Vector matching is an initially confusing topic; to clarify it, let’s consider examples for the three cases of vector matching – one-to-one, one-to-many/many-to-one, and many-to-many. 

By default, when combining vectors, all label names and values are matched. This means that for each element of the vector, the operator will try to find a single matching element from the second vector.  

Let’s consider a simple example: 

• Vector A: 

10{color=blue,smell=ocean} 

31{color=red,smell=cinnamon} 

27{color=green,smell=grass} 

• Vector B: 

19{color=blue,smell=ocean} 

8{color=red,smell=cinnamon} ‚ 14{color=green,smell=jungle} 

• A{} + B{}: 

29{color=blue,smell=ocean} 

39 {color=red,smell=cinnamon} 

• A{} + on (color) B{} or A{} + ignoring (smell) B{}: 

29{color=blue} 

39{color=red} 

41{color=green} 

When color=blue and smell=ocean, A{} + B{} gives 10 + 19 = 29, and when color=red and smell=cinnamon, A{} + B{} gives 31 + 8 = 29. The other elements do not match the two vectors so are ignored. 

When we sum the vectors using on (color), we will only match on the color label; so now, the two green elements match and are summed. 

This example works when there is a one-to-one relationship of labels between vector A and vector B. However, sometimes there may be a many-to-one or one-to-many relationship – that is, vector A or vector B may have more than one element that matches the other vector. In these cases, Prometheus will give an error, and grouping syntax must be used. Let’s look at another example to illustrate this: 

• Vector A: 

7{color=blue,smell=ocean} 

5{color=red,smell=cinamon} 

2{color=blue,smell=powder} 

• Vector B: 

20{color=blue,smell=ocean} 

8{color=red,smell=cinamon} ‚ 14{color=green,smell=jungle} 

• A{} + on (color) group_left  B{}: 

27{color=blue,smell=ocean} 

13{color=red,smell=cinamon} 

22{color=blue,smell=powder} 

Now, we have two different elements in vector A with color=blue. The group_left command will use the labels from vector A but only match on color. This leads to the third element of the combined vector having a value of 22, when the item matching in vector B has a different smell. The group_right operator will behave in the opposite direction. 

The final option is a many-to-many vector match. These matches use the logical operators and, unless, and or to combine parts of vectors A and B. Let’s see some examples: 

• Vector A: 

10{color=blue,smell=ocean} 

31{color=red,smell=cinamon} 

27{color=green,smell=grass} 

• Vector B: 

19{color=blue,smell=ocean} 

8{color=red,smell=cinamon} ‚ 14{color=green,smell=jungle} 

• A{} and B{}: 

10{color=blue,smell=ocean} 

31{color=red,smell=cinamon} 

• A{} unless B{}: 

27{color=green,smell=grass} 

• A{} or B{}: 

10{color=blue,smell=ocean} 

31{color=red,smell=cinamon} 

27{color=green,smell=grass} 

14{color=green,smell=jungle} 

Unlike the previous examples, mathematical operators are not being used here, so the values of the elements are the values from vector A, but only the elements of A that match the logical condition in B are returned. 

Conclusion

PromQL is an essential component of Prometheus, offering users a flexible and powerful means of querying and analyzing time-series data. By understanding its data types and operators, users can craft complex queries that provide deep insights into system performance. The language supports a variety of data selection and comparison operations, allowing for precise monitoring and alerting. Whether working with instant vectors, range vectors, or scalars, PromQL enables developers and operators to optimize their use of Prometheus for monitoring and alerting, ensuring systems remain performant and reliable. As organizations continue to embrace cloud-native architectures, mastering PromQL becomes increasingly vital for maintaining robust and efficient systems. 

Author Bio

Rob Chapman is a creative IT engineer and founder at The Melt Cafe, with two decades of experience in the full application life cycle. Working over the years for companies such as the Environment Agency, BT Global Services, Microsoft, and Grafana, Rob has built a wealth of experience on large complex systems. More than anything, Rob loves saving energy, time, and money and has a track record for bringing production-related concerns forward so that they are addressed earlier in the development cycle, when they are cheaper and easier to solve. In his spare time, Rob is a Scout leader, and he enjoys hiking, climbing, and, most of all, spending time with his family and six children.

Peter Holmes is a senior engineer with a deep interest in digital systems and how to use them to solve problems. With over 16 years of experience, he has worked in various roles in operations. Working at organizations such as Boots UK, Fujitsu Services, Anaplan, Thomson Reuters, and the NHS, he has experience in complex transformational projects, site reliability engineering, platform engineering, and leadership. Peter has a history of taking time to understand the customer and ensuring Day-2+ operations are as smooth and cost-effective as possible.