In practice, Grafana has not been great at backward compatibility

May 26, 2023

We started our Prometheus and Grafana based metrics setup in late 2018. Although many of our
Grafana dashboards weren’t
created immediately, the majority of them were probably built by
the middle of 2019. Based on release history, we probably started
somewhere around v6.4.0 and had
many dashboards done by the time v7.0.0 came out.
We’re currently frozen on v8.3.11, having
tried v8.4.0
and rejected it and all subsequent versions. The reason for this
is fairly straightforward; from v8.4.0 onward, Grafana broke too
many of our dashboards. The breakage didn’t start in 8.4, to be
honest. For us, things started to degrade from the change between
the 7.x series and 8.0, but 8.4 was the breaking point where too
much was off or not working.

(I’ve done experiments with Grafana v9.0 and onward, and it had
more issues over the latest 8.x releases. In one way this isn’t too
surprising, since it is a new major release.)

I’ve encountered issues in several areas in Grafana during upgrades.
Grafana’s handling of null results from Pro