RFC: deprecation of codahale metrics

[szuecs]

In skipper we do not intend to break compatibility for our users (library or binary/container), but we found that the codahale metrics library was archived and https://github.com/rcrowley/go-metrics?tab=readme-ov-file#archived-as-of-april-1-2025 tells about Prometheus or OTel metrics being replacements, which aligns with our thoughts and likely it aligns with expectations of our community.

For this change I would also re-think the general metrics.Metrics package. In order to support custom Prometheus labels we hacked around the fact that "name" and "key" of a label have to be static assigned value, because of limitations of the Prometheus client library.
For example:

skipper_custom_duration_seconds_bucket{key="routes.update_latency",le="300"} 420
skipper_custom_duration_seconds_bucket{key="routes.update_latency",le="600"} 420
skipper_custom_duration_seconds_bucket{key="routes.update_latency",le="+Inf"} 420
skipper_custom_duration_seconds_sum{key="routes.update_latency"} 4.013618892000004
skipper_custom_duration_seconds_count{key="routes.update_latency"} 420

skipper_custom_gauges{key="fifo.kube_rg__example_https_redirect.active"} 0
skipper_custom_gauges{key="fifo.kube_rg__example_https_redirect.queued"} 0
skipper_custom_gauges{key="listener.accepted.connections"} 158
skipper_custom_gauges{key="listener.queued.connections"} 0
skipper_custom_gauges{key="routes.total"} 390
skipper_custom_gauges{key="routes.updated_timestamp"} 1.749711365e+09
skipper_custom_gauges{key="swarm.redis.hits"} 119640
skipper_custom_gauges{key="swarm.redis.idleconns"} 25
skipper_custom_gauges{key="swarm.redis.misses"} 0
skipper_custom_gauges{key="swarm.redis.shards"} 1
skipper_custom_gauges{key="swarm.redis.staleconns"} 0
skipper_custom_gauges{key="swarm.redis.timeouts"} 0
skipper_custom_gauges{key="swarm.redis.totalconns"} 26

skipper_custom_total{key="apiUsageMonitoring.custom.foo-service.foo-api.bc49b29a-4369-4992-b7b6-21c05a4c0cb2.POST.foo-record.*.*.http_count",start="1749653057607434800"} 1.55290383e+08
skipper_custom_total{key="incoming.HTTP/1.0",start="1749689607197280215"} 3
skipper_custom_total{key="incoming.HTTP/1.1",start="1749653018593631296"} 3.66307672e+08
skipper_custom_total{key="incoming.HTTP/2.0",start="1749700125765944559"} 3
skipper_custom_total{key="lb-conn-active",start="1749653018593616574"} 3.66309187e+08
skipper_custom_total{key="lb-conn-closed",start="1749653018593809620"} 4.6581183e+07
skipper_custom_total{key="lb-conn-idle",start="1749653033270628954"} 3.6618313e+08
skipper_custom_total{key="lb-conn-new",start="1749653018593545892"} 4.6581341e+07
skipper_custom_total{key="outgoing.HTTP/1.1",start="1749653056748749795"} 3.66169275e+08
skipper_custom_total{key="passive-health-check.endpoints.dropped",start="1749676349093237242"} 131
skipper_custom_total{key="passive-health-check.requests.passed",start="1749676349093238318"} 131

These kind of metrics could be more expressive, if we would be able to dynamically create names that are not dynamic as we have skipper_custom_ as prefix with label key key.

I propose to write our own library in skipper to support this and be able to export different formats like Prometheus or OTel.

[szuecs]

From discussion in slack we can see that no one is using codahale metrics.
So we will accept our breaking change, because the community seems to have no problem with it.