Crashing on scraping

[michael-bud]

We're getting this error in logs when prometheus is scraping. It occurs to 3 of our 5 clustered RabbitMQ nodes and once it starts happening will happen until RabbitMQ is restarted.

2019-07-02 17:42:28.130 [error] <0.2418.0> CRASH REPORT Process <0.2418.0> with 0 neighbours crashed with reason: no function clause matching proplists:get_value(get, 0, undefined) line 215
2019-07-02 17:42:28.131 [error] <0.2417.0> Ranch listener rabbit_web_dispatch_sup_15672, connection process <0.2417.0>, stream 1 had its request process <0.2418.0> exit with reason function_clause and stacktrace [{proplists,get_value,[get,0,undefined],[{file,"proplists.erl"},{line,215}]},{prometheus_rabbitmq_core_metrics_collector,'-collect_metrics/2-lc$^0/1-0-',3,[{file,"src/collectors/prometheus_rabbitmq_core_metrics_collector.erl"},{line,177}]},{prometheus_model_helpers,create_mf,5,[{file,"/home/dead/Projects/rabbitmq/prometheus_rabbitmq_exporter/deps/prometheus/src/model/prometheus_model_helpers.erl"},{line,127}]},{prometheus_rabbitmq_core_metrics_collector,'-mf/3-lc$^2/1-1-',3,[{file,"src/collectors/prometheus_rabbitmq_core_metrics_collector.erl"},{line,169}]},{prometheus_rabbitmq_core_metrics_collector,'-collect_mf/2-lc$^0/1-0-',2,[{file,"src/collectors/prometheus_rabbitmq_core_metrics_collector.erl"},{line,153}]},{prometheus_rabbitmq_core_metrics_collector,collect_mf,2,[{file,"src/collectors/prometheus_rabbitmq_core_metrics_collector.erl"},{line,154}]},{prometheus_collector,collect_mf,3,[{file,"/home/dead/Projects/rabbitmq/prometheus_rabbitmq_exporter/deps/prometheus/src/prometheus_collector.erl"},{line,141}]},{prometheus_registry,'-collect/2-lc$^0/1-0-',3,[{file,"/home/dead/Projects/rabbitmq/prometheus_rabbitmq_exporter/deps/prometheus/src/prometheus_registry.erl"},{line,86}]}]

[MitchDart]

I have the same issue. I used "Google click to deploy" RabbitMQ cluster for GKE. No changes to the configs and yet I get this crash every time.

[arthurdarcet]

same, seeing this in a new cluster running on kubernetes with the default values from the "click to deploy". All three nodes of our cluster are showing this error, and it starts appearing immediately after restart if there is activity on the node.

[jasonjamet]

Hi !
I had the same issue with the exporter at version 3.7.2.5.
Apparently the v3.7.9.1 fix this error:
https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/tag/v3.7.9.1

[hrobertson]

We started getting a 500 on scrapes from all 3 nodes with this error:

2020-03-21 06:34:16.748 [error] <0.15910.45> CRASH REPORT Process <0.15910.45> with 0 neighbours crashed with reason: no case clause matching stopped in prometheus_rabbitmq_queues_collector:'-collect_mf/2-fun-3-'/1 line 75
2020-03-21 06:34:16.748 [error] <0.15909.45> Ranch listener rabbit_web_dispatch_sup_15672, connection process <0.15909.45>, stream 1 had its request process <0.15910.45> exit with reason {case_clause,stopped} and stacktrace [
    {prometheus_rabbitmq_queues_collector,'-collect_mf/2-fun-3-',1,    [{file,"src/collectors/prometheus_rabbitmq_queues_collector.erl"},{line,75}]},
    {prometheus_rabbitmq_queues_collector,'-collect_metrics/2-lc$^1/1-0-',3,    [{file,"src/collectors/prometheus_rabbitmq_queues_collector.erl"},{line,102}]},
    {prometheus_rabbitmq_queues_collector,'-collect_metrics/2-lc$^1/1-0-',3,    [{file,"src/collectors/prometheus_rabbitmq_queues_collector.erl"},{line,102}]},
    {prometheus_model_helpers,create_mf,5,      [{file,"src/model/prometheus_model_helpers.erl"},{line,127}]},
    {prometheus_rabbitmq_queues_collector,mf,3,    [{file,"src/collectors/prometheus_rabbitmq_queues_collector.erl"},{line,94}]},
    {prometheus_rabbitmq_queues_collector,'-collect_mf/2-lc$^4/1-2-',3,    [{file,"src/collectors/prometheus_rabbitmq_queues_collector.erl"},{line,75}]},
    {prometheus_rabbitmq_queues_collector,collect_mf,2,    [{file,"src/collectors/prometheus_rabbitmq_queues_collector.erl"},{line,75}]},
    {prometheus_collector,collect_mf,3,    [{file,"src/prometheus_collector.erl"},{line,171}]}
    ]

Disabled/reenabled the plugin on all nodes - No effect
Restarted RabbitMQ on all nodes - No effect
Restarted the VM of one node - Now metrics working on all nodes 🤷‍♂️