Consensus stopped due to syncing issue

[junwei0117]

Bug description

We noticed that the consensus in our testnet occasionally stops for several minutes, causing no new checkpoints to be created. From the metrics, it’s clear that the number of missing blocks is higher than normal during these periods when consensus stops.

The following warning appears in the logs during this time::

WARN consensus_core::synchronizer: No results returned while requesting missing blocks

Explore-logs-2024-12-17 18_21_17.txt

[junwei0117]

We just encountered another temporary consensus stop again

2024-12-20T09:23:36.466720Z  WARN consensus_core::authority_service: Errored while trying to fetch missing ancestors via synchronizer: Synchronizer for fetching blocks directly from I is saturated
2024-12-20T09:23:36.468730Z  WARN consensus_core::authority_service: Errored while trying to fetch missing ancestors via synchronizer: Synchronizer for fetching blocks directly from P is saturated
2024-12-20T09:23:36.728751Z  WARN consensus_core::authority_service: Errored while trying to fetch missing ancestors via synchronizer: Synchronizer for fetching blocks directly from F is saturated
2024-12-20T09:23:37.241722Z  WARN consensus_core::authority_service: Errored while trying to fetch missing ancestors via synchronizer: Synchronizer for fetching blocks directly from D is saturated
2024-12-20T09:23:40.128437Z  WARN consensus_core::synchronizer: No results returned while requesting missing blocks
2024-12-20T09:23:46.420191Z  WARN consensus_core::synchronizer: Max retries 6 reached while trying to fetch blocks from peer F.

Explore-logs-2024-12-20 20_46_02.txt

You can check more metrics on Grafana within this time range with CET

{"from":"2024-12-20 10:21:20","to":"2024-12-20 10:24:54"}

[junwei0117]

And this one, although the consensus didn't stop, but the consensus latency is still getting very high, you can notice that this happened after the node came back online from being offline.

WARN consensus_core::synchronizer: No results returned while requesting missing blocks

Explore-logs-2024-12-20 20_45_11.txt

You can check more metrics on Grafana within this time range with CET

{"from":"2024-12-20 10:21:20","to":"2024-12-20 10:24:54"}