snuba-subscription-consumer-transactions error tick_partition

[andrsp]

Hi!

Tired of debugging and begging for help.

After upgrading from 24.10 to 24.11.1 snuba-subscription-consumer-transactions constantly failing with error:

Double checked snuba-transactions-commit-log topic in kafka, it has exactly 1 partition, subscription-consumer run in only 1 replica.

What else i can check to fix this error?

    Command:
      snuba
      subscriptions-scheduler-executor
      --dataset=transactions
      --entity=transactions
      --consumer-group=snuba-transactions-subscriptions-consumers
      --followed-consumer-group=transactions_group
      --schedule-ttl=60
      --stale-threshold-seconds=900

2024-12-09 07:38:35,397 Initializing Snuba...
2024-12-09 07:38:37,847 Snuba initialization took 2.4424702310934663s
{"module": "builtins", "event": "Checking Clickhouse connections", "severity": "info", "timestamp": "2024-12-09T07:38:37.876650Z"}
2024-12-09 07:38:37,877 Connecting. Database: default. User: sentry
2024-12-09 07:38:37,877 Connecting to default-sentry-clickhouse-server:9000
2024-12-09 07:38:37,881 Connected to ClickHouse server version 23.10.4, revision: 54466
2024-12-09 07:38:37,890 Starting
2024-12-09 07:38:40,907 New partitions assigned: {Partition(topic=Topic(name='snuba-transactions-commit-log'), index=0): 57550522}
2024-12-09 07:38:40,907 Member id: 'rdkafka-bbfd1d83-5550-4bf6-a459-c94f97e34623'
2024-12-09 07:38:40,907 Initialized processing strategy: <snuba.subscriptions.scheduler_processing_strategy.TickBuffer object at 0x753476927610>
2024-12-09 07:38:40,955 Waited 0.0000 seconds for offsets to be committed to <snuba.subscriptions.scheduler_consumer.CommitLogTickConsumer object at 0x753476c26150>.
2024-12-09 07:38:40,956 Caught exception, shutting down...
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/arroyo/processing/processor.py", line 323, in run
    self._run_once()
  File "/usr/local/lib/python3.11/site-packages/arroyo/processing/processor.py", line 411, in _run_once
    self.__processing_strategy.submit(message)
  File "/usr/src/snuba/snuba/subscriptions/scheduler_processing_strategy.py", line 257, in submit
    self.__buffers[tick_partition].append(message)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 5
2024-12-09 07:38:40,960 Terminating <snuba.subscriptions.scheduler_processing_strategy.TickBuffer object at 0x753476927610>...
2024-12-09 07:38:40,960 Closing <snuba.subscriptions.scheduler_consumer.CommitLogTickConsumer object at 0x753476c26150>...
2024-12-09 07:38:40,961 Partitions to revoke: [Partition(topic=Topic(name='snuba-transactions-commit-log'), index=0)]
2024-12-09 07:38:40,962 Partition revocation complete.
2024-12-09 07:38:40,964 Processor terminated
Traceback (most recent call last):
  File "/usr/local/bin/snuba", line 33, in <module>
    sys.exit(load_entry_point('snuba', 'console_scripts', 'snuba')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/snuba/snuba/cli/subscriptions_scheduler_executor.py", line 153, in subscriptions_scheduler_executor
    processor.run()
  File "/usr/local/lib/python3.11/site-packages/arroyo/processing/processor.py", line 323, in run
    self._run_once()
  File "/usr/local/lib/python3.11/site-packages/arroyo/processing/processor.py", line 411, in _run_once
    self.__processing_strategy.submit(message)
  File "/usr/src/snuba/snuba/subscriptions/scheduler_processing_strategy.py", line 257, in submit
    self.__buffers[tick_partition].append(message)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 5

[untitaker]

I think this may be a duplicate of #5855 (comment)