Implement a highly available metadata store

[tillrohrmann]

To make Restate fully highly available, we also need the metadata store to be highly available. We either find a kv store that provides us with linearizable reads and writes or we need to build it ourselves.

Tasks

Preview Give feedback

1. External metadata store #1829
   +0
   
2. Raft-based embedded metadata store #1831
   0 of 7
   +0
3. Object store based metadata store #1882
   +0
   
4. Etcd external metadata store assumes aligned versions which makes it broken #2204
   bug +1
5. Document how to configure object store based metadata store #2379
   documentation +1
6. Bootstrap embedded metadata store #2408
   1 of 7
   +0
   

[tillrohrmann]

DynamoDB does not seem to work as a geo replicated highly consistent external meta data store. The problem is that there is no synchronous replication between table running in different regions (https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/V2globaltables_HowItWorks.html).

[tillrohrmann]

What might work for an external metadata store is Google Cloud Spanner (not working for AWS), Azure Cosmos DB with strongest consistency settings (not working for AWS) (https://learn.microsoft.com/en-us/azure/cosmos-db/consistency-levels) or CockroachDB (https://cockroachlabs.cloud/).