Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallel synchronization #78

Merged
merged 98 commits into from
Oct 19, 2023
Merged

Parallel synchronization #78

merged 98 commits into from
Oct 19, 2023

Conversation

jeromegn
Copy link
Member

Changes:

  • Sync protocol is now driven by the client. This allows syncing in parallel with multiple nodes and optimizing throughput.
  • Adds OpenTelemetry instrumentation (optional)
  • Many more metrics
  • Uses a patches quinn that does not pre-allocate streams, saving hundreds of MBs of memory
  • Fixes deadlocks when accessing the in-memory bookkeeping
  • Batches "cleared" versions inserts to get at actual "current" versions faster
  • Reduced max foca packet size. TODO: This should be configurable in the future.
  • Fix connection pool locking madness

jeromegn added 30 commits October 6, 2023 17:19
@jeromegn
smaller channel for changes, using too much memory right now
c5ab6dc
@jeromegn
track QUIC connection metrics, in aggregate
47b5c30
@jeromegn
easier to query these metrics if they have the same name but differen…
a61664e 
…t labels
@jeromegn
higher limit of max unidirectional QUIC streams
4320746
@jeromegn
limit number of concurrent syncs to 3
f5d7220
@jeromegn
pool unidirectional streams so we don't create nearly as many
34ee8e8
@jeromegn
don't pool unidirectional send streams
62329e1
@jeromegn
finish unidirectional streams when sending data
c247b3c
@jeromegn
emit tokio runtime metrics
53fd61b
@jeromegn
track cwdn per addr
6a56f11
@jeromegn
track congestion events and blackhols per addr
efddae3
@jeromegn
retry opening a unidirectional stream once if it failed when trying t…
a5747c6 
…o send a unidirectional payload
@jeromegn
split booked versions in their state (cleared, current, partial)
5f49d56
@jeromegn
first part of syncing more efficiently between nodes
06bf45f
@jeromegn
fixed sync mistake
cabceda
@jeromegn
move comment
76501e2
@jeromegn
wip parallel sync
d081764
@jeromegn
hopefully massive improvements on sync speed, especially from scratch
f0bb3ee
@jeromegn
patch upgrade quinn-proto to use less memory
ed895a6
@jeromegn
observe member syncing
c173a7c
@jeromegn
something is weird here, syncing should be a lot faster
59300e5
@jeromegn
hopefully fix tests, add cluster inspection commands
37a7a54
@jeromegn
measure how many sent sync requests
52762b1
@jeromegn
send more often
3438e90
@jeromegn
process completed empties in much smaller batches
359939e
@jeromegn
log a bit more
49ae55d
@jeromegn
send much smaller chunks
e282a72
@jeromegn
possibly faster cleared empties processing
9a714bb
@jeromegn
need to acquire an async read lock here
960afd0
@jeromegn
don't timeout on receive
efbf224
jeromegn added 28 commits October 17, 2023 10:18
@jeromegn
remove span across awaits
a2d8f0e
@jeromegn
propagate trace context when starting sync
2a06e9a
@jeromegn
more instrumentation for sync process, both ways
05e6cd5
@jeromegn
less instrumentation for stuff that happens too often
bf330ee
@jeromegn
try to get rid of more traces
054e241
@jeromegn
log needs len
01875ac
@jeromegn
don't block when sending into empties channel
15cbe87
@jeromegn
seqs start at 0
79091b8
@jeromegn
add a timeout to quic connect
c19d2f2
@jeromegn
warn if sending too large datagram, log its size
7da737a
@jeromegn
also log max datagram size
966aa23
@jeromegn
set a smaller max_packet_size for foca, appears we are limited by pat…
66fd461 
…h mtu
@jeromegn
don't require locking the bookie to get at the registry
2de4bf4
@jeromegn
fix a deadlock
fb2248e
@jeromegn
allow far more unidirectional streams
a4b38c9
@jeromegn
try to simplify reading from unidirectional stream
e8f631b
@jeromegn
use multiple endpoints to parallelize sending data
35d0e19
@jeromegn
use seahash instead of manually modulo-ing to figure out endpoint index
3abf6fc
@jeromegn
restore corro_peer_stream_bytes_recv_total metric
08d5493
@jeromegn
reduce number of max concurrent uni streams
db1bca1
@jeromegn
record gossip recv lag
94adf28
@jeromegn
don't send empty trace states
444c017
@jeromegn
more prometheus histogram buckets for more precision
143ad89
@jeromegn
fix panic, somehow it received a 0-value head version in the sync gen…
f94b0fa 
…erated from another actor, added a log to investigate next time it happens
@jeromegn
update CHANGELOG
04586cc
@jeromegn
upgrade mdbook-admonish
c667075
@jeromegn
probably fix rustflags on CI
ba4b29e
@jeromegn
try sccache instead fo rust-cache
4ca321f
@jeromegn jeromegn merged commit 622c343 into main Oct 19, 2023
2 of 3 checks passed
@jeromegn jeromegn deleted the sync-mem-tweaks branch October 19, 2023 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@jeromegn