allow reading and saving (save_object()
) csv draws with DuckDB to avoid memory explosion
#1041
Labels
feature
New feature or request
For large models (~13K paramemeters) the procedure
Takes an incredible amount of memory.
When loaded into memory, the
cmdstanr_fit
is around ~8Gb, and the operations pick at 54Gb.This could be solved by handling ".csv" draws on-disk using DuckDB technology (toggling it with a parameter?)
Here an example of csv handling by DuckDB.
https://stackoverflow.com/questions/77797976/working-with-large-csv-file-using-duckdb-or-arrow-in-r
The text was updated successfully, but these errors were encountered: