Checkpoints

Checkpoints.jl allows packages to register checkpoints which can serialize objects to disk during the execution of an application program, if the application program configures them.

A minimal working example consists of the package:

module MyPackage

using Checkpoints

MODULE = "MyPackage"

__init__() = Checkpoints.register(MODULE, ["foo", ])

function foo(x)
    with_checkpoint_tags(:foo1 => 1, :foo2 => 2) do
        checkpoint(MODULE, "foo", :data => 2x)
    end
    return 2x
end

end

and the application program:

using Checkpoints

Checkpoints.config("MyPackage.foo", "./path/to/checkpoints")

for i in 1:2
    with_checkpoint_tags(:iteration => i) do
        MyPackage.foo(1.0)
    end
end

which results in recorded checkpoints at

./path/to/checkpoints/iteration=1/foo1=1/foo2=2/MyPackage/foo.jlso
./path/to/checkpoints/iteration=2/foo1=1/foo2=2/MyPackage/foo.jlso

You can use index_checkpoint_files to get an index of the files, which is a Tables.jl table and so can e.g. be passed to DataFrame (and then you can do things like groupby etc):

julia> using DataFrames

julia> DataFrame(index_checkpoint_files("./path/to/checkpoints/"))
2×6 DataFrame
 Row │ prefixes        checkpoint_name  iteration   foo1        foo2        checkpoint_path
     │ Tuple…          SubString…       SubString…  SubString…  SubString…  PosixPath…
─────┼────────────────────────────────────────────────────────────────────────────────────────────────────────
   1 │ ("MyPackage",)  foo              1           1           2           ./path/to/checkpoints/iteration=…
   2 │ ("MyPackage",)  foo              2           1           2           ./path/to/checkpoints/iteration=…

or worked with directly:

julia> [checkpoint_path(out) for out in index_checkpoint_files("./path/to/checkpoints/") if out.iteration=="1"]
1-element Array{FilePathsBase.PosixPath,1}:
 p"./path/to/checkpoints/iteration=1/foo1=1/foo2=2/MyPackage/foo.jlso"