Ranking plot making job stability

[Tomoya-Iizawa]

Hi,

I am making ranking plots, but the job to make a runking plot is not so stable, it fails frequently.
For example, if I run with systematic sources 1-10 or 11-20, the job works, but if I run with all 1-20, the job fails.
If I change the fit range, it helps sometimes, but I would like to include far more sources > 100.
(On the other hand, the job to make a pull plot works far more stable)

Is there any reason of instability, and is there any way to improve the stability?

Best Regards,
Tomoya

[alexander-held]

Hi Tomoya, the current ranking interface unfortunately does not provide too much control. Just to make sure I understand the setup correctly:

run with systematic sources 1-10 or 11-20

means presumably that you only add some systematics to your workspace? Or are you dynamically disabling them by setting the parameters to constant? Or do you have some other method here?

the job fails

I guess the part that fails are the fits here? Unfortunately there is currently no protection built-in against this. That is something that needs to be improved, the easiest solution might be to catch a failed fit and to just return a NaN impact when that happens. Not sure how to best visualize that on the related figure, ideas are welcome there. If failed fits are the problem, I can put together a branch to circumvent the crash.

If you're looking to do this with >100 parameters, the next thing that needs to be improved is parallelization: currently all fits run sequentially, which will become very slow. I think a solution similar to #421 makes sense. This requires a bit of work (probably not all that much actually) and there is a bit of a backlog of open pull request to get through, so it might take me a bit to get to implementing this (contributions are very welcome though if anyone wants to give this a try).

[Tomoya-Iizawa]

and is there any way to set the number of sources shown in the ranking plot?

[Tomoya-Iizawa]

maybe fit unstability is due to bad modeling... I need to confirm further, but sorry for bothering you.
anyways, parallelization is still interesting for me, please let me know when it is ready, I would like to try with my fit model.

[Tomoya-Iizawa]

Hi,
is there any way to set the number of sources shown in the ranking plot?

and is there any way to select sources shown in the correlation plot? for example, I would like to remove stat error from the correlation plot.

[alexander-held]

is there any way to set the number of sources shown in the ranking plot?

Yes, the max_pars option does that, see https://cabinetry.readthedocs.io/en/latest/api.html#cabinetry.visualize.ranking.

is there any way to select sources shown in the correlation plot?

Not currently. #342 adds something like this for pulls, which could also be applied to correlations. At the moment the way to accomplish this is to pass a modified FitResults object into visualize.correlation_matrix. Here's an example:

import numpy as np

# assumes you have some object fit_results with your fit results
# remove all parameters with "staterror" in the name
delete_indices = [idx for idx, label in enumerate(fit_results.labels) if "staterror" in label]

fit_results_pruned = cabinetry.fit.FitResults(
    np.delete(fit_results.bestfit, delete_indices),
    np.delete(fit_results.uncertainty, delete_indices),
    np.delete(fit_results.labels, delete_indices),
    np.delete(np.delete(fit_results.corr_mat, delete_indices, axis=1), delete_indices, axis=0),
    fit_results.best_twice_nll,
    fit_results.goodness_of_fit,
    fit_results.minos_uncertainty,
)

cabinetry.visualize.correlation_matrix(fit_results_pruned)

parallelization is still interesting for me, please let me know when it is ready, I would like to try with my fit model.

I just created #442 to track this. It would be great if you could subscribe to that issue. Once something is available it would also be great if you could give it a try! That still might take a bit of time unfortunately.