Séminaire IMAGE : « Navigating the Evaluation Landscape: From Scalar Metrics to Parametrized Curves in Generative Model Assessment » (Benjamin Sykes)

While it seems fairly easy to evaluate the « quality » of a generative model’s output, finding a metric that intrinsically quantifies the performance of the model remains a difficult task, let alone defining what a « performant generative model » in fact means.

Whereas most generative models outputs are still evaluated with some scalar values such as Fréchet Inception Distance (FID) or Inception Score (IS), in the last years (Sajjadi et al., 2018) proposed a definition of precision and recall for distributions (PRD) with an interpretation as a precision-recall curve, therefore transposing a concept already used and accepted in binary classification. Since then, various different approaches to precision and recall have seen the light (Kynkaanniemi et al., 2019), (Park et al., 2023), (Naeem et al., 2020). While those implementations of precision and recall tend to jointly simplify the computations, give an intuition of the algorithms and aim at reducing the effects of outliers, they do have noticeable short-comings such as the fact they are based solely on some intuition and they overlook the initial and mathematically grounded definition of precision and recall as a trade-off curve.

After presenting the generative model evaluation landscape by presenting the existing evaluation metrics (FID, IS, PRD, Improved Precision Recall, Density Coverage). We will then motivate the use of precision and recall as parametrised curves as opposed to two scalar metrics.  We will finally present our unifying setting in which we extend common scalar PR metrics to PRD curves using the binary classification approach from (Simon et al., 2019).