How to choose the appropriate aggregate sharpness metric

[djgagne]

Thinking about the construction of a summary sharpness metric, we should be thoughtful about how we are comparing the distributions of our chosen measures of sharpness between the predicted and observed image. For a sharpness metric like gradients, we want to reward images that have a similar distribution of gradients to the target image, but we don't care that the gradients are in exactly the same spot. Thus we should use metrics that compare the predicted and observed gradient distributions aggregated across each image or over a reasonably large region of an image.

The most distribution agnostic way to compare two images would be to calculate the empirical CDF of the gradients for both images and then take either the total difference between the two CDFs (similar to CRPS) or the max difference (similar to the K-S test). The max difference would be the most aggressive metric for sharpness.

Would anyone be interested in implementing this or should I take a stab at it?

[zyjux]

This makes a lot of sense to me - as far aggregating over local regions, we could make use of the heatmap code to compute across tiles, although it may be that we want to be able to tune the tile sizes separately for each metric - this would be easy enough to implement, although it might slow the implementation down a little bit (right now, when computing all metrics, it only has to subset the tiles once then compute everything on each tile, where with varying tile sizes we'd need to recreate the tiles for each metric).

Also, I believe that the Histogram of Oriented Gradients (HOG) method that @jsschreck implemented in the gradients module has some of this idea, in that it computes a distribution of gradients for each image and compares those, so it'd be interesting to see that idea expanded to other metrics.