FEL Complexity Misclassification Due to Metadata-Based Analysis

[teymourb]

Description

For Full Enhancement Layer (FEL) complexity determination, HDR10 MaxCLL and Dolby Vision RPU L1 MaxCLL are imperfect representatives of the HDR10 Base Layer (BL) peak luminance and the HDR12 (BL+FEL) peak luminance, respectively. This can lead to false negatives (Complex detected as Simple, i.e., brightness expansion not detected when it should be) and false positives (Simple detected as Complex, i.e., nonexistent brightness expansion detected).

Steps to Reproduce

1. Run the inspect command on the Nosferatu (2024) 4K Blu-ray remux.
2. Run the inspect command on the Jack Reacher (2012) 4K Blu-ray remux.

Expected Behavior

Nosferatu should be detected as Complex when inspected. Comparing BL vs. BL+FEL screenshots shows the brightness expansion.
Jack Reacher should be detected as Complex when inspected. Comparing BL vs. BL+FEL screenshots shows the brightness expansion.

Actual Behavior

Nosferatu is detected as Simple when inspected.
Jack Reacher is detected as Simple when inspected. This is because the maximum DV L1 MaxCLL is lower than the actual peak luminance found using madVR, and more importantly ends up being lower than the peak luminance value decided for the BL (here the default 1000 nits was used since HDR10 MaxCLL is missing).

Suggestions

An accurate reading of frame luminance, instead of relying on metadata, should be used. MadVR is a popular option used in DoVi_Scripts.

To avoid using intensive methods when not necessary, one approach would be to initially rely on the DV L1 MaxCLL metadata for BL+FEL luminance, while using madVR for HDR10 BL luminance, to filter out false positives (Simple detected as Complex). With this approach, a title detected as Complex is guaranteed to be Complex, under the assumption that DV L1 MaxCLL is always equal to or lower than the actual maximum luminance of the frame or scene. This assumption appears to hold for all titles I have examined.

If the title is found to be Simple at this stage, madVR should then be used to calculate the peak luminance of the BL+FEL and perform a second examination. A luminance difference tolerance level (e.g., 1%) should likely be considered when determining whether a title is Simple or not. Since the purpose of Simple vs. Complex determination is to know whether the FEL can be safely discarded when creating a Profile 8 file, it is often difficult to state with certainty that a title does not expand brightness, while it is more often possible to state with certainty that it does expand brightness.

[cryptochrome]

Hey,

thanks for your thoughts. You hit the nail on the head. Relying purely on metadata (which can be wrong, or missing) is a massive compromise.

I have been thinking about this for some time. Unfortunately, dovi_script's approach is not something I can easily replicate. Tools like MadVR are Windows only, and not open source, so I won't/can't use them.

I have been thinking about an approach where I look at the RPU, find 3-5 of the highest peaks, and then use ffmpeg to analyze the BL luminance at the same timestamp (instead of looking at the reported maxCLL metadata). This should, in theory, at least rule out the false Simple FELs. If we assume something like:

IF RPU L1 = 2500 nits (at timestamp x) AND measured BL luminance (same timestamp) = 600 nits THEN verdict = complex.

The BL measurement would be done with ffmpeg signalstat(looking for YMAX).

Any thoughts on this?

I am aware this would still leave us with the problem of potentially unreliable RPU L1 metadata, but I'd rather have some false complex positives than false simple positives.

[teymourb]

This should, in theory, at least rule out the false Simple FELs.

That is not accurate. What the suggested method can do is ensure that an initial Complex verdict is actually a Complex file without the need for analysis of the BL+FEL frame.

IF RPU L1 = 2500 nits (at timestamp x) AND measured BL luminance (same timestamp) = 600 nits THEN verdict = complex. And you can be sure that the title is Complex because we assume that the real peak luminance of BL+FEL is equal or higher to RPU L1 MaxCLL, so it will always be higher than 600 nits. That assumption seems to be true as the Dolby Vision L1 analysis is more conservative.

However, IF RPU L1 = 550 nits (at timestamp x) AND measured BL luminance (same timestamp) = 600 nits THEN verdict = simple. But you cannot be sure that the title is Simple because the real peak luminance or BL+FEL can be and is often higher than the RPU L1 MaxCLL, so it could be higher than 600 nits, in which case the verdict would be wrong, i.e. false negative (Complex detected as Simple).

So that is all to say that avoiding real measurement of the BL+FEL is only useful in the case that a Complex verdict is already found when using RPU L1 MaxCLL metadata with real peak luminance of BL. So it doesn't help you to find Simple FEL titles, only helps to more easily classify some Complex ones.

The BL measurement would be done with ffmpeg signalstat(looking for YMAX).

I don't know much about this tool but as long as you test its output with known values (measurements in DaVinci Resolve or madVR) and it produces accurate results, then it should be good.

I am aware this would still leave us with the problem of potentially unreliable RPU L1 metadata, but I'd rather have some false complex positives than false simple positives.

Yes, but in the proposed solution, you cannot rule out false negatives (Simple detected as Complex) at all which means that you cannot confidently convert any FEL title. An accurate reading of both BL and BL+FEL is necessary to give a Simple verdict. The simplified approach stated above only helps to avoid extra processing in the case of certain Complex verdicts.

Another approach would be to understand whether brightness expansion only occurs in the case where mapping was also applied. From there, there may be some markers that are a clear indicator of an FEL title expanding brightness or not and that could be used. You might at least manage to confidently confirm some more titles as Complex with that method. I personally have not found an obvious pattern but there may be one. I think Nosferatu is a good title to examine because it isn't clear whether mapping was done, yet FEL expands brightness in certain shots even if they are relatively dark.

[cryptochrome]

If only madVR were available for Linux/macOS. If anyone has any ideas on how to tackle this properly, I am all ears. If we have tools that can do that, I can wrap them into dovi_convert.

I'll explore some more, too.

[teymourb]

Here is the DoVi_Scripts author, R3S3t9999, explaining how FEL brightness expansion detection is implemented in their tool: https://forum.doom9.org/showthread.php?p=2027783#post2027783. It doesn’t seem overly intensive. As long as you can validate your implementation against known titles and their classifications and get accurate results, this seems like a solid approach. You can always refine it later as edge cases come up.