Inquiry Regarding Target Distribution in Model Training

[choidaedae]

Hello,

Thank you for your valuable work!

I have been implementing the training code for your model, but I encountered some unclear aspects in the implementation, so I created this issue.
Since the model was trained using cross-entropy loss, I assume that the softmax output of the model logits should resemble the ground truth (GT) distribution, right?
However, I noticed that the GT distribution used in the training procedure appears to be quite different from the configuration described in your paper.
For instance, when the input image’s orientation is estimated as [0, 90, 90], the model's softmax output is as follows:

Meanwhile, the GT distribution I implemented based on your paper is:

Apart from the polar distribution, there seems to be a significant difference between the model output (after softmax) and the GT distribution. In particular, the azimuth distribution values are very similar to each other, which might not contribute effectively to training.
According to your paper, the target distribution is described as follows:

To summarize:
Azimuth: Circular distribution with std = 20°
Polar: Gaussian distribution with std = 2°
Rotation: Circular distribution with std = 1°.

For azimuth and rotation, the GT distribution I implemented is quite different from the model's output, so I believe I may have implemented it incorrectly. I would like to know what I did wrong.

For your understanding, I share my implementation of the target distribution for reference. Could you provide the corresponding code via email (daehyeonchoi@kaist.ac.kr) or in this issue?

import torch
import numpy as np

def orientation_to_distribution(orientation):
    """
    Generate a Gaussian distribution for the given angle.
    
    Args:
        theta (int): The ground truth angle in degrees.
        sigma (float): Variance for the Gaussian distribution.
        angle_range (int): The maximum angle value (e.g., 180 or 360).
        
    Returns:
        np.ndarray: Discretized probability distribution.
    """

    azimuth, polar, rotation = orientation
    
    polar_sigma = 2.0
    polar_range = 180
    angles = np.arange(1, polar_range + 1)
    polar_distribution = np.exp(-((angles - polar) ** 2) / (2 * polar_sigma ** 2))
    polar_distribution = torch.tensor(polar_distribution / np.sum(polar_distribution), dtype=torch.float32)

    from scipy.special import i0  # Import zero-order modified Bessel function

    azimuth_sigma = 20.0
    azimuth_range = 360
    angles = np.arange(1, azimuth_range + 1)
    azimuth_distribution = np.exp(np.cos(np.radians(angles - azimuth)) / azimuth_sigma ** 2)
    azimuth_distribution  /= (2 * np.pi * i0(1 / azimuth_sigma ** 2))  # Normalize with the Bessel function
    azimuth_distribution = torch.tensor(azimuth_distribution  / np.sum(azimuth_distribution), dtype=torch.float32)

    rotation_sigma = 1.0
    rotation_range = 180
    angles = np.arange(1, rotation_range + 1)
    rotation_distribution = np.exp(np.cos(np.radians(angles - rotation)) / rotation_sigma ** 2)
    rotation_distribution  /= (2 * np.pi * i0(1 / rotation_sigma ** 2))  # Normalize with the Bessel function
    rotation_distribution = torch.tensor(rotation_distribution  / np.sum(rotation_distribution), dtype=torch.float32)

    return azimuth_distribution, polar_distribution, rotation_distribution

Looking forward to your response. Thanks!

Best regards,
Daehyeon

[zhang-ziang]

Hello, here is our implementation of normal distribution and circular normal distribution(von Mises distribution).

import numpy as np
from scipy.stats import vonmises
# normal distribution
mean = 0
sigma = 2
x = np.linspace(-90, 90, 180)
y = (1 / (sigma * np.sqrt(2 * np.pi))) * np.exp(-0.5 * ((x - mean) / sigma) ** 2)

# circular normal distribution
mean =  0
sigma = 20*np.pi/180
kappa = 1/(sigma**2)
x = np.linspace(-np.pi, np.pi, 360)
y = vonmises.pdf(x, kappa, loc=mean)
# then trans angle unit to degree
...

When,
Azimuth: Circular distribution with std = 20°
Polar: Gaussian distribution with std = 2°
Rotation: Circular distribution with std = 1°.
The resulting target distribution is shown in the figure below

Hope this reply can help!

[akhilgvss]

Hey @choidaedae ,

Were you able to recreate the plots visualized by @zhang-ziang ? I’m also facing the same issue: in my case, the ground truth distribution for rotation is very widespread, while the predicted distribution is much narrower.

Could you please share how you managed to correct the distributions? If you have any code for this, I’d appreciate it if you could send it to me.

Below is my ground truth distribution:

And here is my predicted distribution

For the image:

[choidaedae]

Hi @akhilgvss ,
I'm not sure it is the original training procedure, I empirically found that model's prediction among all orientations (azimuth, polar, rotation) is similar with (circular) gaussian distribution, not a circular distribution in authors' mention.

If you change polar and rotation distribution from circular to gaussian, you can see that it aligns well with Model's predicted logit.

Hope that it helps you!

[akhilgvss]

Thank you for the reply, @choidaedae.

I have a question: In the paper, we have the polar angle modelled as a normal (Gaussian) distribution, right? Whereas the rotational and azimuth angles are modelled as circular (Gaussian) distributions.

I used a formula to get the predicted distribution, while in the above function author has mentioned he uses an imported function. Which one did you use, and how did you decide which distribution to use for each angle?

This is how I am building the function using these methods:

def create_polar_target_distribution(center, sigma, num_bins=180):
    bins = np.arange(num_bins)
    unnormalized = np.exp(-(bins - center)**2 / (2 * sigma**2))
    return unnormalized / np.sum(unnormalized)

def create_azimuth_rotational_target_distribution(center, sigma, num_bins=360):
    bins = np.arange(num_bins)
    kappa = 1.0 / (sigma**2)
    diff = np.radians(bins - center)
    
    normalization = 2 * np.pi * i0(kappa)
    
    unnormalized = np.exp(kappa * np.cos(diff))
    return unnormalized / normalization

def compute_cross_entropy(target_dist, pred_dist,EPSILON=1e-15):
    pred_clipped = np.clip(pred_dist, EPSILON, 1.0)
    ce = -np.sum(target_dist * np.log(pred_clipped))
    return ce / len(target_dist)

Thank you.

[choidaedae]

I mean I used same distribution with polar distribution (divided by sum of all values but no zeroth order bessel function in denominator) for all dimension, and I used same variance mentioned in paper.