How does the training of gligen work?

[Putzzmunta]

Hi everybody,
I have some hard time understanding how and why the training script of the gligen example works.

In the StableDiffusionGLIGENTextImagePipeline there are the following attributes listed for the cross attention:

{
            "boxes": boxes,
            "masks": masks,
            "phrases_masks": phrases_masks,
            "image_masks": image_masks,
            "phrases_embeddings": phrases_embeddings,
            "image_embeddings": image_embeddings,
}

Whereas in the `train_gligen_text.py the following ones are listed:

cross_attention_kwargs["gligen"] = {
                    "boxes": batch["boxes"],
                    "positive_embeddings": batch["text_embeddings_before_projection"],
                    "masks": batch["masks"],
}

How is this working, when the attributes are different?

And then i have another question:
Since I have a custom dataset I needed some adjustments in the example code. There I stumbled over the way the gligen-phrases get encoded.
I wrote my own custom method, that i called within the dataset getItem method to encode the phrases for each box:

def encode_gligen_phrases(self, gligen_phrases, text_encoder, tokenizer, max_boxes_per_data=10):
        result = torch.zeros(max_boxes_per_data, 768)
        for i, phrase in enumerate(gligen_phrases):
            if len(phrase) > 0:
                #from the orginal gligen code.
                # inputs = tokenizer(phrase, return_tensors="pt", padding=True)
                # outputs = text_encoder(**inputs)
                # feature = outputs.pooler_output.squeeze(0)
                # result[i] = feature
                with torch.no_grad():
                    caption_tokenized = tokenizer(
                    phrase,
                    padding="max_length",
                    max_length=tokenizer.model_max_length,
                    truncation=True,
                    return_tensors="pt",
                )
                    caption_tokenized = {k: v.cpu() for k, v in caption_tokenized.items()}
                    text_embeddings_before_projection = text_encoder(**caption_tokenized).pooler_output.squeeze(0).cpu()
                result[i] = text_embeddings_before_projection
        return result

I don't understand why there is a difference in how the gligen pipeline does this encoding and how its done for the example project.

Thank you all in advance :)