The dimension about the Av and Ao

[bnuss98]

Hello, it's very kind and excellent of you to share your work! The logic of this article is very clear, but as a green-hand, I am confused about the Nv and No in Av and Ao. The Nv and No are the numbers of verbs and objects about which dataset or things else? Looking forward to your answer~ ^_^

[zhihou7]

Hi, N_v and N_o are the numbers of verb categories and object categories respectively. For example, on HICO-DET, N_v is 117 and N_o is 80. A_v is the verb-HOI co-occurrence matrix and A_o is the object-HOI co-occurrence matrix. You can see this line of the code:

HOI-CL/lib/networks/tools.py

Line 88 in 7d2b559

def get_convert_matrix(verb_class_num=117, obj_class_num=80):

For example, if the i HOI category is composed by verb j and object k, we have A_v[j][i] == 1 and A_o[k][I] == 1.

Regards,

[bnuss98]

Thanks! I am also confused that lo is the label of an object, is it also an one-hot label? But what is the label dimension for a new object? y = (loAo)&(lvAv), so the lo should be 1*80, but the new object is not in HOI's obj_class.

[bnuss98]

    Sorry to bother you again. In "Visual Compositional Learning for Human-Object Interaction Detection", according to my own understanding, I calculated according to equation (1) and (2), but it seems that only the composited HOIs which have appeared in the list of datasets' HOIs will be valid. For example, if the composited HOI is 'play basketball', it will be valid only when there is an annotation of 'play basketball' in the list of datasets' HOIs. But if there has 'play baseball' in the datasets' annotations, and there is no 'play basketball',  the composited HOI of 'play basketball' will be invalid, right? Could you please tell me about it in detail? Thank you very much !

…

------------------ 原始邮件 ------------------ 发件人: "zhihou7/HOI-CL" ***@***.***>; 发送时间: 2021年5月24日(星期一) 下午3:12 ***@***.***>; ***@***.******@***.***>; 主题: Re: [zhihou7/HOI-CL] The dimension about the Av and Ao (#4) Hi, N_v and N_o are the numbers of verb categories and object categories respectively. For example, on HICO-DET, N_v is 117 and N_o is 80. A_v is the verb-HOI co-occurrence matrix and A_o is the object-HOI co-occurrence matrix. You can see this line of the code: https://github.com/zhihou7/HOI-CL/blob/7d2b55947f8e5a75d32345508f70c929c8d4e8d1/lib/networks/tools.py#L88 For example, if the i HOI category is composed by verb j and object k, we have A_v[j][i] == 1 and A_o[k][I] == 1. Regards, — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[zhihou7]

Thanks! I am also confused that lo is the label of an object, is it also an one-hot label? But what is the label dimension for a new object? y = (loAo)&(lvAv), so the lo should be 1*80, but the new object is not in HOI's obj_class.

Currently, we only consider the objects whose categories are the same or a subset of object categories of HOI dataset. Sorry for getting you latter. I did not receive the email notification.

[zhihou7]

Yes, it is right. Thanks for your nice question!

However, you can simply set the co-occurrence matrix by hand to enable the composite HOI of 'play basketball'. Then, the model can learn 'play basketball' although there are no 'play basketball' instance in HICO-DET. Well, I guess this also answer your question about composing HOIs with novel types of objects. In fact, our next work also address your problem. we'll release it once it is accepted.

regards,

[bnuss98]

thank you^-^

…

---Original--- From: ***@***.***> Date: Wed, May 26, 2021 22:15 PM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [zhihou7/HOI-CL] The dimension about the Av and Ao (#4) Yes, it is right. Thanks for your nice question! However, you can simply set the co-occurrence matrix by hand to enable the composite HOI of 'play basketball'. Then, the model can learn 'play basketball' although there are no 'play basketball' instance in HICO-DET. regards, — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[rouge012]

Yes, it is right. Thanks for your nice question!

However, you can simply set the co-occurrence matrix by hand to enable the composite HOI of 'play basketball'. Then, the model can learn 'play basketball' although there are no 'play basketball' instance in HICO-DET. Well, I guess this also answer your question about composing HOIs with novel types of objects. In fact, our next work also address your problem. we'll release it once it is accepted.

regards,

Hi! How can I set the co-occurrence matrix?

[zhihou7]

hi @rouge012,
the co-occurrence matrix $A\in R^{N_v \times N_o} $ is a two dimension matrix, where $N_v$ indicates the length of verb categories and $N_o$ indicate the length of object categories. We can initialize $A$ as a zero matrix. For each object, there are annotated verbs. We can set the corresponding position of the matrix $A$ as 1. For each example, if the apple is combinable with "eat", "cut" in the dataset, we set corresponding position of and in $A$ as 1.

Feel free to post if you have further questions

Regards,

[rouge012]

hi @rouge012, the co-occurrence matrix $A\in R^{N_v \times N_o} $ is a two dimension matrix, where Nv indicates the length of verb categories and No indicate the length of object categories. We can initialize A as a zero matrix. For each object, there are annotated verbs. We can set the corresponding position of the matrix A as 1. For each example, if the apple is combinable with "eat", "cut" in the dataset, we set corresponding position of and in A as 1.

Feel free to post if you have further questions

Regards,

Thanks for the quick and detailed clarification, and I am wondering where I can find the code for setting the co-occurrence matrix. Thank you!

[zhihou7]

Hi @rouge012,

I have posted in #25. If you need to generate a matrix by yourself, you can just open the three files "https://github.com/zhihou7/HOI-CL/blob/master/misc/hico_list_hoi.txt", "https://github.com/zhihou7/HOI-CL/blob/master/misc/hico_list_obj.txt", "https://github.com/zhihou7/HOI-CL/blob/master/misc/hico_list_vb.txt", and link them together according to the object name and verb name. I think it requires just several lines of code.