This paper has problems and there is information leakage.

[Doke543]

This paper has problems. After carefully checking the code, I found that there was information leakage. In the 183rd line of the "evaluate_models_utils.py" file, the random_projections.update function is responsible for adding the samples from the validation set and the test set to the model. However, theoretically, there were no separate positive and negative samples added to the model at this time. That is to say, at this point, the true labels in the validation set and the test set have been leaked to the model for training. If this random_projections.update is removed, the performance of the model in this paper will significantly drop.
It is suggested that future researchers should not make comparisons with this paper. This paper is clearly a case of cheating.

[Doke543]

Based on the UCI datasets under the transductive setting, the AP value before information leakage was 94.87, and after leakage it was 97.35. The difference is quite significant.

[lxd99]

The update is performed in strict chronological order. Specifically, when predicting the t-th batch, we use the representations obtained from the first t−1 batches (i.e., the random projections maintained up to that point) for prediction. After finishing the prediction for the current batch, the random projections are then updated using the new interactions.

This design follows the standard workflow of memory-based temporal graph models, where the model first makes predictions using the current memory and then updates the memory afterward. You can also see this update logic in other memory-based methods (e.g., TGN, JODIE, DyRep)
https://github.com/lxd99/TPNet/blob/main/models/MemoryModel.py#L184-L205

If you are interested in the motivation behind this design, you may find it helpful to look at the original TGN paper. It may also be useful to check the implementations of memory-based methods in other repositories (e.g., TGB or DyGLib), which follow a similar prediction-then-update paradigm.

[Doke543]

Yes, I'm not disputing your temporal ordering, which I find acceptable. However, the main issue is that you're only updating the memory with positive samples, without any negative samples. You're not even using your model's prediction results, but rather the ground truth (batch_src_node_ids, batch_dst_node_ids), which is unacceptable as it constitutes information leakage. Alternatively, you could try using your model's prediction results.

In the 183rd line of the "evaluate_models_utils.py"
“if random_projections is not None:
random_projections.update(src_node_ids=batch_src_node_ids, dst_node_ids=batch_dst_node_ids,
node_interact_times=batch_node_interact_times)
”

[lxd99]

For other memory-based methods, the update mechanism is implemented in the same way. You can refer to the part of the code I mentioned above, especially the section starting with (https://github.com/lxd99/TPNet/blob/main/models/MemoryModel.py#L184-L205) =>"if edges_are_positive:"
where the memory is updated after the prediction step.

If this design is still unclear, it may be helpful to look at the implementations of other memory-based methods and read the original TGN paper.

[Doke543]

I have checked the link you provided to me. I suggest you also take a look at it. Regarding the chronological order, you and I are in agreement. I have never raised any doubts about this. However, regarding the issue of updating the memory using the prediction results or the ground truth (batch_src_node_ids, batch_dst_node_ids), you have different opinions.

Additionally, it can also be said in this way. Even if you have found that the previous literature was used in this way, is it still correct to do so?

”if edges_are_positive:
assert edge_ids is not None
# if the edges are positive, update the memories for source and destination nodes (since now we have new messages for them)
self.update_memories(node_ids=node_ids, node_raw_messages=self.memory_bank.node_raw_messages)

        # clear raw messages for source and destination nodes since we have already updated the memory using them
        self.memory_bank.clear_node_raw_messages(node_ids=node_ids)

        # compute new raw messages for source and destination nodes
        unique_src_node_ids, new_src_node_raw_messages = self.compute_new_node_raw_messages(
            src_node_ids=src_node_ids,
            dst_node_ids=dst_node_ids,
            dst_node_embeddings=dst_node_embeddings,
            node_interact_times=node_interact_times,
            edge_ids=edge_ids)
        unique_dst_node_ids, new_dst_node_raw_messages = self.compute_new_node_raw_messages(
            src_node_ids=dst_node_ids,
            dst_node_ids=src_node_ids,
            dst_node_embeddings=src_node_embeddings,
            node_interact_times=node_interact_times,
            edge_ids=edge_ids)

        # store new raw messages for source and destination nodes
        self.memory_bank.store_node_raw_messages(node_ids=unique_src_node_ids,
                                                 new_node_raw_messages=new_src_node_raw_messages)
        self.memory_bank.store_node_raw_messages(node_ids=unique_dst_node_ids,
                                                 new_node_raw_messages=new_dst_node_raw_messages)

        # updating the temporal walk matrices
        if self.model_name == 'PINT':
            self.matrix_memory.update(src_node_ids=src_node_ids, dst_node_ids=dst_node_ids)“

[lxd99]

This will be my final response to this issue. I do not intend to waste more time arguing about it.

First, your statement in the issue — “It is suggested that future researchers should not make comparisons with this paper. This paper is clearly a case of cheating” — is a very serious accusation. However, judging from your question, it is clear that you do not even understand how memory-based methods update memory during inference. Making such accusations without understanding the basic mechanism only demonstrates a lack of fundamental understanding of the field and a lack of basic academic professionalism. If you are still doing research, I do not believe this attitude will lead to meaningful work. If you genuinely do not understand the design choice, you could simply ask a question for clarification. If I have time, I am willing to explain.

Second, I will explain clearly why the design is made this way. Consider a real-world link prediction system like recommendation system as an example. Suppose a user (u) may listen to a song at time (t+1). At time (t), we do not know which song the user will actually listen to. For example, we might consider two candidate songs (v_1) and (v_2), and the model needs to predict:

$$ [ (u, v_1, t+1),\quad (u, v_2, t+1) ] $$

At time (t), these are only predictions — the system simply assigns scores to candidate interactions.

When time advances to (t+1), we observe the real outcome. Suppose the user actually listened to (v_1). Then ((u, v_1, t+1)) becomes a positive interaction, while ((u, v_2, t+1)) is simply an interaction that did not happen.

The key question is: when updating the memory of the model at t+1, should we feed the model the interactions that actually happened, or also include the negative samples that were only constructed during prediction?

In a real system, only observed interactions exist. Negative samples were merely introduced during prediction for scoring purposes. Therefore, the memory update should only use the actual observed edge ((u, v_1, t+1)), rather than artificially feeding the negative samples back into the system.

This is exactly the motivation behind the design.

If you still believe there is a problem with this design, then you are free to hold your opinion.

[Doke543]

This is a purely academic discussion, and I firmly hold my viewpoint. First of all, you need to carefully read my description. What I'm questioning is not this memory update mechanism; it's a common learning method. Instead, when you were conducting the sample test using the testset, you also performed memory updates. This is actually acceptable. But the most important thing is that when you performed the memory update, you did not use the model's prediction results, but rather naturally used the actual links. This was actually invisible during the testing process. In the field of dynamic networks, making such a setting is undoubtedly unfair to other researchers.

Secondly, my intention in making this statement is actually to provide a reminder, and it is aimed at promoting the healthy development of the community. There is no element of accusation. If anyone feels offended, I sincerely apologize.

I am always open to your academic discussions. I believe that all of us are rational and calm. Errors are inevitable during the research process, and there is no need to avoid them.