Speed improvement #218
Speed improvement #218
Conversation
|
Hi! |
|
@moinfar sorry for being slow on this. Have you rerun the scpoli tutorials with this PR to check that everything is fine? |
| @@ -76,27 +76,31 @@ def __init__(self, | |||
| self.cell_types = np.stack(self.cell_types).T | |||
| self.cell_types = torch.tensor(self.cell_types, dtype=torch.long) | |||
|
|
|||
| def __getitem__(self, index): | |||
| def __getitems__(self, indices): | |||
There was a problem hiding this comment.
Why introduce a new name here? __getitem__ in this implementation simply calls this method. Maybe it is better not to duplicate it in such case. indices can simply be a Union[int, Collection[int]]
There was a problem hiding this comment.
The core implementation is not duplicated. __getitem__ now calls __getitems__ which accepts both list indices and single indices.
The reason that we have to maintain both interfaces is that pytorch data loader originally uses __getitem__. But if you also provide __getitems__, then it will use it, and we benefit from the parallel fetching of samples. I am not sure about the newer versions of pytorch but the one scArches is using does not automatically fetch multiple samples if __getitems__ interface is not available.
You can also check this issue in pytorch repo.
There was a problem hiding this comment.
I see, thanks! Didn't know it about pytorch
Speed improvement in:
MultiConditionAnnotatedDatasetrather than cell-by-cell and then combining them incustom_collate. This may result in significant dataloader speedup, especially when loading sparse data.