optimize the performance of FlashBert Path for HPU #575
Conversation
Signed-off-by: Liu, Kaixuan <[email protected]>
| @@ -323,19 +323,21 @@ def batch_type(self) -> Union[FlashBatch, PaddedBatch]: | |||
| def embed(self, batch: Union[FlashBatch, PaddedBatch]) -> List[Embedding]: | |||
| if isinstance(batch, PaddedBatch): | |||
| input_lens = batch.attention_mask.cumsum(-1)[:, -1].to(torch.int32) | |||
| max_input_lens = input_lens.max().item() | |||
| max_input_lens = 0 # This value will not be used | |||
There was a problem hiding this comment.
| max_input_lens = 0 # This value will not be used |
NIT
There was a problem hiding this comment.
Hi, sorry , there may be misunderstanding. Here I commented "This value will not be used" means this variable can be any value, but we need to keep it here, as we need to pass it to L352
There was a problem hiding this comment.
@Narsil , can you help double check?
There was a problem hiding this comment.
I guess there are cases where the forward of the model does need a right value for this right? Otherwise why not removing it there?
There was a problem hiding this comment.
Well, this is a common file shared by CPU/XPU andd HPU devices. On CPU/XPU, we do need this variable with exact meaning, while on HPU, we do not have real varlen_attention API, so we pass attn_mask to replace its functionality. Here we just need to set a random value for max_input_lens. This line cannot be deleted, as we need to pass it to L352.
Use
WhereIsAI/UAE-Large-V1model to do benchmark, below is the throughput(seq/s) comparison: