Refactor barcodeBERT tokenizer: move to dataset and fix performance issue #34
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…erge_with_BarcodeBERT_5M
Add bioscan_bert_checkpoint_trained_with_canada_1_5_m and bioscan_bert_checkpoint_trained_with_bioscan_5_m
correct max_length of tokenizer
…o .hidden_states[-1].mean(dim=1). This commit is for debugging.
…erge_with_BarcodeBERT_5M
…erge_with_BarcodeBERT_5M
…erge_with_BarcodeBERT_5M
…o accept bert-small as valid name.
…factor to also remove that in the code.
…id for ICLR final.
…d token type ids (which is not being used) for the old BarcodeBERT model. Based on the old KmerTokenizer, new one will have attention masks to indicate the pad tokens (just 'N').
…and 1 inference and evaluation epoch.
…e take as a pad token.
…re_merge_with_BarcodeBERT_5M
- Fixed `Dataset` class in dataset.py to support the new BarcodeBERT tokenizer - Modified `__getitem__` method to ensure proper tensor handling when processing DNA input - Removed redundant device movement of DNA inputs in inference_epoch.py - Added wrapper for NewKmerTokenizer to improve compatibility with existing code - Ensured consistent tensor shapes to prevent "storage not resizable" errors during batching
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Refactored the tokenizer integration by moving it from the epoch file to the dataset module for better modularity and maintainability. Also fixed an issue where tokenization was significantly slowing down the data loading process.
Changes
dataset.py