updated train for computing per token loss - #40
Open
adamimos wants to merge 1 commit into
Open
Conversation
had to explicitly calculate loss for this
Collaborator
|
I'm confused. Why are we calculating loss per token if we're not logging the value? |
Owner
Author
|
The main thing I was thinking is that this was the more general thing, that would work for cases where we did or did not want to log by token. I was going to implement the logging by token in a different pull request. A secondary issue was that the way transformerlens deals with loss is to not compute it on the final token since there's no data for what that prediction should be. So I at least wanted to manually do cross-entropy loss for the logits in order to get that (but this is a seperate issue for the per token thing). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
had to explicitly calculate loss for this