include inference time in performance report only when not-mocked #1572
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dafnapension
force-pushed
the
polish_performance
branch
from
c988f5e to
fe4dd35
Compare
dafnapension
changed the title
dafnapension
force-pushed
the
polish_performance
branch
3 times, most recently
from
94bc4ec to
7ca3758
Compare
…cause inference is mocked, and evaluation is not indicative if runs over mocked predictions Signed-off-by: dafnapension <[email protected]>
… mocked Signed-off-by: dafnapension <[email protected]>
* minor bug fixes * adding anls * Added LMMS_eval template, exact_match_mm metric, RGB image augmentor. * Added qa.with_context_multiple choice lmms_eval template * Added relaxed_correctness metric, edited info_vqa card, edit lmms_eval template * minor script changes * minor run changes * Added default template * Added lmms-lab version of chartqa with default template and relaxed correctness metric and with_type task * Added lmms-lab card of info_vqa. added default template to info_vqa and docvqa * Wevsrc new with_domain task, new metric. WIP * websrc working * discarding not important changes from main * workaround for llama_vision with WML * WML vs local UNITXT reproductions * Added llama vision benchmark * Changes for commit * fix unitxt typo * ruff * Update inference engine model and adjust expected targets in tests Signed-off-by: elronbandel <[email protected]> * Fix WML Inference Engine tests for images Signed-off-by: elronbandel <[email protected]> * Update tests Signed-off-by: elronbandel <[email protected]> * Enhance CSV loader with low_memory option and update inference engine tests for set equality Signed-off-by: elronbandel <[email protected]> * Add error handling in Loader class to raise UnitxtError on load_iterables failure Signed-off-by: elronbandel <[email protected]> --------- Signed-off-by: elronbandel <[email protected]>
Signed-off-by: dafnapension <[email protected]>
Signed-off-by: dafnapension <[email protected]>
dafnapension
force-pushed
the
polish_performance
branch
from
7ca3758 to
4d6db03
Compare
|
covered by #1600 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
because inference is mocked, and evaluation is not indicative if runs over mocked predictions