Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a triton backend to kernel bench. To invoke it simply add backend="triton" to the following 4 scripts (use them as normal otherwise)
This PR also adds a
{error_type}_name
into the eval json. The reason for this is that it makes classifying errors (especially for triton) much easier. For example, from the error log it isn't obvious what an error is (ie. you might getat 37:15:\n h_start = pooled_row * stride - padding\n w_start = pooled_col * stride - padding\n\n # Initialize the max value\n max_val = tl.full((1,), float('-inf'), tl.float32)\n\n # Itera...
). But if the error name istriton.compiler.errors.UnsupportedLanguageConstruct
it's a lot more obvious.Testing: I've tested the 4 scripts in both the triton and cuda variants and they seem to work normally. (outside of
scripts/generate_and_eval_single_sample_modal.py
which should be equivalent toscripts/generate_and_eval_single_sample.py
)Todo:
Below is the github copilot generated summary which is honestly pretty useful for navigating large PRs.
==========================================================================================
This pull request includes several changes to improve code readability and add new functionality to the
scripts/eval_from_generations.py
andscripts/generate_and_eval_single_sample.py
files. The most notable changes include reformatting code for better readability, adding a new backend configuration option, and enhancing error logging.Code readability improvements:
scripts/eval_from_generations.py
: Reorganized import statements and reformatted multiple lines of code to follow PEP 8 guidelines.scripts/generate_and_eval_single_sample.py
: Reorganized import statements and reformatted multiple lines of code to follow PEP 8 guidelines.New functionality:
scripts/eval_from_generations.py
: Added a new configuration optionbackend
to specify the backend for kernel implementation (cuda or triton).Error logging enhancements:
scripts/eval_from_generations.py
: Enhanced error logging by adding more detailed error messages and including error names in the metadata. [1] [2]Miscellaneous:
scripts/eval_from_generations.py
: Added indentation to JSON output inadd_to_eval_results_file
for better readability.scripts/generate_and_eval_single_sample.py
: Added thebackend
configuration option to theEvalConfig
class.