Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add rephrasal parser option to run_experiment command #112

Merged
merged 3 commits into from
Nov 26, 2024

Conversation

rchan26
Copy link
Collaborator

@rchan26 rchan26 commented Nov 25, 2024

For #111. Can now run a parser on the rephrasal output by adding a function in src/rephrasal_parser.py and to the PARSER_FUNCTIONS dictionary in that script.

When running an experiment, you can use the --rephrase-parser argument, e.g. to run an experiment with rephraser and judge, you can use:

prompto_run_experiment \
    --file xstest_small.jsonl \
    --max-queries 6 \
    --rephrase-folder rephrase \
    --rephrase-templates template.txt \
    --rephrase-model ollama-mistral \
    --rephrase-parser split_numbered_list \
    --judge-folder judge \
    --judge ollama-llama3-2

split_numbered_list is a simple function to parse a string which has numbered lists. It will create a list of the numbered items, removing quotation marks and things with brackets which are often provided by models to give disclaimers.

@rchan26 rchan26 requested a review from fedenanni November 25, 2024 14:26
@codecov-commenter
Copy link

codecov-commenter commented Nov 25, 2024

Codecov Report

Attention: Patch coverage is 53.12500% with 30 lines in your changes missing coverage. Please review.

Project coverage is 65.10%. Comparing base (52e4a39) to head (53a37ee).

Files with missing lines Patch % Lines
src/prompto/rephrasal_parser.py 27.58% 21 Missing ⚠️
src/prompto/rephrasal.py 71.42% 8 Missing ⚠️
src/prompto/scripts/run_experiment.py 80.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #112      +/-   ##
==========================================
- Coverage   65.51%   65.10%   -0.42%     
==========================================
  Files          41       42       +1     
  Lines        2439     2496      +57     
==========================================
+ Hits         1598     1625      +27     
- Misses        841      871      +30     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.


🚨 Try these New Features:

src/prompto/rephrasal.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@fedenanni fedenanni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rchan26 all looks good - i just left a quick question, but it's all good to go

@rchan26 rchan26 merged commit 61d0e7d into main Nov 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants