Skip to content

Dpatel enable tests on spyre #77

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
Closed

Dpatel enable tests on spyre #77

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
1e2a600
- accept model path,model list, backend type from env variables to pa…
dpatel-ops Jan 30, 2025
faa9ff4
fix typo
dpatel-ops Feb 3, 2025
421fb72
Merge pull request #1 from IBM/main
dpatel-ops Feb 6, 2025
705669e
fix file name in run command
dpatel-ops Feb 10, 2025
59a86da
- update variable names
dpatel-ops Feb 12, 2025
6e3083b
Signed-off-by: Dhruval Patel <[email protected]>
dpatel-ops Feb 15, 2025
df1261a
Signed-off-by: Dhruval Patel <[email protected]>
dpatel-ops Feb 16, 2025
d40d69b
Signed-off-by: Dhruval Patel <[email protected]>
dpatel-ops Feb 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions tests/spyre/test_spyre_basic.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,19 +10,36 @@
generate_spyre_vllm_output)

from vllm import SamplingParams
import os

# get model directory path from env, if not set then default to "/models".
model_dir_path = os.environ.get("SPYRE_TEST_MODEL_DIR", "/models")
# get model backend from env, if not set then default to "eager"
# For multiple values, export SPYRE_TEST_MODEL_DIR="eager,inductor"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# For multiple values, export SPYRE_TEST_MODEL_DIR="eager,inductor"
# For multiple values, export SPYRE_TEST_BACKEND_TYPE="eager,inductor"

backend_type = os.environ.get("SPYRE_TEST_BACKEND_TYPE", "eager")
# get model names from env, if not set then default to "llama-194m"
# For multiple values, export SPYRE_TEST_MODEL_DIR="llama-194m,all-roberta-large-v1"

Check failure on line 21 in tests/spyre/test_spyre_basic.py

View workflow job for this annotation

GitHub Actions / ruff (3.12)

Ruff (E501) 

tests/spyre/test_spyre_basic.py:21:81: E501 Line too long (84 > 80)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# For multiple values, export SPYRE_TEST_MODEL_DIR="llama-194m,all-roberta-large-v1"
# For multiple values, export SPYRE_TEST_MODEL_LIST="llama-194m,all-roberta-large-v1"

user_test_model_list = os.environ.get("SPYRE_TEST_MODEL_LIST","llama-194m")
test_model_list, test_backend_list = [],[]

@pytest.mark.parametrize("model", ["/models/llama-194m"])
for model in user_test_model_list.split(','):
test_model_list.append(f"{model_dir_path}/{model.strip()}")

for backend in backend_type.split(','):
test_backend_list.append(backend.strip())

@pytest.mark.parametrize("model", test_model_list)
@pytest.mark.parametrize("prompts", [[
"Provide a list of instructions for preparing"
" chicken soup for a family of four.", "Hello",
"What is the weather today like?", "Who are you?"
]])
@pytest.mark.parametrize("warmup_shape", [(64, 20, 4), (64, 20, 8),
(128, 20, 4), (128, 20, 8)]
# @pytest.mark.parametrize("warmup_shape", [(64, 20, 1), (128, 20, 1)]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this line be removed?

) # (prompt_length/new_tokens/batch_size)
@pytest.mark.parametrize("backend",
["eager"]) #, "inductor", "sendnn_decoder"])
test_backend_list) #, "inductor", "sendnn_decoder"])
def test_output(
model: str,
prompts: List[str],
Expand Down
19 changes: 17 additions & 2 deletions tests/spyre/test_spyre_embeddings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,24 @@
import pytest
from spyre_util import (compare_embedding_results, spyre_vllm_embeddings,
st_embeddings)
import os
# get model directory path from env, if not set then default to "/models".
model_dir_path = os.environ.get("SPYRE_TEST_MODEL_DIR", "/models")
# get model backend from env, if not set then default to "eager"
# For multiple values, export SPYRE_TEST_MODEL_DIR="eager,inductor"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# For multiple values, export SPYRE_TEST_MODEL_DIR="eager,inductor"
# For multiple values, export SPYRE_TEST_BACKEND_TYPE="eager,inductor"

backend_type = os.environ.get("SPYRE_TEST_BACKEND_TYPE", "eager")
# get model names from env, if not set then default to "llama-194m"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# get model names from env, if not set then default to "llama-194m"
# get model names from env, if not set then default to "all-roberta-large-v1"

# For multiple values, export SPYRE_TEST_MODEL_DIR="llama-194m,all-roberta-large-v1"

Check failure on line 18 in tests/spyre/test_spyre_embeddings.py

View workflow job for this annotation

GitHub Actions / ruff (3.12)

Ruff (E501) 

tests/spyre/test_spyre_embeddings.py:18:81: E501 Line too long (84 > 80)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# For multiple values, export SPYRE_TEST_MODEL_DIR="llama-194m,all-roberta-large-v1"
# For multiple values, export SPYRE_TEST_EMBEDDING_MODEL_LIST="llama-194m,all-roberta-large-v1"

user_test_model_list = os.environ.get("SPYRE_TEST_EMBEDDING_MODEL_LIST","all-roberta-large-v1")

Check failure on line 19 in tests/spyre/test_spyre_embeddings.py

View workflow job for this annotation

GitHub Actions / ruff (3.12)

Ruff (E501) 

tests/spyre/test_spyre_embeddings.py:19:81: E501 Line too long (95 > 80)
test_model_list, test_backend_list = [],[]

for model in user_test_model_list.split(','):
test_model_list.append(f"{model_dir_path}/{model.strip()}")

@pytest.mark.parametrize("model", ["/models/all-roberta-large-v1"])
for backend in backend_type.split(','):
test_backend_list.append(backend.strip())

@pytest.mark.parametrize("model", test_model_list)
@pytest.mark.parametrize("prompts", [[
"The capital of France is Paris."
"Provide a list of instructions for preparing"
Expand All @@ -21,7 +36,7 @@
[(64, 4), (64, 8), (128, 4),
(128, 8)]) # (prompt_length/new_tokens/batch_size)
@pytest.mark.parametrize("backend",
["eager"]) #, "inductor", "sendnn_decoder"])
test_backend_list) #, "inductor", "sendnn_decoder"])
def test_output(
model: str,
prompts: List[str],
Expand Down
23 changes: 21 additions & 2 deletions tests/spyre/test_spyre_max_prompt_length.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,25 @@
from transformers import AutoTokenizer

from vllm import SamplingParams
import os

# get model directory path from env, if not set then default to "/models".
model_dir_path = os.environ.get("SPYRE_TEST_MODEL_DIR", "/models")
# get model backend from env, if not set then default to "eager"
# For multiple values, export SPYRE_TEST_MODEL_DIR="eager,inductor"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# For multiple values, export SPYRE_TEST_MODEL_DIR="eager,inductor"
# For multiple values, export SPYRE_TEST_BACKEND_TYPE="eager,inductor"

backend_type = os.environ.get("SPYRE_TEST_BACKEND_TYPE", "eager")
# get model names from env, if not set then default to "llama-194m"
# For multiple values, export SPYRE_TEST_MODEL_DIR="llama-194m,all-roberta-large-v1"

Check failure on line 22 in tests/spyre/test_spyre_max_prompt_length.py

View workflow job for this annotation

GitHub Actions / ruff (3.12)

Ruff (E501) 

tests/spyre/test_spyre_max_prompt_length.py:22:81: E501 Line too long (84 > 80)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# For multiple values, export SPYRE_TEST_MODEL_DIR="llama-194m,all-roberta-large-v1"
# For multiple values, export SPYRE_TEST_MODEL_LIST="llama-194m,all-roberta-large-v1"

user_test_model_list = os.environ.get("SPYRE_TEST_MODEL_LIST","llama-194m")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a note: you named it SPYRE_TEST_MODEL_LIST because multiple models can be tested, but you named SPYRE_TEST_BACKEND_TYPE and not SPYRE_TEST_BACKEND_TYPE_LIST. Same for the other files.

test_model_list, test_backend_list = [],[]

@pytest.mark.parametrize("model", ["/models/llama-194m"])
for model in user_test_model_list.split(','):
test_model_list.append(f"{model_dir_path}/{model.strip()}")

for backend in backend_type.split(','):
test_backend_list.append(backend.strip())

@pytest.mark.parametrize("model", test_model_list)
@pytest.mark.parametrize("prompts", [
7 * [
"Hello",
Expand All @@ -27,9 +43,12 @@
])
@pytest.mark.parametrize("warmup_shapes",
[[(64, 20, 4)], [(64, 20, 4), (128, 20, 4)]]
# ) # (prompt_length/new_tokens/batch_size)
# @pytest.mark.parametrize("warmup_shapes",
# [[(64, 20, 1)], [(128, 20, 1)]]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as before: should it be removed?

) # (prompt_length/new_tokens/batch_size)
@pytest.mark.parametrize("backend",
["eager"]) #, "inductor", "sendnn_decoder"])
test_backend_list) #, "inductor", "sendnn_decoder"])
def test_output(
model: str,
prompts: List[str],
Expand Down
20 changes: 18 additions & 2 deletions tests/spyre/test_spyre_seed.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,25 @@
from spyre_util import generate_spyre_vllm_output

from vllm import SamplingParams
import os

# get model directory path from env, if not set then default to "/models".
model_dir_path = os.environ.get("SPYRE_TEST_MODEL_DIR", "/models")
# get model backend from env, if not set then default to "eager"
# For multiple values, export SPYRE_TEST_MODEL_DIR="eager,inductor"
backend_type = os.environ.get("SPYRE_TEST_BACKEND_TYPE", "eager")
# get model names from env, if not set then default to "llama-194m"
# For multiple values, export SPYRE_TEST_MODEL_DIR="llama-194m,all-roberta-large-v1"

Check failure on line 21 in tests/spyre/test_spyre_seed.py

View workflow job for this annotation

GitHub Actions / ruff (3.12)

Ruff (E501) 

tests/spyre/test_spyre_seed.py:21:81: E501 Line too long (84 > 80)
user_test_model_list = os.environ.get("SPYRE_TEST_MODEL_LIST","llama-194m")
test_model_list, test_backend_list = [],[]

@pytest.mark.parametrize("model", ["/models/llama-194m"])
for model in user_test_model_list.split(','):
test_model_list.append(f"{model_dir_path}/{model.strip()}")

for backend in backend_type.split(','):
test_backend_list.append(backend.strip())

@pytest.mark.parametrize("model", test_model_list)
@pytest.mark.parametrize("prompt", [
"Provide a list of instructions for preparing"
" chicken soup for a family of four."
Expand All @@ -23,7 +39,7 @@
(128, 20, 4), (128, 20, 8)]
) # (prompt_length/new_tokens/batch_size)
@pytest.mark.parametrize("backend",
["eager"]) #, "inductor", "sendnn_decoder"])
test_backend_list) #, "inductor", "sendnn_decoder"])
def test_seed(
model: str,
prompt: str,
Expand Down
22 changes: 19 additions & 3 deletions tests/spyre/test_spyre_tensor_parallel.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,20 +10,36 @@
generate_spyre_vllm_output)

from vllm import SamplingParams
import os

# get model directory path from env, if not set then default to "/models".
model_dir_path = os.environ.get("SPYRE_TEST_MODEL_DIR", "/models")
# get model backend from env, if not set then default to "eager"
# For multiple values, export SPYRE_TEST_MODEL_DIR="eager,inductor"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# For multiple values, export SPYRE_TEST_MODEL_DIR="eager,inductor"
# For multiple values, export SPYRE_TEST_BACKEND_TYPE="eager,inductor"

backend_type = os.environ.get("SPYRE_TEST_BACKEND_TYPE", "eager")
# get model names from env, if not set then default to "llama-194m"
# For multiple values, export SPYRE_TEST_MODEL_DIR="llama-194m,all-roberta-large-v1"

Check failure on line 21 in tests/spyre/test_spyre_tensor_parallel.py

View workflow job for this annotation

GitHub Actions / ruff (3.12)

Ruff (E501) 

tests/spyre/test_spyre_tensor_parallel.py:21:81: E501 Line too long (84 > 80)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# For multiple values, export SPYRE_TEST_MODEL_DIR="llama-194m,all-roberta-large-v1"
# For multiple values, export SPYRE_TEST_MODEL_LIST="llama-194m,all-roberta-large-v1"

user_test_model_list = os.environ.get("SPYRE_TEST_MODEL_LIST","llama-194m")
test_model_list, test_backend_list = [],[]

@pytest.mark.parametrize("model", ["/models/llama-194m"])
for model in user_test_model_list.split(','):
test_model_list.append(f"{model_dir_path.strip()}/{model.strip()}")

for backend in backend_type.split(','):
test_backend_list.append(backend.strip())

@pytest.mark.parametrize("model", test_model_list)
@pytest.mark.parametrize("prompts", [[
"Provide a list of instructions for preparing"
" chicken soup for a family of four.", "Hello",
"What is the weather today like?", "Who are you?"
]])
@pytest.mark.parametrize("warmup_shapes", [[(64, 20, 4)]]
@pytest.mark.parametrize("warmup_shapes", [[(64, 20, 1)]]
) #,[(64,20,8)],[(128,20,4)],[(128,20,8)]])
# (prompt_length/new_tokens/batch_size)
@pytest.mark.parametrize("tp_size", [2])
@pytest.mark.parametrize("backend",
["eager"]) #, "inductor", "sendnn_decoder"])
test_backend_list) #, "inductor", "sendnn_decoder"])
def test_output(
model: str,
prompts: List[str],
Expand Down
22 changes: 19 additions & 3 deletions tests/spyre/test_spyre_warmup_shapes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,25 @@
generate_spyre_vllm_output)

from vllm import SamplingParams
import os

# get model directory path from env, if not set then default to "/models".
model_dir_path = os.environ.get("SPYRE_TEST_MODEL_DIR", "/models")
# get model backend from env, if not set then default to "eager"
# For multiple values, export SPYRE_TEST_MODEL_DIR="eager,inductor"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# For multiple values, export SPYRE_TEST_MODEL_DIR="eager,inductor"
# For multiple values, export SPYRE_TEST_BACKEND_TYPE="eager,inductor"

backend_type = os.environ.get("SPYRE_TEST_BACKEND_TYPE", "eager")
# get model names from env, if not set then default to "llama-194m"
# For multiple values, export SPYRE_TEST_MODEL_DIR="llama-194m,all-roberta-large-v1"

Check failure on line 21 in tests/spyre/test_spyre_warmup_shapes.py

View workflow job for this annotation

GitHub Actions / ruff (3.12)

Ruff (E501) 

tests/spyre/test_spyre_warmup_shapes.py:21:81: E501 Line too long (84 > 80)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# For multiple values, export SPYRE_TEST_MODEL_DIR="llama-194m,all-roberta-large-v1"
# For multiple values, export SPYRE_TEST_MODEL_LIST="llama-194m,all-roberta-large-v1"

user_test_model_list = os.environ.get("SPYRE_TEST_MODEL_LIST","llama-194m")
test_model_list, test_backend_list = [],[]

@pytest.mark.parametrize("model", ["/models/llama-194m"])
for model in user_test_model_list.split(','):
test_model_list.append(f"{model_dir_path.strip()}/{model.strip()}")

for backend in backend_type.split(','):
test_backend_list.append(backend.strip())

@pytest.mark.parametrize("model", test_model_list)
@pytest.mark.parametrize("prompts", [
7 * [
"Hello",
Expand All @@ -25,9 +41,9 @@
]
])
@pytest.mark.parametrize("warmup_shapes", [[(64, 20, 8), (128, 20, 4)]]
) # (prompt_length/new_tokens/batch_size)
) # (prompt_length/new_tokens/batch_size)
@pytest.mark.parametrize("backend",
["eager"]) #, "inductor", "sendnn_decoder"])
test_backend_list) #, "inductor", "sendnn_decoder"])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

open question: should we still keep the #, "inductor", "sendnn_decoder"]) part if we use a list of backends? Maybe we can put that in the header part?

def test_output(
model: str,
prompts: List[str],
Expand Down
Loading