test: intermittent failures from vllm tests on LSF cluster

[planetf1]

I'm seeing intermittent failures from vllm tests on lsf cluster when run with

uv run --all-extras --all-groups pytest --isolate-heavy -v

For example:

==== 723 passed, 142 skipped, 2 xfailed, 90 warnings in 1572.83s (0:26:12) =====

when all worked well, and

FAILED test/backends/test_openai_vllm.py::test_instruct - openai.NotFoundErro...
FAILED test/backends/test_openai_vllm.py::test_multiturn - openai.NotFoundErr...
FAILED test/backends/test_openai_vllm.py::test_chat - openai.NotFoundError: E...
FAILED test/backends/test_openai_vllm.py::test_chat_stream - openai.NotFoundE...
FAILED test/backends/test_openai_vllm.py::test_format - openai.NotFoundError:...
FAILED test/backends/test_openai_vllm.py::test_generate_from_raw - openai.Not...
FAILED test/backends/test_openai_vllm.py::test_generate_from_raw_with_format
= 7 failed, 716 passed, 142 skipped, 2 xfailed, 90 warnings in 1409.38s (0:23:29) =

at other times.

Success seems about 50-75% failure from running multiple times

On further investigation the underlying error for all these cases is:

E               openai.NotFoundError: Error code: 404 - {'error': {'message': 'The model `ibm-granite/granite-4.0-micro` does not exist.', 'type': 'NotFoundError', 'param': None, 'code': 404}}

Question to persue -- How is the vllm server initialized when tests are run with uv on a GPU enabled cluster - clearly sometimes we get access to a vllm environment with the right model, othertimes we don't

[planetf1]

Analysis from logs:

The vLLM server is reachable but serving the wrong model.

Root cause: port 8000 collision

The GPU test isolation framework runs test_vllm.py in a separate subprocess before the main session. That test starts a vLLM server on port 8000 serving QWEN3_0_6B. When the subprocess exits, the vLLM server process may not be fully torn down.

When the main session reaches test_openai_vllm.py, its vllm_process fixture tries to start a new server (vllm serve ibm-granite/granite-4.0-micro) on the same port 8000. This new process fails to bind (port taken), but the fixture's readiness check (GET http://127.0.0.1:8000/ping) succeeds against the old Qwen server. The fixture therefore does not skip — it yields. All API calls then hit the Qwen server, which returns 404 for the Granite model ID.

Evidence

• test_vllm.py fixture uses model_ids.QWEN3_0_6B on port 8000
• test_openai_vllm.py fixture uses IBM_GRANITE_4_MICRO_3B.hf_model_name (ibm-granite/granite-4.0-micro) on port 8000
• Tests fail (not skip), meaning the health check passed — server was already up
• Error is 404 model-not-found, not a connection error

Possible fixes

1. Use a different port in test_openai_vllm.py (e.g. 8001) to avoid the collision.
2. Verify the loaded model in the vllm_process fixture by querying /v1/models after the ping check and asserting the expected model ID is present — this would cause a skip rather than a spurious pass-through.
3. Ensure full teardown of the vLLM process tree in the GPU isolation framework before returning to the main session.

[planetf1]

We can also consider how to manage vllm more generally, and our strategy for test execution