Raising HTTPexception on single request from encode_request or decode_request raise same for all request in batch

[bimarshak7]

🐛 Bug

To Reproduce

Setup litserve api with batch processing (max_batch_size>1) and raise HTTPexception (from fastapi) either in encode_request or decode_request.

Code sample

def decode(self, output):
        if "error" in output:
            raise HTTPException(status_code=500, detail="Error processing file")

        
        return output

Expected behavior

Only the request with error should return 500 and other request should return output as expected.

[bimarshak7]

@Paidinful I changed my code to handle batch inputs. the pipeline includes batching and unbatching in the middle few times and in the process some input may be unprocessable which i remove from batch and continue processing. Even the output passed to decode function is correct. But if any request should be given 500, then all request will get 500 in response

[aniketmaurya]

hey @bimarshak7, LitServe currently process the whole decode - predict - encode pipeline together and failure of a single item leads to failure of all the requests in that batch. Thanks for reporting and we will let you know when we fix this.

[bimarshak7]

@aniketmaurya I am waiting for the update

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[bhimrazy]

Let's keep this open — I believe the issue still exists.

Hi @bimarshak7, would you be interested in opening a PR for this?

[aniketmaurya]

This is already possible! Here is a simple example to showcase how you can do this.

import litserve as ls
from litserve.utils import PickleableHTTPException
import numpy as np

class InferencePipeline(ls.LitAPI):
    def setup(self, device):
        self.model = lambda x: x**2

    def predict(self, requests):
        x = np.asarray([request["input"] for request in requests])
        a = self.model(x)
        c = a + 1
        results = c.tolist()
        for i in range(len(results)):
            if results[i] > 100:
                raise PickleableHTTPException(status_code=400, detail=f"c is too large: {results[i]}")
        return results

if __name__ == "__main__":
    server = ls.LitServer(InferencePipeline(max_batch_size=4))
    server.run(port=8000)

client

import requests

response = requests.post("http://127.0.0.1:8000/predict", json={"input": 4})
print(f"Status: {response.status_code}\nResponse:\n {response.text}")

Output

[bimarshak7]

@aniketmaurya I tried to replicate your exact code. But on the client side, I used python multiprocessing to make parallel requests so that litserve receives batch of request.

import requests
import multiprocessing

def test(n):
    response = requests.post("http://127.0.0.1:8000/predict", json={"input": n})
    return f"Input: {n}\nStatus: {response.status_code}\nResponse:\n{response.text}\n"

if __name__ == "__main__":
    with multiprocessing.Pool(processes=5) as pool:
        results = pool.map(test, [100, 200, 50, 20, 600])
    
    for result in results:
        print(result)

Below is the response i got.
Input: 100
Status: 400
Response:
{"detail":"c is too large: 401"}

Input: 200
Status: 400
Response:
{"detail":"c is too large: 40001"}

Input: 50
Status: 400
Response:
{"detail":"c is too large: 2501"}

Input: 20
Status: 400
Response:
{"detail":"c is too large: 401"}

Input: 600
Status: 400
Response:
{"detail":"c is too large: 360001"}

It works well only when all the input either satisfy failure condition or all the input satisfy success condition.