-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ClearML serving design v2 #17
Comments
@bmartinn nice works.
|
I'm hoping that late next week I will be able to push a new dev branch with new code to play around with
Good point on latency, and obviously this is by choice. The feedback we received, and feel free to add some more, is that pre/post python callbacks are really necessary for a lot of use cases. We opted for this design as it allows users to very easily add pre/post python functions and still use serving engines for the model inference heavy lifting. A good example for preprocessing function would be if the input is a
Yes this exactly what we have in mind. Specifically in the diagram the "3rd party plugin" would be an integration to model drift / anomaly detection, either running on the same machine, or sent to an external service.
😍
Yes that would be the easiest out of the box solution, I think the major drawback is the ability to easily configure it in realtime?!
Feel free to post a link to a gir repo / snippet :)
I might have failed to illustrate it in the diagram, the idea is Not to store another copy, just reference the Model entity in
Right now we use additional Tasks to do that, every time the "sidecar" spins, it creates a new Task this identifies the serving engine itself (and thus we can store metrics on the serving engine instance performance). wdyt? |
Expanding the original post
|
@bmartinn here's the work I am doing which focuses on the "clearml-sidecar". I have termed it "triton-proxy" as I am focusing on triton as the serving engine. https://github.com/okyspace/clearml-serving/tree/triton_proxy/clearml_serving |
@okyspace Quick update, things are finally wrapping up, here is the latest branch: Next main step is getting the statistics merged as well :) |
@bmartinn Thanks for the new design for ClearML serving. |
Perfect! Thanks @jkhenning! much appreciated. |
ClearML serving design document v2.0
Goal: Create a simple interface to serve multiple models with scalable serving engines on top of Kubernetes
Design Diagram (edit here)
Features
Modules
Usage Example
The text was updated successfully, but these errors were encountered: