Add self-detecting on-the-fly bfloat16->float16 conversion pass #741
Conversation
mklimenk
changed the title
mklimenk
force-pushed
the
private/mklimenk/bfloat16_fix_implicit
branch
from
a02a919 to
c594c4d
Compare
mklimenk
changed the title
| @@ -453,6 +465,16 @@ BackendManager::GetModelProtoFromFusedNode(const onnxruntime::Node& fused_node, | |||
| DumpOpenVINOEPModel(onnx_model_path_name, model_proto.get(), fused_node); | |||
| ORT_ENFORCE(status.IsOK(), status.ErrorMessage()); | |||
| return model_proto; | |||
| } else if (HasBf16(subgraph)) { | |||
There was a problem hiding this comment.
Is a check needed for enable_qdq_optimizer . Should you check for GPU here ? Please let me know if you support ep context graphs
There was a problem hiding this comment.
Not necessarily, this is a universal pass, which works for all the IPs.
UPD to an edited comment: this is an else condition for all the qdq_scales-related graph modifications. Overall, qdq_scales and bfloat16 are mutually exclusive, so the current logic is the following: if qdq_scaling pass is requested, we go there with two different paths for NPU and GPU. Else, if the model has bfloat16 initializers, we convert them to fp16 in this pass. Otherwise, we just transfer the model directly to openvino.
Regarding ep context graphs: no, they're not supported, since they're basically an encapsulated OVIR and we can only redirect it to OV, nothing more. So if there is a request from a customer to work with bfloat16 ep context models, we'd need to solve it on the OV side.
|
Please update branch. |
|
@mklimenk |
A follow-up to #740 with changed logic. Instead of relying on an external configuration key, perform bfloat16->float16 conversion in case there is at least one tensor in bfloat16 in the model.
https://jira.devtools.intel.com/browse/CVS-170592