Skip to content

Commit ead15b3

Browse files
mzeglamzegla
authored
Initial Go client sample (openvinotoolkit#1014)
1 parent 3b9f4f4 commit ead15b3

File tree

13 files changed

+1778
-1
lines changed

13 files changed

+1778
-1
lines changed

‎example_client/go/Dockerfile

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
#
2+
# Copyright (c) 2021 Intel Corporation
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
#
16+
17+
FROM golang:latest
18+
ARG http_proxy
19+
20+
RUN echo "Acquire::http::Proxy \"$http_proxy\";" > /etc/apt/apt.conf.d/proxy.conf
21+
RUN apt-get update && \
22+
apt-get -y install git unzip build-essential autoconf libtool protobuf-compiler libprotobuf-dev
23+
RUN go get google.golang.org/grpc
24+
RUN go get github.com/golang/protobuf/protoc-gen-go
25+
26+
# Install Go OpenCV (to simplify postprocessing)
27+
RUN apt-get install -y sudo && \
28+
git clone https://github.com/hybridgroup/gocv.git && \
29+
cd gocv && \
30+
make install
31+
32+
RUN mkdir /app
33+
COPY . /app
34+
WORKDIR /app
35+
36+
# Compile API
37+
RUN protoc -I apis/ apis/tensorflow_serving/apis/*.proto --go_out=plugins=grpc:.
38+
RUN protoc -I apis/ apis/tensorflow/core/framework/*.proto --go_out=plugins=grpc:.
39+
40+
# Move compiled protos under GOROOT
41+
RUN mv tensorflow /usr/local/go/src/
42+
RUN mv tensorflow_serving /usr/local/go/src/
43+
44+
## we run go build to compile the binary
45+
RUN go mod init ovmsclient
46+
RUN go mod tidy
47+
RUN go build .
48+
49+
50+
ENTRYPOINT ["/app/ovmsclient"]

‎example_client/go/README.md

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# Go client for prediction
2+
3+
This client has been created to demostrate how to interact with OpenVINO Model Server prediction endpoint from Go application. Presented example demonstrates end to end flow for running classification on JPG/PNG image using ResNet50 model. For simplicity of the environment setup, the example is run in the Docker container.
4+
5+
6+
## Get the model
7+
8+
To run end to end flow and get correct results, please download `resnet-50-tf` model and convert it to IR format by following [instructions available on the OpenVINO Model Zoo page](https://docs.openvinotoolkit.org/latest/omz_models_model_resnet_50_tf.html)
9+
10+
Place converted model files (XML and BIN) under the following path: `<PATH_TO_MODELS>/resnet/1`
11+
12+
Where `PATH_TO_MODELS` is the path to the directory with models on the host filesystem.
13+
14+
For example:
15+
```
16+
/home/user/models/resnet/1/resnet-50-tf.xml
17+
/home/user/models/resnet/1/resnet-50-tf.bin
18+
```
19+
20+
## Build Go client docker image
21+
22+
To build the docker image and tag it `ovmsclient` run:
23+
```
24+
docker build . -t ovmsclient
25+
```
26+
27+
## Start OpenVINO Model Server with ResNet model
28+
29+
Before running the client launch OVMS with prepared ResNet model. You can do that with a command similar to:
30+
31+
```
32+
docker run -d --rm -p 9000:9000 -v <PATH_TO_MODELS>/resnet:/models/resnet openvino/model-server:latest --model_name resnet --model_path /models/resnet --port 9000 --layout NHWC
33+
```
34+
35+
**Note** Changing the layout with `--layout NHWC` option is necessary in this example, so the model will accept binary input generated by the client. See [binary inputs](../../docs/binary_input.md) doc if you want to learn more about this feature.
36+
37+
## Run prediction with Go client
38+
39+
In order to run prediction on the model served by the OVMS using Go client run the following command:
40+
41+
`docker run --net=host --rm ovmsclient --serving-address localhost:9000 zebra.jpeg`
42+
43+
Command explained:
44+
- `--net=host` option is required so the container with the client can access container with the model server via host network (localhost),
45+
- `--serving-address` parameter defines the address of the model server gRPC endpoint,
46+
- the last part in the command is a path to the image that will be send to OVMS for prediction. The image must be accessible from the inside of the container (could be mounted). Single zebra picture - `zebra.jpeg` - has been embedded in the docker image to simplify the example, so above command would work out of the box. If you wish to use other image you need to provide it to the container and change the path.
47+
48+
You can also choose if the image should be sent as binary input (raw JPG or PNG bytes) or should be converted on the client side to the data array accepted by the model.
49+
To send raw bytes just add `--binary-input` flag like this:
50+
51+
`docker run --net=host --rm ovmsclient --serving-address localhost:9000 --binary-input zebra.jpeg`
52+
53+
### Exemplary output:
54+
55+
If the client successfully prepared and sent the request and then received a valid response, the output of this command should look somewhat like this:
56+
```
57+
$ docker run --net=host --rm ovmsclient --serving-address localhost:9000 zebra.jpeg
58+
2021/08/30 15:46:40 Request sent successfully
59+
Predicted class: zebra
60+
Classification confidence: 98.353996%
61+
```

‎example_client/go/apis/tensorflow/core/framework/resource_handle.proto

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
syntax = "proto3";
2+
3+
package tensorflow;
4+
5+
import "tensorflow/core/framework/tensor_shape.proto";
6+
import "tensorflow/core/framework/types.proto";
7+
8+
option cc_enable_arenas = true;
9+
option java_outer_classname = "ResourceHandle";
10+
option java_multiple_files = true;
11+
option java_package = "org.tensorflow.framework";
12+
option go_package = "tensorflow/core/framework";
13+
14+
// Protocol buffer representing a handle to a tensorflow resource. Handles are
15+
// not valid across executions, but can be serialized back and forth from within
16+
// a single run.
17+
message ResourceHandleProto {
18+
// Unique name for the device containing the resource.
19+
string device = 1;
20+
21+
// Container in which this resource is placed.
22+
string container = 2;
23+
24+
// Unique name of this resource.
25+
string name = 3;
26+
27+
// Hash code for the type of the resource. Is only valid in the same device
28+
// and in the same execution.
29+
uint64 hash_code = 4;
30+
31+
// For debug-only, the name of the type pointed to by this handle, if
32+
// available.
33+
string maybe_type_name = 5;
34+
35+
// Protocol buffer representing a pair of (data type, tensor shape).
36+
message DtypeAndShape {
37+
DataType dtype = 1;
38+
TensorShapeProto shape = 2;
39+
}
40+
41+
// Data types and shapes for the underlying resource.
42+
repeated DtypeAndShape dtypes_and_shapes = 6;
43+
44+
reserved 7;
45+
}

‎example_client/go/apis/tensorflow/core/framework/tensor.proto

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
syntax = "proto3";
2+
3+
package tensorflow;
4+
5+
import "tensorflow/core/framework/resource_handle.proto";
6+
import "tensorflow/core/framework/tensor_shape.proto";
7+
import "tensorflow/core/framework/types.proto";
8+
9+
option cc_enable_arenas = true;
10+
option java_outer_classname = "TensorProtos";
11+
option java_multiple_files = true;
12+
option java_package = "org.tensorflow.framework";
13+
option go_package = "tensorflow/core/framework";
14+
15+
// Protocol buffer representing a tensor.
16+
message TensorProto {
17+
DataType dtype = 1;
18+
19+
// Shape of the tensor. TODO(touts): sort out the 0-rank issues.
20+
TensorShapeProto tensor_shape = 2;
21+
22+
// Only one of the representations below is set, one of "tensor_contents" and
23+
// the "xxx_val" attributes. We are not using oneof because as oneofs cannot
24+
// contain repeated fields it would require another extra set of messages.
25+
26+
// Version number.
27+
//
28+
// In version 0, if the "repeated xxx" representations contain only one
29+
// element, that element is repeated to fill the shape. This makes it easy
30+
// to represent a constant Tensor with a single value.
31+
int32 version_number = 3;
32+
33+
// Serialized raw tensor content from either Tensor::AsProtoTensorContent or
34+
// memcpy in tensorflow::grpc::EncodeTensorToByteBuffer. This representation
35+
// can be used for all tensor types. The purpose of this representation is to
36+
// reduce serialization overhead during RPC call by avoiding serialization of
37+
// many repeated small items.
38+
bytes tensor_content = 4;
39+
40+
// Type specific representations that make it easy to create tensor protos in
41+
// all languages. Only the representation corresponding to "dtype" can
42+
// be set. The values hold the flattened representation of the tensor in
43+
// row major order.
44+
45+
// DT_HALF, DT_BFLOAT16. Note that since protobuf has no int16 type, we'll
46+
// have some pointless zero padding for each value here.
47+
repeated int32 half_val = 13 [packed = true];
48+
49+
// DT_FLOAT.
50+
repeated float float_val = 5 [packed = true];
51+
52+
// DT_DOUBLE.
53+
repeated double double_val = 6 [packed = true];
54+
55+
// DT_INT32, DT_INT16, DT_INT8, DT_UINT8.
56+
repeated int32 int_val = 7 [packed = true];
57+
58+
// DT_STRING
59+
repeated bytes string_val = 8;
60+
61+
// DT_COMPLEX64. scomplex_val(2*i) and scomplex_val(2*i+1) are real
62+
// and imaginary parts of i-th single precision complex.
63+
repeated float scomplex_val = 9 [packed = true];
64+
65+
// DT_INT64
66+
repeated int64 int64_val = 10 [packed = true];
67+
68+
// DT_BOOL
69+
repeated bool bool_val = 11 [packed = true];
70+
71+
// DT_COMPLEX128. dcomplex_val(2*i) and dcomplex_val(2*i+1) are real
72+
// and imaginary parts of i-th double precision complex.
73+
repeated double dcomplex_val = 12 [packed = true];
74+
75+
// DT_RESOURCE
76+
repeated ResourceHandleProto resource_handle_val = 14;
77+
78+
// DT_VARIANT
79+
repeated VariantTensorDataProto variant_val = 15;
80+
81+
// DT_UINT32
82+
repeated uint32 uint32_val = 16 [packed = true];
83+
84+
// DT_UINT64
85+
repeated uint64 uint64_val = 17 [packed = true];
86+
}
87+
88+
// Protocol buffer representing the serialization format of DT_VARIANT tensors.
89+
message VariantTensorDataProto {
90+
// Name of the type of objects being serialized.
91+
string type_name = 1;
92+
// Portions of the object that are not Tensors.
93+
bytes metadata = 2;
94+
// Tensors contained within objects being serialized.
95+
repeated TensorProto tensors = 3;
96+
}

‎example_client/go/apis/tensorflow/core/framework/tensor_shape.proto

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
// Protocol buffer representing the shape of tensors.
2+
3+
syntax = "proto3";
4+
option cc_enable_arenas = true;
5+
option java_outer_classname = "TensorShapeProtos";
6+
option java_multiple_files = true;
7+
option java_package = "org.tensorflow.framework";
8+
option go_package = "tensorflow/core/framework";
9+
10+
package tensorflow;
11+
12+
// Dimensions of a tensor.
13+
message TensorShapeProto {
14+
// One dimension of the tensor.
15+
message Dim {
16+
// Size of the tensor in that dimension.
17+
// This value must be >= -1, but values of -1 are reserved for "unknown"
18+
// shapes (values of -1 mean "unknown" dimension). Certain wrappers
19+
// that work with TensorShapeProto may fail at runtime when deserializing
20+
// a TensorShapeProto containing a dim value of -1.
21+
int64 size = 1;
22+
23+
// Optional name of the tensor dimension.
24+
string name = 2;
25+
};
26+
27+
// Dimensions of the tensor, such as {"input", 30}, {"output", 40}
28+
// for a 30 x 40 2D tensor. If an entry has size -1, this
29+
// corresponds to a dimension of unknown size. The names are
30+
// optional.
31+
//
32+
// The order of entries in "dim" matters: It indicates the layout of the
33+
// values in the tensor in-memory representation.
34+
//
35+
// The first entry in "dim" is the outermost dimension used to layout the
36+
// values, the last entry is the innermost dimension. This matches the
37+
// in-memory layout of RowMajor Eigen tensors.
38+
//
39+
// If "dim.size()" > 0, "unknown_rank" must be false.
40+
repeated Dim dim = 2;
41+
42+
// If true, the number of dimensions in the shape is unknown.
43+
//
44+
// If true, "dim.size()" must be 0.
45+
bool unknown_rank = 3;
46+
};

‎example_client/go/apis/tensorflow/core/framework/types.proto

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
syntax = "proto3";
2+
3+
package tensorflow;
4+
option cc_enable_arenas = true;
5+
option java_outer_classname = "TypesProtos";
6+
option java_multiple_files = true;
7+
option java_package = "org.tensorflow.framework";
8+
option go_package = "tensorflow/core/framework";
9+
10+
// (== suppress_warning documentation-presence ==)
11+
// LINT.IfChange
12+
enum DataType {
13+
// Not a legal value for DataType. Used to indicate a DataType field
14+
// has not been set.
15+
DT_INVALID = 0;
16+
17+
// Data types that all computation devices are expected to be
18+
// capable to support.
19+
DT_FLOAT = 1;
20+
DT_DOUBLE = 2;
21+
DT_INT32 = 3;
22+
DT_UINT8 = 4;
23+
DT_INT16 = 5;
24+
DT_INT8 = 6;
25+
DT_STRING = 7;
26+
DT_COMPLEX64 = 8; // Single-precision complex
27+
DT_INT64 = 9;
28+
DT_BOOL = 10;
29+
DT_QINT8 = 11; // Quantized int8
30+
DT_QUINT8 = 12; // Quantized uint8
31+
DT_QINT32 = 13; // Quantized int32
32+
DT_BFLOAT16 = 14; // Float32 truncated to 16 bits. Only for cast ops.
33+
DT_QINT16 = 15; // Quantized int16
34+
DT_QUINT16 = 16; // Quantized uint16
35+
DT_UINT16 = 17;
36+
DT_COMPLEX128 = 18; // Double-precision complex
37+
DT_HALF = 19;
38+
DT_RESOURCE = 20;
39+
DT_VARIANT = 21; // Arbitrary C++ data types
40+
DT_UINT32 = 22;
41+
DT_UINT64 = 23;
42+
43+
// Do not use! These are only for parameters. Every enum above
44+
// should have a corresponding value below (verified by types_test).
45+
DT_FLOAT_REF = 101;
46+
DT_DOUBLE_REF = 102;
47+
DT_INT32_REF = 103;
48+
DT_UINT8_REF = 104;
49+
DT_INT16_REF = 105;
50+
DT_INT8_REF = 106;
51+
DT_STRING_REF = 107;
52+
DT_COMPLEX64_REF = 108;
53+
DT_INT64_REF = 109;
54+
DT_BOOL_REF = 110;
55+
DT_QINT8_REF = 111;
56+
DT_QUINT8_REF = 112;
57+
DT_QINT32_REF = 113;
58+
DT_BFLOAT16_REF = 114;
59+
DT_QINT16_REF = 115;
60+
DT_QUINT16_REF = 116;
61+
DT_UINT16_REF = 117;
62+
DT_COMPLEX128_REF = 118;
63+
DT_HALF_REF = 119;
64+
DT_RESOURCE_REF = 120;
65+
DT_VARIANT_REF = 121;
66+
DT_UINT32_REF = 122;
67+
DT_UINT64_REF = 123;
68+
}
69+
// LINT.ThenChange(
70+
// https://www.tensorflow.org/code/tensorflow/c/tf_datatype.h,
71+
// https://www.tensorflow.org/code/tensorflow/go/tensor.go,
72+
// https://www.tensorflow.org/code/tensorflow/core/framework/tensor.cc,
73+
// https://www.tensorflow.org/code/tensorflow/core/framework/types.h,
74+
// https://www.tensorflow.org/code/tensorflow/core/framework/types.cc,
75+
// https://www.tensorflow.org/code/tensorflow/python/framework/dtypes.py,
76+
// https://www.tensorflow.org/code/tensorflow/python/framework/function.py)
77+
78+
// For identifying the underlying type of a variant. For variants, the types
79+
// listed here are a subset of the types in the variant type registry,
80+
// corresponding to commonly used variants which must occasionally be
81+
// special-cased.
82+
enum SpecializedType {
83+
// Invalid/unknown specialized type.
84+
ST_INVALID = 0;
85+
// "tensorflow::TensorList" in the variant type registry.
86+
ST_TENSOR_LIST = 1;
87+
// "tensorflow::data::Optional" in the variant type registry.
88+
ST_OPTIONAL = 2;
89+
}

0 commit comments

Comments
 (0)