Vald Search APIs

Overview

Search Service is responsible for searching vectors similar to the user request vector from vald-agent.

service Search {
+  rpc Search(payload.v1.Search.Request) returns (payload.v1.Search.Response) {}
+
+  rpc SearchByID(payload.v1.Search.IDRequest)
+      returns (payload.v1.Search.Response) {}
+
+  rpc StreamSearch(stream payload.v1.Search.Request)
+      returns (stream payload.v1.Search.StreamResponse) {}
+
+  rpc StreamSearchByID(stream payload.v1.Search.IDRequest)
+      returns (stream payload.v1.Search.StreamResponse) {}
+
+  rpc MultiSearch(payload.v1.Search.MultiRequest)
+      returns (payload.v1.Search.Responses) {}
+
+  rpc MultiSearchByID(payload.v1.Search.MultiIDRequest)
+      returns (payload.v1.Search.Responses) {}
+
+  rpc LinearSearch(payload.v1.Search.Request) returns (payload.v1.Search.Response) {}
+
+  rpc LinearSearchByID(payload.v1.Search.IDRequest)
+      returns (payload.v1.Search.Response) {}
+
+  rpc StreamLinearSearch(stream payload.v1.Search.Request)
+      returns (stream payload.v1.Search.StreamResponse) {}
+
+  rpc StreamLinearSearchByID(stream payload.v1.Search.IDRequest)
+      returns (stream payload.v1.Search.StreamResponse) {}
+
+  rpc MultiLinearSearch(payload.v1.Search.MultiRequest)
+      returns (payload.v1.Search.Responses) {}
+
+  rpc MultiLinearSearchByID(payload.v1.Search.MultiIDRequest)
+      returns (payload.v1.Search.Responses) {}
+}
+

Search RPC

Search RPC is the method to search vector(s) similar to the request vector.

Input

  • the scheme of payload.v1.Search.Request

    message Search {
+  message Request {
+    repeated float vector = 1 [ (validate.rules).repeated .min_items = 2 ];
+    Config config = 2;
+  }
+
+  message Config {
+    string request_id = 1;
+    uint32 num = 2 [ (validate.rules).uint32.gte = 1 ];
+    float radius = 3;
+    float epsilon = 4;
+    int64 timeout = 5;
+    Filter.Config ingress_filters = 6;
+    Filter.Config egress_filters = 7;
+    uint32 min_num = 8;
+    AggregationAlgorithm aggregation_algorithm = 9;
+  }
+}
+
+enum AggregationAlgorithm {
+  Unknown = 0;
+  ConcurrentQueue = 1;
+  SortSlice = 2;
+  SortPoolSlice = 3;
+  PairingHeap = 4;
+}
+

    • Search.Request

      fieldtypelabelrequireddescription
      vectorfloatrepeated(Array[float])*The vector data. Its dimension is between 2 and 65,536.
      configConfig*The configuration of the search request.

    • Search.Config

      fieldtypelabelrequireddescription
      request_idstringUnique request ID.
      numuint32*The maximum number of results to be returned.
      radiusfloat*The search radius.
      epsilonfloat*The search coefficient (default value is 0.1).
      timeoutint64Search timeout in nanoseconds (default value is 5s).
      ingress_filtersFilter.ConfigIngress Filter configuration.
      egress_filtersFilter.ConfigEgress Filter configuration.
      min_numuint32The minimum number of results to be returned.
      aggregation_algorithmAggregationAlgorithmThe search aggregation algorithm option (default value is ConcurrentQueue).

Output

  • the scheme of payload.v1.Search.Response.

    message Search {
+  message Response {
+    string request_id = 1;
+    repeated Object.Distance results = 2;
+  }
+}
+
+message Object {
+  message Distance {
+    string id = 1;
+    float distance = 2;
+  }
+}
+

    • Search.Response

      fieldtypelabeldescription
      request_idstringThe unique request ID.
      resultsObject.Distancerepeated(Array[Object.Distance])Search results.

    • Object.Distance

      fieldtypelabeldescription
      idstringThe vector ID.
      distancefloatThe distance between the result vector and the request vector.

Status Code

codename
0OK
1CANCELLED
3INVALID_ARGUMENT
4DEADLINE_EXCEEDED
5NOT_FOUND
10ABORTED
13INTERNAL

Please refer to Response Status Code for more details.

Troubleshooting

The request process may not be completed when the response code is NOT 0 (OK).

Here are some common reasons and how to resolve each error.

namecommon reasonhow to resolve
CANCELLEDExecuted cancel() of rpc from client/server-side or network problems between client and server.Check the code, especially around timeout and connection management, and fix if needed.
INVALID_ARGUMENTThe Dimension of the request vector is NOT the same as Vald Agent’s config, or some request payload is invalid.Check Agent config, request payload, and fix request payload or Agent config.
DEADLINE_EXCEEDEDThe RPC timeout setting is too short on the client/server side.Check the gRPC timeout setting on both the client and server sides and fix it if needed.
NOT_FOUNDSearch result is empty or insufficient to request result length.Send a request with another vector or set min_num to a smaller value.
INTERNALTarget Vald cluster or network route has some critical error.Check target Vald cluster first and check network route including ingress as second.

SearchByID RPC

SearchByID RPC is the method to search similar vectors using a user-defined vector ID.
The vector with the same requested ID should be indexed into the vald-agent before searching.

Input

  • the scheme of payload.v1.Search.IDRequest

    message Search {
+  message IDRequest {
+    string id = 1;
+    Config config = 2;
+  }
+
+  message Config {
+    string request_id = 1;
+    uint32 num = 2 [ (validate.rules).uint32.gte = 1 ];
+    float radius = 3;
+    float epsilon = 4;
+    int64 timeout = 5;
+    Filter.Config ingress_filters = 6;
+    Filter.Config egress_filters = 7;
+    uint32 min_num = 8;
+    AggregationAlgorithm aggregation_algorithm = 9;
+  }
+}
+
+enum AggregationAlgorithm {
+  Unknown = 0;
+  ConcurrentQueue = 1;
+  SortSlice = 2;
+  SortPoolSlice = 3;
+  PairingHeap = 4;
+}
+

    • Search.IDRequest

      fieldtypelabelrequireddescription
      idstring*The vector ID to be searched.
      configConfig*The configuration of the search request.

    • Search.Config

      fieldtypelabelrequireddescription
      request_idstringUnique request ID.
      numuint32*The maximum number of results to be returned.
      radiusfloat*The search radius.
      epsilonfloat*The search coefficient (default value is 0.1).
      timeoutint64Search timeout in nanoseconds (default value is 5s).
      ingress_filtersFilter.ConfigIngress Filter configuration.
      egress_filtersFilter.ConfigEgress Filter configuration.
      min_numuint32The minimum number of results to be returned.
      aggregation_algorithmAggregationAlgorithmThe search aggregation algorithm option (default value is ConcurrentQueue).

Output

  • the scheme of payload.v1.Search.Response.

    message Search {
+  message Response {
+    string request_id = 1;
+    repeated Object.Distance results = 2;
+  }
+}
+
+message Object {
+  message Distance {
+    string id = 1;
+    float distance = 2;
+  }
+}
+

    • Search.Response

      fieldtypelabeldescription
      request_idstringThe unique request ID.
      resultsObject.Distancerepeated(Array[Object.Distance])Search results.

    • Object.Distance

      fieldtypelabeldescription
      idstringThe vector ID.
      distancefloatThe distance between the result vector and the request vector.

Status Code

codename
0OK
1CANCELLED
3INVALID_ARGUMENT
4DEADLINE_EXCEEDED
5NOT_FOUND
10ABORTED
13INTERNAL

Please refer to Response Status Code for more details.

Troubleshooting

The request process may not be completed when the response code is NOT 0 (OK).

Here are some common reasons and how to resolve each error.

namecommon reasonhow to resolve
CANCELLEDExecuted cancel() of rpc from client/server-side or network problems between client and server.Check the code, especially around timeout and connection management, and fix if needed.
INVALID_ARGUMENTThe Requested vector’s ID is empty, or some request payload is invalid.Check request payload and fix request payload.
DEADLINE_EXCEEDEDThe RPC timeout setting is too short on the client/server side.Check the gRPC timeout setting on both the client and server sides and fix it if needed.
NOT_FOUNDThe Requested ID is not inserted on the target Vald cluster, or the search result is insufficient to the required result length.Send a request with another vector or set min_num to a smaller value.
INTERNALTarget Vald cluster or network route has some critical error.Check target Vald cluster first and check network route including ingress as second.

StreamSearch RPC

StreamSearch RPC is the method to search vectors with multi queries(vectors) using the bidirectional streaming RPC.
Using the bidirectional streaming RPC, the search request can be communicated in any order between the client and server. +Each Search request and response are independent.

Input

  • the scheme of payload.v1.Search.Request stream

    message Search {
+  message Request {
+    repeated float vector = 1 [ (validate.rules).repeated .min_items = 2 ];
+    Config config = 2;
+  }
+
+  message Config {
+    string request_id = 1;
+    uint32 num = 2 [ (validate.rules).uint32.gte = 1 ];
+    float radius = 3;
+    float epsilon = 4;
+    int64 timeout = 5;
+    Filter.Config ingress_filters = 6;
+    Filter.Config egress_filters = 7;
+    uint32 min_num = 8;
+    AggregationAlgorithm aggregation_algorithm = 9;
+  }
+}
+
+enum AggregationAlgorithm {
+  Unknown = 0;
+  ConcurrentQueue = 1;
+  SortSlice = 2;
+  SortPoolSlice = 3;
+  PairingHeap = 4;
+}
+

    • Search.Request

      fieldtypelabelrequireddescription
      vectorfloatrepeated(Array[float])*The vector data. Its dimension is between 2 and 65,536.
      configConfig*The configuration of the search request.

    • Search.Config

      fieldtypelabelrequireddescription
      request_idstringUnique request ID.
      numuint32*The maximum number of results to be returned.
      radiusfloat*The search radius.
      epsilonfloat*The search coefficient (default value is 0.1).
      timeoutint64Search timeout in nanoseconds (default value is 5s).
      ingress_filtersFilter.ConfigIngress Filter configuration.
      egress_filtersFilter.ConfigEgress Filter configuration.
      min_numuint32The minimum number of results to be returned.
      aggregation_algorithmAggregationAlgorithmThe search aggregation algorithm option (default value is ConcurrentQueue).

Output

  • the scheme of payload.v1.Search.StreamResponse.

    message Search {
+  message StreamResponse {
+    oneof payload {
+      Response response = 1;
+      google.rpc.Status status = 2;
+    }
+  }
+
+  message Response {
+    string request_id = 1;
+    repeated Object.Distance results = 2;
+  }
+}
+
+message Object {
+  message Distance {
+    string id = 1;
+    float distance = 2;
+  }
+}
+

    • Search.StreamResponse

      fieldtypelabeldescription
      responseResponseThe search result response.
      statusgoogle.rpc.StatusThe status of Google RPC.

    • Search.Response

      fieldtypelabeldescription
      request_idstringThe unique request ID.
      resultsObject.Distancerepeated(Array[Object.Distance])Search results.

    • Object.Distance

      fieldtypelabeldescription
      idstringThe vector ID.
      distancefloatThe distance between the result vector and the request vector.

Status Code

codename
0OK
1CANCELLED
3INVALID_ARGUMENT
4DEADLINE_EXCEEDED
5NOT_FOUND
10ABORTED
13INTERNAL

Please refer to Response Status Code for more details.

Troubleshooting

The request process may not be completed when the response code is NOT 0 (OK).

Here are some common reasons and how to resolve each error.

namecommon reasonhow to resolve
CANCELLEDExecuted cancel() of rpc from client/server-side or network problems between client and server.Check the code, especially around timeout and connection management, and fix if needed.
INVALID_ARGUMENTThe Dimension of the request vector is NOT the same as Vald Agent’s config, or some request payload is invalid.Check Agent config, request payload, and fix request payload or Agent config.
DEADLINE_EXCEEDEDThe RPC timeout setting is too short on the client/server side.Check the gRPC timeout setting on both the client and server sides and fix it if needed.
NOT_FOUNDSearch result is empty or insufficient to request result length.Send a request with another vector or set min_num to a smaller value.
INTERNALTarget Vald cluster or network route has some critical error.Check target Vald cluster first and check network route including ingress as second.

StreamSearchByID RPC

StreamSearchByID RPC is the method to search vectors with multi queries(IDs) using the bidirectional streaming RPC.
Using the bidirectional streaming RPC, the search request can be communicated in any order between the client and server. +Each SearchByID request and response are independent.

Input

  • the scheme of payload.v1.Search.IDRequest stream

    message Search {
+  message IDRequest {
+    string id = 1;
+    Config config = 2;
+  }
+
+  message Config {
+    string request_id = 1;
+    uint32 num = 2 [ (validate.rules).uint32.gte = 1 ];
+    float radius = 3;
+    float epsilon = 4;
+    int64 timeout = 5;
+    Filter.Config ingress_filters = 6;
+    Filter.Config egress_filters = 7;
+    uint32 min_num = 8;
+    AggregationAlgorithm aggregation_algorithm = 9;
+  }
+}
+
+enum AggregationAlgorithm {
+  Unknown = 0;
+  ConcurrentQueue = 1;
+  SortSlice = 2;
+  SortPoolSlice = 3;
+  PairingHeap = 4;
+}
+

    • Search.IDRequest

      fieldtypelabelrequireddescription
      idstring*The vector ID to be searched.
      configConfig*The configuration of the search request.

    • Search.Config

      fieldtypelabelrequireddescription
      request_idstringUnique request ID.
      numuint32*The maximum number of results to be returned.
      radiusfloat*The search radius.
      epsilonfloat*The search coefficient (default value is 0.1).
      timeoutint64Search timeout in nanoseconds (default value is 5s).
      ingress_filtersFilter.ConfigIngress Filter configuration.
      egress_filtersFilter.ConfigEgress Filter configuration.
      min_numuint32The minimum number of results to be returned.
      aggregation_algorithmAggregationAlgorithmThe search aggregation algorithm option (default value is ConcurrentQueue).

Output

  • the scheme of payload.v1.Search.StreamResponse.

    message Search {
+  message StreamResponse {
+    oneof payload {
+      Response response = 1;
+      google.rpc.Status status = 2;
+    }
+  }
+
+  message Response {
+    string request_id = 1;
+    repeated Object.Distance results = 2;
+  }
+}
+
+message Object {
+  message Distance {
+    string id = 1;
+    float distance = 2;
+  }
+}
+

    • Search.StreamResponse

      fieldtypelabeldescription
      responseResponseThe search result response.
      statusgoogle.rpc.StatusThe status of Google RPC.

    • Search.Response

      fieldtypelabeldescription
      request_idstringThe unique request ID.
      resultsObject.Distancerepeated(Array[Object.Distance])Search results.

    • Object.Distance

      fieldtypelabeldescription
      idstringThe vector ID.
      distancefloatThe distance between the result vector and the request vector.

Status Code

codename
0OK
1CANCELLED
3INVALID_ARGUMENT
4DEADLINE_EXCEEDED
5NOT_FOUND
10ABORTED
13INTERNAL

Please refer to Response Status Code for more details.

Troubleshooting

The request process may not be completed when the response code is NOT 0 (OK).

Here are some common reasons and how to resolve each error.

namecommon reasonhow to resolve
CANCELLEDExecuted cancel() of rpc from client/server-side or network problems between client and server.Check the code, especially around timeout and connection management, and fix if needed.
INVALID_ARGUMENTThe Requested vector’s ID is empty, or some request payload is invalid.Check request payload and fix request payload.
DEADLINE_EXCEEDEDThe RPC timeout setting is too short on the client/server side.Check the gRPC timeout setting on both the client and server sides and fix it if needed.
NOT_FOUNDThe Requested ID is not inserted on the target Vald cluster, or the search result is insufficient to the required result length.Send a request with another vector or set min_num to a smaller value.
INTERNALTarget Vald cluster or network route has some critical error.Check target Vald cluster first and check network route including ingress as second.

MultiSearch RPC

MultiSearch RPC is the method to search vectors with multiple vectors in 1 request.

gRPC has a message size limitation.
Please be careful that the size of the request exceeds the limit.

Input

  • the scheme of payload.v1.Search.MultiRequest

    message Search {
+  message MultiRequest {
+    repeated Request requests = 1;
+  }
+
+  message Request {
+    repeated float vector = 1 [ (validate.rules).repeated .min_items = 2 ];
+    Config config = 2;
+  }
+
+  message Config {
+    string request_id = 1;
+    uint32 num = 2 [ (validate.rules).uint32.gte = 1 ];
+    float radius = 3;
+    float epsilon = 4;
+    int64 timeout = 5;
+    Filter.Config ingress_filters = 6;
+    Filter.Config egress_filters = 7;
+    uint32 min_num = 8;
+    AggregationAlgorithm aggregation_algorithm = 9;
+  }
+}
+
+enum AggregationAlgorithm {
+  Unknown = 0;
+  ConcurrentQueue = 1;
+  SortSlice = 2;
+  SortPoolSlice = 3;
+  PairingHeap = 4;
+}
+

    • Search.MultiRequest

      fieldtypelabelrequireddescription
      requestsrepeated(Array[Request])*The search request list.

    • Search.Request

      fieldtypelabelrequireddescription
      vectorfloatrepeated(Array[float])*The vector data. Its dimension is between 2 and 65,536.
      configConfig*The configuration of the search request.

    • Search.Config

      fieldtypelabelrequireddescription
      request_idstringUnique request ID.
      numuint32*The maximum number of results to be returned.
      radiusfloat*The search radius.
      epsilonfloat*The search coefficient (default value is 0.1).
      timeoutint64Search timeout in nanoseconds (default value is 5s).
      ingress_filtersFilter.ConfigIngress Filter configuration.
      egress_filtersFilter.ConfigEgress Filter configuration.
      min_numuint32The minimum number of results to be returned.
      aggregation_algorithmAggregationAlgorithmThe search aggregation algorithm option (default value is ConcurrentQueue).

Output

  • the scheme of payload.v1.Search.Responses.

    message Search {
+  message Responses {
+    repeated Response responses = 1;
+  }
+
+  message Response {
+    string request_id = 1;
+    repeated Object.Distance results = 2;
+  }
+}
+
+message Object {
+  message Distance {
+    string id = 1;
+    float distance = 2;
+  }
+}
+

    • Search.Responses

      fieldtypelabeldescription
      responsesResponserepeated(Array[Response])The list of search results response.

    • Search.Response

      fieldtypelabeldescription
      request_idstringThe unique request ID.
      resultsObject.Distancerepeated(Array[Object.Distance])Search results.

    • Object.Distance

      fieldtypelabeldescription
      idstringThe vector ID.
      distancefloatThe distance between the result vector and the request vector.

Status Code

codename
0OK
1CANCELLED
3INVALID_ARGUMENT
4DEADLINE_EXCEEDED
5NOT_FOUND
10ABORTED
13INTERNAL

Please refer to Response Status Code for more details.

Troubleshooting

The request process may not be completed when the response code is NOT 0 (OK).

Here are some common reasons and how to resolve each error.

namecommon reasonhow to resolve
CANCELLEDExecuted cancel() of rpc from client/server-side or network problems between client and server.Check the code, especially around timeout and connection management, and fix if needed.
INVALID_ARGUMENTThe Dimension of the request vector is NOT the same as Vald Agent’s config, or some request payload is invalid.Check Agent config, request payload, and fix request payload or Agent config.
DEADLINE_EXCEEDEDThe RPC timeout setting is too short on the client/server side.Check the gRPC timeout setting on both the client and server sides and fix it if needed.
NOT_FOUNDSearch result is empty or insufficient to request result length.Send a request with another vector or set min_num to a smaller value.
INTERNALTarget Vald cluster or network route has some critical error.Check target Vald cluster first and check network route including ingress as second.

MultiSearchByID RPC

MultiSearchByID RPC is the method to search vectors with multiple IDs in 1 request.

gRPC has a message size limitation.
Please be careful that the size of the request exceeds the limit.

Input

  • the scheme of payload.v1.Search.MultiIDRequest stream

    message Search {
+
+  message MultiIDRequest {
+      repeated IDRequest requests = 1;
+  }
+
+  message IDRequest {
+    string id = 1;
+    Config config = 2;
+  }
+
+  message Config {
+    string request_id = 1;
+    uint32 num = 2 [ (validate.rules).uint32.gte = 1 ];
+    float radius = 3;
+    float epsilon = 4;
+    int64 timeout = 5;
+    Filter.Config ingress_filters = 6;
+    Filter.Config egress_filters = 7;
+    uint32 min_num = 8;
+    AggregationAlgorithm aggregation_algorithm = 9;
+  }
+}
+
+enum AggregationAlgorithm {
+  Unknown = 0;
+  ConcurrentQueue = 1;
+  SortSlice = 2;
+  SortPoolSlice = 3;
+  PairingHeap = 4;
+}
+

    • Search.MultiIDRequest

      fieldtypelabelrequireddescription
      requestsIDRequestrepeated(Array[IDRequest])*The searchByID request list.

    • Search.IDRequest

      fieldtypelabelrequireddescription
      idstring*The vector ID to be searched.
      configConfig*The configuration of the search request.

    • Search.Config

      fieldtypelabelrequireddescription
      request_idstringUnique request ID.
      numuint32*The maximum number of results to be returned.
      radiusfloat*The search radius.
      epsilonfloat*The search coefficient (default value is 0.1).
      timeoutint64Search timeout in nanoseconds (default value is 5s).
      ingress_filtersFilter.ConfigIngress Filter configuration.
      egress_filtersFilter.ConfigEgress Filter configuration.
      min_numuint32The minimum number of results to be returned.
      aggregation_algorithmAggregationAlgorithmThe search aggregation algorithm option (default value is ConcurrentQueue).

Output

  • the scheme of payload.v1.Search.Responses.

    message Search {
+  message Responses {
+    repeated Response responses = 1;
+  }
+
+  message Response {
+    string request_id = 1;
+    repeated Object.Distance results = 2;
+  }
+}
+
+message Object {
+  message Distance {
+    string id = 1;
+    float distance = 2;
+  }
+}
+

    • Search.Responses

      fieldtypelabeldescription
      responsesResponserepeated(Array[Response])The list of search results response.

    • Search.Response

      fieldtypelabeldescription
      request_idstringThe unique request ID.
      resultsObject.Distancerepeated(Array[Object.Distance])Search results.

    • Object.Distance

      fieldtypelabeldescription
      idstringThe vector ID.
      distancefloatThe distance between the result vector and the request vector.

Status Code

codename
0OK
1CANCELLED
3INVALID_ARGUMENT
4DEADLINE_EXCEEDED
5NOT_FOUND
10ABORTED
13INTERNAL

Please refer to Response Status Code for more details.

Troubleshooting

The request process may not be completed when the response code is NOT 0 (OK).

Here are some common reasons and how to resolve each error.

namecommon reasonhow to resolve
CANCELLEDExecuted cancel() of rpc from client/server-side or network problems between client and server.Check the code, especially around timeout and connection management, and fix if needed.
INVALID_ARGUMENTThe Requested vector’s ID is empty, or some request payload is invalid.Check request payload and fix request payload.
DEADLINE_EXCEEDEDThe RPC timeout setting is too short on the client/server side.Check the gRPC timeout setting on both the client and server sides and fix it if needed.
NOT_FOUNDThe Requested ID is not inserted on the target Vald cluster, or the search result is insufficient to the required result length.Send a request with another vector or set min_num to a smaller value.
INTERNALTarget Vald cluster or network route has some critical error.Check target Vald cluster first and check network route including ingress as second.

LinearSearch RPC

LinearSearch RPC is the method to linear search vector(s) similar to the request vector.

Input

  • the scheme of payload.v1.Search.Request

    message Search {
+  message Request {
+    repeated float vector = 1 [ (validate.rules).repeated .min_items = 2 ];
+    Config config = 2;
+  }
+
+  message Config {
+    string request_id = 1;
+    uint32 num = 2 [ (validate.rules).uint32.gte = 1 ];
+    int64 timeout = 5;
+    Filter.Config ingress_filters = 6;
+    Filter.Config egress_filters = 7;
+    uint32 min_num = 8;
+    AggregationAlgorithm aggregation_algorithm = 9;
+  }
+}
+
+enum AggregationAlgorithm {
+  Unknown = 0;
+  ConcurrentQueue = 1;
+  SortSlice = 2;
+  SortPoolSlice = 3;
+  PairingHeap = 4;
+}
+

    • Search.Request

      fieldtypelabelrequireddescription
      vectorfloatrepeated(Array[float])*The vector data. Its dimension is between 2 and 65,536.
      configConfig*The configuration of the search request.

    • Search.Config

      fieldtypelabelrequireddescription
      request_idstringUnique request ID.
      numuint32*The maximum number of results to be returned.
      timeoutint64Search timeout in nanoseconds (default value is 5s).
      ingress_filtersFilter.ConfigIngress Filter configuration.
      egress_filtersFilter.ConfigEgress Filter configuration.
      min_numuint32The minimum number of results to be returned.
      aggregation_algorithmAggregationAlgorithmThe search aggregation algorithm option (default value is ConcurrentQueue).

Output

  • the scheme of payload.v1.Search.Response.

    message Search {
+  message Response {
+    string request_id = 1;
+    repeated Object.Distance results = 2;
+  }
+}
+
+message Object {
+  message Distance {
+    string id = 1;
+    float distance = 2;
+  }
+}
+

    • Search.Response

      fieldtypelabeldescription
      request_idstringThe unique request ID.
      resultsObject.Distancerepeated(Array[Object.Distance])Search results.

    • Object.Distance

      fieldtypelabeldescription
      idstringThe vector ID.
      distancefloatThe distance between the result vector and the request vector.

Status Code

codename
0OK
1CANCELLED
3INVALID_ARGUMENT
4DEADLINE_EXCEEDED
5NOT_FOUND
10ABORTED
13INTERNAL

Please refer to Response Status Code for more details.

Troubleshooting

The request process may not be completed when the response code is NOT 0 (OK).

Here are some common reasons and how to resolve each error.

namecommon reasonhow to resolve
CANCELLEDExecuted cancel() of rpc from client/server-side or network problems between client and server.Check the code, especially around timeout and connection management, and fix if needed.
INVALID_ARGUMENTThe Dimension of the request vector is NOT the same as Vald Agent’s config, or some request payload is invalid.Check Agent config, request payload, and fix request payload or Agent config.
DEADLINE_EXCEEDEDThe RPC timeout setting is too short on the client/server side.Check the gRPC timeout setting on both the client and server sides and fix it if needed.
NOT_FOUNDSearch result is empty or insufficient to request result length.Send a request with another vector or set min_num to a smaller value.
INTERNALTarget Vald cluster or network route has some critical error.Check target Vald cluster first and check network route including ingress as second.

LinearSearchByID RPC

LinearSearchByID RPC is the method to linear search similar vectors using a user-defined vector ID.
The vector with the same requested ID should be indexed into the vald-agent before searching. +You will get a NOT_FOUND error if the vector isn’t stored.

Input

  • the scheme of payload.v1.Search.IDRequest

    message Search {
+  message IDRequest {
+    string id = 1;
+    Config config = 2;
+  }
+
+  message Config {
+    string request_id = 1;
+    uint32 num = 2 [ (validate.rules).uint32.gte = 1 ];
+    int64 timeout = 5;
+    Filter.Config ingress_filters = 6;
+    Filter.Config egress_filters = 7;
+    uint32 min_num = 8;
+    AggregationAlgorithm aggregation_algorithm = 9;
+  }
+}
+
+enum AggregationAlgorithm {
+  Unknown = 0;
+  ConcurrentQueue = 1;
+  SortSlice = 2;
+  SortPoolSlice = 3;
+  PairingHeap = 4;
+}
+

    • Search.IDRequest

      fieldtypelabelrequireddescription
      idstring*The vector ID to be searched.
      configConfig*The configuration of the search request.

    • Search.Config

      fieldtypelabelrequireddescription
      request_idstringUnique request ID.
      numuint32*The maximum number of results to be returned.
      timeoutint64Search timeout in nanoseconds (default value is 5s).
      ingress_filtersFilter.ConfigIngress Filter configuration.
      egress_filtersFilter.ConfigEgress Filter configuration.
      min_numuint32The minimum number of results to be returned.
      aggregation_algorithmAggregationAlgorithmThe search aggregation algorithm option (default value is ConcurrentQueue).

Output

  • the scheme of payload.v1.Search.Response.

    message Search {
+  message Response {
+    string request_id = 1;
+    repeated Object.Distance results = 2;
+  }
+}
+
+message Object {
+  message Distance {
+    string id = 1;
+    float distance = 2;
+  }
+}
+

    • Search.Response

      fieldtypelabeldescription
      request_idstringThe unique request ID.
      resultsObject.Distancerepeated(Array[Object.Distance])Search results.

    • Object.Distance

      fieldtypelabeldescription
      idstringThe vector ID.
      distancefloatThe distance between the result vector and the request vector.

Status Code

codename
0OK
1CANCELLED
3INVALID_ARGUMENT
4DEADLINE_EXCEEDED
5NOT_FOUND
10ABORTED
13INTERNAL

Please refer to Response Status Code for more details.

Troubleshooting

The request process may not be completed when the response code is NOT 0 (OK).

Here are some common reasons and how to resolve each error.

namecommon reasonhow to resolve
CANCELLEDExecuted cancel() of rpc from client/server-side or network problems between client and server.Check the code, especially around timeout and connection management, and fix if needed.
INVALID_ARGUMENTThe Requested vector’s ID is empty, or some request payload is invalid.Check request payload and fix request payload.
DEADLINE_EXCEEDEDThe RPC timeout setting is too short on the client/server side.Check the gRPC timeout setting on both the client and server sides and fix it if needed.
NOT_FOUNDThe Requested ID is not inserted on the target Vald cluster, or the search result is insufficient to the required result length.Send a request with another vector or set min_num to a smaller value.
INTERNALTarget Vald cluster or network route has some critical error.Check target Vald cluster first and check network route including ingress as second.

StreamLinearSearch RPC

StreamLinearSearch RPC is the method to linear search vectors with multi queries(vectors) using the bidirectional streaming RPC.
Using the bidirectional streaming RPC, the linear search request can be communicated in any order between the client and server. +Each LinearSearch request and response are independent.

Input

  • the scheme of payload.v1.Search.Request stream

    message Search {
+  message Request {
+    repeated float vector = 1 [ (validate.rules).repeated .min_items = 2 ];
+    Config config = 2;
+  }
+
+  message Config {
+    string request_id = 1;
+    uint32 num = 2 [ (validate.rules).uint32.gte = 1 ];
+    int64 timeout = 5;
+    Filter.Config ingress_filters = 6;
+    Filter.Config egress_filters = 7;
+    uint32 min_num = 8;
+    AggregationAlgorithm aggregation_algorithm = 9;
+  }
+}
+
+enum AggregationAlgorithm {
+  Unknown = 0;
+  ConcurrentQueue = 1;
+  SortSlice = 2;
+  SortPoolSlice = 3;
+  PairingHeap = 4;
+}
+

    • Search.Request

      fieldtypelabelrequireddescription
      vectorfloatrepeated(Array[float])*The vector data. Its dimension is between 2 and 65,536.
      configConfig*The configuration of the search request.

    • Search.Config

      fieldtypelabelrequireddescription
      request_idstringUnique request ID.
      numuint32*The maximum number of results to be returned.
      timeoutint64Search timeout in nanoseconds (default value is 5s).
      ingress_filtersFilter.ConfigIngress Filter configuration.
      egress_filtersFilter.ConfigEgress Filter configuration.
      min_numuint32The minimum number of results to be returned.
      aggregation_algorithmAggregationAlgorithmThe search aggregation algorithm option (default value is ConcurrentQueue).

Output

  • the scheme of payload.v1.Search.StreamResponse.

    message Search {
+  message StreamResponse {
+    oneof payload {
+      Response response = 1;
+      google.rpc.Status status = 2;
+    }
+  }
+
+  message Response {
+    string request_id = 1;
+    repeated Object.Distance results = 2;
+  }
+}
+
+message Object {
+  message Distance {
+    string id = 1;
+    float distance = 2;
+  }
+}
+

    • Search.StreamResponse

      fieldtypelabeldescription
      responseResponseThe search result response.
      statusgoogle.rpc.StatusThe status of Google RPC.

    • Search.Response

      fieldtypelabeldescription
      request_idstringThe unique request ID.
      resultsObject.Distancerepeated(Array[Object.Distance])Search results.

    • Object.Distance

      fieldtypelabeldescription
      idstringThe vector ID.
      distancefloatThe distance between the result vector and the request vector.

Status Code

codename
0OK
1CANCELLED
3INVALID_ARGUMENT
4DEADLINE_EXCEEDED
5NOT_FOUND
10ABORTED
13INTERNAL

Please refer to Response Status Code for more details.

Troubleshooting

The request process may not be completed when the response code is NOT 0 (OK).

Here are some common reasons and how to resolve each error.

namecommon reasonhow to resolve
CANCELLEDExecuted cancel() of rpc from client/server-side or network problems between client and server.Check the code, especially around timeout and connection management, and fix if needed.
INVALID_ARGUMENTThe Dimension of the request vector is NOT the same as Vald Agent’s config, or some request payload is invalid.Check Agent config, request payload, and fix request payload or Agent config.
DEADLINE_EXCEEDEDThe RPC timeout setting is too short on the client/server side.Check the gRPC timeout setting on both the client and server sides and fix it if needed.
NOT_FOUNDSearch result is empty or insufficient to request result length.Send a request with another vector or set min_num to a smaller value.
INTERNALTarget Vald cluster or network route has some critical error.Check target Vald cluster first and check network route including ingress as second.

StreamLinearSearchByID RPC

StreamLinearSearchByID RPC is the method to linear search vectors with multi queries(IDs) using the bidirectional streaming RPC.
Using the bidirectional streaming RPC, the linear search request can be communicated in any order between the client and server. +Each LinearSearchByID request and response are independent.

Input

  • the scheme of payload.v1.Search.IDRequest stream

    message Search {
+  message IDRequest {
+    string id = 1;
+    Config config = 2;
+  }
+
+  message Config {
+    string request_id = 1;
+    uint32 num = 2 [ (validate.rules).uint32.gte = 1 ];
+    int64 timeout = 5;
+    Filter.Config ingress_filters = 6;
+    Filter.Config egress_filters = 7;
+    uint32 min_num = 8;
+    AggregationAlgorithm aggregation_algorithm = 9;
+  }
+}
+
+enum AggregationAlgorithm {
+  Unknown = 0;
+  ConcurrentQueue = 1;
+  SortSlice = 2;
+  SortPoolSlice = 3;
+  PairingHeap = 4;
+}
+

    • Search.IDRequest

      fieldtypelabelrequireddescription
      idstring*The vector ID to be searched.
      configConfig*The configuration of the search request.

    • Search.Config

      fieldtypelabelrequireddescription
      request_idstringUnique request ID.
      numuint32*The maximum number of results to be returned.
      timeoutint64Search timeout in nanoseconds (default value is 5s).
      ingress_filtersFilter.ConfigIngress Filter configuration.
      egress_filtersFilter.ConfigEgress Filter configuration.
      min_numuint32The minimum number of results to be returned.
      aggregation_algorithmAggregationAlgorithmThe search aggregation algorithm option (default value is ConcurrentQueue).

Output

  • the scheme of payload.v1.Search.StreamResponse.

    message Search {
+  message StreamResponse {
+    oneof payload {
+      Response response = 1;
+      google.rpc.Status status = 2;
+    }
+  }
+
+  message Response {
+    string request_id = 1;
+    repeated Object.Distance results = 2;
+  }
+}
+
+message Object {
+  message Distance {
+    string id = 1;
+    float distance = 2;
+  }
+}
+

    • Search.StreamResponse

      fieldtypelabeldescription
      responseResponseThe search result response.
      statusgoogle.rpc.StatusThe status of Google RPC.

    • Search.Response

      fieldtypelabeldescription
      request_idstringThe unique request ID.
      resultsObject.Distancerepeated(Array[Object.Distance])Search results.

    • Object.Distance

      fieldtypelabeldescription
      idstringThe vector ID.
      distancefloatThe distance between the result vector and the request vector.

Status Code

codename
0OK
1CANCELLED
3INVALID_ARGUMENT
4DEADLINE_EXCEEDED
5NOT_FOUND
10ABORTED
13INTERNAL

Please refer to Response Status Code for more details.

Troubleshooting

The request process may not be completed when the response code is NOT 0 (OK).

Here are some common reasons and how to resolve each error.

namecommon reasonhow to resolve
CANCELLEDExecuted cancel() of rpc from client/server-side or network problems between client and server.Check the code, especially around timeout and connection management, and fix if needed.
INVALID_ARGUMENTThe Requested vector’s ID is empty, or some request payload is invalid.Check request payload and fix request payload.
DEADLINE_EXCEEDEDThe RPC timeout setting is too short on the client/server side.Check the gRPC timeout setting on both the client and server sides and fix it if needed.
NOT_FOUNDThe Requested ID is not inserted on the target Vald cluster, or the search result is insufficient to the required result length.Send a request with another vector or set min_num to a smaller value.
INTERNALTarget Vald cluster or network route has some critical error.Check target Vald cluster first and check network route including ingress as second.

MultiLinearSearch RPC

MultiLinearSearch RPC is the method to linear search vectors with multiple vectors in 1 request.

gRPC has a message size limitation.
Please be careful that the size of the request exceeds the limit.

Input

  • the scheme of payload.v1.Search.MultiRequest

    message Search {
+  message MultiRequest {
+    repeated Request requests = 1;
+  }
+
+  message Request {
+    repeated float vector = 1 [ (validate.rules).repeated .min_items = 2 ];
+    Config config = 2;
+  }
+
+  message Config {
+    string request_id = 1;
+    uint32 num = 2 [ (validate.rules).uint32.gte = 1 ];
+    int64 timeout = 5;
+    Filter.Config ingress_filters = 6;
+    Filter.Config egress_filters = 7;
+    uint32 min_num = 8;
+    AggregationAlgorithm aggregation_algorithm = 9;
+  }
+}
+
+enum AggregationAlgorithm {
+  Unknown = 0;
+  ConcurrentQueue = 1;
+  SortSlice = 2;
+  SortPoolSlice = 3;
+  PairingHeap = 4;
+}
+

    • Search.MultiRequest

      fieldtypelabelrequireddescription
      requestsrepeated(Array[Request])*The search request list

    • Search.Request

      fieldtypelabelrequireddescription
      vectorfloatrepeated(Array[float])*The vector data. Its dimension is between 2 and 65,536.
      configConfig*The configuration of the search request.

    • Search.Config

      fieldtypelabelrequireddescription
      request_idstringUnique request ID.
      numuint32*The maximum number of results to be returned.
      timeoutint64Search timeout in nanoseconds (default value is 5s).
      ingress_filtersFilter.ConfigIngress Filter configuration.
      egress_filtersFilter.ConfigEgress Filter configuration.
      min_numuint32The minimum number of results to be returned.
      aggregation_algorithmAggregationAlgorithmThe search aggregation algorithm option (default value is ConcurrentQueue).

Output

  • the scheme of payload.v1.Search.Responses.

    message Search {
+  message Responses {
+    repeated Response responses = 1;
+  }
+
+  message Response {
+    string request_id = 1;
+    repeated Object.Distance results = 2;
+  }
+}
+
+message Object {
+  message Distance {
+    string id = 1;
+    float distance = 2;
+  }
+}
+

    • Search.Responses

      fieldtypelabeldescription
      responsesResponserepeated(Array[Response])The list of search results response

    • Search.Response

      fieldtypelabeldescription
      request_idstringThe unique request ID.
      resultsObject.Distancerepeated(Array[Object.Distance])Search results.

    • Object.Distance

      fieldtypelabeldescription
      idstringThe vector ID.
      distancefloatThe distance between the result vector and the request vector.

Status Code

codename
0OK
1CANCELLED
3INVALID_ARGUMENT
4DEADLINE_EXCEEDED
5NOT_FOUND
10ABORTED
13INTERNAL

Please refer to Response Status Code for more details.

Troubleshooting

The request process may not be completed when the response code is NOT 0 (OK).

Here are some common reasons and how to resolve each error.

namecommon reasonhow to resolve
CANCELLEDExecuted cancel() of rpc from client/server-side or network problems between client and server.Check the code, especially around timeout and connection management, and fix if needed.
INVALID_ARGUMENTThe Dimension of the request vector is NOT the same as Vald Agent’s config, or some request payload is invalid.Check Agent config, request payload, and fix request payload or Agent config.
DEADLINE_EXCEEDEDThe RPC timeout setting is too short on the client/server side.Check the gRPC timeout setting on both the client and server sides and fix it if needed.
NOT_FOUNDSearch result is empty or insufficient to request result length.Send a request with another vector or set min_num to a smaller value.
INTERNALTarget Vald cluster or network route has some critical error.Check target Vald cluster first and check network route including ingress as second.

MultiLinearSearchByID RPC

MultiLinearSearchByID RPC is the method to linear search vectors with multiple IDs in 1 request.

gRPC has a message size limitation.
Please be careful that the size of the request exceeds the limit.

Input

  • the scheme of payload.v1.Search.MultiIDRequest stream

    message Search {
+
+  message MultiIDRequest {
+      repeated IDRequest requests = 1;
+  }
+
+  message IDRequest {
+    string id = 1;
+    Config config = 2;
+  }
+
+  message Config {
+    string request_id = 1;
+    uint32 num = 2 [ (validate.rules).uint32.gte = 1 ];
+    int64 timeout = 5;
+    Filter.Config ingress_filters = 6;
+    Filter.Config egress_filters = 7;
+    uint32 min_num = 8;
+    AggregationAlgorithm aggregation_algorithm = 9;
+  }
+}
+
+enum AggregationAlgorithm {
+  Unknown = 0;
+  ConcurrentQueue = 1;
+  SortSlice = 2;
+  SortPoolSlice = 3;
+  PairingHeap = 4;
+}
+

    • Search.MultiIDRequest

      fieldtypelabelrequireddescription
      requestsIDRequestrepeated(Array[IDRequest])*The searchByID request list.

    • Search.IDRequest

      fieldtypelabelrequireddescription
      idstring*The vector ID to be searched.
      configConfig*The configuration of the search request.

    • Search.Config

      fieldtypelabelrequireddescription
      request_idstringUnique request ID.
      numuint32*The maximum number of results to be returned.
      timeoutint64Search timeout in nanoseconds (default value is 5s).
      ingress_filtersFilter.ConfigIngress Filter configuration.
      egress_filtersFilter.ConfigEgress Filter configuration.
      min_numuint32The minimum number of results to be returned.
      aggregation_algorithmAggregationAlgorithmThe search aggregation algorithm option (default value is ConcurrentQueue).

Output

  • the scheme of payload.v1.Search.Responses.

    message Search {
+  message Responses {
+    repeated Response responses = 1;
+  }
+
+  message Response {
+    string request_id = 1;
+    repeated Object.Distance results = 2;
+  }
+}
+
+message Object {
+  message Distance {
+    string id = 1;
+    float distance = 2;
+  }
+}
+

    • Search.Responses

      fieldtypelabeldescription
      responsesResponserepeated(Array[Response])The list of search results response.

    • Search.Response

      fieldtypelabeldescription
      request_idstringThe unique request ID.
      resultsObject.Distancerepeated(Array[Object.Distance])Search results.

    • Object.Distance

      fieldtypelabeldescription
      idstringThe vector ID.
      distancefloatThe distance between the result vector and the request vector.

Status Code

codename
0OK
1CANCELLED
3INVALID_ARGUMENT
4DEADLINE_EXCEEDED
5NOT_FOUND
10ABORTED
13INTERNAL

Please refer to Response Status Code for more details.

Troubleshooting

The request process may not be completed when the response code is NOT 0 (OK).

Here are some common reasons and how to resolve each error.

namecommon reasonhow to resolve
CANCELLEDExecuted cancel() of rpc from client/server-side or network problems between client and server.Check the code, especially around timeout and connection management, and fix if needed.
INVALID_ARGUMENTThe Requested vector’s ID is empty, or some request payload is invalid.Check request payload and fix request payload.
DEADLINE_EXCEEDEDThe RPC timeout setting is too short on the client/server side.Check the gRPC timeout setting on both the client and server sides and fix it if needed.
NOT_FOUNDThe Requested ID is not inserted on the target Vald cluster, or the search result is insufficient to the required result length.Send a request with another vector or set min_num to a smaller value.
INTERNALTarget Vald cluster or network route has some critical error.Check target Vald cluster first and check network route including ingress as second.

See also

Insert

Insert vectors to the Vald cluster

Update

Update vectors indexed in the Vald cluster

Upsert

Update or Insert vectors to the Vald cluster

Remove

Remove indexes from the Vald cluster