Worker Indexing Changes

The goal of Project Stein is to shift the functions indexing responsibility from the host to the worker. Some changes were made to the host to enable this new role, so this wiki page will go over the details of what changes were made and why they were necessary.

Current host flow without worker indexing:

At ScriptHost startup, the host requests for HostFunctionMetadataProvider to index the functions.

---

New host flow with worker indexing:

At ScriptHost startup, the host requests for the worker to index the functions and send back the function metadata through WorkerFunctionMetadataProvider.

---

Changes that were made

• New metadata provider: WorkerFunctionMetadataProvider communicates with the worker to get the function metadata. It also validates the received metadata before sending it back to FunctionMetadataManager

• New metadata provider factory: FunctionMetadataProviderFactory was created to provide a specific metadata provider to FunctionMetadataManager depending on if the host was indexing or if the worker was indexing. Also, to make the distinction between each type of metadata provider more clear, the original FunctionMetadataProvider.cs file was renamed to HostFunctionMetadataProvider.

• Edited FunctionMetadataManager constructor: added the metadata provider factory to the constructor for later use in LoadFunctionMetadata (getting the appropriate provider and calling GetFunctionMetadata on it).

• New GRPC message types: WorkerFunctionMetadataRequest, WorkerFunctionMetadataResponse, WorkerFunctionMetadata

  WorkerFunctionMetadataRequest sends the function app directory to the worker.

  WorkerFunctionMetadataResponse includes a list of WorkerFunctionMetadata messages and an overall StatusResult attribute.

  WorkerFunctionMetadata includes all information necessary to populate FunctionMetadata and convey the indexing status for each function (ie. success, failure, etc.). Each of these messages corresponds to a single function.

• Added new initialization code path to RpcFunctionInvocationDispatcher: new functions break up the original InitializeAsync logic to allow the host to send the WorkerMetadataRequest in the middle of the channel initialization process. The logic also needed to be broken up because the invocation buffers and invocation requests can only be made once the host has metadata to operate with.

• Added feature flag, FeatureFlagEnableWorkerIndexing, in front of new worker indexing code path

• Added WorkerCapabilities service to store capabilities for each worker runtime in a central structure. Every time a new worker channel sends its capabilities to the host, we update the _workerCapabilities structure in WorkerCapabilities that can later be queried through GetCapabilityValue(runtimeLanguage, capabilityName)