More Communication Protocols

[djungelorm]

The underlying server communication protocol should be moddable, to allow communication via other means than just google's protocol buffers.

Then we can support, for example:

• http://en.wikipedia.org/wiki/SOAP to make it easy to interact with kRPC from web apps (e.g. JavaScript)
• CCSDS protocol???

[thesamprice]

Send me an email, I could probably help with the javascript / CCSDS protocol stuff.
My C# skills are pretty bad, but I know javascript,c, python.

[djungelorm]

This is fairly low on my priority list right now, but when I do get round to it I'll put together a prototype and if you could try out the SOAP interface from JS that'd be great (I don't know JS very well...)

I was thinking we could allow additional DLLs to be placed in kRPC's plugin directory, which would provide additional communication layers. The in-game server interface could then allow you to choose which communication protocol you want the server to use.

[kleiram]

Quick suggestion: REST instead of SOAP, since it's also HTTP but a lot easier to interface with!

[Outurnate]

GRPC might be cheap to generate code for

[sgayda2]

GRPC would solve a lot of your issues, you are already doing what grpc does, proto messages over ipc but it will do a lot more as well, and make it more reliable and stable.

[gitanat]

Seconding GRPC, it seems well maintained and solves the problem of having to write client libraries.

Unfortunately it won't help for getting data in the browser, as GRPC has no defined mechanism for interacting with it (like websockets).

[djungelorm]

I've only had a quick look at GRPC, but here are two issues that came to mind compared to the current server protocol:

• I'm not sure GRPC would allow for kRPC "streams". GRPC has "streams" but they sound a bit different as "the client completes once it has all the server's responses."
• I don't think dynamic clients are possible with GRPC? The kRPC Lua and Python clients currently get a list of available RPCs from the server and create the stubs dynamically. This keeps their implementation nicely separated from the server.

Also, I've started working on making the server protocol used by kRPC extensible. My vision is that the user will be able to start one or more servers, and each server can be of a different 'type'. I see no reason why GRPC couldn't be included as one of these 'server types'. Then GRPC can even run alongside the current server protocol (with its streams and dynamic client support). A REST 'server type' is also planned.

[djungelorm]

In case you're interested, the changes are on this branch: https://github.com/krpc/krpc/tree/feature/server-refactoring

[sgayda2]

I have used GRPC streaming in a project and it worked by having the client connect to the server to get data. The server would never finish sending data so the stream was maintained until the client decided to end the connection.

[gitanat]

I don't think dynamic clients are possible with GRPC? The kRPC Lua and Python clients currently get a list of available RPCs from the server and create the stubs dynamically. This keeps their implementation nicely separated from the server.

Indeed, GRPC does force you to have a well defined interface, and compiling protobufs at runtime is not easy.

Is that added flexibility useful in practice, though? Doesn't the current code have to agree on the protobufs ahead of time anyway?

If we do get gRPC, REST would come for free, if we are willing to run a external program

[djungelorm]

Yes the current code does have to agree on the protobufs, but not the RPCs. The protobufs just define what a request/response message looks like, not what the RPCs are. So you can add/change RPCs on the server and the python/lua clients automatically pick up the changes.

The existing C++, C# and Java clients have generated stubs though, so these could be just be replaced with gRPC.

[Lokaltog]

Let me know if you need help testing a websockets/REST API, I'll definitely port KeRD to kRPC as soon as this is ready.

[djungelorm]

I've now got the server refactored into a state where I can start adding additional communication protocols, and I'm going to start by adding ReST+websockets. Here my thoughts on a potential design. Comments welcome!

RPC Server

Uses ReST over HTTP. Requests are sent to the server as GET requests.

• Requests to call static methods are of the form: /service-name/method?arg1=value&arg2=value
• Requests to call object methods are the form: /service-name/object-<objectid>/method?arg1=value

Some examples:

• Get the active vessel: /space-center/get-active-vessel
  (Returns a response containing the vessels object id number)
• Get a vessels position: /space-center/vessel-4/get-position
  (where 4 is the object id of the vessel in question)
• Get the control object for a vessel: /space-center/vessel-4/control
• Enable RCS: /space-center/control-3/set-rcs?value=true using the id returned by the previous request
• Create a maneuver node: /space-center/control-3/add-node?ut=12345&prograde=100

Responses are sent as JSON formatted strings, following the JSON API specification: http://jsonapi.org/

Stream server

For this, I don't think ReST over HTTP is enough, as the server needs to send data to the client continuously. I think websockets would be more suitable? The client could connect to the websocket, send a request (formatted in the same manner as above, but just putting the URI in a text frame?) and the server will then send repeated messages containing the return value of the request. If the client sends additional requests, their results could be added to the response messages sent by the server, so that clients can have multiple streams over the same TCP connection.

Now I'm going to go ponder how to fit gRPC into all of this!

[Lokaltog]

I'm personally not sure if a separate REST API is required for this plugin, or at least if it should be prioritized over WS. After playing around with the Python API, I'd prefer a very similar RPC API for JS using only websockets. The reason for this is that there's much more overhead requesting stuff via HTTP, so polling some resource at e.g. 10hz would be unreasonably resource intensive with a REST-based API, but would work just fine with websockets.

A REST API would in my opinion be better suited for applications where the data doesn't change as frequently, or in this case maybe limited to actions instead of data retrieval - e.g. for RCS like you mentioned PUT /space-center/control/3/rcs with data enabled=true or something like that.

Just a point regarding REST APIs: they're usually implemented using existing HTTP methods instead of having method names in the URI, so data retrieval would use e.g. GET /space-center/active-vessel as opposed to GET /space-center/get-active-vessel, changing data would use the PUT method, creating data the POST method and deleting data the DELETE method.

I was looking around for a simple temporary workaround to start playing around with websockets, and noticed that the websockify project might be able to accomplish what I think would work great as a JS API. Basically it would expose the gRPC/protobuf(?) API directly to JS via websockets, then something like protobuf.js could possibly be used to decode and encode communications with kRPC. The JS API would then expose actions and data through objects similar to e.g. the Python API. By doing it this way you'll also be able to keep the JS API similar to the other RPC APIs which is a plus.

What do you think?

[djungelorm]

You make a good point about performance. Having to create a new tcp connection for every request would be painful!

I guess what we are really after then is a JS client with some code to wrap server-side object ids up as JS objects. This is separate to the actual transport layer underneath, so could be done via gRPC (or the current protocol based on protobuf, but would take more work to implement the client). Then there's no need for websockets or REST over HTTP at all.

I don't know if gRPC can provide this object id wrapping though?

[Lokaltog]

You'd still need websockets support in kRPC (currently no browser can open regular TCP connections), but I'm sure there's some library or wrapper for C# that can expose the kRPC socket as a websocket (Telemachus has a simple WS server implementation, and here's more info related to writing WS servers). If you're able to implement a way to connect to kRPC via websockets I can definitely help writing the JS bindings if needed.

[djungelorm]

Ah I see. Sorry my knowledge of browser based programming is rather lacking. Didn't know you can't just open a TCP connection!

So I guess option 1 is to add a websockets server to kRPC, and use the same protobuf based serialization over it just like the current TCP based protocol. Then implement a JS client (just like the current clients).

Or option 2: we add a gRPC server to kRPC and just use their JS client. But I guess this would not work in a browser if it needs to open a TCP connection? Or can we do gRPC over websockets?

Also, some other thoughts on gRPC: I've been having a read of the gRPC documentation, and it doesn't look like it can provide the same "remote objects" that the current kRPC clients do. It follows the design philosophy of "services not objects and messages not references". While adding a gRPC server would give us a JS and many other clients for free (which is a good reason to add support for it) those clients would feel very different to the current clients. You would need to pass object ids around as plain old integers in the gRPC messages, rather than being able to make method calls on objects.

So option 2b is to add a gRPC server to kRPC, then build a JS client on top of the gRPC JS client that provides these "remote objects". Writing such a client would probably be less implementation with than writing a JS client for the current protocol, plus we get all the gRPC clients for free :)

I think I prefer option 2b if gRPC can run in a browser - less implementaion work and lots of free clients :)

[djungelorm]

To answer my own question, gRPC doesn't work in the browser. I guess we need option 1 for KeRD to work with kRPC.

[djungelorm]

There is also an official JS protobuf library we could use (instead of protobuf.js) for the encoding and decoding and it works in the browser: https://github.com/google/protobuf/tree/master/js

[Lokaltog]

Yep, option 1 sounds like the only real option (at least if you want to support browser based kRPC clients), even if it means more work implementing the JS bindings. Google's protobuf library sounds good! I can start working on JS bindings as soon as a websocket server is added to kRPC, do you know if it will require a lot of work implementing a WS server?

[djungelorm]

No, it should be fairly straight forward. I imagine I'll be able to get a prototype done over the weekend for you to try out.

[Lokaltog]

Awesome, I can't wait to check it out.

[gitanat]

This is looking great!

I just want to point out that, although browser-based clients can't possibly support gRPC (because of the lack of TCP support), there are some projects around that act as bridges (https://github.com/gengo/grpc-gateway), so we could get REST for free, if there's a way of running that from inside Kerbal (can you launch processes within unity?).

Technically, we could also get gRPC support from websockets using this project (https://github.com/kanaka/websockify), but we'd have to re-implement gRPC in javascript...

Having to create a new tcp connection for every request would be painful!

With HTTP (1.1), you can reuse the same connection. The overhead in HTTP really comes from the fact that it's request/response based - streams of data are not well supported, so the client has to constantly issue new requests.

[djungelorm]

@Lokaltog I've got a websockets server ready for you to play with :D

You can download it from here: https://krpc.s3.amazonaws.com/deploy/feature/websockets/375.1/krpc-0.3.0-30-g65ca751.zip

To use it, just install the plugin as usual, fire up KSP, choose a port number and hit start server. Then send KRPC.Request messages to it and receive KRPC.Response protobuf mesages from it using websockets frames with opcode = binary. Note that you don't need to send the size of the protobuf message like you do with the other protocol. The size is include in the ws frame header, so you just need to send the protobuf message data.

Here's an example script (in python) that I used to test it: https://gist.github.com/djungelorm/6372d31acc515fc63f18c889a8f5f631#file-ws-client-py

I haven't done much testing so expect bugs. I'll be doing some more testing tomorrow, so let me know if you have any issues!

If your interested in the server code it's here: 856f819b7fed55064e2094ee05d30ed0d48a6665

[Lokaltog]

Sweet! I just downloaded and installed it, and your test script works fine. I'll start working on some proof-of-concept JS code right away.

[Lokaltog]

I've reimplemented the basic example using webpack and proto-loader which uses protobuf.js to compile .proto files to JSON and handle the message encoding and decoding. Here's the example script in JS:

var ProtoBuf = require('protobufjs')
var proto = ProtoBuf.loadJson(require('krpc.proto')).build()
var sock = new WebSocket('ws://localhost:50000')
sock.binaryType = 'arraybuffer'

sock.onopen = (ev) => {
    let req = new proto.krpc.schema.Request('KRPC', 'GetStatus')
    sock.send(req.toArrayBuffer())
}

sock.onmessage = (ev) => {
    let resp = proto.krpc.schema.Response.decode(ev.data)
    let status = proto.krpc.schema.Status.decode(resp.return_value)
    console.log(status)
}

This prints the same status object as the Python script. I'll start reading up on how the kRPC Python bindings are actually implemented, then I'll start working on the JS bindings. I'll use protobuf.js for now, as Google's JS implementation is currently in a poorly documented beta.

[Lokaltog]

I'm currently attempting to set the client name and receive the client identifier like this, but I'm not receiving any response from kRPC when sending the hello message and client ID. The connection is also left open so I think some error may be occuring in kRPC. Is this feature not implemented yet?

[eXigentCoder]

I have managed to get a very basic node client library up and running if anyone feels like playing around, there are still a bunch of things to do like:

• Adding more tests
• Documentation on how to use the api
• Examples of how to use the api
• Decoders for return types of [double, float, sInt32, sInt64, uInt32, uInt64, bool, bytes, enum] see ./decoders.js file
• Check if adding and removing streams actually works
• Code clean up

I started off building code manually (see the apis/krpc.js file) but then realised I could generate it all from the getServices call, so switched to that (Code in the ./utilities/generate-service.js file).

So far I have tested:

• krpc.getStatus
• krpc.getServices
• krpc.getClients
• krpc.getCurrentGameScene
• spaceCenter.clearTarget

[gitanat]

Great stuff, thanks, ExigentCoder!
@djungelorm
Would it be possible to have this javascript api be part of krpc - much like the existing python and C++ clients?

[djungelorm]

Yeah it's possible. Although for now I think it would be easier for @eXigentCoder to work on it in its own repo?

Also, having it in the main kRPC release wouldn't really gain much from a users perspective. They would just be doing npm install krpc-node anyway, which is unaffected by where the source code is actually held. However, it would be good to have it documented on http://krpc.github.io/krpc

[eXigentCoder]

Yeah, I think with npm it should be fine to leave the source where it is. Especially since it's currently working with Travis to run CI & deployment. Once the documentation is done, would we just need to add a link from http://krpc.github.io/krpc to the node documentation file (would most likely be markdown) or would your rather have it as html on that site?

[djungelorm]

I don't mind if you want to host the documentation yourself - probably makes more sense as you could maintain it more easily. And I'm happy to link to it from all the various places.

[djungelorm]

I have now merged the websockets branch into the v0.4.0 branch, and have revamped the UI so now you can create multiple servers, and choose which protocol they use!

Latest build is here: http://krpc.s3-website-us-east-1.amazonaws.com/deploy/v0.4.0/692.1/

And here's a screenshot. The UI is functionally complete, although it could probably do with some visual improvements. Feedback welcome - you can probably tell I am not a UX designer!

[gitanat]

Fantastic news, thanks for all the work djungelorm!
Regarding the UI: is there a reason not to have all the servers active all the time? Does it consume a lot of memory / cpu?
I would expect most users to not know about TCP/Websockets. Developers will, of course, but many people would just use kRPC to get some external running.

[eXigentCoder]

I was having some issues with the release I was using and like to keep updated so I tried to use the latest 0.4.0 build krpc-0.3.6-73-g42d47a0 since I see the websockets branch has been merged in.
However my client no longer connects.

I went back in the websockets release branch and tried out the krpc-0.3.6-69-g49af2e4 build and it seems to have the same problem.
krpc-0.3.6-68-ge3e0eab works fine though, any idea what might have changed between the two releases so I can fix my code?
Thanks again!
My server connections are included in case I'm doing something silly there:

[djungelorm]

Oooo there is a nasty bug! Nothing to do with your client - the UI changes introduced an issue where reconfiguring a server doesn't update its event handlers, leading to it not starting properly.

Should be fixed in this version: http://krpc.s3-website-us-east-1.amazonaws.com/deploy/v0.4.0/706.1/

[eXigentCoder]

Awesome thanks, will test it out this evening!

[nevercast]

Hi @djungelorm

What is the status of WebSockets? Is it implemented and does it just wrap Protobuf?

Regards, NeverCast.

[eXigentCoder]

I've got most of the node.js library working (At least I think so, need to add a bunch more tests), however one part that definitely isn't working is connecting to the stream server.

Am I right in thinking this endpoint isn't upgraded yet to use WebSockets or have I just broken something?
The error I get when attempting to connect is:

WebSocket {
     domain: null,
     _events: 
      { error: [Object],
        close: [Object],
        message: [Object],
        open: [Object] },
     _eventsCount: 4,
     _maxListeners: undefined,
     _socket: null,
     _ultron: null,
     _closeReceived: false,
     bytesReceived: 0,
     readyState: 0,
     supports: { binary: true },
     extensions: {},
     _binaryType: 'arraybuffer',
     _isServer: false,
     url: 'ws://127.0.0.1:50001',
     protocolVersion: 13 } } Error: unexpected server response (400)

This is while using the v0.4.0- 711.1 build

[djungelorm]

@nevercast websockets is fully implemented and working in the upcoming v0.4.0 release (pre-release is available with it working). It does indeed use protobuf to serialize messages within the websockets messages.

@eXigentCoder The stream server should work. I'll have a look.

[nevercast]

@djungelorm Thanks excellent for my HTML UI needs. Cheers!

[magwo]

Really looking forward to the websocket support as I'm trying to create a browser-based (for mobile) throttle controller.

The idea is to have the iPhone sitting to the left of my keyboard with a throttle interface, instead of a HOTAS throttle.

[eXigentCoder]

@djungelorm sorry, I was on a bit of a break but am back now and trying to get everything working. I got the latest release of the 0.4 branch (0.4.0-122) and most of the existing code I had is still working except for streams.

I am trying to connect to the stream server using ws://127.0.0.1:50001 but the server returns a 400 Bad Request still.

Does anything there seem obviously wrong to you?

[djungelorm]

@eXigentCoder To connect to the stream server, you need to pass the RPC clients identifier in the connection string (as a base64 encoded string). For example: ws://127.0.0.1:50001?id=8qY/Dfa+lESsDIC2jGat7Q==

This is used to "link" the two connections together in the server side logic. Unfortunately I neglected to add a way for a websockets client to get hold of this id! I've added an RPC called KRPC.GetClientID to allow this.

Here's the latest build with this addition:
http://krpc.s3-website-us-east-1.amazonaws.com/deploy/v0.4.0/822.1/

And an example script that connects to the stream server:
https://gist.github.com/djungelorm/90fd690db0af5d01f4a31a3b811165b2

[eXigentCoder]

Thanks a lot, your code to get the client Id works perfectly, and I am now able to successfully establish a connection to the stream server!

I noticed in the 0.3x documentation, previously the AddStream method was expecting a Request object:

after taking a look at the source code it looks like it has been changed to take in a ProcedureCall:

So I sent through a ProcedureCall object to add stream and get back a successful response with the id and the stream socket is receiving data however I am unable to decode it. At the moment I'm trying to decode it to a StreamResult:

The buffer library throws the error Illegal group end indicator for Message .krpc.schema.StreamResult: 1 (not a group)

Am I trying to decode to the wrong type maybe?

[djungelorm]

Yeah you are decoding the wrong type. The stream server sends a sequence of StreamUpdate messages, so that's the type you want to decode.

[eXigentCoder]

Ah yes that makes more sense. I updated the code to decode to StreamUpdate but I get a similar error Illegal group end indicator for Message .krpc.schema.StreamUpdate: 1 (not a group).

Sorry to keep pestering you, will keep trying to get it working this side but if you have an idea of where to look I'd appreciate it.

[djungelorm]

There was a bug... The server was writing the size of the message, followed by the message data. I've fixed it to only write the message, so your code should hopefully now work.

Updated version available here:
http://krpc.s3-website-us-east-1.amazonaws.com/deploy/v0.4.0/827.1/

And here's the script I used to test it: https://gist.github.com/djungelorm/fb09f4683d8e988eebf4da63a2e340c7
It connects to both servers, sets up a stream for KRPC.GetStatus then prints out all the stream updates that it receives.

And no worries - happy to help!

[eXigentCoder]

Thanks so much, had a quick window to test out and it's working now, time to refactor it to be pretty and easier to use!

[djungelorm]

Quick update from me: I've now "officially" released v0.4.0 with the websockets server included. I also have a serial I/O protocol in the works for release very soon, along with a client library written in C using nanopb targetting embedded devices so you can use kRPC from an Arduino/whatever. I will have another look into adding gRPC support after that is released.

[djungelorm]

There has been some discussion on the discord regarding gRPC. While great to add it in theory, it isn't practical to do so. Here's a summary:

• gRPC requires HTTP/2 (if we want bidirectional streaming, which we do)
• HTTP/2 implementation is not available in Unity
• gRPC also requires .NET Core 3.0 and up which Unity does not support.
• There is an older gRPC implementation that might work, BUT it uses a native library for its transport implementation. This would cause headaches for cross platform support. Plus this implementation is no longer supported.

More info here: grpc/grpc-dotnet#1309