-
Notifications
You must be signed in to change notification settings - Fork 1
SensorWeb cloud
Some of the tasks I can think of for this are:
- Design and documentation of the API. So far we have only designed and documented the API for clients. We have the SensorThings API as a good base for our own public API, but we still need to define an API for all the other stuff surrounding it. This involves some deep reading and understanding of the user stories and the SensorThings API.
- API endpoints implementation. Just implementing the barebones of the API. No handling logic.
- SensorUp API wrapper implementation. Make our public API handling logic use SensorUp's API sandbox.
Estimation: 3 weeks.
There's a specification for a suite of conformance tests. It would be great to have them in place before starting the work in our own SensorThings API implementation. And it makes sense to have them after building the SensorUp sandbox wrapper, so we can test our own tests.
Estimation: 1 week.
This task needs a deeper analysis of the API specification that I didn't do yet, so I can't really tell about the smaller tasks.
Estimation: 12 weeks
Even if we don't use an external IdP for the MVP, we need to implement the concept of SensorWeb user. Without IdP these will be anonymous users. For each user we will be storing things like push endpoints, session tokens, the list of watched sensors, etc.
Estimation: 2 weeks without IdP. +3 weeks for IdP integration.
This requires dealing with two external APIs (Apple's and Googles's one) and implementing the logic to handle and using the different push endpoints per user.
Estimation: 3 weeks.
Estimation: 2 weeks.
This one is hard to estimate and even if it listed after all the other tasks, its priority should also be high. This should be a continuous effort, just like testing. Just to give a number, I would say that we can spend 5 weeks polishing all the security related stuff (TLS, client credentials, tokens, scopes, security reviews, etc.).
Estimation: 5 weeks.
This one depends on how much functionality we want to add here. Since this is an internal tool and there are no user stories specifically related to this, we have more flexibility here. IMHO at the very least we need a way to generate and revoke API credentials (the basics of this is already done), manage users (if we finally have IdP) and sensors. Basic authentication is also done, but we need to move to something more solid (likely FxA OAuth, with session management).
Estimation: 6 weeks.
There are some other basic things that we need to do for any cloud service.
- Backoff protocol
- Load tests
- Proper logging
- Cache
- Telemetry
- Load balance strategy. We will be periodically receiving requests from sensor stations every X seconds. We should implement a load balance strategy to spread sensor requests in different temporal windows.
This still concerns me. If we want to have data privacy, I think we need something like what I suggest here. We will need to check with rfkelly how much effort would be required to implement something like that or if he can think of an alternative and simpler approach.
TBD
TBD
TBD
Given that we control part of the clients that will be making requests to our cloud services, we can implement some strategies to reduce the number of these requests.
-
Sensor stations may implement different strategies to limit the number of requests to the SW services. For example, a sensor station may choose to batch observations during night time or if no change is found with respect to a previous observation.
-
The cloud service may also implement a protocol to balance the load by spreading the number of requests coming from sensor stations in time. Because we know the number of active sensor stations at any given time and we know the ideal refresh rate, we can create and provide different time windows for different sensor stations to make their requests. This way we will avoid the worst case scenario where all the sensor stations push their observation data at the same exact time. For example, let's say that we expect to have 1K active sensor stations. And we want to get observations every 20 seconds. We will be creating for example 5 different windows for observations. Each sensor station will need to ask to the cloud service for one of these time windows to push its observations. With these numbers, if we follow this protocol, we would be constantly receiving 50 requests every second.
-
While sensor stations may batch requests or have a very high refresh rate, we still want to provide close to real time data. To achieve that, the cloud service could request a sensor station to send its latest queued observations by sending a push notification (via MQTT?).
- The SW web app can implement offline support (via Service Workers and Cache APIs) to reduce the number of requests required to the cloud service.