LWM2M: send arbitrary data

[jvermillard]

Is your feature request related to a problem? Please describe.

I work on a system where I need to collect various resources according to specific filtering rules (configured using an LWM2M object). After some time, I want to send the accumulated data using an LWM2M send operation.

The problem with the current Zephyr LWM2M API is that you are bound to the values resource changes in the cache system, and trigger the sending. If you want to filter the data according to specific business rules, it's not possible.

The LWM2M 1.1 Core spec says: "The Resources and Resource Instances to send are implementation specific." But the way it's implemented in Zephyr, you will collect all the value changes for the "cached" resources.

Describe the solution you'd like

Either a lower-level send API that permits sending any timeseries payload, or a way to filter the values going into the cache before triggering the LWM2M send operation.

[Edit]

I put more thought into what a solution would look like:

First, manage a sample ring in the application side and not in the Zehpyr LWM2M client, so the application would have the requested flexibility to accumulate and drop samples as the business logic would require.
Now, on the Zephyr LWM2M client side, we would need a new API, allowing us to provide a list of samples in the form of:

struct lwm2m_ts_sample {
    struct lwm2m_obj_path path;   /*  /object/instance/resource[ /res‑inst ] */
    enum lwm2m_value_type type; 
    int64_t timestamp_ms;         /* Epoch milliseconds (UTC)  */
    union {
        int64_t  v_int;
        double   v_float;
        bool     v_bool;
        struct {
            const uint8_t *ptr;
            uint16_t len;
        } v_buf;
    } value;
};

The Zephyr LWM2M client would be responsible for encoding data as SenML (CBOR/JSON) and handling the CoAP blockwise transfer.
Of course, this mechanism would be in parallel with the current cache-based implementation, and the code for encoding/sending the senML payload would be refactored to be used for both code paths.

Describe alternatives you've considered

Using observer (or compose observe) is not an option for me because it generates more packets, so larger payloads and longer on-air time, on a battery-sensitive application.

Contribution?

If I could provide the effort to spearhead this development, would the project be interested in such a contribution?

[rlubos]

Hi @jvermillard,

If I could provide the effort to spearhead this development, would the project be interested in such a contribution?

If you're willing to put effort into implementing such functionality I don't see why not. Shall I reassign the issue to you?

One option could indeed be providing a custom lwm2m_send_*() API for application-provided timeseries data, however, my concern here is that the LwM2M encoders are quite tightly integrated with the LwM2M registry, I'm not sure how much code reusability could be achieved here beyond the most basic struct lwm2m_writer APIs. But encoding seems to be the biggest challenge here (unless I miss something), other than that it should integrate with block transfer and SEND notification mechanism easily I think.

I'm wondering however about the alternate solution you've described - cache filtering. Perhaps one option could also be providing an alternate set of lwm2m_set_*() APIs, lwm2m_set_*_no_cache() so that the application can decide whether to cache specific data sample or not. Under the hood it should be a pretty straightforward change to lwm2m_engine_set() if I'm not mistaken, and there wouldn't be a need to modify the existing message encoding logic at all? Not sure however if I'm not missing something here, the LwM2M client library has grown into a pretty complex beast over time.

[rlubos]

Adding @SeppoTakalo for visibility.

[SeppoTakalo]

I feel that touching the send API would be a bit wrong on this case.
lwm2m_send() should send exactly what user requested. If filtering is wanted, it must be done on the application level, before feeding the path list to lwm2m_send().

But I think the cache-filtering makes sense.
If I can propose, you could develop this kind of API:

/**
 * @brief Callback function type for filtering LwM2M cache entries.
 *
 * This callback is used to determine whether a specific LwM2M cache entry
 * should be included or.
 *
 * @param path   The LwM2M resource path.
 * @param element data element to be filtered.
 *
 * @return true if the entry should be included, false if it should be dropped.
 */
typedef bool (*lwm2m_cache_filter_cb_t)(struct lwm2m_obj_path *path,
					struct lwm2m_time_series_elem *element);

/**
 * @brief Set a filter callback for LwM2M cache entries.
 *
 * This function allows the application to register a filter callback that will be used
 * to determine whether specific cache entries should be included or dropped.
 *
 * @param path LwM2M path to resource as a struct
 * @param filter_cb Callback function for filtering cache entries
 *
 * @return 0 for success or negative in case of error.
 */
int lwm2m_set_cache_filter(const struct lwm2m_obj_path *path, lwm2m_cache_filter_cb_t filter_cb);

The cache filter callback would then be called from lwm2m_cache_write() and it will drop any entries that filter is requesting to drop. Pointer to callback can be stored in struct lwm2m_time_series_resource so overall this requires minimal changes to any lwm2m registry logic.

[jvermillard]

Thanks for the feedback. After some thought, I prefer a lwm2m_send function where you provide the samples you want to send. Filtering using a callback still enforces the ring buffer on each resource. In my case, I want to have a single global bucket to accumulate my samples and drop based on the age.

[jvermillard]

Sorry for the delay; I needed to follow up with my customer on this. We agree with the filtering trade-off proposed by @SeppoTakalo :

• it keep the current cache system and doesn't create a new mechanism for send operations
• simpler to implement (famous last word in software engineering..)
• but doesn't allow us to shape our caching strategy exactly like we wanted, but that's fine

To clarify the context, the goal is to create an object ("send scheduler") to accumulate samples based on a list of rules (like observation attributes) and connect&push to the server based on some connection rules (max number of samples collected, maximum time,..).
@rlubos you can assign it to me, but don't expect me to start soon, it's an early project