Selective events / data stored in redis

[vlotech]

For our use-case we've switched of the persist option in core-data. The high event rate was flooding the database and letting it crash after a while.

Cleaning up through the scheduler would be an option, although that doesn't give proper control over which data to keep and what to clean.

Therefore, I was looking into means to write specific streams / topics / profiles to core-data.
I managed to build up a new Event, made a eventRequest and used the eventClient to send the event. I do see the event on the redis
Pub/Sub but it nevers ends at the key/value store.

	eventClient := service.EventClient()

	if eventClient != nil {

		event := dtos.NewEvent("testProfile", "ObjectParser", "Packet")
		event.Id = "d5471d59-2810-419a-8744-18eb8fa03465"

		event.AddObjectReading("Packet", packet)
		eventRequest := requests.NewAddEventRequest(event)

		bgctx := context.Background()
		eventClient.Add(bgctx, "app-object-parser", eventRequest)
	}

What would be the right way to solve this? Is there an API to write directly into redis or would a second database be the only solution?

[cloudxxx8]

The usage of event client is correct. Did you rewrite the edgex-go core-data persistence layer? What is the new database? If the message bus (redis pub/sub) receives the events, the core-data works fine.
If you modified the persistence layer, the Redis key/value store won't persist any new events.

Another solution is the Core Data Retention feature. We just finished the implementation and under review:
https://docs.edgexfoundry.org/3.0/design/ucr/Core-Data-Retention/
edgexfoundry/edgex-go#4668

This feature will be available in v3.1

[vlotech]

I was using the default redis message bus / database.
I disabled the persistence in core-data using the config flag, so no data is stored. Some data (from the final processing step) I do want to store. That is the challenge. To select specific data streams to be stored and others not.
The feature as described does seem to help in that way.
I'm looking forward testing it.

[cloudxxx8]

You should write an App Service to filter and store the events to the database you want.

[vlotech]

If I want to reuse the redis database for it, can I use the core-data API or should i communicate directly with Redis (or any other database, as that goes beyond the edgeX core functions?

[cloudxxx8]

Yes, you can enable the Core Data persistence, but modify the MessageQueue.SubscribeEnabled = false if you are using 2.3.
In this case, Core Data will only accept your REST call and store the Events.
If you are using v3, there is no way to disable the subscription, but you can modify MessageBus.BaseTopicPrefix to an irrelavent topic to prevent Core Data to receive any message coming from Device Service directly.

[vlotech]

As I'm running on v3, I'll try the proposed solution.