Question: Does it flag 'deletion/removal' of entities as changes?

[veenarm]

I'm curious if a component/entity etc was deleted, would it detect that it was removed and send a remove notification to the end point?

This is the only thing I need to ensure data integrity across the systems, just curious before I start testing it out.

[jgentes]

Sorry for not responding @veenarm I apparently need to adjust my notification settings!

Unfortunately today it does not know when a component is deleted, since it actually disappears from the Backstage catalog. I'm not sure why Backstage doesn't use some kind of archived status rather than removing it, but that's where the problem lies.

The plugin does maintain a cache, which is how it determines what changes to send, however that cache is ephemeral and disappears when a reset is triggered from Compass, or if Backstage restarts. A solution might be to store a more persistent version of the cache, then compare a full sync with the cache, and if something disappears, we could send some indication of a deletion.

[veenarm]

Could it be modified to use the in built cache service? This would put it in redis or infinispan etc and relatively easy to setup. Well simpler than keeping it in the db?

[jgentes]

It actually does already use the backstate cache service: https://github.com/ForgedApps/backstage-catalog-webhook-plugin/blob/main/src/cache.ts

So if you've configured backstage to use a persistent cache, then I suppose it would work. Is that how you've configured Backstage?

[veenarm]

Yeah we use infinispan as a clustered cache service across data centres with replication. It's not persisted to disk but it's. Distributed cache so can handle several failures before data loss. We do plan to add other type of storage as well.

[jgentes]

Ok, I'll take a look at adding this. I'm thinking it would simply add a deleted: true key to the entity, but curious if you have any thoughts on how you would like to receive the information.

Since the cache only includes the uid from the backstage entity, I think I would need to update the cache to include the entityRef as well, otherwise it may be hard to determine what disappeared unless the uid is also being stored on the target.

I built this to sync with Compass, which doesn't track the uid so at least in my use case, I would need the entityRef as well to find the right component to archive.

[veenarm]

I'd say the entity ref and action type how you've mentioned sounds good. If you get something working I'm happy to test it out or if you need a or or some work from me (which I'm happy to do) may take a little longer sorry working on a few other things semi related :)

[jgentes]

No problem, I'll let you know when I have something ready.