[DRAFT] statement-store: allow controlling of eviction policy

[alexggh]

Description

When a new statement is submitted the priority and channel is used to determine which statements get evicted when the used storage is above the account quota. The two are used like that:

• Submitting a new statement with a higher priority and the same channel automatically triggers the eviction of old statements.
• If not enough storage space has been created, either because channel is not specified, or because there is no statement already using the channel or because the new statement has higher storage requirements than the old statement, then we start evicting statements with lower priorities, starting from the lowest.

Problem

The limitation of this approach is that it does not allow applications for a way to decide which statements they still want to keep in the store. For example if an app submits A, B, C, D, E, F with increasing priorities and they know B and C are not needed anymore, they have a few options:

1. They can only replace with a new statement G just one of them by using the same channel, but they will be limited to how big B and C are. There is a trick around that by submitting an empty statement on B channel(EmptyB) with higher priority, and then submitting G on C's channel. This is really problematic because statement-store protocol does not have ordering guarantees, so on some nodes G could still arrive before EmptyB and trigger eviction of other statements.
2. Re-submit A with a higher priority than B and C, effectively making B and C the lowest priorities, then submit G, this is logically sound, but it has the big downside that it is inefficient and creates a lot of extra work in the network, imagine A, B, C, D, E, F, G, H, I, J, K, and you want to replace J and K.

Credit to @gui1117 for these ideas.

Proposal

Change the API to explicitly allow the App to state their replacement preference, the proposed way to achieve that is like this:

1. Repurpose the first u8 from the channel to identify a storage slot, the rest remains at the discretion of the App. The channel identifier is still given by the storage slot + the rest.
2. Add a bitmask field on the Statement called replacement_preference_mask, which allows App to identify storage slots that can be evicted.

The eviction policy becomes like this:

• First evict statements with lower priority and the same channel identifier.
• Then evict statements with lower priority with their slots specified in replacement_preference_mask.
• If we still did not create enough storage space then evict statements with lower priority.

[gui1117]

I think it should work good, I wonder if 256 slots is enough. probably it is.

[gui1117]

it will be limited to a maximum capacity of 256 (the number of slots), it sounds reasonable.

[alexggh]

Yeah, at least the usecases I saw 256, should be enough.

[alexggh]

I'm trying to be frugal with the space, do you think we need more than that ?

[gui1117]

256 slots seems enough, at least for now. Or should we expect more than 256 statements for a single account?

[gui1117]

I guess we can have this as the field number 9.

[alexggh]

Yeah, it shouldn't be a problem.