Using Redis with SimpleFeed

Going to Production

In a real-world implementation scenario, you will likely need to setup a high-availability Redis cluster (or an instance) and make it available to your application.

This Redis cluster will be used to store each individual user's activity feeds. SimpleFeed stores events in an Ordered Set data type supported by Redis, one set of fixed length for each user. The default is 1000, so that each user will be able to page over most recent 1000 events. You can change this limit to a higher number, but do some math on how much extra memory will that take.

Implementation Notes: to render individual user's feed, we must instantiate the user's SimpleFeed.feed(:myfeed).activity(@user.id) object, and then query it via the #paginate method.

Sharding

To scale SimpleFeed you can shard Redis into hundreds of shards by using a twemproxy sharding proxy. But you may not need to do this until you are serving activity feeds for millions of unique users, because the storage requirements are rather small (assuming you keep small values in SimpleFeed).

Persistence

Your Redis cluster, ideally, is configured with disk persistence enabled.

I personally prefer the RDB method of persistence (instead of the append log), with each 15 minute sync to disk interval. This provides a nice balance — if you loose the machine, but your RDB backups were on a networked drive (or were being sync'd to other machine) — you can always bring another Redis instance up, possibly on a different server, as long as when it starts Redis has access to the RDB file backup from the other machine. In my past I've achieved this by using a shared network storage, and a Chef recipe that would copy this RDB file from the network server to the local disk as it was provisioning Redis instance. This is probably too much detail, since most of you will be using something like ElasticCache on AWS, but I simply recommend having a disk backup so that you can always rebuild Redis backend and populate it with at most 15-minute old backups.

Your application requirements will dictate the consequence of loosing activity feed for a portion of users, and the acceptable time to rebuild it. If the time to rebuild it must be very short, then the RDB backup + copy to network drive is the best strategy I've seen. If the time to rebuild is allowed to be longer, ie, hours, then you can run Redis without any persistence at all, and simply have an ability to rebuild activity feed from a database table, such as Events.