Postgresql database as Moneta backend ?

[solyarisoftware]

Hi Daniel, all

Moneta seem a superb level of abstraction!
Here minor points, not issues! just feedbacks:

1. Postgresql as relational db supported backend

I read you mention Sqlite3 (with multiprocess limitations you point out in the comparison table)
I suppose I'd use Postgres instead, maybe through the beloved Sequel ORM adapter,

Probably a dedicated Postgres adapter could benefit by the HSTORE (http://www.postgresql.org/docs/9.4/static/hstore.html) or JSONB (http://www.postgresql.org/docs/9.4/static/datatype-json.html) in last release database features, just to store hashes/unstructured data.

I didn't yet read the Sequel code of Moneta Sequel backend adapter but I presume that's generic (not using any specific feature of Posgres); that's correct ?

2. Oj as JSON Moneta::Transformer

I see in the list mentioned Ox (I presume that's Peter Ohler optimizer for XMLs adata)
Do you also foresee superfast Oj (https://github.com/ohler55/oj) as JSON optimizer ?

3. Performances/Benchmarks

Do you pheraps did some performance benchmarks comparing different backends (with disk persistence) ?
BTW, I found interesting this: https://gist.github.com/stephan-nordnes-eriksen/6c9c56f63f36d5d100b2

Sorry for many topics and to not yet browsed the project code
respect
giorgio

[minad]

• Benchmarks: https://travis-ci.org/minad/moneta/jobs/29347732 see also the bechmark script
• Sure, dedicated postgres adapter would be possible and could take advantage of hstore
• Oj is taken into account my MultiJson

[solyarisoftware]

• Thanks for benchmark link/script! :)
  LevelDB backend seems to me the fastest with disk persistence (If I well read your report)
• I'll study more in deep your great project and possibly propose a postgres-hstore adapter.
• Any occasion is good for me to say thank you also to Peter Ohler :)

[asppsa]

I've added some optimisations to the Sequel adapter in master to use the INSERT ... ON CONFLICT and RETURNING features of Postgres, and I've also added some travis testing using postgres. No hstore support as-yet though.

[asppsa]

I've now added an hstore implementation too. If you are interested in trying it out, you can do something like:

store = Moneta::Adapters::Sequel.new(
  db: 'posgres://...',
  table: :some_table,
  hstore: 'key value')

store['x'] = 'y' 
# etc

On store creation, a table called :some_table gets added to your database with a string column k and an hstore column v. On each write to the store, a row gets added with k = 'key value' if it doesn't already exist, and the pair gets merged into the v column.

It all seems to work fine, but its definitely slower in some cases than using a row for every pair. In particular, concurrent writes are extremely slow. Still might be useful in some cases ...

[bjeanes]

I wat thinking about adding a dedicated Postgres adapter which could be a bit tighter than the existing ActiveRecord one by leaning on facilities like ON CONFLICT.

However, since you've already special-cased Sequel for this, would you consider special-casing ActiveRecord too?

[bjeanes]

That being said, I was thinking about this in the context of adding native expires support, and I think that would be problematic. I'm not sure if you can use ON CONFLICT to handle this case so cleanly. You can use ON CONFLICT with an index conditions, but indices can't be dependent on the current time, so that feels like a non-starter...

EDIT: Never-mind, this should be possible, by adding a WHERE condition to a DO UPDATE:

INSERT INTO moneta (k, v, expire_at)
  VALUES ('key', 'v', now() + interval '1 minute')
  ON CONFLICT (k)
  DO UPDATE 
    SET v = 'v'
      , expire_at = now() + interval '1 minute'
    WHERE moneta.expire_at < now();

[asppsa]

@bjeanes if you were to subclass the ActiveRecord adapter in a similar way to the Sequel adapter, that contribution would very gratefully accepted!

[bjeanes]

If I end up doing this work, I will absolutely contribute it back. As it stands, I want to stress test the existing one to make sure it is as resilient as it claims. Performance isn't my immediate concern, but resilience is. That being said, native expires (esp in the context of the mutex API changes discussed in #156) would be excellent.

EDIT: But my intention wasn't to use ActiveRecord necessarily, as it's mostly an unnecessary dependency. If special-casing to a specific DB, might as well use the DB driver directly to cut down on dependencies.