Move qiskit.Qobj to model-based approach

[diego-plan9]

What is the expected enhancement?

The qiskit.Qobj class is currently the only item in the schema that does not fully use the qiskit.validation mechanism - it uses the previous iteration of the idea, which is similar in its usage but lacks a number of features. Ideally, refactoring it would allow us to take advantage of those (more fine-grained validation and informative error handling, better control of the schema, more robust serialization and deserialization). It is a tad more complex than the rest of the models, not only due to the schema itself but due to other implications (backend.validation(), how "schema inheritance" should be modeled, and probably the need for all backends to support qobj to ease implementation).

The move would ideally take care of a number of issues (the list is open - please double check and add as needed, some of them might be rendered obsolete):
#676
#721
#753
#851
#1057 (note: this one required a deeper look and has implications)
#1092

[dongreenberg]

I'd like to float the idea of moving to protocol buffer or a similar library for the qobj, for the following reasons:

1. Protobuffers are higher performance than Json in serialization and packet size
2. Validation is built in, so we don't need to implement it from scratch, including providing meaningful error messaging or performance optimization
3. Using protobufs gives us access to other high performance libraries for related tasks, such as gRPC.
4. Explicit and clear schemata - the proto schema declaration is succint, explicit, and readable, while still hierarchical and flexible

Downsides:

1. Binary protos are not human readable, you have to compile to a human-readable proto to troubleshoot
2. Another library

I've discussed this Ismael and Andrew, and they are also of the opinion that proto or a similar solution would be more convenient than json. I'd also note that my pony in this race is that the qobj packaging and schema valuation multiplies my vqe runtime by a factor of 2-3.

[delapuente]

Thank you @dongreenberg for noticing this. I'm a 100% supporter of the idea of generated code based on an intermediate IDL such as proto3 and choosing a binary representation for the transportation layer.

Even if switching to an alternative representation, what we don't want is to continue coupling the object model with the transportation format so we should evaluate how separable is proto3 from the output proto buffer binary format.

Before switching to a different IDL + serialization format, let's finish with this PoC consisting of Marshmallow models + JSON.

[taalexander]

I would also like to chime in that I am a supporter of moving to a new serialized data format such as protobuffers. I do believe we would have to do a proper survey of the field before settling on a format.

[nonhermitian]

Yes. If you guys want to make a
Big change right on the heels of qobj release, you better have some timing results that justify the work involved. Also what about HDF5?

[taalexander]

Hence why we would need to do this properly. Built-in validation and accessors would also remove many pain-points with the backends. HDF5 only gives us the serialized format, not the other benefits.

[dongreenberg]

Agreed that this needs to be fleshed out, options explored, and to be properly justified. Is there a Qobj v0 design doc floating around somewhere? This smells like a Qobj v1 design doc.

[jaygambetta]

@diego-plan9 can this be done for 0.8