[vstempen]

Motivation

Before client like roc-Optiq uses profiler hub to open database, it needs a tool to check if database is supported by profiler hub.

Technical Details

Database probing rejects an unsupported trace before any reader is opened.

probe_database() and ph_probe_database() open the file read-only and do not create or modify it. The probe checks, in order:

1. The file exists and is readable.
2. The header is SQLite (SQLite format 3), not a generic SQL file.
3. There is exactly one rocpd_metadata_ table.
4. The schema version is one profiler-hub can read. Schema 3 is accepted.

Version tags are read from that metadata table. If schema_version_major, schema_version_minor, and schema_version_patch are all present, those values are used. Older traces that only have schema_version are still accepted: "3" is 3.0.0, "3.0" is 3.0.0, and "3.0.1" is 3.0.1.

ph_ctx_create() runs this probe before the connection pool opens. A rejected file returns one of the new result codes instead of failing later inside a reader:

PH_RESULT_UNSUPPORTED_DATABASE
PH_RESULT_INVALID_DATABASE
PH_RESULT_METADATA_MISSING
PH_RESULT_UNSUPPORTED_SCHEMA_VERSION

Another database format is added by registering a detector. Another schema is added by registering the versions its reader accepts.

[adanalis-amd]

Probe result in the C API

1. ph_database_probe_t exposes the status but not the reason. The C++ result carries message and uuid, but C callers (roc-Optiq, per the PR motivation) can only show something like "metadata_malformed", not why. ph_ctx_create() is coarser still: to_ph_result() turns file-not-found, unreadable, malformed and ambiguous into a single PH_RESULT_INVALID_DATABASE. Consider passing the message through, either as a caller-provided buffer on the probe or with a ph_probe_database_message()-style accessor.

2. ph_database_probe_t has no struct_size or version field, so adding message or uuid later changes its size and breaks existing callers. Since the struct is new, now is the cheap time to add one, e.g. a leading size_t size that the caller sets to sizeof(ph_database_probe_t).

[adanalis-amd]

The schema version is now read in accept_trace(), so this "Reading schema version" progress step no longer does anything. It can be dropped.

[vstempen]

I had to add more changes to the PR, because I needed cumulative change. With these changes plus changes in Optiq I can load timeline and histogram. Everything looks fine and works quite fast, thanks to Alexandar. All table and event operations are not yet working. It shouldn't be much efforts adding event operations, like flow and stack, but much more efforts to add table data and searches.
I'll look into the comments tomorrow and try to fix them.

[adjordje-amd]

Every row type in this header is named *_result. Rename to interval_result.

[adjordje-amd]

query_intervals prepares a statement and executes it in the same call, from a raw SQL string built in histogram.cpp. Move the SQL here and create the interval statements (one or more, with ? parameters) in the constructor like the other statements, expose them as statement_func_t, and let the histogram call them to fetch the data.

[adjordje-amd]

row_id exposes a rocpd table id through the public API. It exists only because ph_stream_t.pid, ph_queue_t.pid and nid also carry database ids instead of public identifiers. It is specific to this database, so it becomes a problem as soon as another format is added, which has no such row id. Express those relationships with public identifiers (the same applies to ph_process_t.row_id and ph_thread_t.row_id) and drop row_id.

[vstempen]

Ok, I'll do that. I would use ph_agent_t.id, but I think it's not right to take it from static_caststd::uint32_t(agent->type_index). We may have database with CPU and GPU agents in the same table. So we need to take it from agent->absolute_index to make unique identifier.

[vstempen]

Without these IDs we are loosing topology relations. We will need to synthesize row ids and replace them in all information tables. So, yeah, I disagree we are removing them. And that creates event more inconsistency between databases. For example, some databases may have process row id equal to PID, some may just have sequential number. Not talking about table view. For example I get table row as is, with indexes, no strings. Now I want to reach to information table and get a name of corresponding to the index. Without proper row_id I cannot do that. So it's not private ID anymore after you agreed to give us information tables. I'm considering not implementing this change and leave row_id.

[adjordje-amd]

Parsing "3", "3.0" or "3.0.1" into a version_t is not specific to the schema. Move it to a private version_parser.hpp so any version text reuses it, and keep only the tag resolution (resolve_schema_version) schema-specific, in schema_version.hpp. This also removes the confusing schema_version_text.hpp name.