SnowflakeOperatorAsync executes queries in parallel

[ReadytoRocc]

Currently, the SnowflakeOperatorAsync operator submits queries in parallel. If the queries require order within the session (i.e. CREATE -> INSERT INTO), there is no guarantee that the first query finishes successfully before the next query is run. Sequentiality is guaranteed by the SnowflakeOperator, as the queries are run synchronously in the same process.

I believe we should rename the SnowflakeOperatorAsync to the SnowflakeParallelQueryOperatorAsync to denote the behavior of this operator, and document the differences.

As a note, the Snow SQL API allows for submitting multiple SQL statements in a single request. In combination with aiohttp, this may be an option for creating a SnowflakeOperatorAsync that matches the query submission behavior of the SnowflakeOperator.

[phanikumv]

Agree with you @ReadytoRocc , I think this is a great idea to rename the current version of SnowflakeOperatorAsync to SnowflakeParallelQueryOperatorAsync and document the differences.

But we need to further analyse the feasibility of executing multiple SQL statements in a single request, @bharanidharan14 can you take a look in the next sprint?

[bharanidharan14]

@ReadytoRocc Thanks for the suggestion, we will get back with some analyse on executing multiple SQL statements in a single request

[phanikumv]

This is addressed in #478

[pankajkoti]

Looks like it is addressed in #478 . Closing this discussion.

[ReadytoRocc]

Hi @pankajkoti, has this part been addressed?

I believe we should rename the SnowflakeOperatorAsync to the SnowflakeParallelQueryOperatorAsync to denote the behavior of this operator, and document the differences.

Does the SnowflakeOperatorAsync now provide the ability to submit queries sequentially?

[pankajkoti]

hi @ReadytoRocc , I saw a comment above which says #478 addresses the question. I see an example DAG in the PR https://github.com/astronomer/astronomer-providers/pull/478/files#diff-1777bc841029df4ee527bbdd430c64bfd02204ff2684bcfd6898f91c547532b3 which seems to submit multiple queries. Is there something more needed on this?

[ReadytoRocc]

That is correct, the SnowflakeSqlApiAsync provides this capability. But if the SnowflakeOperatorAsync does not, my suggestion is that we should note and/or document that it will submit queries in parallel.