Update conf column in dag_run table type from bytes to JSON

[vatsrahul1001]

closes: #43933

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[vatsrahul1001]

While Testing this I noticed an issue in the downgrade case. We are not removing the data from downgrade and just perform below conversion.

          ALTER TABLE dag_run
          ALTER COLUMN conf TYPE BYTEA
          USING CASE
              WHEN conf IS NOT NULL THEN CONVERT_TO(conf::TEXT, 'UTF8')
              ELSE NULL
          END
 

I am getting UnpicklingError error with this when I try to open DAG page.

Should we also move data to the archive table in case of a downgrade as well?
cc: @kaxil

[kaxil]

While Testing this I noticed an issue in the downgrade case. We are not removing the data from downgrade and just perform below conversion.

          ALTER TABLE dag_run
          ALTER COLUMN conf TYPE BYTEA
          USING CASE
              WHEN conf IS NOT NULL THEN CONVERT_TO(conf::TEXT, 'UTF8')
              ELSE NULL
          END
 

I am getting UnpicklingError error with this when I try to open DAG page.

Should we also move data to the archive table in case of a downgrade as well? cc: @kaxil

Yeah, it is different than XCom, because XCom code had handling of both JSON & pickle type -- for dagrun conf it would be different. For downgrade you might just want to insert all records from the archive table back here.

[kaxil]

Do we not need any more handling than these?

[kaxil]

Check if the DagRun edit view works in the FAB UI

[kaxil]

I think you might need to change this line:

airflow/airflow/www/forms.py

Lines 131 to 132 in 3c1124e

	if item.conf:
	item.conf = json.loads(item.conf)

[kaxil]

You will also need to add a newsfragment, something like:

https://github.com/astronomer/airflow/blob/feb39ed3788e22fd83a1cb294a824e60f4c13187/newsfragments/aip-72.significant.rst#L21-L37

[kaxil]

fyi: the following is what I had to do for XComs:

PRs:

1. Remove XCom pickling
2. Migrate pickled data & change XCom value type to JSON
3. Handle XCom value errors in FAB's XCom list view
4. Use JSONB type for XCom.value column in PostgreSQL

[jscheffl]

I don't see why the two failing tests are failing - they are really unrelated to the PR - but they are green on canary builds.
I think the comment from @kaxil requires some double check, else from point of data and migration I am good with it.

[potiuk]

I don't see why the two failing tests are failing - they are really unrelated to the PR - but they are green on canary builds. I think the comment from @kaxil requires some double check, else from point of data and migration I am good with it.

Those tests are run with full tests needed so they are very likely going to fail in canary (and for others tests as well after this one is merged. @vatsrahul1001 - if you can rebase it again and ping me when it fails, I might want to have a deeper look or maybe @dabla might take a look.

It definitely comes from #41327 becasue that's where get_primary_keys was introduced.

[potiuk]

This is a very interesting one. It looks like for some reason the compat tests were running the version of tests that were there before #41327 - which it should not - tests should always be taken from main.

First time where #45287 might prove to be useful @gopidesupavan !

[potiuk]

Ah yeah. I think it just needs rebase.

[vatsrahul1001]

@potiuk I have rebased. Test are running now.

[jscheffl]

Oh, we had another issue fixed in #45347 - maybe you need to re-base another time, not generating the same again :-(

[potiuk]

Can you rebase AGAIN @vatsrahul1001 :) -> we found and issue with @jscheffl with the new caching scheme - fixed in #45347 that would run "main" version of the tests. So I think that could be the reason

[potiuk]

:D @jscheffl -> same thought

[gopidesupavan]

This is a very interesting one. It looks like for some reason the compat tests were running the version of tests that were there before #41327 - which it should not - tests should always be taken from main.

First time where #45287 might prove to be useful @gopidesupavan !

I think we are uploading some images differently by seeing workflows?

[potiuk]

I think we are uploading some images differently by seeing workflows?

We were not uploading them AT ALL

[potiuk]

😱

[potiuk]

I am sorry @vatsrahul1001 -> you have to rebase AGAIN - the #45347 caused that #45335 was merged as "green" but it had some issues. It' s now reverted so after you rebase .... you should get a clean green state (minus flaky asyncio test we know about but this one needs further investigation and mostly happens in Python 3.12

[vatsrahul1001]

thanks @jedcunningham. @potiuk all test passed now :)

[potiuk]

thanks @jedcunningham. @potiuk all test passed now :)

UFF! Goood.... That was it then :)

[potiuk]

BTW. @vatsrahul1001 I also submitted this apache/infrastructure-actions#88 so that in the futur we can protect against similar mistakes (of mine this time). That was a good learning :)