migrate to AWX 21.6 / operator 0.29 ; migrations are broken #1064

lerminou · 2022-09-23T22:24:22Z

Please confirm the following

I agree to follow this project's code of conduct.
I have checked the current issues for duplicates.
I understand that AWX is open source software provided for free and that I might not receive a timely response.

Bug Summary

upgrading a AWX instance to 0.29 is broken with the migrations

AWX version

operator 0.29
previous awx 21.1 / operator 0.22
new awx 21.6

Select the relevant components

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

No response

Steps to reproduce

install a AWX operator 0.29 with a previous version (in my case 0.21)

Expected results

a working instance :)

Additional information

maybe due to this commit: ansible/awx@c467b6e

As a workaround I needed to first edit the awx deployment in the running operator to force to have 21.5 version
because from the operator, the AWX version is hardcoded and cannot be modified

After thet I manually run the migrations directly in the pod
awx-manage migrate
To force to install the migration 165.
Then reset the awx version to 21.6

Why do I needed to change the AWX vesion and not install the operator 0.28 ? Just because AWX-operator only keep 1 release in the history ! #1053

Actual results

exception during migrations:

  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedColumn: ERREUR:  la colonne main_project.signature_validation_credential_id n'existe pas
LINE 1: ..._cache_timeout", "main_project"."allow_override", "main_proj...
                                                             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/awx-manage", line 8, in <module>
    sys.exit(manage())
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py", line 201, in manage
    execute_from_command_line(sys.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/migrations/0164_remove_inventorysource_update_on_project_update.py", line 20, in forwards
    proj.save(update_fields=['scm_update_on_launch'])
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/base.py", line 787, in save_base
    post_save.send(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/dispatch/dispatcher.py", line 180, in send
    return [
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/dispatch/dispatcher.py", line 181, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/fields.py", line 289, in _post_save
    latest_instance = Model.objects.get(pk=instance.pk)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/query.py", line 431, in get
    num = len(clone)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/polymorphic/query.py", line 58, in _polymorphic_iterator
    o = next(base_iter)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/query.py", line 51, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: ERREUR:  la colonne main_project.signature_validation_credential_id n'existe pas
LINE 1: ..._cache_timeout", "main_project"."allow_override", "main_proj...
                                                             ^

bash-5.1$ awx-manage migrate main 0166_alter_jobevent_host
Operations to perform:
  Target specific migration: 0166_alter_jobevent_host, from main
Running migrations:
  Applying main.0164_remove_inventorysource_update_on_project_update...2022-09-23 22:14:42,151 INFO     [-] awx Setting update_on_launch to True for InventorySource object (12)
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedColumn: ERREUR:  la colonne main_project.signature_validation_credential_id n'existe pas
LINE 1: ..._cache_timeout", "main_project"."allow_override", "main_proj...
                                                             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/awx-manage", line 8, in <module>
    sys.exit(manage())
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py", line 201, in manage
    execute_from_command_line(sys.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/migrations/0164_remove_inventorysource_update_on_project_update.py", line 20, in forwards
    proj.save(update_fields=['scm_update_on_launch'])
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/base.py", line 787, in save_base
    post_save.send(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/dispatch/dispatcher.py", line 180, in send
    return [
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/dispatch/dispatcher.py", line 181, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/fields.py", line 289, in _post_save
    latest_instance = Model.objects.get(pk=instance.pk)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/query.py", line 431, in get
    num = len(clone)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/polymorphic/query.py", line 58, in _polymorphic_iterator
    o = next(base_iter)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/query.py", line 51, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: ERREUR:  la colonne main_project.signature_validation_credential_id n'existe pas
LINE 1: ..._cache_timeout", "main_project"."allow_override", "main_proj...

The text was updated successfully, but these errors were encountered:

HGS9761 · 2022-10-17T16:02:14Z

There is something strange happening

I have turned on logging in Postgres and all statements are being logged to the logfile.

When I upgrade from 0.21 to 0.29 or 0.30 at once the following is not executed:
ALTER TABLE "main_project" ADD COLUMN "signature_validation_credential_id" integer NULL CONSTRAINT
"main_project_signature_validation_41e77a69_fk_main_cred" REFERENCES
"main_credential"("id") DEFERRABLE INITIALLY DEFERRED; SET CONSTRAINTS "main_project_signature_validation_41e77a69_fk_main_cred" IMMEDIATE

This results in the following error
ERROR: column main_project.signature_validation_credential_id does not exist at character 1299

When I upgrade first to 0.28 and then to either 0.29 or 0.30 this does not happen.
The statement :
022-10-17 15:03:52.111 UTC [19135] [10.42.0.54(51022)] LOG: statement: ALTER TABLE "main_project" ADD COLUMN "signature_validation_credential_id" integer NULL CONSTRAINT "main_project_signature_validation_41e77a69_fk_main_cred" REFERENCES "main_credential"("id") DEFERRABLE INITIALLY DEFERRED; SET CONSTRAINTS "main_project_signature_validation_41e77a69_fk_main_cred" IMMEDIATE
Is executed during the upgrade to 0.29 or 0.30.

jweiher · 2022-11-03T17:09:38Z

This is caused by the migration 0164_remove_inventorysource_update_on_project_update, when you already have a newer version of the application.
Workaround: Deploy 21.5.0 first, and the latest version after that.

HGS9761 · 2022-11-04T09:03:08Z

Can this be fixed ?
In general the documentation suggests that an indirect upgrade is not needed.
If fixing is not feasible, the workaround should be mentioned in the documentation.

akus062381 · 2022-12-13T19:49:47Z

Hi @lerminou,

Thank you very much for for this issue. It means a lot to us that you have taken time to contribute by opening this report.

On this issue, there were comments added but it has been some time since then without response. At this time we are closing this issue. If you get time to address the comments we can reopen the issue if you can contact us by using any of the communication methods listed in the page below:

https://github.com/ansible/awx/#get-involved

Thank you once again for this and your interest in AWX!

github-actions bot added needs_triage type:bug Something isn't working labels Sep 23, 2022

lerminou changed the title ~~migrate to AWX 29 migration are broken~~ migrate to AWX 21.6 / operator 0.29 ; migrations are broken Sep 23, 2022

fosterseth transferred this issue from ansible/awx Sep 28, 2022

akus062381 closed this as completed Dec 13, 2022

akus062381 removed the needs_triage label Dec 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

migrate to AWX 21.6 / operator 0.29 ; migrations are broken #1064

migrate to AWX 21.6 / operator 0.29 ; migrations are broken #1064

lerminou commented Sep 23, 2022 •

edited

Loading

HGS9761 commented Oct 17, 2022

jweiher commented Nov 3, 2022

HGS9761 commented Nov 4, 2022

akus062381 commented Dec 13, 2022

migrate to AWX 21.6 / operator 0.29 ; migrations are broken #1064

migrate to AWX 21.6 / operator 0.29 ; migrations are broken #1064

Comments

lerminou commented Sep 23, 2022 • edited Loading

Please confirm the following

Bug Summary

AWX version

Select the relevant components

Installation method

Modifications

Ansible version

Operating system

Web browser

Steps to reproduce

Expected results

Additional information

Actual results

HGS9761 commented Oct 17, 2022

jweiher commented Nov 3, 2022

HGS9761 commented Nov 4, 2022

akus062381 commented Dec 13, 2022

lerminou commented Sep 23, 2022 •

edited

Loading