Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix weird flake in integration tests with sqlite #17036

Closed
zzstoatzz opened this issue Feb 7, 2025 · 7 comments · Fixed by #17044
Closed

fix weird flake in integration tests with sqlite #17036

zzstoatzz opened this issue Feb 7, 2025 · 7 comments · Fixed by #17044
Labels
development Tech debt, refactors, CI, tests, and other related work.

Comments

@zzstoatzz
Copy link
Collaborator

zzstoatzz commented Feb 7, 2025 •
edited
Loading

Bug summary

see https://github.com/PrefectHQ/prefect/actions/runs/13191059627/job/36823978425#step:3:2242

  File "/usr/local/lib/python3.9/site-packages/prefect/server/database/_migrations/env.py", line 159, in do_run_migrations
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.9/site-packages/alembic/runtime/environment.py", line 946, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.9/site-packages/alembic/runtime/migration.py", line 611, in run_migrations
    for step in self._migrations_fn(heads, self):
  File "/usr/local/lib/python3.9/site-packages/alembic/command.py", line 442, in downgrade
    return script._downgrade_revs(revision, rev)
  File "/usr/local/lib/python3.9/site-packages/alembic/script/base.py", line 476, in _downgrade_revs
    return [
  File "/usr/local/lib/python3.9/site-packages/alembic/script/base.py", line 476, in <listcomp>
    return [
  File "/usr/local/lib/python3.9/site-packages/alembic/script/revision.py", line 814, in iterate_revisions
    revisions, heads = fn(
  File "/usr/local/lib/python3.9/site-packages/alembic/script/revision.py", line 1339, in _collect_downgrade_revisions
    branch_label, target_revision = self._parse_downgrade_target(
  File "/usr/local/lib/python3.9/site-packages/alembic/script/revision.py", line 1211, in _parse_downgrade_target
    return branch_label, self.get_revision(symbol)
  File "/usr/local/lib/python3.9/site-packages/alembic/script/revision.py", line 584, in get_revision
    resolved_id, branch_label = self._resolve_revision_number(id_)
  File "/usr/local/lib/python3.9/site-packages/alembic/script/revision.py", line 768, in _resolve_revision_number
    self._revision_map
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 1145, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/usr/local/lib/python3.9/site-packages/alembic/script/revision.py", line 245, in _revision_map
    down_revision = map_[downrev]
KeyError: '354f1ede7e9f'

Version info

main

Additional context

No response
@zzstoatzz zzstoatzz added the development Tech debt, refactors, CI, tests, and other related work. label Feb 7, 2025
@cicdw
Copy link
Member

cicdw commented Feb 7, 2025

I was just looking at this!! Yea it seems to have cropped up consistently.

@aaazzam aaazzam mentioned this issue Feb 7, 2025
Merged
@cicdw
Copy link
Member

cicdw commented Feb 7, 2025

Doesn't always happen on older versions of sqlite: https://github.com/PrefectHQ/prefect/actions/runs/13203592361/job/36861326261?pr=17033

@cicdw cicdw mentioned this issue Feb 7, 2025
Merged
@cicdw
Copy link
Member

cicdw commented Feb 8, 2025

This is still very much alive, my proposed fix didn't change anything

@cicdw cicdw reopened this Feb 8, 2025
@zzstoatzz zzstoatzz changed the title fix weird flake in integration tests with old sqlite fix weird flake in integration tests with sqlite Feb 8, 2025
@marcm-ml
Copy link

marcm-ml commented Feb 14, 2025 •
edited
Loading

i have this error too in my cicd :)

I am using the prefect docker image with tag "3.2.2-python3.12" and with minimal custom changes (nothing related to versions).

I am starting the ephemeral sever via the prefect_test_harness method

08:17:42.562 | INFO    | prefect - Starting temporary server on http://127.0.0.1:8693/
See https://docs.prefect.io/3.0/manage/self-host#self-host-a-prefect-server for more information on running a dedicated Prefect server.
/usr/local/lib/python3.12/site-packages/alembic/script/revision.py:241: UserWarning: Revision 8644a9595a08 referenced from 8644a9595a08 -> cc510aec4689 (head), Automation event follower is not present
  util.warn(
08:17:56.725 | ERROR   | uvicorn.error - Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 693, in lifespan
    async with self.lifespan_context(app) as maybe_state:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/server/api/server.py", line 606, in lifespan
    await run_migrations()
  File "/usr/local/lib/python3.12/site-packages/prefect/server/api/server.py", line 586, in run_migrations
    await db.create_db()
  File "/usr/local/lib/python3.12/site-packages/prefect/server/database/interface.py", line 77, in create_db
    await self.run_migrations_upgrade()
  File "/usr/local/lib/python3.12/site-packages/prefect/server/database/interface.py", line 85, in run_migrations_upgrade
    await run_sync_in_worker_thread(alembic_upgrade)
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 233, in run_sync_in_worker_thread
    result = await anyio.to_thread.run_sync(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2461, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 962, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 243, in call_with_mark
    return call()
           ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/server/database/alembic_commands.py", line 36, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/server/database/alembic_commands.py", line 72, in alembic_upgrade
    alembic.command.upgrade(alembic_config(), revision, sql=dry_run)
  File "/usr/local/lib/python3.12/site-packages/alembic/command.py", line [40](https://git.tech.rz.db.de/house-of-ai/ki-dispo/mlops/prefect-kidispo/-/jobs/307958789#L40)6, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.12/site-packages/alembic/script/base.py", line 586, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
    module = load_module_py(module_id, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 999, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/usr/local/lib/python3.12/site-packages/prefect/server/database/_migrations/env.py", line 201, in <module>
    run_async_from_worker_thread(apply_migrations)
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 254, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/from_thread.py", line 59, in run
    return async_backend.run_async_from_thread(func, args, token=token)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2501, in run_async_from_thread
    return f.result()
           ^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2488, in task_wrapper
    return await func(*args)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/server/database/_migrations/env.py", line 190, in apply_migrations
    await connection.run_sync(do_run_migrations)
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/ext/asyncio/engine.py", line 887, in run_sync
    return await greenlet_spawn(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 203, in greenlet_spawn
    result = context.switch(value)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/server/database/_migrations/env.py", line 159, in do_run_migrations
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.12/site-packages/alembic/runtime/environment.py", line 946, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.12/site-packages/alembic/runtime/migration.py", line 611, in run_migrations
    for step in self._migrations_fn(heads, self):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/alembic/command.py", line 395, in upgrade
    return script._upgrade_revs(revision, rev)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/alembic/script/base.py", line 462, in _upgrade_revs
    for script in reversed(list(revs))
                           ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/alembic/script/revision.py", line 814, in iterate_revisions
    revisions, heads = fn(
                       ^^^
  File "/usr/local/lib/python3.12/site-packages/alembic/script/revision.py", line 1447, in _collect_upgrade_revisions
    for rev in self._parse_upgrade_target(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/alembic/script/revision.py", line 1234, in _parse_upgrade_target
    return self.get_revisions(target)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/alembic/script/revision.py", line 544, in get_revisions
    resolved_id, branch_label = self._resolve_revision_number(id_)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/alembic/script/revision.py", line 768, in _resolve_revision_number
    self._revision_map
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 1145, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
                                           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/alembic/script/revision.py", line 245, in _revision_map
    down_revision = map_[downrev]
                    ~~~~^^^^^^^^^
KeyError: '8644a9595a08'
08:17:56.737 | ERROR   | uvicorn.error - Application startup failed. Exiting.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/builds/marcml/integration.py", line 48, in <module>
    main()
  File "/builds/marcml/integration.py", line 18, in main
    with prefect_test_harness():
         ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/testing/utilities.py", line 166, in prefect_test_harness
    test_server.start(
  File "/usr/local/lib/python3.12/site-packages/prefect/server/api/server.py", line 832, in start
    raise RuntimeError(error_message)
RuntimeError: Timed out while attempting to connect to ephemeral Prefect API server.
08:18:13.506 | INFO    | prefect - Stopping temporary server on http://127.0.0.1:8735
/usr/local/lib/python3.12/site-packages/alembic/script/revision.py:2[41](https://git.tech.rz.db.de/house-of-ai/ki-dispo/mlops/prefect-kidispo/-/jobs/307958789#L41): UserWarning: Revision 8644a9595a08 referenced from 8644a9595a08 -> cc510aec4689 (head), Automation event follower is not present
  util.warn(
08:18:15.359 | ERROR   | uvicorn.error - Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 693, in lifespan
    async with self.lifespan_context(app) as maybe_state:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/server/api/server.py", line 606, in lifespan
    await run_migrations()
  File "/usr/local/lib/python3.12/site-packages/prefect/server/api/server.py", line 586, in run_migrations
    await db.create_db()
  File "/usr/local/lib/python3.12/site-packages/prefect/server/database/interface.py", line 77, in create_db
    await self.run_migrations_upgrade()
  File "/usr/local/lib/python3.12/site-packages/prefect/server/database/interface.py", line 85, in run_migrations_upgrade
    await run_sync_in_worker_thread(alembic_upgrade)
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 233, in run_sync_in_worker_thread
    result = await anyio.to_thread.run_sync(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2461, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 962, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 2[43](https://git.tech.rz.db.de/house-of-ai/ki-dispo/mlops/prefect-kidispo/-/jobs/307958789#L43), in call_with_mark
    return call()
           ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/server/database/alembic_commands.py", line 36, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/server/database/alembic_commands.py", line 72, in alembic_upgrade
    alembic.command.upgrade(alembic_config(), revision, sql=dry_run)
  File "/usr/local/lib/python3.12/site-packages/alembic/command.py", line 406, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.12/site-packages/alembic/script/base.py", line 586, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
    module = load_module_py(module_id, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 999, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/usr/local/lib/python3.12/site-packages/prefect/server/database/_migrations/env.py", line 201, in <module>
    run_async_from_worker_thread(apply_migrations)
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 254, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/from_thread.py", line 59, in run
    return async_backend.run_async_from_thread(func, args, token=token)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2501, in run_async_from_thread
    return f.result()
           ^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2488, in task_wrapper
    return await func(*args)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/server/database/_migrations/env.py", line 190, in apply_migrations
    await connection.run_sync(do_run_migrations)
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/ext/asyncio/engine.py", line 887, in run_sync
    return await greenlet_spawn(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 203, in greenlet_spawn
    result = context.switch(value)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/server/database/_migrations/env.py", line 159, in do_run_migrations
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.12/site-packages/alembic/runtime/environment.py", line 946, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.12/site-packages/alembic/runtime/migration.py", line 611, in run_migrations
    for step in self._migrations_fn(heads, self):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/alembic/command.py", line 395, in upgrade
    return script._upgrade_revs(revision, rev)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/alembic/script/base.py", line 462, in _upgrade_revs
    for script in reversed(list(revs))
                           ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/alembic/script/revision.py", line 814, in iterate_revisions
    revisions, heads = fn(
                       ^^^
  File "/usr/local/lib/python3.12/site-packages/alembic/script/revision.py", line 1[44](https://git.tech.rz.db.de/house-of-ai/ki-dispo/mlops/prefect-kidispo/-/jobs/307958789#L44)7, in _collect_upgrade_revisions
    for rev in self._parse_upgrade_target(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/alembic/script/revision.py", line 1234, in _parse_upgrade_target
    return self.get_revisions(target)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/alembic/script/revision.py", line 544, in get_revisions
    resolved_id, branch_label = self._resolve_revision_number(id_)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/alembic/script/revision.py", line 768, in _resolve_revision_number
    self._revision_map
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 11[45](https://git.tech.rz.db.de/house-of-ai/ki-dispo/mlops/prefect-kidispo/-/jobs/307958789#L45), in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
                                           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/alembic/script/revision.py", line 245, in _revision_map
    down_revision = map_[downrev]
                    ~~~~^^^^^^^^^
KeyError: '8644a9595a08'

@FHU-yezi
Copy link

Got the same issue in my Docker deployment, and confirmed this is only happend on v3.2.2 with SQLite as database.

I got KeyError: '8644a9595a08'.

@zzstoatzz zzstoatzz mentioned this issue Feb 19, 2025
Closed
@cicdw
Copy link
Member

cicdw commented Feb 20, 2025

Hi @FHU-yezi and @marcm-ml : this may have been fixed in the latest 3.2.6 release with #17193; could you upgrade and let us know at your convenience?

@FHU-yezi
Copy link

FHU-yezi commented Feb 21, 2025 •
edited
Loading

Hi @FHU-yezi and @marcm-ml : this may have been fixed in the latest 3.2.6 release with #17193; could you upgrade and let us know at your convenience?

Confirm fixed.

@cicdw cicdw closed this as completed Feb 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
development Tech debt, refactors, CI, tests, and other related work.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix migration downgrades on deployment schedule table PrefectHQ/prefect
4 participants
@cicdw @zzstoatzz @FHU-yezi @marcm-ml