Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NXDRIVE-2967: Upgrade the deprecated datetime adapter.. #5445

Open
wants to merge 15 commits into
base: wip-NXDRIVE-2929-upgrade-python-from-3.9.5-to-3.12.3
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/dead_code.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: 3.9 # XXX_PYTHON
python-version: 3.12 # XXX_PYTHON
- uses: actions/cache@v4
with:
path: ~/.cache/pip
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/functional_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: 3.9 # XXX_PYTHON
python-version: 3.12 # XXX_PYTHON
- uses: actions/cache@v4
with:
path: ~/.cache/pip
Expand Down Expand Up @@ -63,7 +63,7 @@ jobs:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: 3.9 # XXX_PYTHON
python-version: 3.12 # XXX_PYTHON
- uses: actions/cache@v4
with:
path: ~/Library/Caches/pip
Expand Down Expand Up @@ -93,7 +93,7 @@ jobs:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: 3.9 # XXX_PYTHON
python-version: 3.12 # XXX_PYTHON
- uses: actions/cache@v4
with:
path: ~\AppData\Local\pip\Cache
Expand Down Expand Up @@ -129,7 +129,7 @@ jobs:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: 3.9 # XXX_PYTHON
python-version: 3.12 # XXX_PYTHON
- uses: actions/cache@v4
with:
path: ~/.cache/pip
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/functional_tests_2023.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: 3.9 # XXX_PYTHON
python-version: 3.12 # XXX_PYTHON
- uses: actions/cache@v4
with:
path: ~/.cache/pip
Expand Down Expand Up @@ -63,7 +63,7 @@ jobs:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: 3.9 # XXX_PYTHON
python-version: 3.12 # XXX_PYTHON
- uses: actions/cache@v4
with:
path: ~/Library/Caches/pip
Expand Down Expand Up @@ -93,7 +93,7 @@ jobs:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: 3.9 # XXX_PYTHON
python-version: 3.12 # XXX_PYTHON
- uses: actions/cache@v4
with:
path: ~\AppData\Local\pip\Cache
Expand Down Expand Up @@ -123,7 +123,7 @@ jobs:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: 3.9 # XXX_PYTHON
python-version: 3.12 # XXX_PYTHON
- uses: actions/cache@v4
with:
path: ~/.cache/pip
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/integration_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: 3.9 # XXX_PYTHON
python-version: 3.12 # XXX_PYTHON
architecture: "x86"
- uses: actions/cache@v4
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: 3.9 # XXX_PYTHON
python-version: 3.12 # XXX_PYTHON
- uses: actions/cache@v4
with:
path: ~/.cache/pip
Expand Down
Empty file added .github/workflows/main.yml
Empty file.
2 changes: 1 addition & 1 deletion .github/workflows/spell.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: 3.9 # XXX_PYTHON
python-version: 3.12 # XXX_PYTHON
- uses: actions/cache@v4
with:
path: ~/.cache/pip
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/style.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: 3.9 # XXX_PYTHON
python-version: 3.12 # XXX_PYTHON
- uses: actions/cache@v4
with:
path: ~/.cache/pip
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/translations.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: 3.9 # XXX_PYTHON
python-version: 3.12 # XXX_PYTHON
- uses: actions/cache@v4
with:
path: ~/.cache/pip
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/types.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: 3.9 # XXX_PYTHON
python-version: 3.12 # XXX_PYTHON
- uses: actions/cache@v4
with:
path: ~/.cache/pip
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/unit_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: 3.9 # XXX_PYTHON
python-version: 3.12 # XXX_PYTHON
- uses: actions/cache@v4
with:
path: ~/.cache/pip
Expand Down Expand Up @@ -56,7 +56,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: 3.9 # XXX_PYTHON
python-version: 3.12 # XXX_PYTHON
- uses: actions/cache@v4
with:
path: ~/Library/Caches/pip
Expand Down Expand Up @@ -87,7 +87,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: 3.9 # XXX_PYTHON
python-version: 3.12 # XXX_PYTHON
architecture: "x86"
- uses: actions/cache@v4
with:
Expand Down
2 changes: 1 addition & 1 deletion docs/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ Executing the script without argument will setup.update the isolated environment

[//]: # (XXX_PYTHON, XXX_INNO_SETUP)

- [Python 3.9.5](https://www.python.org/ftp/python/3.9.5/python-3.9.5.exe).
- [Python 3.12.3](https://www.python.org/ftp/python/3.12.3/python-3.12.3.exe).
- [Inno Setup 6.1.2](http://www.jrsoftware.org/isdl.php) to create the installer.

### Troubleshooting
Expand Down
2 changes: 1 addition & 1 deletion docs/gnu_linux.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ sudo mount -o remount,user_xattr /dev/sda3

[//]: # (XXX_PYTHON)

Nuxeo Drive is officially supported on **Python 3.9.1+**.
Nuxeo Drive is officially supported on **Python 3.12.3+**.

## Installation

Expand Down
2 changes: 1 addition & 1 deletion docs/support.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ It may evolve quickly without notification as we are following the Python develo

[//]: # (XXX_PYTHON)

As of now, we are using the __Python 3.9.5__.
As of now, we are using the __Python 3.12.3__.

History:

Expand Down
6 changes: 3 additions & 3 deletions nxdrive/client/local/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
import unicodedata
import uuid
from contextlib import suppress
from datetime import datetime
from datetime import datetime, timezone
from logging import getLogger
from pathlib import Path
from tempfile import mkdtemp
Expand Down Expand Up @@ -263,12 +263,12 @@
stat_info = os_path.stat()
size = 0 if folderish else stat_info.st_size
try:
mtime = datetime.utcfromtimestamp(stat_info.st_mtime)
mtime = datetime.fromtimestamp(stat_info.st_mtime, tz=timezone.utc)
except (ValueError, OverflowError, OSError) as e:
log.warning(
f"{e} file path: {os_path}. st_mtime value: {stat_info.st_mtime}"
)
mtime = datetime.utcfromtimestamp(0)
mtime = datetime.fromtimestamp(0, tz=timezone.utc)

Check warning on line 271 in nxdrive/client/local/base.py

View check run for this annotation

Codecov / codecov/patch

nxdrive/client/local/base.py#L271

Added line #L271 was not covered by tests
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Error handling should use consistent timezone approach

Both the try and except blocks now use timezone-aware timestamps, but the error handling could be more consistent by using a common approach or constant for the fallback time.

Suggested change
mtime = datetime.fromtimestamp(0, tz=timezone.utc)
EPOCH = datetime(1970, 1, 1, tzinfo=timezone.utc)
mtime = EPOCH


# TODO Do we need to load it every time ?
remote_ref = self.get_remote_id(ref)
Expand Down
11 changes: 9 additions & 2 deletions nxdrive/dao/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
Query formatting in this file is based on http://www.sqlstyle.guide/
"""

from datetime import datetime, timezone
import sys
from contextlib import suppress
from logging import getLogger
Expand All @@ -19,14 +20,20 @@

log = getLogger(__name__)


class AutoRetryCursor(Cursor):
def adapt_datetime_iso(self, val):
return datetime.fromtimestamp(val.strftime('%s'), tz=timezone.utc)

Check warning on line 25 in nxdrive/dao/base.py

View check run for this annotation

Codecov / codecov/patch

nxdrive/dao/base.py#L25

Added line #L25 was not covered by tests
def execute(self, sql: str, parameters: Iterable[Any] = ()) -> Cursor:
count = 1
while True:
count += 1
try:
return super().execute(sql, parameters)
import sqlite3
# return super().execute(sql, parameters)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: The datetime parameter conversion logic should be simplified and the original super().execute() call restored

The current implementation with register_adapter could have unintended side effects. Consider moving the datetime conversion to a separate method and maintaining the original execution flow.

# new_param = tuple( datetime.fromtimestamp(param, tz=timezone.utc) if isinstance(param, datetime) else param for param in parameters )
new_param = tuple( sqlite3.register_adapter(param, self.adapt_datetime_iso) if isinstance(param, datetime) else param for param in parameters )

return super().execute(sql, new_param)
except OperationalError as exc:
log.info(
f"Retry locked database #{count}, {sql=}, {parameters=}",
Expand Down
12 changes: 6 additions & 6 deletions nxdrive/dao/engine.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
import os
import shutil
from contextlib import suppress
from datetime import datetime
from datetime import datetime, timezone
from logging import getLogger
from os.path import basename
from pathlib import Path
Expand Down Expand Up @@ -1503,7 +1503,7 @@ def increase_error(
self, row: DocPair, error: str, /, *, details: str = None, incr: int = 1
) -> None:
with self.lock:
error_date = datetime.utcnow()
error_date = datetime.now(tz=timezone.utc)
c = self._get_write_connection().cursor()
c.execute(
"UPDATE States"
Expand Down Expand Up @@ -1592,7 +1592,7 @@ def unsynchronize_state(
" error_count = 0,"
" last_sync_error_date = NULL"
" WHERE id = ?",
(datetime.utcnow(), last_error, row.id),
(datetime.now(tz=timezone.utc), last_error, row.id),
)

def unset_unsychronised(self, row: DocPair, /) -> None:
Expand All @@ -1617,7 +1617,7 @@ def unset_unsychronised(self, row: DocPair, /) -> None:
row.local_state,
row.remote_state,
row.pair_state,
datetime.utcnow(),
datetime.now(tz=timezone.utc),
f"{adapt_path(row.local_path)}%",
),
)
Expand Down Expand Up @@ -1659,7 +1659,7 @@ def synchronize_state(
row.remote_state,
row.pair_state,
row.local_digest,
datetime.utcnow(),
datetime.now(tz=timezone.utc),
row.id,
version,
),
Expand Down Expand Up @@ -1687,7 +1687,7 @@ def synchronize_state(
row.local_state,
row.remote_state,
row.pair_state,
datetime.utcnow(),
datetime.now(tz=timezone.utc),
row.id,
row.local_path,
row.remote_name,
Expand Down
6 changes: 4 additions & 2 deletions nxdrive/engine/engine.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
import datetime
import json
import os
import os.path
import shutil
from contextlib import suppress
from dataclasses import dataclass
from datetime import datetime, timezone
from functools import partial
from logging import getLogger
from pathlib import Path
Expand Down Expand Up @@ -1182,7 +1182,9 @@ def _check_last_sync(self) -> None:
log.debug(f"Emitting syncPartialCompleted for engine {self.uid}")
self.syncPartialCompleted.emit()
else:
self.dao.update_config("last_sync_date", datetime.datetime.utcnow())
self.dao.update_config(
"last_sync_date", datetime.datetime.now(tz=timezone.utc)
)
log.debug(f"Emitting syncCompleted for engine {self.uid}")
self._sync_started = False
self.syncCompleted.emit()
Expand Down
4 changes: 2 additions & 2 deletions nxdrive/engine/watcher/remote_watcher.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import os
from datetime import datetime
from datetime import datetime, timezone
from logging import getLogger
from operator import attrgetter, itemgetter
from time import monotonic, sleep
Expand Down Expand Up @@ -116,7 +116,7 @@ def scan_remote(self, *, from_state: DocPair = None) -> None:

# Recursive update
self._do_scan_remote(from_state, remote_info)
self._last_remote_full_scan = datetime.utcnow()
self._last_remote_full_scan = datetime.now(tz=timezone.utc)
self.dao.update_config("remote_last_full_scan", self._last_remote_full_scan)
self.dao.clean_scanned()

Expand Down
10 changes: 5 additions & 5 deletions nxdrive/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
import sys
from configparser import DEFAULTSECT, ConfigParser
from copy import deepcopy
from datetime import datetime
from datetime import datetime, timezone
from functools import lru_cache
from itertools import islice
from logging import getLogger
Expand Down Expand Up @@ -646,7 +646,7 @@ def get_certificate_details(
Note: This function uses a undocumented method of the _ssl module.
It is continuously tested in our CI to ensure it still
available after any Python upgrade.
Certified working as of Python 3.9.5.
Certified working as of Python 3.12.3.
"""

import ssl
Expand Down Expand Up @@ -1287,7 +1287,7 @@ def today_is_special() -> bool:
"""This beautiful day is special, isn't it? As all other days, right? :)"""
return (
os.getenv("I_LOVE_XMAS", "0") == "1"
or int(datetime.utcnow().strftime("%j")) >= 354
or int(datetime.now(tz=timezone.utc).strftime("%j")) >= 354
)


Expand All @@ -1301,7 +1301,7 @@ def get_current_locale() -> str:
else:
import locale

encoding = locale.getdefaultlocale()[1] or ""
encoding = locale.getlocale()[1] or ""

# Guess the current locale name
if WINDOWS:
Expand All @@ -1315,7 +1315,7 @@ def get_current_locale() -> str:
l10n_code = NSLocale.currentLocale()
l10n = NSLocale.localeIdentifier(l10n_code)
else:
l10n = locale.getdefaultlocale()[0] or ""
l10n = locale.getlocale()[1] or ""

return ".".join([l10n, encoding])

Expand Down
Loading
Loading