[master] Fix Python3.13 compatibility regarding urllib.parse module

[marmarek]

What does this PR do?

Python 3.13 fixed handling relative paths in urllib.parse module. Specifically, relative file URL is now constructed as file:path instead of converting it to absolute file:///path. This breaks salt.utils.url.create which expects file:/// specifically. The mismatch results in for example changing salt://top.sls into salt://.sls and thus not finding the top file.

Fix this by handling both prefixes.

Relevant python change: python/cpython#85110

What issues does this PR fix or reference?

Fixes: #66898

Merge requirements satisfied?

[NOTICE] Bug fixes or features added to Salt require tests.

• Changelog - https://docs.saltproject.io/en/master/topics/development/changelog.html
• Tests written/updated (current tests in tests/unit/utils/test_url.py are enough when will start running on Python 3.13)

Commits signed with GPG?

Yes

Please review Salt's Contributing Guide for best practices, including the
PR Guidelines.

See GitHub's page on GPG signing for more information about signing commits with GPG.

[twangboy]

Could you please write a test for this

[dmurphy18]

Needs tests written in pytest

[marmarek]

Could you please write a test for this

The existing tests/unit/utils/test_url.py already covers this. And indeed, without my change, it fails already on Python 3.13rc2 (Fedora 41 beta) or Python 3.12.6 (Fedora 40). The change that broke did got backported to Python 3.12.6 (released about 2 weeks ago). I see you have Fedora 40 in CI already, but it seems to run use onedir (and Python 3.10.x inside) instead of Python from Fedora. Is there any CI job that tests on Python 3.12.x (or 3.13)?

[twangboy]

We currently only support up to Python 3.10. There are dependency issues we need to get figured out across all the Operating systems we support before we can officially support anything newer.

[marmarek]

In absence of newer Python to actually test the behavior there, do you want me to mimic its new behavior by patching urllib.parse.urlunparse then (I find it weird, but if you insist I can add it)? The code is already tested that it still works on Python 3.10. And once you will be adding support for Python 3.12, you will have one thing less to fix :)

BTW, does the CI report somewhere test coverage? From what I can see, coverage-related jobs got skipped...

[slimlv]

how many "If/return"s you going to insert ?
note that changes in python code was introduced in minor release 3.12.6

try to make smart bulletproof improvements , for example

url = salt.utils.data.decode(urlunparse(("file", "", "/" + path, "", query, "")))

[twangboy]

I think, just a unit test to make sure that if the URL starts with file:/// it gets converted to 'salt://`

[bdrx312]

This create function should not be doing path = path.replace("\\", /) and probably not

if salt.utils.platform.is_windows():
path = salt.utils.path.sanitize_win_path(path)

Instead it should use urllib.request.pathname2url or some other python builtin functionality.

Especially since the sanitize_win_path function is clearly miswritten:

def sanitize_win_path(winpath):
    """
    Remove illegal path characters for windows
    """
    intab = "<>:|?*"
    if isinstance(winpath, str):
        winpath = winpath.translate({ord(c): "_" for c in intab})
    elif isinstance(winpath, str):

Notice the duplicate isinstance(winpath, str) in the elif that will never occur.

I would prefer instead of assuming that the string starts with file: or file:/// we do a regex search replace like re.sub(r"^file:(?:///)?", "", url) Also use f string instead of format.

return f'salt://{re.sub(r"^file:(?:///)?", "", url)}'

[slimlv]

return f'salt://{re.sub(r"^file:(?:/+)?", "", url)}'

[bdrx312]

return f'salt://{re.sub(r"^file:(?:/+)?", "", url)}'

If you are going to do zero or more you don't + and ? nor parenthesis you just use *

return f'salt://{re.sub(r"^file:/*", "", url)}'

[bdrx312]

and if you want to do 0 to 3 / then it would be

return f'salt://{re.sub(r"^file:/{0,3}", "", url)}'

or

return f'salt://{re.sub(r"^file:/{,3}", "", url)}'

but I think the explicit 0 is more readable.

[marmarek]

I think, just a unit test to make sure that if the URL starts with file:/// it gets converted to 'salt://`

Most tests in tests/unit/utils/test_url.py do that already, for example test_create_url. Those tests do fail when running on Python 3.13 (or Python 3.12.6), so this regression is covered already. Your CI will notice it as soon as you start testing on such Python version.
And BTW, the file:// prefix is added only in salt.utils.url.create(), just to be replaced with salt:// - it doesn't exist in the function call argument.

[yangskyboxlabs]

I recommend just not using schema when calling unparse/unsplit:

Suggested change

	url = salt.utils.data.decode(urlunparse(("file", "", path, "", query, "")))
	return f"salt://{salt.utils.data.decode(urlunsplit(("", "", path, query, "")))}"

[bdrx312]

I just read through the source code of urlunparse/urlunsplit and it looks like there is not currently any special handling for the file scheme so I agree that this is probably a better solution.

[marmarek]

Ok, lets see how this works. Tests on Python 3.12.6 are green with this version, I'll let CI check older ones.

[bdrx312]

Using urlunparse is okay, but I think using urlunsplit as @yangskyboxlabs is more ideal since it passes one less unused argument.

[jfindlay]

In my opinion, having written both the original salt.utils.url and its accompanying tests, I agree the existing tests already cover the code well enough because obviously I write the most amazing test cases. I don't think it is necessary to cover the case that upstream will again change the behavior of code that hasn't been modified in ~25 years.

It is a subtle breaking change for a use case they very likely didn't expect. Regexing file:/// for salt:// is a yolo move, whereas subclassing the urlparse code to add a salt proto would have been more proper. (Maybe the upstream code doesn't care because you already can sub your own proto?) If anything a rare bug in old code is an indictment of the poor usecase. Not that urlparse.urlunparse's api is user friendly (pass the url components in a fixed length anonymous list; give me a break). There are still many occurrences in the codebase that handcraft salt:// URLs anyway.

Ironically, many years ago when I worked on this, Tom said "I wish we had gone with salt:///".

[dead10ck]

I'd like to remind the maintainers that this bug completely and utterly breaks salt on many systems, especially because this "fix" has been backported to Python 3.12.

I understand that 3.10 is currently the officially supported version, but the reality is that a lot of folks are not, and never will, use the onedir package, so this is high impact regardless of official compatibility guarantees.

I understand the desire for tests with changes in the general case, but I think folks have to keep in mind that that should be a rule of thumb, not a hard requirement in every case no matter what the cost. Sometimes fixing high impact bugs, level of effort to benefit ratio, and potential risk for introducing new regressions in more strongly supported environments should weigh into the decision to block merging unless there are tests. It's been established that there is significant preexisting coverage on this code path, so blocking on adding more tests is not doing anyone any good.