fix: Support Pydantic v2 #1229

pawamoy · 2023-07-02T13:45:03Z

Closes #1225.

codecov · 2023-07-02T13:49:20Z

Codecov Report

Merging #1229 (f520785) into master (3a76e66) will increase coverage by 0.01%.
The diff coverage is 93.75%.

@@            Coverage Diff             @@
##           master    #1229      +/-   ##
==========================================
+ Coverage   96.75%   96.77%   +0.01%     
==========================================
  Files          47       47              
  Lines        3944     3935       -9     
==========================================
- Hits         3816     3808       -8     
+ Misses        128      127       -1

Flag	Coverage Δ
unittests	`96.77% <93.75%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
copier/errors.py	`91.11% <71.42%> (-4.45%)`	⬇️
copier/main.py	`95.66% <100.00%> (ø)`
copier/types.py	`89.47% <100.00%> (+3.47%)`	⬆️
copier/user_data.py	`94.65% <100.00%> (+0.06%)`	⬆️
tests/test_config.py	`99.08% <100.00%> (ø)`

sisp

LGTM! 🎉 The code coverage reduction seems irrelevant to me.

henryiii · 2023-07-03T05:53:53Z

Could you bump the pydantic version in the CI too? Looks like it’s pinned and/or cached too:

 Installing pydantic (1.10.9): Skipped for the following reason: Already installed

lhupfeldt · 2023-07-03T08:07:53Z

Please release this a a quick fix for both 7.x an 8.x unless #1220 is released.
We use copier as part of our CI pipeline. Implementing the suggested workarounds is not feasible for us.

pawamoy · 2023-07-03T09:23:06Z

@henryiii, good catch. I'm personally not used to track lock files for libraries, even when they provide a CLI 😄 @sisp should I update the lock file in this PR or another?

@lhupfeldt yes, don't worry, we'll surely merge and release this. We can't go faster than what our free time allows. Waiting for either:

a second approval by @yajo
a fix in the merge permissions, so that @sisp's approval is enough and allows me to merge

lhupfeldt · 2023-07-03T09:26:35Z

@henryiii, good catch. I'm personally not used to track lock files for libraries, even when they provide a CLI 😄 @sisp should I update the lock file in this PR or another?

@lhupfeldt yes, don't worry, we'll surely merge and release this. We can't go faster than what our free time allows. Waiting for either:
* a second approval by @yajo

* a fix in the merge permissions, so that @sisp's approval is enough and allows me to merge

Thank you, I'm aware it is a volunteer effort and it is much appreciated.
Note that I'm really hoping for a backport of the fix to 7.x, because I now have two issues :)

pawamoy · 2023-07-03T09:35:42Z

Note that I'm really hoping for a backport of the fix to 7.x, because I now have two issues :)

The fix is simple, so I think that's reasonable 🙂

sisp · 2023-07-03T14:38:46Z

We use copier as part of our CI pipeline. Implementing the suggested workarounds is not feasible for us.

@lhupfeldt Why not? You should be able to cap Pydantic as a temporary workaround on your end.

I'm personally not used to track lock files for libraries, even when they provide a CLI 😄

@pawamoy Slightly off-topic, but how do you ensure deterministic dev and test environments then? 🤔

@sisp should I update the lock file in this PR or another?

@pawamoy I think in this PR to check that the fix actually works for Pydantic v2, although this means we'll be developing against Pydantic v2 then since that'll be the installed version; meaning we can't test against v1 anymore, but the same is true vice versa. I assume Pydantic won't keep supporting v1 anyway. To test against both v1 and v2 we'd need something like Tox which doesn't work well with lock files though. 😵‍💫

I think @henryiii has been thinking about dependencies, constraints, upper bounds, etc. a lot (fantastic article by the way 👏). Any advice from your side?

pawamoy · 2023-07-03T14:56:55Z

Slightly off-topic, but how do you ensure deterministic dev and test environments then?

I don't. Each dev has their own lock file. I let things break as new versions of dependencies come out. It sometimes break my own CI (for PRs on my projects), but I'm fine with that as it warns me early that something is broken. Locally, I can update my lock file to test on latest dependencies versions, without worrying about reverting the lock file before commits.

Also, allowing each dev to have their own lock file integrates better with projects offering insiders versions (Material for MkDocs, my own projects 😉), as these versions can be fetched from local, self-hosted PyPI servers. With a tracked lock file, devs using insiders versions have to re-lock, test/check then revert each time. It's also easier for devs that do not have access to the same index as the one that was used to lock (users whose government blocks pypi.org for example, and who have to use mirrors).

we'd need something like Tox

Not sure about Poetry, but PDM can be used with both Nox and Tox.

henryiii · 2023-07-03T15:21:56Z

I'm fine with that as it warns me early that something is broken

Yes, it would have been nice for this breakage to have been caught here, rather than in all the downstream projects that weren't using lock files. Same reasoning, I'd much rather get early warning of a breakage than have stable CI that doesn't tell me that my users are currently broken. :) (For a library, not a deployment of something, obviously)

However, the caching is nice to have. I wonder if enabling dev versions would generally help? There were a lot of dev releases of Poetry. Not all packages do that, but locking + periodic automated updates + dev versions would be really a good way to go if packages did that often enough.

On important projects, I usually test against minimum supported versions too - I have a test/constraints.txt file with all my specified minimums, and have a job or two dedicated to testing that.

Not sure about Poetry, but PDM can be used with both Nox and Tox.

Huge fan of PDM. It supports all PEP 621 backends, so you can use hatchling, flit-core, setuptools, one of the compiled ones, etc. - not just pdm-backend. And because you are using standards instead of custom Poetry config, Black and Ruff can pick up your minimum version of Python directly from your config, no need to set it for each tool! :)

Hatch is nice too, and has multiple environment support, but doesn't support locking yet, which is the main reason I would use a tool like that.

pawamoy · 2023-07-03T15:31:04Z

For reference, @yajo previously tried to migrate to PDM but faced some obstacles. I believe all of them are resolved nowadays (except maybe for a missing pdm2nix tool?). See #357

henryiii · 2023-07-03T15:37:38Z

Yes, looks like someone has mentioned it at nix-community/poetry2nix#1183. Not familiar at all with nix. The GitHub dependency graph supports PEP 621 now. Not sure about dependabot, but it's easy to write a job that runs pdm update (though making CI run on that job requires an extra step).

sisp · 2023-07-03T17:04:10Z

This is what Poetry's docs say about committing a lock file for a library: https://python-poetry.org/docs/basic-usage/#committing-your-poetrylock-file-to-version-control

lhupfeldt · 2023-07-03T19:27:57Z

@lhupfeldt Why not? You should be able to [cap Pydantic as a temporary workaround] (#1225 (comment)) on your end.

Feasible was probably not the right word. Just some pinning, committing, PR review, build, and then undoing it all which I would rather not have to do:) We are generally setup for CI. Pinning stuff that breaks should be easier than it is, but that is work in progress.

majidaldo · 2023-07-07T17:18:27Z

Yes, looks like someone has mentioned it at nix-community/poetry2nix#1183. Not familiar at all with nix. The GitHub dependency graph supports PEP 621 now. Not sure about dependabot, but it's easy to write a job that runs pdm update (though making CI run on that job requires an extra step).

👋

Have you guys seen rye ? it creates two lockfiles, one for running and another for dev'ing.

From there, the path to nix I think would be pip2nix.

pawamoy · 2023-07-07T17:21:31Z

PDM is also able to create/use multiple lockfiles with specific groups of dependencies 🙂

lhupfeldt · 2023-07-08T12:30:18Z

Now that #1220 has been merged, it is probably not necessary to backport this fix to 7.x.

sisp · 2023-07-08T13:26:59Z

@pawamoy Could you update the locked Pydantic version to the latest v2, so that also CI can verify the correctness of the fix?

poetry update pydantic

pawamoy · 2023-07-08T13:30:30Z

Oh yes, thanks for the reminder! Done.

sisp

@pawamoy Could you please update your Poetry version to the most recent? The lock file nkw contains many entries category = "..." again which got dropped in Poetry v1.5.

sisp · 2023-07-08T13:51:08Z

Oh, it seems there are more incompatibilities with Pydantic v2.

pawamoy · 2023-07-08T13:58:29Z

Done. Let me know if I should rework the history to keep the lock file commits apart from the fix. Otherwise feel free to squash them all.

pawamoy · 2023-07-08T13:59:25Z

Also I'll take a look at the other errors 🙂

pawamoy · 2023-07-08T14:45:18Z

Ha, too bad, bump-pydantic didn't update anything 🤔
I pushed other fixes, but it's not finished yet. I converted the PR back to draft.
Also, do we want to keep support for Pydantic v1? I'm afraid I don't have enough time to work on this in a serious way...

yajo · 2023-07-10T16:13:27Z

Hi folks! Many things popped up here, sorry to be late to the party 😆

I see a bunch of off-topic discussions and I think they are interesting, but let me kindly suggest you to move them to the forum. Regarding the PR, I think the important part here is:

do we want to keep support for Pydantic v1?

IMHO no. They broke compatibility. We should just update the dependency and forget Pydantic v1.

sisp · 2023-07-11T16:05:36Z

@pawamoy Is it okay for you if I push commits to this PR? I have finished the Pydantic v2-only adaptations and would like to (a) retain your credit for your work on this and (b) avoid creating another PR.

pawamoy · 2023-07-11T16:21:38Z

Of course, no need to ask :)

sisp

Pydantic v2 is working now, only the flake-check jobs are failing with some error seemingly related to Nix that I don't understand yet.

sisp · 2023-07-11T16:38:56Z

tests/demo_migrations/copier.yaml

@@ -16,7 +16,7 @@ _exclude:

 _tasks:
  - "python [[ _copier_conf.src_path / 'tasks.py' ]] 1"
-  - [python, "[[ _copier_conf.src_path / 'tasks.py' ]]", 2]
+  - [python, "[[ _copier_conf.src_path / 'tasks.py' ]]", "2"]


Pydantic v2 doesn't parse an integer as a string anymore. But I think changing this tests isn't terrible because an integer was never officially supported according to the type hint of a task and the docs. Just FYI.

sisp · 2023-07-11T18:38:11Z

I suspect that annotated-types needs to be added to https://github.com/nix-community/poetry2nix/blob/master/overrides/build-systems.json, but I'm not familiar enough with Nix and poetry2nix. Do you know, @yajo?

yajo · 2023-07-14T05:19:10Z

Yes, that's it. I'll see if I can do that later, it's just a routine task.

yajo

The rest of the code looks great. I'll push later the Nix build fix so we can merge.

fix: Stop importing private exception from Pydantic

c0a2802

pawamoy mentioned this pull request Jul 2, 2023

fix: Stop importing private exception from Pydantic #1228

Closed

sisp approved these changes Jul 2, 2023

View reviewed changes

henryiii mentioned this pull request Jul 3, 2023

Pydantic 2.0 (complete rewrite) breaks copier #1225

Closed

chore: Update lock file (Pydantic)

4672c9b

Merge branch 'master' into support-pydantic-v2

353d1f3

sisp requested changes Jul 8, 2023

View reviewed changes

chore: Relock deps with Poetry 1.5.1

7119b4f

pawamoy marked this pull request as draft July 8, 2023 14:43

pawamoy changed the title ~~fix: Stop importing private exception from Pydantic~~ fix: Support Pydantic v2 Jul 8, 2023

fix: More fixes to support Pydantic v2

e73ec2e

yajo mentioned this pull request Jul 10, 2023

fix(pydantic): add upper dependency bound to fix unlocked installations #1244

Merged

Merge branch 'master' into support-pydantic-v2

de61758

sisp added 2 commits July 11, 2023 18:35

fix: support only Pydantic v2

9ab2388

style: revert pyproject.toml auto-formatting

025b437

sisp reviewed Jul 11, 2023

View reviewed changes

yajo approved these changes Jul 14, 2023

View reviewed changes

build(nix): fix pydantic 2

f520785

yajo marked this pull request as ready for review July 14, 2023 08:03

yajo enabled auto-merge (squash) July 14, 2023 08:04

sisp approved these changes Jul 14, 2023

View reviewed changes

yajo merged commit a42a3a7 into copier-org:master Jul 14, 2023
19 of 20 checks passed

sisp mentioned this pull request Nov 20, 2023

build(deps): bump pygments from 2.16.1 to 2.17.1 #1418

Closed

sisp mentioned this pull request Apr 1, 2024

⚠ CRITICAL: Copier 9.1.1 broken by too loose pyyaml-include requirement ⚠ #1568

Closed

pawamoy deleted the support-pydantic-v2 branch April 2, 2024 10:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Support Pydantic v2 #1229

fix: Support Pydantic v2 #1229

pawamoy commented Jul 2, 2023

codecov bot commented Jul 2, 2023 •

edited

Loading

sisp left a comment

henryiii commented Jul 3, 2023

lhupfeldt commented Jul 3, 2023 •

edited

Loading

pawamoy commented Jul 3, 2023

lhupfeldt commented Jul 3, 2023

pawamoy commented Jul 3, 2023

sisp commented Jul 3, 2023 •

edited

Loading

pawamoy commented Jul 3, 2023

henryiii commented Jul 3, 2023 •

edited

Loading

pawamoy commented Jul 3, 2023

henryiii commented Jul 3, 2023

sisp commented Jul 3, 2023 •

edited

Loading

lhupfeldt commented Jul 3, 2023 •

edited

Loading

majidaldo commented Jul 7, 2023

pawamoy commented Jul 7, 2023

lhupfeldt commented Jul 8, 2023

sisp commented Jul 8, 2023

pawamoy commented Jul 8, 2023

sisp left a comment

sisp commented Jul 8, 2023

pawamoy commented Jul 8, 2023

pawamoy commented Jul 8, 2023

pawamoy commented Jul 8, 2023 •

edited

Loading

yajo commented Jul 10, 2023

sisp commented Jul 11, 2023

pawamoy commented Jul 11, 2023

sisp left a comment

sisp Jul 11, 2023

sisp commented Jul 11, 2023 •

edited

Loading

yajo commented Jul 14, 2023

yajo left a comment

fix: Support Pydantic v2 #1229

fix: Support Pydantic v2 #1229

Conversation

pawamoy commented Jul 2, 2023

codecov bot commented Jul 2, 2023 • edited Loading

Codecov Report

sisp left a comment

Choose a reason for hiding this comment

henryiii commented Jul 3, 2023

lhupfeldt commented Jul 3, 2023 • edited Loading

pawamoy commented Jul 3, 2023

lhupfeldt commented Jul 3, 2023

pawamoy commented Jul 3, 2023

sisp commented Jul 3, 2023 • edited Loading

pawamoy commented Jul 3, 2023

henryiii commented Jul 3, 2023 • edited Loading

pawamoy commented Jul 3, 2023

henryiii commented Jul 3, 2023

sisp commented Jul 3, 2023 • edited Loading

lhupfeldt commented Jul 3, 2023 • edited Loading

majidaldo commented Jul 7, 2023

pawamoy commented Jul 7, 2023

lhupfeldt commented Jul 8, 2023

sisp commented Jul 8, 2023

pawamoy commented Jul 8, 2023

sisp left a comment

Choose a reason for hiding this comment

sisp commented Jul 8, 2023

pawamoy commented Jul 8, 2023

pawamoy commented Jul 8, 2023

pawamoy commented Jul 8, 2023 • edited Loading

yajo commented Jul 10, 2023

sisp commented Jul 11, 2023

pawamoy commented Jul 11, 2023

sisp left a comment

Choose a reason for hiding this comment

sisp Jul 11, 2023

Choose a reason for hiding this comment

sisp commented Jul 11, 2023 • edited Loading

yajo commented Jul 14, 2023

yajo left a comment

Choose a reason for hiding this comment

codecov bot commented Jul 2, 2023 •

edited

Loading

lhupfeldt commented Jul 3, 2023 •

edited

Loading

sisp commented Jul 3, 2023 •

edited

Loading

henryiii commented Jul 3, 2023 •

edited

Loading

sisp commented Jul 3, 2023 •

edited

Loading

lhupfeldt commented Jul 3, 2023 •

edited

Loading

pawamoy commented Jul 8, 2023 •

edited

Loading

sisp commented Jul 11, 2023 •

edited

Loading