Add defaults for parameters

[yangdanny97]

Closes #1292

This PR uses a semi-automated/LLM-assisted method to add a few hundred parameter defaults to pandas-stubs, replacing what was previously ....

There were 3 annotations that had to change because their type signatures are not compatible with the default.

The codemod that made these transformations is here: https://gist.github.com/yangdanny97/0d81e16f6374582413dd5bcf90d902fd

How it works is:

1. parse pandas-stubs and find the module, method/function, & parameter name for any params w/ default ...
2. parse the docstring of the method/function in pandas, and if the parameter's docstring contains "default" extract the part after that keyword
3. look up the type annotation in a mapping of the extracted snippet to type annotation
4. use libCST to apply the type annotation to the parameter

The main thing that needs careful reviewing is the default_type_map which maps the docstring snippet to the type annotation we replace the ... with. That part was generated with chatGPT's help (I gave it every snippet I got after step 2), and I commented out the mappings that seemed wrong or unclear. So, even though this was "LLM-assisted" the script is deterministic and will always give you the same thing each time, and the LLM is only involved in a one-time manual step.

[loicdiridollou]

Just for my own knowledge, what is adding the defaults instead of the ellipsis trying to achieve? Is it to help when looking at the docs in the IDE, for pyrefly, or something else?

[yangdanny97]

Just for my own knowledge, what is adding the defaults instead of the ellipsis trying to achieve? Is it to help when looking at the docs in the IDE, for pyrefly, or something else?

It's to help with looking at docs/signature in the IDE. This work is unrelated to Pyrefly.

Here's a thread on typeshed w/ some discussion on the pros and cons of defaults: python/typeshed#8988

The consensus seems to be that if the default is simple (like None, or some literal) then it's fine, but if it's complex we use .... Mypy's stub generator has been updated to emit stubs containing simple defaults.

The second reason I made the PR is just to test out this way of doing LLM-assisted codemods. I have another PR in the works that uses the same approach to add a few hundred types for un-annotated parameters and returns, but that one requires more manual filtering to make sure the generated types are good.

[Dr-Irv]

I checked every method you changed by looking in the docs. It picks up things that we need to delete or modify anyway. Some general comments:

1. I use the documented signatures to determine default values, not just what the docs say, because the docs don't always indicate the correct default value, but the signature does because it is picked up in the docs from the actual python declaration.
2. The copy argument often has a default value of None, so I had to change the declaration a lot of times in your PR, and probably missed a few.

It might be worth modifying your script to look at the default values that are in the signatures rather than the docs. If you do that, you'll pick up a lot more defaults (and probably catch a bunch that I caught via this review).

This PR points out that the pandas docs could be improved, but creating an issue for each improvement would be a lot of work.

When you update, please don't force push, and don't resolve conversations, because then I can see if everything I found was addressed, and can just concentrate on what is changed in your next commit(s).

Thanks for doing this.

[Dr-Irv]

2. The copy argument often has a default value of None, so I had to change the declaration a lot of times in your PR, and probably missed a few.

I looked further into the copy argument issue. Eventually, that argument will be removed. So I think it would be best to not change copy: _bool = False or copy: _bool = True to be copy: _bool | None = None, because None won't be accepted in the future. So ignore those requests in my review.

[yangdanny97]

Thanks for the detailed review!

That's a good point about taking the default values directly from the function declaration. I've pushed a commit doing that using https://gist.github.com/yangdanny97/170f82ee5389584f8b6292bc4ea9c24d

I'll go thru the comments one by one and address them over the next few days, and keep track of which followup issues we need to file.

[yangdanny97]

Ok, I think all the comments have been addressed.

[Dr-Irv]

I found a few things in your latest changes. I resolved conversation on all my previous comments except for ones that I didn't see addressed. I think the open ones will still show when you load the github page.

[Dr-Irv]

All looks good - just need to resolve conflicts.

[Dr-Irv]

Looks like you have to carefully look at the version of shift in main and the version of shift in your PR and reconcile the differences. There was a recent change to the type of the freq argument.

[Dr-Irv]

Looks like there are more conflicts for you to resolve.