py.typed present, but not all methods have types

[allisonkarlitskaya]

We're using Pillow from our strictly-typed Python code. We recently updated to Fedora 40, which brought us python3-pillow-10.3.0-1.fc40.x86_64.

This version of Pillow ships a py.typed file, which was not present in the version in Fedora 39 (python3-pillow-10.2.0-1.fc39.x86_64).

Previously, our mypy setup was ignoring our import of PIL.Image from a typing perspective, but with the py.typed file, it tries to check types. This is problem, because we call PIL.Image.Image.open() which (among many other function) is untyped:

Pillow/src/PIL/Image.py

Line 837 in e542c9f

def load(self):

The following example produces no errors on Fedora 39, but fails on Fedora 40:

from PIL import Image

with open('file.png', 'rb') as file:
    Image.open(file).load()

$ mypy --strict bzzt.py
bzzt.py:4: error: Call to untyped function "load" in typed context  [no-untyped-call]
Found 1 error in 1 file (checked 1 source file)

Output from python3 -m PIL --report:

--------------------------------------------------------------------
Pillow 10.3.0
Python 3.12.2 (main, Feb 21 2024, 00:00:00) [GCC 14.0.1 20240217 (Red Hat 14.0.1-0)]
--------------------------------------------------------------------
Python executable is /usr/bin/python3
System Python files loaded from /usr
--------------------------------------------------------------------
Python Pillow modules loaded from /usr/lib64/python3.12/site-packages/PIL
Binary Pillow modules loaded from /usr/lib64/python3.12/site-packages/PIL
--------------------------------------------------------------------
--- PIL CORE support ok, compiled for 10.3.0
--- TKINTER support ok, loaded 8.6
--- FREETYPE2 support ok, loaded 2.13.2
--- LITTLECMS2 support ok, loaded 2.16
--- WEBP support ok, loaded 1.3.2
--- WEBP Transparency support ok
--- WEBPMUX support ok
--- WEBP Animation support ok
--- JPEG support ok, compiled for libjpeg-turbo 3.0.2
--- OPENJPEG (JPEG2000) support ok, loaded 2.5.2
--- ZLIB (PNG/ZIP) support ok, loaded 1.3.0.zlib-ng
--- LIBTIFF support ok, loaded 4.6.0
--- RAQM (Bidirectional Text) support ok, loaded 0.8.0
--- LIBIMAGEQUANT (Quantization method) support ok, loaded 4.2.2
--- XCB (X protocol) support ok
--------------------------------------------------------------------

Out of curiosity, I got the current main branch of Pillow and added

[tool.mypy]
disallow_untyped_defs = true

to pyproject.toml. There's quite a lot of untyped defs:

Found 853 errors in 58 files (checked 102 source files)

I'd like to propose a PR, but I'm afraid I don't have the time to work on such a large change. Even the Image class has quite a lot of untyped methods (load(), verify(), draft(), expand(), filter(), and many others).

In the meantime, I think the most appropriate course of action may be to temporarily remove the py.typed file.

[johnthagen]

We hit a large number of new Mypy issues when we upgraded to Pillow 10.3.0, to the point where we may pin to 10.2.0 until they are addressed and all the missing type hints added (or py.typed is removed until this is done).

Some of the errors can be suppressed in the Mypy config:

[tool.mypy]
untyped_calls_exclude = [
    "PIL",
]

One example of a function missing type hints is PIL.Image.fromarray.

[radarhere]

While I don't feel comfortable making the decision to remove py.typed myself, I can help add further type hints. I've created #8030 to add verify(), draft() and _expand() (I think you meant this, as ImageOps.expand() is already typed in 10.3.0).

Edit: I've created #8042 for filter().

#8032 has been created to add type hints for load().

I know that to a certain extent, these were just examples you were mentioning, and there is plenty more work to go.

we call PIL.Image.Image.open() which (among many other function) is untyped

This should already be partially type hinted in 10.3.0.

Pillow/src/PIL/Image.py

Line 3225 in 5c89d88

def open(fp, mode="r", formats=None) -> Image:

Thanks to #7944, that has been improved in main.

Pillow/src/PIL/Image.py

Lines 3250 to 3254 in ddbf08f

	def open(
	fp: StrOrBytesPath | IO[bytes],
	mode: Literal["r"] = "r",
	formats: list[str] | tuple[str, ...] | None = None,
	) -> ImageFile.ImageFile:

One example of a function missing type hints is from PIL.Image.fromarray.

This has actually already been added to main in #7936

The next release of Pillow is scheduled for July 1st, so if there are any more specific methods you would like to see typed before then, feel free to request them.

[hugovk]

I'd like to propose a PR, but I'm afraid I don't have the time to work on such a large change. Even the Image class has quite a lot of untyped methods (load(), verify(), draft(), expand(), filter(), and many others).

Smaller PRs are very welcome and also preferred, they're easier to review and quicker to merge.

[radarhere]

Out of curiosity, I got the current main branch of Pillow and added

[tool.mypy]
disallow_untyped_defs = true

to pyproject.toml. There's quite a lot of untyped defs:

Found 853 errors in 58 files (checked 102 source files)

As a progress update, that number is now halved - we're down to

Found 421 errors in 40 files (checked 102 source files)

[radarhere]

Pillow 10.4.0 has now been released, with type hints for all of the methods explicitly mentioned in this issue.

[johnthagen]

@radarhere 10.4.0 allowed us to remove a lot of type: ignore statements, thanks! ❤️

In our particular codebase, the only remaining untyped call we hit after upgrading to 10.4.0 is:

Pillow/src/PIL/ImageTk.py

Line 99 in 8c26622

def __init__(self, image=None, size=None, **kw):

[radarhere]

#8191 has now typed that method.

[radarhere]

We're now down to

Found 97 errors in 16 files (checked 283 source files)

[akx]

Looks like

Success: no issues found in 283 source files

to me on 71876cf :)

[radarhere]

If you're happy with that, great! However, the OP wanted us to meet a different standard - not showing any errors when disallow_untyped_defs = true was enabled.

With that setting, we're still at

Found 57 errors in 11 files (checked 283 source files)

[akx]

Ah, I thought we were going with what make mypy checks. My bad!

[johnthagen]

To avoid regressions, I'd recommend turning this on in this repo once the remaining missing types are fixed.

[radarhere]

typeshed's Pillow stubs have now been removed in favour of ours.

In main, we're now down to

Found 5 errors in 2 files (checked 294 source files)

Those improvements will be part of Pillow 11.0.0, due out on the 15th of this month.

#8362 and #8227 are open to handle the remaining errors. After that, sure, adding disallow_untyped_defs = true to prevent regressions sounds great.

[radarhere]

Pillow 11.0.0 has now been released, with

Found 3 errors in 1 file (checked 294 source files)

#8362 will handle the remaining errors, and also add disallow_untyped_defs = true to prevent regressions, resolving this.