OME-TIFF: `Result::unwrap()` on an `Err` value: General("invalid utf-8 sequence of 1 bytes from index 2144")

[weiji14]

FYI I decided to try the OME-TIFF file from https://www.cellimagelibrary.org/images/40613 and encountered another panic on opening (xref virtual-zarr/virtual-tiff#32).

Originally posted by @maxrjones in #97

Code to reproduce panic:

import asyncio
from async_tiff.store import HTTPStore

from async_tiff import TIFF

async def test_ome_tiff():
    store = HTTPStore(url="https://cildata.crbs.ucsd.edu/")
    path = "media/images/40613/40613.tif"

    tiff = await TIFF.open(path=path, store=store)
    return tiff

if __name__ == "__main__":
    asyncio.run(test_ome_tiff())

Full traceback:

thread 'tokio-runtime-worker' panicked at python/src/tiff.rs:42:18:
called `Result::unwrap()` on an `Err` value: General("invalid utf-8 sequence of 1 bytes from index 2144")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "/home/user/projects/async-tiff/python/tests/test_ome_tiff.py", line 58, in <module>
    asyncio.run(test_ome_tiff())
  File "/home/user/.local/share/uv/python/cpython-3.12.7-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/uv/python/cpython-3.12.7-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/uv/python/cpython-3.12.7-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/user/Documents/github/async-tiff/python/tests/test_ome_tiff.py", line 49, in test_ome_tiff
    tiff = await TIFF.open(path=path, store=store)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pyo3_async_runtimes.RustPanic: rust future panicked: unknown error

Panic is caused by this .unwrap() here:

async-tiff/python/src/tiff.rs

Lines 39 to 42 in 6b7df90

	let ifds = metadata_reader
	.read_all_ifds(&metadata_fetch)
	.await
	.unwrap();

[kylebarron]

Thanks! If it's easy it would be great to have these reproductions on the Rust side so we get fuller tracebacks for the panics.

[weiji14]

The cause of the panic was some non-UTF8 characters in the ImageDescription tag in the https://cildata.crbs.ucsd.edu/media/images/40613/40613.tif file. Actual error is from this line:

async-tiff/src/metadata/reader.rs

Lines 617 to 619 in 6b7df90

	Ok(Value::Ascii(
	String::from_utf8(out).map_err(|err| AsyncTiffError::General(err.to_string()))?,
	))

If I change that line to Ok(Value::Ascii(String::from_utf8_lossy(&out).into_owned())) based on this stackoverflow answer, we can get the actual output XML (formatted below):

<?xml version=\"1.0\" encoding=\"UTF-8\"?><!-- Warning: this comment is an OME-XML metadata block, which contains crucial dimensional parameters and other important metadata. Please edit cautiously (if at all), and back up the original data before doing so. For more information, see the OME-TIFF web site: http://loci.wisc.edu/ome/ome-tiff.html. -->
<OME xmlns:AML=\"http://www.openmicroscopy.org/Schemas/AnalysisModule/2008-09\" xmlns:Bin=\"http://www.openmicroscopy.org/Schemas/BinaryFile/2008-09\" xmlns:MLI=\"http://www.openmicroscopy.org/Schemas/MLI/2008-09\" xmlns:SA=\"http://www.openmicroscopy.org/Schemas/SA/2008-09\" xmlns:OME=\"http://www.openmicroscopy.org/Schemas/OME/2008-09\" xmlns:SPW=\"http://www.openmicroscopy.org/Schemas/SPW/2008-09\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:CA=\"http://www.openmicroscopy.org/Schemas/CA/2008-09\" xmlns:STD=\"http://www.openmicroscopy.org/Schemas/STD/2008-09\" xmlns:CLI=\"http://www.openmicroscopy.org/Schemas/CLI/2008-09\" xmlns=\"http://www.openmicroscopy.org/Schemas/OME/2008-09\" xsi:schemaLocation=\"http://www.openmicroscopy.org/Schemas/OME/2008-09 http://www.openmicroscopy.org/Schemas/OME/2008-09/ome.xsd http://www.openmicroscopy.org/Schemas/BinaryFile/2008-09 http://www.openmicroscopy.org/Schemas/BinaryFile/2008-09/BinaryFile.xsd http://www.openmicroscopy.org/Schemas/CA/2008-09 http://www.openmicroscopy.org/Schemas/CA/2008-09/CA.xsd http://www.openmicroscopy.org/Schemas/SPW/2008-09 http://www.openmicroscopy.org/Schemas/SPW/2008-09/SPW.xsd http://www.openmicroscopy.org/Schemas/STD/2008-09 http://www.openmicroscopy.org/Schemas/STD/2008-09/STD.xsd\" UUID=\"3af39f55-0ac0-431a-bc60-8f9c3e782b85\">
<Experimenter ID=\"urn:lsid:export.openmicroscopy.org:Experimenter:46be26c5-9fcf-497e-a913-48513759f00b_3\"/>
<Group ID=\"urn:lsid:export.openmicroscopy.org:ExperimenterGroup:46be26c5-9fcf-497e-a913-48513759f00b_54:3259518\"/>
<Image DefaultPixels=\"urn:lsid:export.openmicroscopy.org:Pixels:46be26c5-9fcf-497e-a913-48513759f00b_40613:24177120\" ID=\"urn:lsid:export.openmicroscopy.org:Image:46be26c5-9fcf-497e-a913-48513759f00b_40613:25467732\" Name=\"IM_20100715_Nikitina_3_Niki_6_004_001_2_����������_lg.jpg\">
<CreationDate>2012-03-25 21:26:29.0</CreationDate>
<ExperimenterRef ID=\"urn:lsid:export.openmicroscopy.org:Experimenter:46be26c5-9fcf-497e-a913-48513759f00b_3\"/>
<GroupRef ID=\"urn:lsid:export.openmicroscopy.org:ExperimenterGroup:46be26c5-9fcf-497e-a913-48513759f00b_54:3259518\"/>
<LogicalChannel ID=\"urn:lsid:export.openmicroscopy.org:LogicalChannel:46be26c5-9fcf-497e-a913-48513759f00b_81798:24177119\" Name=\"Red\" SamplesPerPixel=\"1\"/>
<LogicalChannel ID=\"urn:lsid:export.openmicroscopy.org:LogicalChannel:46be26c5-9fcf-497e-a913-48513759f00b_81799:24177119\" Name=\"Green\" SamplesPerPixel=\"1\"/>
<LogicalChannel ID=\"urn:lsid:export.openmicroscopy.org:LogicalChannel:46be26c5-9fcf-497e-a913-48513759f00b_81800:24177119\" Name=\"Blue\" SamplesPerPixel=\"1\"/>
<Pixels BigEndian=\"true\" DimensionOrder=\"XYCZT\" ID=\"urn:lsid:export.openmicroscopy.org:Pixels:46be26c5-9fcf-497e-a913-48513759f00b_40613:24177120\" PixelType=\"uint8\" SizeC=\"3\" SizeT=\"1\" SizeX=\"1024\" SizeY=\"943\" SizeZ=\"1\">
<TiffData>
<UUID FileName=\"__omero_export__2069104425008571311.ome.tiff\">3af39f55-0ac0-431a-bc60-8f9c3e782b85</UUID>
</TiffData>
</Pixels>
</Image>
</OME>

Notice the � (U+FFFD, REPLACEMENT CHARACTER) on one of the lines where the non-UTF8 characters have been replaced. Technically the OME-TIFF spec does say that the XML should be UTF-8 encoded, but returning those replacement characters might be ok too?

I was gonna add a test for this, but encountered a stange issue when using that String::from_utf8_lossy function. Things work ok if I test that https://cildata.crbs.ucsd.edu/media/images/40613/40613.tif file downloaded locally, but cargo test will get stuck if I try to stream that file from a remote HTTP connection, not sure why.

Anyways, I've added a test to read a single-channel OME-TIFF in #102 for now, will try to do a proper fix for those non-UTF8 characters later.

[kylebarron]

but cargo test will get stuck if I try to stream that file from a remote HTTP connection, not sure why

It's possible that has something to do with the hangs I've gotten separately #89

In any case, best to check in a tiny copy of that file into the test directory to use for a test case.

[kylebarron]

Back to the core issue; yeah, I guess an open question is what to do with invalid input. Because according to the spec, that string should be valid UTF8, right? I think using from_utf8_lossy here is a reasonable solution.