testing pancel.csv

[Milad4849]

It appears that the output of imc.create_panel_from_imc_panel should be tested against imc_test_data_steinbock_path/panel.csv in the test dataset folder. However the outcome of imc.create_panel_from_imc_panel(imc_test_data_steinbock_path / "raw" / "panel.csv") does not match the file. Am I missing something here?

[jwindhager]

Could you diff the two files to see where they don't match?

[Milad4849]

Comparing them as dataframes.

[jwindhager]

Yes, that makes sense: the only missing column seems to be cellpose, which was introduced with steinbock v0.16.0. The test data has not been updated in a while, so likely requires a new release.

[Milad4849]

Apart from the column, other differences also make direct comparison not possible. Shall I run the test data through the pipeline and produce new output files for a new release of the test data?

[jwindhager]

What are those other differences? Yes, it would be good to update the TestData @nilseling

[Milad4849]

That fixes the column datatype issue, the 'NA' vs 'NaN' and the indexes remain. It didnt seem to me that I could specify options differently to read them without these differences.

[jwindhager]

Thats because create_panel_from_imc_panel interprets the ilastik/deepcell/cellpose columns as nullable pd.UInt8Dtype:

steinbock/steinbock/preprocessing/imc.py

Lines 73 to 79 in 66e9717

	if "ilastik" not in panel:
	panel["ilastik"] = pd.Series(dtype=pd.UInt8Dtype())
	panel.loc[panel["keep"], "ilastik"] = range(1, panel["keep"].sum() + 1)
	if "deepcell" not in panel:
	panel["deepcell"] = pd.Series(dtype=pd.UInt8Dtype())
	if "cellpose" not in panel:
	panel["cellpose"] = pd.Series(dtype=pd.UInt8Dtype())

but pd.read_csv used by read_panel reads optional columns as pandas default data type (non-nullable float):

steinbock/steinbock/io.py

Lines 74 to 85 in 66e9717

	panel = pd.read_csv(
	panel_file,
	sep=",|;",
	dtype={
	"channel": pd.StringDtype(),
	"name": pd.StringDtype(),
	"keep": pd.BooleanDtype(),
	},
	engine="python",
	true_values=["1"],
	false_values=["0"],
	)

As said, on disk the files should be the same, and the index is irrelevant.

[jwindhager]

It may make sense to unify this behavior (either use non-nullable or nullable column types), but I'd be careful with touching read_panel, since its used a lot. Maybe rather adapt create_panel_from_imc_panel (i.e., don't use nullable column types anywhere in the code), but this is something I would not spend too much time on, since conversion from IMC Segmentation Pipeline panels to steinbock panels should be considered a legacy feature IMHO.

[Milad4849]

All right, it is clear now, thanks for the help!

[Milad4849]

#175 should fix it