Can segmentation crop images rather than output the segmented regions?

[sripar2000]

Hello, I'm using RootPainter to segment rhizotron/rhizobox images and it's working great (also really appreciate the incredibly clear documentation and instructions). However, the segmentation tends to overestimate the root area, so if I were to send it to RhizoVision, the diameter, network area, etc would be greater than true values. This is fine for treatment differences but I'm looking to calibrate 3d models using this data.

Is there a feature for the segmentation to crop out the areas of interest so that I can more manually threshold in ImageJ? Thanks in advance.

[Abe404]

Hi,

Thanks for the positive comments and question!

the segmentation tends to overestimate the root area

If you are having trouble with over segmentation i.e segmentation going outside the border of the root, then I think the best way to fix this is with annotation. When interactively training the model keep your foreground (red) annotation within the root. What I mean by this is leave some space between your foreground annotation and the edge of the root. Even just a couple of pixels should be enough. This should help prevent annotation errors where foreground has gone over the background i.e outside of the root (encouraging the model to over segment roots).

Then with the background (green) annotation you can push the boundary inwards to make sure the root is not over-segmented. Unfortunately you'd probably want to attempt this approach from the beginning of a project, as existing projects would be significantly influenced by the annotations you have done so far.

If that's not clear I could add some images to help explain.

Is there a feature for the segmentation to crop out the areas of interest so that I can more manually threshold in ImageJ? Thanks in advance.

I'm not 100% sure I understand what you mean by 'crop out the areas of interest'. Could you provide more details of how you would want this to work?

I have an idea for one feature that might help:

Instead of providing the binary segmentation from RootPainter i.e each pixel being labelled as foreground or background, RootPainter could could provide you with the prediction 'probabilities', where each pixel is a value between 0 and 1. i.e it could be 0.1 or 0.8 etc. Then you could experiment with thresholding the images yourself (converting to binary) based on your own desired probability threshold. I'm open to adding an option to output the 'probabilities' so you could perform this thresholding yourself instead of using 0.5 which is the current default.

But, my suggestion for now would be to experiment with being more conservative with your foreground annotation during the interactive training process as I suspect this could address the problem without modifications to the software.

Fixing the problem by altering the annotation would also provide you with feedback during the annotation, so you can see if it is fixed.

Kind regards,
Abraham

[Abe404]

BTW if you do just want to crop to areas of interest, I'd suggest doing this as a pre-processing step. If your data is small enough then doing this manually works fine (I have used preview in the past). For larger datasets a more automatic approach would be required i.e a python/ImageJ script.

If you want to use RootPainter to crop to areas of interest, before you generate a more fine-detailed segmentation then I can also provide advice on this (perhaps this is what you meant anyway?).

There's a lot of different ways to do this but what I advise in this situation is to train two models. First a localisation model on a reduced resolution version of your image, say at 1/4 or 1/8. This model can be trained to predict the general area where the root is (don't worry about exact boundaries) and then to exclude the larger background region. It's actually pretty quick to train this type of localisation model. Then you'd use the structure predicted from this model to generate crop coordinates that would be used for the generation of a dataset that only included your regions of interest. Then a high resolution model could be trained on just these areas of interest. Does that make sense?

The above approach can be done now, but may require some scripting outside of RootPainter. I'd be open to better supporting it within the client software itself.

Another (perhaps simpler) approach would be to train one model and then use connected component analysis on the output, so you automatically exclude any structures that are not connected to your largest connected root structure in each image. Perhaps this could also be performed in RVE/ImageJ already without too much trouble?

Kind regards,
Abraham

[sripar2000]

Hi Abraham,

Thanks so much for the replies. I'm replying to the previous answer here as well.

I can try being more precise with my annotations and let you know how it works. I have been pushing it right up to the edge of the roots but hadn't pushed it past that boundary.

I think you addressed my question about cropping areas of interest, apologies for not wording it better. The localisation model sounds like an interesting approach and I'll look more into it. Connected component analysis also sounds like a good approach from what you've explained though it's something I'm less familiar with. I'll do more research on both these approaches and reach out if there's some necessary utility in RootPainter to make them work.

Thanks again for the responses, it was very helpful!

Sriram

[Abe404]

Hi Sriram,

If it helps, in this video (from the linked time point for the next couple of minutes) I discuss a bit about the importance of annotating within the boundaries of the root: https://youtu.be/73u73tBvRO4?t=5605

I am rushing my annotation a bit in the above demo, but the general idea is that the foreground annotation should be within the root boundary.

I also give some extended advice on annotation approaches in this video, and what types of errors to avoid (from the supplementary material of the paper):
https://nph.onlinelibrary.wiley.com/action/downloadSupplement?doi=10.1111%2Fnph.18387&file=nph18387-sup-0002-VideoS1.mp4

I think, in general, you should not have to push the green (background) annotation past the root boundary, but perhaps on blurry boundaries there could be a need for use of background annotation over the blurry boundary region to try to clean up the root segmentation.

In the short term, I suggest performing some trial and error to see if changing your approach to annotation can solve the problem. Longer term, if you are using the metrics plot (available from the extras menu), it is worth taking into consideration how your annotation strategy might effect error quantification as it is assumed that all errors are corrected when computing each individual image error measure.

If you wanted to experiment with connected component analysis, perhaps a function like this might be worth trying: https://scikit-image.org/docs/stable/api/skimage.morphology.html#remove-small-objects but this will require some python scripting (I suspect it's also possible with ImageJ, but I have also not tried ImageJ).

The localisation approach is something I'd be happy to better support within RootPainter (without any extra python scripts) so just let me know if/when you'd like more information on this. Guy's images below are also a great example. Thanks @geezacoleman!

Kind regards,
Abraham

[sripar2000]

Hi Abraham,

Thanks for the further info! I spent some time yesterday playing around with different annotations and I think I'm already seeing improvement. Hopefully some more training will fix my issue but I'll be in touch regarding the localisation approach if it's necessary.

Thanks again, I really appreciate your help!

Sriram

[Abe404]

Hi Sriram,

I just noticed a relevant feature was mentioned in Benjamin Delory's talk : https://youtu.be/c2YjmynhF8U?t=4600

You might want to experiment with 'filter non root objects' in RVE as shown in this screenshot from the video:

[geezacoleman]

Thought I'd share my experience with this - I was having some difficulty on a dataset of cotton/Palmer amaranth plants with multiple plants per image, so tried the the two model approach based on @Abe404's advice.

We trained one model to pick up where the general region of each plant was and then a separate model with higher resolution on the result of that first stage. The process I went through is roughly described below - happy to share more if it's helpful!

Input image:

Stage 1 segmentation output:

Stage 2 input - crop of a region:

Stage 2 output:

[sripar2000]

This is more or less what I'm trying to do as well, except with separating roots from background peat/perlite artifacts. Did you do stage 2 segmentation in ImageJ?

Thanks so much for sharing!

[geezacoleman]

Those peat/perlite artifacts have been a big hassle in this work with plants as well because they find their way to the soil at the top. I ended up sieving soil to reduce the quantity of them, but I imagine that's a bit more difficult with soil focused analyses.

Everything was done in RootPainter - I haven't tried ImageJ before, but I've found all other colour-only methods to be very lacking compared to RootPainter, mostly because I was interested in retaining as much non-green plant material as possible.

[geezacoleman]

To add a bit more detail, both models were trained on about 200 images, so it is a fairly fast process once you have all the image data ready to go and doable within a day I'd estimate.

[sripar2000]

Gotcha, looks like I have a lot more to learn with RootPainter! Thanks again for sharing your process.

[geezacoleman]

No worries, happy to help! It's been something I've been meaning to write up this year with Abe and share as a bit of a guide for less common/slightly different use cases. Hopefully can get it out soon...

[sripar2000]

Hi,

I realized I never updated this discussion thread about what I ended up doing to solve my problem. I used some simple tools from OpenCV to create a mask using the segmented RootPainter output to crop out my areas of interest. Then, some simple thresholding in ImageJ and/or RhizoVision seemed to (mostly) do the trick in refining the root architecture I wanted to isolate.

Sriram

[Abe404]

Thanks for the update. Sounds like a sensible approach.

Just in case you (or anyone else) needs to do this again, I created an issue to implement this as part of RootPainter (#66) which is now complete.

It's available in the latest release (from 0.2.21 onward, but I suggest using the latest release as this will have more bugs fixed).

https://github.com/Abe404/root_painter/releases

The function is available by clicking the 'mask images' option from the extras menu.

[sripar2000]

That's fantastic, thanks for all the quick hands-on updates!

Sriram

[Abe404]

No Problem! Thank you for using RootPainter! I'm looking forward to reading your paper.

Kind regards,
Abraham