Exporting bokeh plots

[NTBiotech]

Dear FlowKit team,

While Bokeh plots allow for interactivity, it is a bit complicated to export them as pdf.

Single plots as generated by the session.plot_gate and session.plot_scatter functions can be converted to svg, read with svglib and then saved as pdf using the renderPDF module.
If plots are combined, this is not possible, as a bokeh.layouts object cannot be converted to svg. Here, the layout can be converted to html and then to pdf using a headless browser setup.
I have build an implementation of these features and would like to add them to the package.
Another option would be to add matplotlib as an alternative to bokeh.

As I am not yet fully familiar with bokeh, I might be missing something.
I am looking forward to your opinions and feedback!

[whitews]

Hi Nicolas,

Some background before getting to the PDF export specifics...

Plotting in FlowKit has always been a compromise. When FlowKit development began, it did use matplotlib & seaborn for plotting. The problem was scalability & performance: matplotlib was incredibly slow for scatter plots once you start getting above 10k points. Researching alternatives, I found Bokeh the clear winner for handling lots of data points. For consistency, visually & API-wise, the rest of the plot functions were converted to Bokeh over time. This migration was complete in FlowKit 1.1 when matplotlib and seaborn were removed as dependencies.

The flip side is that most Python users are more familiar with matplotlib than Bokeh, and there are features (like exporting as PDF) where matplotlib has the advantage. Having both MPL & Bokeh as options, where the user could select a plot back-end of their choice would be nice, but there is a LOT of plot code in FlowKit and it touches nearly every class. There are many core cytometry features that take precedence given my time (e.g. the upcoming 1.3 release will begin support for spectral compensation). Another consideration is that plotting is somewhat of a challenge for maintenance in terms of testing...I've struggled to come up with a good strategy for testing plot code.

OK, for the PDF export. I am interested in your implementation. If you can create a feature branch off of the develop branch and submit a PR, I'm happy to review and have further discussion. I assume this would require additional dependencies like selenium and chromedriver (or geckodriver), which is ok and could be added as optional dependencies.

Thanks for taking the time to implement this and post this issue,
Scott

[whitews]

Hi Nicolas,

I saw a couple PRs come through that were then closed (assume you need to do more revisions). I did get a chance to briefly look at the commits. Before you spend too much time, maybe a discussion is warranted of how you envision the API and the libraries required.

One of the commits indicated the new functionality is not yet implemented in all the plot functions. I'm wondering if adding options to the existing plot methods is necessary, and instead we keep it simple, like a publicly available plot_utils function that the user can then pass a Bokeh Figure object to and it handles the conversion / export.

This has a few advantages. One, it has a lighter footprint on the current code base. This would also greatly reduce the number of additional tests for each modified plot method, we only need coverage for the plot_utils function. Second, it allows users to export any Bokeh Figure object, like a modified Bokeh Figure from FlowKit. I actually modify the FlowKit plots frequently (changing titles, etc.), and this would allow those to also be conveniently exported.

One other note. I saw you are using the pyppeteer library, of which I was not familiar. I looked at the pyppeteer repo and it states it is no longer maintained. They recommend using the playwright library as an alternative. I haven't looked into either of them in depth, but I would not add an unsupported dependency to FlowKit. I wanted to touch base with you before you spend too much time, and see if you could investigate whether playwright can also provide this functionality.

Finally, the more I've thought about this feature, the more I think it would be a great addition to FlowKit. Thanks again for the suggestion and I look forward to hearing your thoughts.

Kind regards,
Scott

[whitews]

Nicolas,

I had a chance to read some of the playwright docs and I don't think that is the best library to use either. Looks like it is geared towards end-to-end testing of headless browser setups, which is related, but that library seems intended for heavy web testing.

After some googling, I put together this minimal working example:

from bokeh.io import export_svgs
from svglib.svglib import svg2rlg
from reportlab.graphics import renderPDF
import flowkit as fk


# Using some of the FlowKit test data, change paths as needed
wsp_path = "../../data/8_color_data_set/8_color_ICS.wsp"
fcs_path = "../../data/8_color_data_set/fcs_files/101_DEN084Y5_15_E01_008_clean.fcs"
sample_id = '101_DEN084Y5_15_E01_008_clean.fcs'
sample_grp = 'DEN'

wsp = fk.Workspace(wsp_path, fcs_samples=fcs_path, load_missing_file_data=False)
wsp.analyze_samples(group_name=sample_grp)

gate_name = 'CD4+'

p = wsp.plot_gate(
    sample_id,
    gate_name=gate_name,
    x_min=0,
    x_max=1,
    y_min=0,
    y_max=1
)

# Set the output backend to SVG and export the plot
p.output_backend = "svg"
export_svgs(p, filename="plot.svg")

# Convert the SVG file to a PDF using svglib and reportlab
drawing = svg2rlg("plot.svg")
renderPDF.drawToFile(drawing, "plot.pdf")

The dependencies to get this working are:

• selenium
• svglib
• reportlab

Note, svglib needed cairo installed on my system (Ubuntu), specifically the libcairo2-dev package.

This seems to work well for my setup, but obviously needs to be tested on other platforms. Also, I haven't investigated how easy it is to specify the size of the exported PDF, presumably there are options for this.

-Scott

[NTBiotech]

Hi Scott,
Sorry for the delay, I was occupied with my studies.

Thank you for the input. If you would like me to, I can implement the example you mentioned. I used this before to export some of my own FACS plots, but I think there is a problem with exporting bokeh layout object to svg. Bokeh natively only supports figure objects. Which is why I used pyppeteer in the first place. Of course adding an unsupported dependency is not ideal, so I can look for some alternatives.

All the best,
Nicolas

[whitews]

Hi Nicolas,

I see, you explained this in the original post but I didn't make the connection with the layouts being the main problem. Looks like Bokeh may have a workaround for this by using the output_backend="svg" argument to Bokeh's figure function. Something like:

p1 = figure(width=300, height=300, output_backend="svg")
p1.circle([1, 2, 3], [2, 5, 8], size=10)

p2 = figure(width=300, height=300, output_backend="svg")
p2.line([1, 2, 3], [8, 5, 2], line_width=2)

layout = column(p1, p2)

export_svg(layout, filename="layout.svg")

This requires the same dependencies as my previous example. If this works, maybe we can add this output_backend kwarg as a pass-through option to the various plot functions. The default value should then be output_backend="canvas" to retain the current interactive behaviour. The user would then have to set this for their plots if they intend to export, but this can be explained in the docstrings and tutorials.

If you can test that this works, maybe this is a solution?

Thanks,
Scott