Graphics details upon export #81
Comments
|
Hi, Nils ... wonderful question. As background, the existing py4cytoscape call (https://py4cytoscape.readthedocs.io/en/latest/reference/generated/py4cytoscape.network_views.toggle_graphics_details.html?highlight=level%20of%20detail#py4cytoscape-network-views-toggle-graphics-details) invokes the long-standing LOD (Level of Detail) operation ... more than you'd ever want to know is here: https://manual.cytoscape.org/en/3.9.1/Rendering_Engine.html?highlight=level%20of%20detail#what-is-level-of-detail-lod I think you're referring to the new 3.10 export, which I have heard raves about. I'll look into it and get back to you. |
|
Hi, Nils ... we're still working on this. There is a new parameter for View Export, and its name is allGraphicsDetails= ... but it's not working when accessed from Cytoscape Automation. It'll likely be at least a week before we know more. |
|
I will add that parameter into my workflow in anticipation of the Automation update. Thank you for your help! |
|
Hi, Nils -- One of the engineers offers some advice as a workaround until Cytoscape itself can be fixed: The values in the image export dialog are saved to a props file in the CytoscapeConfiguration folder. If you export a network image with "Show All Graphics Details" unchecked, then close Cytoscape, the value of the checkbox is saved. If you then restart cytoscape and run the 'view export' command it will be like if you did allGraphicsDetails=false You can see this by going to the CytoscapeConfiguration folder, find the file named writer.org.cytoscape.io.internal.write.graphics.PNGWriter.props, there should be a line allGraphicsDetails=false. This actually isn't a good thing because changing something in the UI shouldn't have the side effect of changing the behaviour of a command. But it might be a good enough workaround for now. |
|
This will work for the time being. I have a singularity container that's starts up a brand new instance of Cytoscape within the container, exports an image, then is closed and the singularity instance is closed. I manually create a props file when building the image so that I can specify the REST API version without having to open the GUI and then close it. I will add this flag to the props file when I build the image.
Thanks so much! I think this should solve our problem until a production version is released. Your responsiveness to my queries is always much appreciated.
Nils
…________________________________
From: Barry Demchak ***@***.***>
Sent: Tuesday, April 26, 2022, 6:10 PM
To: cytoscape/py4cytoscape ***@***.***>
Cc: Oberg, Nils O ***@***.***>; Author ***@***.***>
Subject: Re: [cytoscape/py4cytoscape] Graphics details upon export (Issue #81)
Hi, Nils --
One of the engineers offers some advice as a workaround until Cytoscape itself can be fixed:
The values in the image export dialog are saved to a props file in the CytoscapeConfiguration folder. If you export a network image with "Show All Graphics Details" unchecked, then close Cytoscape, the value of the checkbox is saved. If you then restart cytoscape and run the 'view export' command it will be like if you did allGraphicsDetails=false
You can see this by going to the CytoscapeConfiguration folder, find the file named writer.org.cytoscape.io.internal.write.graphics.PNGWriter.props, there should be a line allGraphicsDetails=false.
This actually isn't a good thing because changing something in the UI shouldn't have the side effect of changing the behaviour of a command. But it might be a good enough workaround for now.
—
Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/cytoscape/py4cytoscape/issues/81*issuecomment-1110333017__;Iw!!DZ3fjg!96XRqVg5j8P4Ki4JXWK6WQL1EbxyjcCPTHsOBTYHIlE5ZluLWOUrmmr2-zFfPfDgil0mYtm0T60Wn6ajipNiukhbpw$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AI3P2JLAGBCGAOMJUWXAKDLVHBZWHANCNFSM5UC5ETNA__;!!DZ3fjg!96XRqVg5j8P4Ki4JXWK6WQL1EbxyjcCPTHsOBTYHIlE5ZluLWOUrmmr2-zFfPfDgil0mYtm0T60Wn6ajipPTSCHwsg$>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
|
Hi, Nils ... very much appreciate your kind words! Would you be interested in telling us more about your Singularity image work? This sounds very broadly useful! |
|
Hi Barry.
The idea behind our code is to start up an instance of Cytsocape using a singularity container, interact with it using the CyREST API using a separate singularity container to export an image of a network.
We are exporting images for tens of thousands (and soon even hundreds of thousands) sequence similarity networks (XGMML files). We do this on a cluster. We’ve currently been using the GUI version of Cytoscape via Xvfb in a job script, then running a singularity image with custom code that wraps py4cytoscape to interact with the instance of Cytoscape to generate the image. This process has a couple of issues:
1. You have to run Cytoscape GUI once to set the props file so that the REST API version is saved
2. Multiple dependencies need to be installed, i.e. Java, Xvfb, Python, etc, and this is not generally applicable (I work with our cluster team to install software as necessary, but would like to avoid having to ask them every time I need something)
3. We have a 15-20% failure rate where Cytoscape just closes without generating an image, and no message appears in the output
4. Sometimes images get exported with a large amount of whitespace around the network
5. Since there are multiple versions of Cytoscape running at any given time in order to batch process the networks, conflicts can occur in the karaf cache data (I am not sure if that’s the proper description), so we have to copy the karaf data from the base CytoscapeConfiguration directory to a temp directory and tell karaf to use that data directory. This happens every time we start up a Cytoscape instance, and then also we have to clean it up.
The idea of wrapping all of this into a container is attractive because it would allow us to easily support various versions of Cytoscape, different memory configs, cleanup code, include various utilties, etc., making running Cytoscape for exporting images tidier. Singularity is the only thing required to run. When building the singularity image, the properties are automatically set (i.e. REST API version, and the flag for image export that we have been discussing). You still have to bind an external directory to the image on run-time in order for Cytoscape to write the cache files to, but this is still cleaner than the approach we’ve been taking.
We also have a py4cytoscape image that interacts with Cytoscape (whether started with the above process or image) to generate the image. This one is very useful because we can wrap all of the dependencies inside of the image without having to worry about Python dependency installation and such.
Here is the repo that contains the code and image definitions if you’re interested:
https://github.com/EnzymeFunctionInitiative/cytoscape-util/tree/dev-local-py4cytoscape
We are currently using a copy of a recent version of the py4cytoscape package. At some point I want to automate the download of the latest version in the repo, because the version of py4cytoscape when doing a pip install is way out of date.
It’s still a work in progress. However, we have successfully done this (with some manual exports) for this site:
https://radicalsam.org/
Go here for an example:
https://radicalsam.org/explore.php?id=cluster-1-1-1&v=3.0&as=110
Navigate through by clicking the Next button to see the various exports.
We have to generate images for some of the larger networks by hand because Cytoscape sometimes provides an image that has a layout that results in a single point. Doing this through the GUI results in better results.
We are getting ready to refresh the site, hoping to do it twice a year, anticipating > 100,000 networks/images. We are also planning on creating sites like the aforementioned one for other protein superfamilies.
We are really looking forward to 3.10, with the hope that this solves some of the issues.
Nils
From: Barry Demchak ***@***.***>
Sent: Wednesday, April 27, 2022 10:08 AM
To: cytoscape/py4cytoscape ***@***.***>
Cc: Oberg, Nils O ***@***.***>; Author ***@***.***>
Subject: Re: [cytoscape/py4cytoscape] Graphics details upon export (Issue #81)
Hi, Nils ... very much appreciate your kind words!
Would you be interested in telling us more about your Singularity image work? This sounds very broadly useful!
—
Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/cytoscape/py4cytoscape/issues/81*issuecomment-1111118588__;Iw!!DZ3fjg!611x8f5V5_EyXk-XarB1UvoWqwM6EXsCUYBKlUJsJEWh_vPVisDnBlliIR3xq4n4Pz3-lWBz4g1TQPF7f_1gMdrtzA$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AI3P2JMLXHN4AF6HQR3L6CDVHFJ4DANCNFSM5UC5ETNA__;!!DZ3fjg!611x8f5V5_EyXk-XarB1UvoWqwM6EXsCUYBKlUJsJEWh_vPVisDnBlliIR3xq4n4Pz3-lWBz4g1TQPF7f_2lRPhExQ$>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
|
Hi, Nils .... Thanks for all of the great detail! We discussed this at our weekly meeting today ... lots to tell. It sounds like one or more of our team members are in contact with either your PI or a team leader on this issue, too. Apparently, they recommended using Solidarity, and I can certainly agree. Good news there. Generally, the developer team agrees that helping you get to a successful point is very important ... certainly by 3.10 if not much sooner. It's entirely likely that nightly Cytoscape builds will get us closer and closer to good solutions. So, we'll keep you in the loop, and good feedback would be very helpful. I think the dependencies issue has also been encountered by others while trying to do a Docker version. So far, multiple groups have limited success ... e.g., Kozo Nashida in Japan, the Toronto Cytoscape group, and the San Francisco Cytoscape group. At some point, it would be good to bring heads together ... maybe soon. The 15-20% failure rate could well be because of multiple Cytoscape instances using the same CytoscapeConfiguration directory. The developers think that new features in Solidarity enable writeable file systems that are valid for the life of the Solidarity instance. That should solve this. There's a different concept in Docker ... an overlay applied to a Docker instance that accomplishes the same thing. This will be an important discussion topic. The developers are also working on a straight forward way to get rid of the initial REST API dialog box. It really messes up Docker/Solidarity. Regarding the whitespace around a network, is it possible that the network includes an extreme outlier, and that fitting the network shrinks the main network so the outlier can fit in, too? Alternately, could it be that a zoom factor has been applied, but the network hasn't be re-laid out? It seems like all of the above would address all of the concerns you indicated. Is that so? Good to get feedback on this. FYI, I'll be releasing an updated py4cytoscape onto PyPI in the next week or two ... but after that, py4cytoscape development will track 3.10 changes so you can take advantage of them. For example, we discovered a better way to accommodate the fast Export you're trying for. We'll make it available as soon as the 3.10 nightly build supports it. Comments? |
|
Sorry for the delay in replying. We are very grateful for our collaboration and your assistance. Thank you! I haven’t tried Solidarity yet. It’s on my list. I am happy to participate in discussions that you feel I would be able to contribute to. In order to avoid the failures, I have made a complicated workaround that does something like this:
This is all managed by a series of scripts that are executed automatically by cron. The whitespace is caused by the way the clusters are laid out, and since I have a fixed image size the images sometimes have a lot of space around them. I wrote a script that crops it. May not be the best way, but it gets the job done. I have tested my code using the Cytoscape 3.10 06-29 snapshot and the latest py4cytoscape, and I am still having image generation issues. Here is some information that may be relevant:
I am wondering if some of these issues are related directly to Cytoscape as opposed to py4cytoscape, so I am going to forward this information to S. M. as well. In case you are interested, here is the py4cytoscape logging code:
|
|
Hi, Nils -- Great to hear from you! I looked into the export zoom not working ... I can duplicate that here. FYI, when I try to duplicate a py4cytoscape problem, I go to the Cytoscape Commands dialog and issue a command that py4cytoscape would generate. If Cytoscape chokes, we know the problem is in Cytoscape or in how I understand the command. For this zoom problem, I can see Cytoscape choking on the zoom parameter, so I have put the question to the Cytoscape developers. As this thread is getting long, best to break into an Issue for each of your problems. That way, they can be tracked separately and all handled. For each of your issues, would you mind creating a reproduce case and a separate Github Issue? Thanks. |
|
Hi, Nils -- More on the Export issue ... our group has put quite a bit of energy into understanding and solving this issue. We have some definite thoughts on this, and are ready to proceed. Is there a particular test case you can give us? ... we'd like to see something that works in v3.8.2 and not in v10. Thanks. |
|
I'm working on submitting separate issues with test cases, etc. I am not able to reliably replicate the most problematic one (the image is rendered zoomed in to the middle of the network). This may somehow be related to running many instances at the same time, but I feel like that workflow is fairly foolproof. I am submitting that process and code example for how I am running multiple instances on the same system. However, this problem never occurs with 3.8. An example of what is happening is attached. The first image is a properly-generated image. The second image is the zoom/crop problem image which I am still working on duplicating.
The workflow for making multiple instances of Cytoscape run simultaneously is as follows:
Modify $TEMP_DIR/cytoscape.sh by replacing the Replace Create a script that does the following: This is then submitted to a cluster, with a unique port. Additional scripts using the CyREST API are then submitted to the cluster with a matching port and node, to generate PNG from given networks. This process runs many instances (up to 50) simultaneously, mostly working. But sometimes the zoom/crop issue appears with 3.10 . I can use the exact same code for 3.8 and I never have the zoom/crop problem. |
|
Hi Barry,
I just realized I didn’t respond to this email. My apologies.
We are very grateful for our collaboration and your assistance. Thank you!
I haven’t tried Solidarity yet. It’s on my list.
I am happy to participate in discussions that you feel I would be able to contribute to.
In order to avoid the failures, I have made a complicated workaround that does something like this:
1. Create a new Cytoscape “install” by symlinking some directories and copying some other files into a unique directory.
2. Copy a vanilla CytoscapeConfiguration folder to the same directory. The vanilla folder has the proper cyrest.version set in cytoscape3.props so that the dialog isn’t required.
3. Modify the cytoscape.sh file in the new “install” changing the cytoscape.home and user.home parameters Java parameters in KARAF_OPTS to use the “install” directory
This is all managed by a series of scripts that are executed automatically by cron.
The whitespace is caused by the way the clusters are laid out, and since I have a fixed image size the images sometimes have a lot of space around them. I wrote a script that crops it. May not be the best way, but it gets the job done.
A couple of things:
We are using the 3.10 snapshots to make images and are running into a few issues:
1. The export_image function in the py4cytoscape code has a zoom parameter. It doesn’t seem to be acknowledged by 3.10 using the py4cytoscape version that I downloaded in April. Images for large networks are generated correctly, but smaller networks are generated with the specified/requested image size, but the image is zoomed in by a large magnitude and the image is cropped to a box in the center.
* Cytoscape 3.8, doesn’t seem to respect that parameter either. There is no apparent effect when I change it. However, the images are fit correctly.
* When I switched to the latest version of py4cytocsape there is no layout, just a single dot. The API calls are given below.
2. We include a node attribute in our xgmml files for coloring the node (node.fillColor). Images exported for large networks in both 3.8 and 3.10 are generated without coloring, like Cytoscape isn’t respecting that node attribute, but networks that are smaller are colored. (Cytoscape problem vs py4cytoscape?)
3. For large clusters, the Prefuse Force-Directed Layout doesn’t actually perform a layout, all that is generated is a single node (or possibly all nodes stacked on top of each other). If the OpenCL version is used instead, the cluster layout is performed correctly. This happens in both versions of Cytoscape. (Cytoscape problem vs py4cytoscape?)
Could I ask if you have any advice for us on how to solve this problem?
Thanks in advance,
Nils
API call sequence:
cytoscape_ping (until alive)
import_network_from_file
get_network_views
create_view (if not created)
layout_network(layout_name="force-directed")
fit_content
toggle_graphics_details
export_image(filename, PNG, pixels, height, width, zoom)
command_quit
From: Barry Demchak ***@***.***>
Sent: Thursday, April 28, 2022 4:58 PM
To: cytoscape/py4cytoscape ***@***.***>
Cc: Oberg, Nils O ***@***.***>; Author ***@***.***>
Subject: Re: [cytoscape/py4cytoscape] Graphics details upon export (Issue #81)
Hi, Nils ....
Thanks for all of the great detail!
We discussed this at our weekly meeting today ... lots to tell.
It sounds like one or more of our team members are in contact with either your PI or a team leader on this issue, too. Apparently, they recommended using Solidarity, and I can certainly agree. Good news there.
Generally, the developer team agrees that helping you get to a successful point is very important ... certainly by 3.10 if not much sooner. It's entirely likely that nightly Cytoscape builds will get us closer and closer to good solutions. So, we'll keep you in the loop, and good feedback would be very helpful.
I think the dependencies issue has also been encountered by others while trying to do a Docker version. So far, multiple groups have limited success ... e.g., Kozo Nashida in Japan, the Toronto Cytoscape group, and the San Francisco Cytoscape group. At some point, it would be good to bring heads together ... maybe soon.
The 15-20% failure rate could well be because of multiple Cytoscape instances using the same CytoscapeConfiguration directory. The developers think that new features in Solidarity enable writeable file systems that are valid for the life of the Solidarity instance. That should solve this. There's a different concept in Docker ... an overlay applied to a Docker instance that accomplishes the same thing. This will be an important discussion topic.
The developers are also working on a straight forward way to get rid of the initial REST API dialog box. It really messes up Docker/Solidarity.
Regarding the whitespace around a network, is it possible that the network includes an extreme outlier, and that fitting the network shrinks the main network so the outlier can fit in, too? Alternately, could it be that a zoom factor has been applied, but the network hasn't be re-laid out?
It seems like all of the above would address all of the concerns you indicated. Is that so? Good to get feedback on this.
FYI, I'll be releasing an updated py4cytoscape onto PyPI in the next week or two ... but after that, py4cytoscape development will track 3.10 changes so you can take advantage of them. For example, we discovered a better way to accommodate the fast Export you're trying for. We'll make it available as soon as the 3.10 nightly build supports it.
Comments?
—
Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/cytoscape/py4cytoscape/issues/81*issuecomment-1112691260__;Iw!!DZ3fjg!-hvBOrsxQAs6BNxh-xkE9dj18PvOBHVUBRhH5Q6NcrNg-iFGNrPobRzA0L6mOd5LYoufSczsSZJYG7C8MsGD6KcNvw$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AI3P2JMVKI7DGQPLFW4QQLTVHMCXNANCNFSM5UC5ETNA__;!!DZ3fjg!-hvBOrsxQAs6BNxh-xkE9dj18PvOBHVUBRhH5Q6NcrNg-iFGNrPobRzA0L6mOd5LYoufSczsSZJYG7C8MsGcI2cmcA$>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
|
Hi, Nils -- Great to hear from you! ... let's handle these one by one.
Any chance of jumping on this soon?? The weekly triage meeting is Thursday, and getting the group's reaction might be informative. Also, can you supply an image that needs cropping? Sometimes, the white space is occupied by a single far-outlier node. Also, if you're doing a Note, too, that the 1.6 Thanks! |
The latest Cytoscape GUI 3.10.0 snapshots include an option for turning off graphics details upon image export. The latest py4cytoscape code includes a toggle_graphics_details function. When I call this before exporting an image using the lastest py4cytoscape code in combination with Cytoscape 3.10.0 snapshot, there isn't any difference. Could I ask if there is a way to export images without applying styling? The image export runs MUCH faster on the snapshot without graphics details during export.
The text was updated successfully, but these errors were encountered: