Release checklist for v7.1

[dondi]

• In the days leading up to the release, all open issues that need to be resolved before the release will be given the label "for next release".
• Any new or changed client-side features need to be documented on the Client Side Testing Document page.
  • the feature JSON files file should be updated
  • the new feature also needs to be added to the Pull Request Checklist
• Test the candidate build (typically the beta branch)
• Merge the branch(es) under development (typically beta) into master
  • Resolve any version control conflicts
  • Test the final merged version
  • Perform hotfixes on master as needed
• Update README.md at top-level directory of master branch, if necessary
  • Update Travis and Coveralls badges to reference the master branch
• Tag the final, tested version as v#.#.# (all three numbers are required, even if 0; see npm step below)
  • Update version numbers in package.json
  • Remember that the tag needs to be pushed to GitHub: git push origin <tag-name>
• Create draft release item on Github repository (name it exactly like the tag)
  • Make release notes: a list of new features/bug fixes with issue numbers covered by this release. Note that this at least will be the ones labeled "for next release", with possible others.
• Publish GitHub release item
• Publish release to GRNsight home page
  • Refresh (pull) the master clone on grnsight.lmucs.org
  • Update the version number and modification date on gh-pages
  • Update the About page with release notes (GitHub release item without issue numbers)
    • Update the browser version compatibility statement with the latest version of Chrome and Firefox for which the new GRNsight version has been tested.
    • Update modification date
  • Add news item about the release to the News page
    • Update modification date
    • Link news item to the corresponding release notes in the About page
  • Update Documentation page with any new features or updates to old features
• Increment beta version
  • Merge released master with beta, especially if hotfixes/merges were made on master
  • Update version number and modified date on gh-pages
  • Update Travis and Coveralls badges to reference the beta branch
  • Refresh (pull) the beta clone on grnsight.cs.lmu.edu
• Publish to npm (npm publish)
  • npm requires all .0s in the semantic version
• Update version number/other relevant information at current bioinformatics listing sites
  • Elixir
    • Then change hyperlink on the GRNsight Links page, if necessary
• Get DOI for new release from Zenodo
  • Verify that the badge in README.md at top level of repo is correct
    (it might be necessary to invoke curl -X PURGE https://camo.githubusercontent.com/camo-GitHub-id in order to update the badge image)
  • Verify that CITATION.md file DOI links to the new release, updating if necessary
  • Verify that "How to cite GRNsight" section on About page links to the new release, updating if necessary

[dondi]

This will be for current beta 7.0.8 once we have fully validated it

[kdahlquist]

Since the idea was that we want to get the changes currently in beta to production, there aren't really any issues to mark "for next release" and we can go on to the next step.

[kdahlquist]

@ntran18 will review issues that were resolved in beta and make a draft of the release notes and post it on this comment.

• Make release notes: a list of new features/bug fixes with issue numbers covered by this release. Note that this at least will be the ones labeled "for next release", with possible others.

[ntran18]

[v7.1.0]

GRNsight v7.1.0 includes the following new features and bug fixes:

• Features:
  • Lock Force Parameters are available for both Force Graph and Grid Layout. (Bug: set layout to grid layout and click on any demo, layout goes back to force graph #1016)
  • Adding Protein-protein interaction (PPI) demo (Add PPI demo #1065)
• Bug Fixes:
  • Remove the checks from Force Graph and Grid Layout options from the menu. (Bug: set layout to grid layout and click on any demo, layout goes back to force graph #1016)
  • The current state of the Force Graph/ Grid Layout is reflected consistently between the menu and the side panel. (Bug: set layout to grid layout and click on any demo, layout goes back to force graph #1016)
  • Selected dataset is centralized to an internal state value, allowing the menu and the side panel to choose and display the same dataset. (Can only change dataset from bottom option, not top #1033)
  • Fixed the button "Restrict Graph to Viewport" from the menu to work correctly (Restrict Graph to Viewport Bug #1034)
  • Fixed the modals not to close when clicking or dragging the modal window with the mouse (Modal closes unexpectedly when click and drag the modal window with the mouse #1029)

[ntran18]

I've shared the details of my release note's draft above. I also make a draft.

[dondi]

Upon reviewing current status, we refreshed our memory to note that:

• We had merged into master but didn't update the server host
• Version number will be 7.1.0 for production and beta will shift to 7.1.1
• We will restart the release procedure since it’s been a while

[ceciliazaragoza]

Here are my additions for the release notes draft:

• Features:
  • D-pad and Zoom are enabled when Restrict Graph to Viewport (Checking restrict graph to viewport makes the D-pad and zoom slider on the viewport disappear #1083)
• Bug Fixes:
  • Fixed Node Coloring Top Menu Options from disappearing when uncheck "node coloring" box on side panel (node coloring top menu availability of options #1081)

[ntran18]

Updated the wiki for the feature JSON files - Pull Request for PPI demo 5.

Cecilia's feature is already in the features list, wondering whether this is a bug fix or a new feature. Refer to Client Side Testing Document with id v4. @ceciliazaragoza

[dondi]

Per our 9/4 meeting, the presence of the PPI network data is a key part of our release. #1106 has a link to the Box folder containing existing data and can be used for comparison

For strategic reasons, let’s go about it in this way:

• @ntran18, @ceciliazaragoza, and @akaiap will run the scripts on their respective machines individually in order to test/validate the correctness of the data
• Loading the scripts in this way will also ensure that everyone on the team is familiar with the process, and sets everyone up for a potential future project to automate the data update
• Once loading is successful on local machines, we can have a session with @dondi to perform the load on the server
• When the server database is updated, we can test beta
• Because the 2024-03-19 load is incomplete, upon successful loading of 2024-09-xx, we will delete 2024-03-19
• For consistency, we will also delete the 2024-03-19 GRN data since the 2024-09-xx version will supersede it anyway

Each individual should post a comment here as an update, indicating if any issues were encountered

[ntran18]

Unfortunately, @ceciliazaragoza and @akaiap won't be able to run the database scripts because the link is down. https://www.yeastgenome.org/blog/yeastmine-shutting-down-july-15. The database is migrated to AllianceMine; however, this website doesn't have a service to download the database.

AllianceMine

• Need to do more research about this website, such as how to build a query to get data.
  • Tried to query gene from AllianceMine using this query

• Have an example code to download data, but the service link provided is not working.

Investigations

Genes

All analysis about genes is done in here

Look at both GRN genes and PPI genes and genes downloaded from AllianceMine.

	GRN	PPI	Alliance
# genes	6514	6713	7336
#genes without display names	6514	5791	5569

	with Alliance data	without Alliance data
# distinct genes	7336	7016
# distinct genes w/o empty display name	6854	6683

The following table shows missing genes between each dataset, comparing GRN, PPI, and Alliance.

	GRN	PPI	Alliance
GRN	0	303	0
PPI	502	0	0
Alliance	822	623	0

Ex: 303 genes exist in the GRN gene table, but not in PPI.

More numbers ...

• 17 gene_display_id in GRN are different from PPI
• 590 gene_display_id in PPI are different from GRN (all of those gene_display_id in PPI are None or empty, whereas GRN has different naming).
• Within 520 addition genes in Alliance compare to PPI and GRN, there are only 171 genes that have display name.

Database Code

I found some errors with the code. I created the union gene table, and the union missing/update gene files. This can only be apply if both PPI and GRN gene table are exactly similar to each other. So I need to update the script to make both PPI and GRN table to have the same value first before I can use the current code.

Proposed Solutions:

Solution 1

In March, I downloaded data to my machine; they are still there, so we can reuse it.
Pros:

• Don't need to download data again

Cons:

• Don't know if there is a problem with the data
• No fresh data
• Need to update current loader_update code.

Solution 2 [Recommended]

Download data directly from AllianceMine
Pros::

• Have data up-to-date
• Have more genes

Cons:

• Clean data
• Check if the data aligns with old data

I think we should go with data from AllianceMine, since all Yeastmine data is transferred to AllienceMine, and we can have data with the newest source. But we also can check whether new genes should be added. See this link for genes tables

[dondi]

After discussing the YeastMine/AllianceMine transition, we will proceed as follows:

• Original Solution 2 will be our long-term approach regardless
• We will go with this revised version of Solution 1 in order to get this imminent production release of GRNsight out:
  • @ntran18 will share the download files w/ @ceciliazaragoza and @akaiap
  • The existing script will be modified to skip the portions that contact YeastMine, and the code can go straight to the point where the files are being read (if commits are desired, they can be placed on a branch)
  • We then proceed as previously planned, with the local databases being looked at and tested (to recap: we want to have complete PPI data for March 2024)
  • Once data validation succeeds on local machines, we can plan a session with @dondi where we update the production database then proceed with software release
• We will start an issue for Solution 2 and that will become one of our fall goals for GRNsight post-release

[ntran18]

Downloaded files are in GRN-Release-7.1 Database Backups

Instructions

1. Switch to ntran-1077 branch

git fetch origin
git switch ntran-1077

2. Download files from Box

Please place the downloaded files into the correct folders as shown in Box.

___ database
      |____ network-database
               |____ script-results
                         |____ processed_loader_files
                                  |____ gene.csv
      |____ protein-protein-database
               |____ script-results
                        |____ processed-loader-files
                                  |____ gene.csv
                                  |____ network.csv 
                                  |____ ....

3. Filtering missing and updated data

After getting all the downloaded files, we need to get missing/ updated genes for both GRN and PPI, and missing/updated proteins for PPI. All the code is writing in filter_update.py. The file need the address to database to access to database and collect all the current genes and proteins that the databases have for comparison. After running this script, you will see missing-genes.csv, update-genes.csv in processed-loader-files for both network-database and protein-protein-database, also missing-protein.csv and update-protein.csv.

For Macbook, please run:

DB_URL="postgresql://[<db_user>:<password>]@<address to database>/<database name>" python3 filter_update.py

Ex:

DB_URL="postgresql://postgres@localhost/postgres" python3 filter_update.py

4. Loading data into the database

The script below will update the database for either GRN or PPI, depending on what we provide in the command.

python3 loader_update.py --network=[GRN/ PPI] | psql postgresql://localhost/postgres

For this time, please try --network=PPI since we want to update the PPI database.

python3 loader_update.py --network=PPI | psql postgresql://localhost/postgres

Also, we want to fix #1106 issue, so we need to update gene table in GRN. In loader_update.py, comment the code in load_grn_data_into_database() like this:

This will make sure that we only update or adding new genes into the genes table, without touching source table or network table.

Now, running this script to update GRN's gene table

python3 loader_update.py --network=GRN | psql postgresql://localhost/postgres

[ntran18]

On Friday, I worked with @ceciliazaragoza and @akaiap to update the database and test the code. Both @ceciliazaragoza and @akaiap can update the GRN gene table and PPI tables.

I will plan a session with @dondi to update production database.

[dondi]

@ntran18 @ceciliazaragoza and @dondi had a Zoom session where we ran the database scripts on the AWS instance. We caught some final bugs in the scripts and @ntran18 committed those fixes to beta

So the current status is that the beta deployment on GRNsight should now have updated PPI network information for March 2024. The beta branch needs to be merged into master but there are some conflicts which could not be resolved on the spot

@dondi will look at these and aim to finish the merge + deploy

[dondi]

Merge has been done (reconciling the conflicting changes in database/README.md took some doing 😅) and 7.1 has now been deployed. We’ll need a re-review of the database README to ensure that I resolved conflicts correctly, plus live testing of production. We can then proceed with finishing up the release

[dondi]

• items relating README.md were checked off but there are potential more extensive changes to do with them (e.g., Migrate CI unit-testing and coverage reports to GitHub Actions #1118, etc.)
• Tag for 7.1.0 already exists but that is now obsolete (from January); we will re-tag after a round of testing in production
• The release notes are current but are linked to the aforementioned obsolete tag; to be safe, I am copying the contents here. We can revisit the final version of the GitHub Release after the final 7.1.0 tag has been made
  • Release notes for 7.0.0 (which was never released) should be integrated into 7.1.0: v7.0.0 Protein-Protein Physical Interactions #1039
• Turned out that @dondi already did the publishing of the software and the Home website; however, the About and News pages need to be updated
• Full documentation update may need to be on-going; checked off for now but will need future review
• Beta update items have not yet been done; for now, beta is identical to master. When master is finalized, we can merge into beta and perform those tasks
• Publication items to Elixir and DOI should update automatically upon publishing to npm

GRNsight v7.1.0 includes the following new features and bug fixes:

• Features:
  • Lock Force Parameters are available for both Force Graph and Grid Layout. (Bug: set layout to grid layout and click on any demo, layout goes back to force graph #1016)
  • Adding Protein-protein interaction (PPI) demo (Add PPI demo #1065)
  • Restrict to Viewport is enabled for D-pad and Zoom (Checking restrict graph to viewport makes the D-pad and zoom slider on the viewport disappear #1083)
• Bug Fixes:
  • Remove the checks from Force Graph and Grid Layout options from the menu. (Bug: set layout to grid layout and click on any demo, layout goes back to force graph #1016)
  • The current state of the Force Graph/ Grid Layout is reflected consistently between the menu and the side panel. (Bug: set layout to grid layout and click on any demo, layout goes back to force graph #1016)
  • Selected dataset is centralized to an internal state value, allowing the menu and the side panel to choose and display the same dataset. (Can only change dataset from bottom option, not top #1033)
  • Fixed the button "Restrict Graph to Viewport" from the menu to work correctly (Restrict Graph to Viewport Bug #1034)
  • Fixed the modals not to close when clicking or dragging the modal window with the mouse (Modal closes unexpectedly when click and drag the modal window with the mouse #1029)
  • Fixed Node Coloring Top Menu Options from disappearing when uncheck "node coloring" box on side panel (node coloring top menu availability of options #1081)

[ntran18]

Updated release notes for version 7.1.0

GRNsight v7.1.0 includes the following new features and bug fixes:

• Features:
  • Lock Force Parameters are available for both Force Graph and Grid Layout. (Bug: set layout to grid layout and click on any demo, layout goes back to force graph #1016)
  • Protein Protein Physical Interactions Database and UI (protein-protein interaction UI refinements #1028, Investigate Yeastmine for information on Protein - Protein Interactions Mode #1020)
  • Adding Protein-protein interaction (PPI) demo (Add PPI demo #1065)
  • Restrict to Viewport is enabled for D-pad and Zoom (Checking restrict graph to viewport makes the D-pad and zoom slider on the viewport disappear #1083)
• Enhancements:
  • Change order of Yeastmine versions in generate GRN modal (Change order of Yeastmine versions in generate GRN modal #1035)
  • Filename and display nomenclature for YeastMine-generated Networks (Filename and display nomenclature for YeastMine-generated Networks #983)
• Bug Fixes:
  • Remove the checks from Force Graph and Grid Layout options from the menu. (Bug: set layout to grid layout and click on any demo, layout goes back to force graph #1016)
  • The current state of the Force Graph/ Grid Layout is reflected consistently between the menu and the side panel. (Bug: set layout to grid layout and click on any demo, layout goes back to force graph #1016)
  • Selected dataset is centralized to an internal state value, allowing the menu and the side panel to choose and display the same dataset. (Can only change dataset from bottom option, not top #1033)
  • Fixed the button "Restrict Graph to Viewport" from the menu to work correctly (Restrict Graph to Viewport Bug #1034)
  • Fixed the modals not to close when clicking or dragging the modal window with the mouse (Modal closes unexpectedly when click and drag the modal window with the mouse #1029)
  • Fixed Node Coloring Top Menu Options from disappearing when uncheck "node coloring" box on side panel (node coloring top menu availability of options #1081)
• Documentation:
  • Database setup in Initial Setup wiki (Database setup in Initial Setup wiki #1011)
  • Database update instruction (#1106 Writing Scripts for Updating Database and Refactor Code #1113)

[ntran18]

• Verified database README.md is correct.
• Verified the database in the production
  • One of the new protein is Ank1p, and it has connection with Bas1p. If you try in 2023 database, there would be no connection between these 2 proteins, but if you try the 2024 database, there will be a connection.
• Investigated more about ERT1 issue and documented in ERT1 not in network database #1106.

[ceciliazaragoza]

• Tested restrict graph to viewport feature on master and created a PR Restrict Graph to Viewport Hotfix: Navbar  #1119 for a small hotfix to refresh the graph when restrict graph to viewport feature clicked in navbar.
• Other than that, everything looks good!

[akaiap]

• Conducted client-side testing on the Beta version.
• Reviewed and investigated previous issue When GRN loaded from file with no expression data, nodes are colored gray instead of white #1074 to determine the next steps.

[dondi]

All software/deployment tasks completed as of 10/2. What remains are:

• documentation/website-related items pertaining to new version
• revisit Zenodo to see if published npm version eventually propagates (and try to fix manually if not)

[dondi]

I checked on Zenodo and it turned out there was an error when it tried to sync up v7.1.1 last week. The error has been fixed but I have not figured out how to get it to retry the sync

As a workaround for now, I published the v7.1.0 release that had been left in draft form. That has been picked up and what is visible in Zenodo

[dondi]

I’m not sure if this is visible without logging in to Zenodo, but this is what its log says:

It can be reached via the My Account > GitHub menu item if one is logged in and configured with our GRNsight repository

[dondi]

Zenodo issue was resolved with a version number bump just for Zenodo; Maika's PR was also merged