Fiji versioning

[grahampugh]

Hi Rich,

I've been taking a look at the Fiji.download recipe. The version that's reported is problematic in that the version you get from https://wsr.imagej.net/notes.html refers to ImageJ, not Fiji specifically. We can get Fiji versions from the tags at https://github.com/fiji/fiji/tags, but these go newer than the releases.

The release at https://downloads.imagej.net/fiji/latest/ is not versioned, but it also appears not to be the latest at all. The latest is available, with version string, from https://downloads.imagej.net/fiji/releases/

However, on further investigation, it seems that pushing the newest version in "releases" is risky.

Finally, I looked at what Homebrew are doing, and they've come up with the idea of grabbing the datestamp of the latest version. They note that a new folder is added to the "archive" folder shortly after the latest release is created, into which that release is copied. This happens around 30 minutes after the date stamp of the latest release, so is a good indicator, and easier to scrape than that actual date stamp.

I've built a recipe based on this for our own purposes but feel free to adapt your "official" one with the logic below.

Input:
  NAME: Fiji

Process:
  - Processor: URLTextSearcher
    Arguments:
      re_pattern: (\d{8}-\d{4})
      url: https://downloads.imagej.net/fiji/archive/?C=N;O=D
      result_output_var_name: version

  - Processor: URLDownloader
    Arguments:
      filename: "%NAME%-%version%.zip"
      url: "https://downloads.imagej.net/fiji/latest/fiji-macosx.zip"

  - Processor: EndOfCheckPhase

[rtrouton]

Thanks! This looks like it works, so I've now rolled this into the Fiji download recipe.

[grahampugh]

Hi Rich, you might need to read my edit. I originally sent the solution that used the releases URL, But then found out that the newest version in "releases" (2.11.0) is in fact newer than the "latest" release (2.9.0), and there is a comment in their GitHub issues that 2.11.0 has problems.

So, I edited my original issue with a new solution based on the release date of the "latest" version. I guess you would not have seen that if you just read the email notification - sorry for the confusion!

[rtrouton]

It appears that using the timestamp is giving pkgbuild some problems:

PkgCreator
{'Input': {'pkg_request': {'chown': [{'group': 'admin',
                                      'path': 'Applications',
                                      'user': 'root'}],
                           'id': 'org.fiji',
                           'options': 'purge_ds_store',
                           'pkgname': 'Fiji-20221201-1017',
                           'scripts': 'Scripts',
                           'version': '20221201-1017'}}}
PkgCreator: Connecting
PkgCreator: Sending packaging request
PkgCreator: Disconnecting
PkgCreator: Failed to close socket: [Errno 9] Bad file descriptor
Traceback (most recent call last):
  File "/Library/AutoPkg/autopkglib/__init__.py", line 840, in process
    self.env = processor.process()
  File "/Library/AutoPkg/autopkglib/__init__.py", line 626, in process
    self.main()
  File "/Library/AutoPkg/autopkglib/PkgCreator.py", line 274, in main
    self.package()
  File "/Library/AutoPkg/autopkglib/PkgCreator.py", line 223, in package
    pkg_path = self.send_request(request)
  File "/Library/AutoPkg/autopkglib/PkgCreator.py", line 263, in send_request
    raise ProcessorError(", ".join([s.replace("ERROR:", "") for s in errors]))
autopkglib.ProcessorError: pkgbuild failed with exit code 1: pkgbuild: error: Cannot write package to "/Users/username/Library/AutoPkg/Cache/com.github.rtrouton.pkg.Fiji/autopkgtmp-62275c7863395c78-Fiji-20221201-1017.pkg". (The operation couldn’t be completed. Operation not permitted)
  File "/Library/AutoPkg/autopkglib/__init__.py", line 840, in process
    self.env = processor.process()
pkgbuild failed with exit code 1: pkgbuild: error: Cannot write package to "/Users/username/Library/AutoPkg/Cache/com.github.rtrouton.pkg.Fiji/autopkgtmp-62275c7863395c78-Fiji-20221201-1017.pkg". (The operation couldn’t be completed. Operation not permitted)
Failed.
Receipt written to /Users/username/Library/AutoPkg/Cache/com.github.rtrouton.pkg.Fiji/receipts/com.github.rtrouton.pkg-receipt-20230627-091141.plist

The following recipes failed:
    com.github.rtrouton.pkg.Fiji
        Error in com.github.rtrouton.pkg.Fiji: Processor: PkgCreator: Error: pkgbuild failed with exit code 1: pkgbuild: error: Cannot write package to "/Users/username/Library/AutoPkg/Cache/com.github.rtrouton.pkg.Fiji/autopkgtmp-62275c7863395c78-Fiji-20221201-1017.pkg". (The operation couldn’t be completed. Operation not permitted)

Full verbose run is attached.
Terminal Saved Output.txt

[rtrouton]

I'm guessing it's the dash in the version information ('version': '20221201-1017')

[grahampugh]

Strangely, I do not get that error. I do get a "Bad descriptor" error, but it doesn't prevent the package from being built.

PkgCreator
{'Input': {'pkg_request': {'chown': [{'group': 'admin',
                                      'path': 'Applications',
                                      'user': 'root'}],
                           'id': 'org.fiji',
                           'options': 'purge_ds_store',
                           'pkgname': 'Fiji-20221201-1017',
                           'version': '20221201-1017'}}}
PkgCreator: Connecting
PkgCreator: Sending packaging request
PkgCreator: Disconnecting
PkgCreator: Failed to close socket: [Errno 9] Bad file descriptor
{'Output': {'new_package_request': True,
            'pkg_creator_summary_result': {'data': {'identifier': 'org.fiji',
                                                    'pkg_path': '/Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.Fiji/Fiji-20221201-1017.pkg',
                                                    'version': '20221201-1017'},
                                           'report_fields': ['identifier',
                                                             'version',
                                                             'pkg_path'],
                                           'summary_text': 'The following '
                                                           'packages were '
                                                           'built:'},
            'pkg_path': '/Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.Fiji/Fiji-20221201-1017.pkg'}}

Same result with AutoPkg versions 2.4.1 and 2.7.

[rtrouton]

I'm on AutoPkg 2.7.2. I can't tell from the output above if that's the same version you were running.

[grahampugh]

Probably not, as I use a custom-build from the dev branch in order to utilise recipe-maps.

Nonetheless, I can't see any changes to PkgCreator in any branches for 2 years. It's dumb that a dash breaks the build, but that is the current state of affairs, at least on your computer, so it's probably worth removing the -.

Given the unlikelihood of Fiji getting two new "latest" releases in a single day, it should work to use https://github.com/autopkg/homebysix-recipes/blob/master/VersionSplitter/VersionSplitter.py. Alternatively https://github.com/autopkg/grahampugh-recipes/blob/master/CommonProcessors/StringReplacer.py can replace one character with a different one so could be used to make 20221201.1017.

[grahampugh]

FWIW, I've used StringReplacer in my processor: https://github.com/eth-its/autopkg-mac-recipes-yaml/blob/main/PKG_Recipes/Fiji.pkg.recipe.yaml

[rtrouton]

OK. For now, I'm reverting yesterday's commit so that the Fiji .download recipe goes back to the way it was. I'll figure out the best way forward.

[rickokid]

I've submitted a pull request with a possible solution - using https://maven.scijava.org/content/groups/public/sc/fiji/fiji/maven-metadata.xml to determine the current Fiji version.