how to copy proj library data to R-package binary on CRAN

[adokter]

Hi developers of sf and terra,

For R package vol2birdR I'm trying to replicate the copying of the proj library data as implemented in sf and terra. My package needs access to the proj.db file of the proj library, which is typically stored in the share/proj of the library (path of this folder can be determined locally with pkg-config --variable=datadir proj).

I was reading through the discussions between @jeroen @rsbivand and @edzer in issues #1605 and #1933 where copying of the proj data is discussed. As I understand the copy occurs in this line:

sf/configure.ac

Line 233 in 3aee3e4

cp -r "${proj_lib}" "${R_PACKAGE_DIR}"

However for this to work both the PROJ_GDAL_DATA_COPY and PROJ_LIB environment variables need to be set during build time on CRAN, see:

sf/configure.ac

Lines 209 to 240 in 3aee3e4

	# Optional local copy of GDAL datadir and PROJ_LIB
	data_copy=no
	if test "${PROJ_GDAL_DATA_COPY}" ; then
	data_copy=yes
	AC_MSG_NOTICE([PROJ_GDAL_DATA_COPY used.])
	else
	AC_ARG_WITH([data-copy],
	AS_HELP_STRING([--with-data-copy=yes/no],[local copy of data directories in package, default no]),
	[data_copy=$withval])
	fi
	if test "${data_copy}" = "yes" ; then
	AC_MSG_NOTICE([Copy data for:])
	proj_lib0="${PROJ_LIB}"
	AC_ARG_WITH([proj-data],
	AS_HELP_STRING([--with-proj-data=DIR],[location of PROJ data directory]),
	[proj_lib1=$withval])
	if test -n "${proj_lib0}" ; then
	proj_lib="${proj_lib0}"
	else
	proj_lib="${proj_lib1}"
	fi
	if test -n "${proj_lib}" ; then
	if test -d "${proj_lib}" ; then
	cp -r "${proj_lib}" "${R_PACKAGE_DIR}"
	AC_MSG_NOTICE([ PROJ: ${proj_lib}])
	else
	AC_MSG_ERROR([PROJ data files not found; set environment variable PROJ_LIB=DIR or --with-proj-data=DIR.])
	fi
	else
	AC_MSG_ERROR([PROJ data files not found; set environment variable PROJ_LIB=DIR or --with-proj-data=DIR.])
	fi

I can find no instance in the sf and terra code base where these environment variables are set.

It's also clear that the package source for sf/terra do not contain the proj library data:
https://cran.r-project.org/src/contrib/sf_1.0-19.tar.gz
only the generated binaries do, e.g.:
https://cran.r-project.org/bin/macosx/big-sur-arm64/contrib/4.4/sf_1.0-19.tgz

If I want to mirror the solution in sf/terra, how do I make sure that the PROJ_GDAL_DATA_COPY and PROJ_LIB environment variables are set on CRAN during the generation of binaries only? Any advice steering me into the right direction would be much appreciated!

[edzer]

Hi developers of sf and terra,

@rhijmans

[jeroen]

You should not set these variables yourself; the owner of the build server sets PROJ_GDAL_DATA_COPY and PROJ_LIB for you to let your package know where it can find those files.

For the r-universe builds that you can find on https://cran.dev/vol2birdR these variables were set by the macos-prep action. But it doesn't matter how they get set, all your package needs to do is respect the values from the environment variables on the current server.

It is correct that no data files are included in the source package, as the version of the data needs to match the version of libproj that your package was linked to. Again, all these things are provided by the build server.

[adokter]

Thanks @jeroen for the quick reply! So I can safely assume PROJ_GDAL_DATA_COPY and PROJ_LIB will be set on the CRAN build servers, these will be generally available for all packages compiled on those servers, nothing I need to do here?

[jeroen]

Yes this variable is set on CRAN servers that produce redistributable standalone binary packages.

However note that MacOS users may also build your package locally from source, dynamically linking to a local libproj. In this case the variable `PROJ_GDAL_DATA_COPY is not set. In this case you should assume the local libproj is be able to locate the data files automatically and it is not needed to copy them in the R package.

[adokter]

Hi @jeroen, I hope I can pick your brain for one more follow up question (let me know if I better ask this to CRAN maintainers or elsewhere instead):

I implemented sf's copy mechanism for the proj library data but am noticing the latest CRAN builds do not include a /proj folder in the Mac binary as expected. Is it possible my package doesn't run the macos-prep action you mentioned and therefore isn't setting thePROJ_DGAL_DATA_COPY env variable? Or is there another step I'm missing, i.e. is copying the folder to the package root by configure insufficient for it to end up in the CRAN binary? Thanks so much for your time!

[jeroen]

Note r-universe and CRAN are different build servers. I can only speak for the binaries you can download on https://adokter.r-universe.dev/vol2birdR and I think those packages include those proj files?

For the CRAN binaries, you can find the build log here, perhaps these can tell you what is happening:

• https://www.r-project.org/nosvn/R.check/r-release-macos-x86_64/vol2birdR-00install.html
• https://www.r-project.org/nosvn/R.check/r-release-macos-arm64/vol2birdR-00install.html

[adokter]

Hi @jeroen, just confirming that the proj files are included in the binaries on https://adokter.r-universe.dev/vol2birdR, but not in the binaries on CRAN

[edzer]

@adokter I had to dig in a little, but you may need to add flags needed for your package to the CRAN macosx build system to override the default flags, see this comment: #1703 (comment) where you see how this is done for a set of other CRAN packages using one or more of GEOS/GDAL/PROJ.

[rsbivand]

Using --configure-args='--with-data-copy=yes --with-proj-data='pkg-config proj --variable=datadir' should work. The environment variables are not set on the server I think, you'd need vol2birdR added to https://svn.r-project.org/R-dev-web/trunk/QA/Simon/packages/flags.inst/ as vol2birdR containing link sf like all the others.

[adokter]

I can confirm the configure arguments mentioned above do the trick, and I've reached out to CRAN and @s-u for vol2birdR to be added to flags.inst. Thank so much everyone for you input, very helpful!

[jeroen]

I also opened R-macos/recipes#69 to see if we can get rid of cran-specific overrides, and instead agree on the environment variable.

[s-u]

@adokter please note that this is really a bug in the package. The package should not rely on files outside of its installation, so it should copy the data at build time - it should not require additional arguments to do so. The additional arguments were a hack for one package way back, but instead of fixing the problem it seems others have copied the same hack. It was never intended to be used by more packages.

This is a good opportunity to finally fix that problem. Any packages that uses proj already knows at build time where the data lives since it has to ask pkg-config anyway, so it should simply copy the data at that point. The other alternative would be to have one package provide the data to all others. Perhaps PROJ would be an obvious candidate? I can reach out to all the maintainers to deprecate the status quo and propose a solution (either make copying the default or use a common source).

[jeroen]

That is IMHO a problem of the packages - why don't they make it the default?

I did not design this, but IIUC the reasoning is that the default behavior assumes the package is built for local use and dynamically linked against a local gdal/proj which can automatically find these data files locally on the system, so we don't need to copy them.

Only in the case where you want to create a binary package which can be redistributed by itself, and you are on a platform that can do this (i.e macos) we need to copy all these extra data files into the package directory. But this is a choice of the builder, not the package author, so that is why it is opt-in using --with-data-copy=yes or PROJ_GDAL_DATA_COPY=true.

Always copying all data files by default on all platforms (especially on Linux), seems like a bad idea because they can be huge, and even on MacOS it is really only needed if you are trying to build something redistributable, which is only the case for CRAN.

[edzer]

The data files in the latest proj are 763M.

I think you're looking at the datum grids; CRAN doesn't distribute those, access from the spatial packages to them is typically through https://cdn.proj.org/ - but you can install them locally and use them from there.

[jeroen]

That is my point exactly, Simon proposes you just unconditionally copy all files from pkg-config --variable=datadir proj into your package. This directory also includes those datum grid files on many platforms. So you end up copying all of that.

[s-u]

@jeroen I was not proposing that at all. The discussion started because right now each package re-using sf configure scripts requires CRAN to add

--configure-args='--with-data-copy=yes --with-proj-data='`pkg-config --variable=prefix proj`/share/proj

manually to the configure overrides. So it's already there, for years, I didn't come up with that. That is a terrible hack - both because it requires manual query of pkg-config (beats me wh the configure can't find it itself) and has to instruct the package to copy the data.

All I'm saying that this should not be necessary, so the packages should sort it out - that's my proposal. I don't really care what the packages deem is the best solution as long as it doesn't involve manual intervention.

As I said, the content copied is tiny, so it doesn't really matter, so I don't see why the above cannot be simply the default. If there is worry about the database not being in sync, then you can check for the system one first and fall back to the copy only if it's clear that there is none. I don't see what's problematic about any of that - it's all very simple and solves the problem of not having to manually tweak each R package that uses proj.

The suggestion about using PROJ was just trying to be helpful in that it seem odd that all those packages link the same library so sharing it and the data would strike me as something that may be worth contemplating, it's not required.

[adokter]

@s-u agree the --with-proj-data argument can easily be removed and resolved in the script. The --with-data-copy argument is essentially used as a flag that tells the configure script whether it is configuring to build a Mac binary or not. Is there a way for the configure script to know when it is making a binary for Macs, are there any other environment variables present on the CRAN build server from which it could grab that information? If that's the case a fix would be easy.

[adokter]

Alternatively, if configure has no way of telling when it's configuring for a mac binary build, we could copy the data folder by default as @s-u suggests, but then the very large datum grids need to be excluded. Would copying only part of the data folder be a solution, e.g. these files:

ls `pkg-config --variable=datadir proj` | grep -v tif$ | grep -v json$ | grep -v txt$ | grep -v pol$`

so always excluding extensions .tif, .json, .txt, .pol. Would that be ok or could that break something? I mostly don't like here that you can't control new large file additions by the proj library that may slip through the filter. Could we copy only proj.db and proj.ini?

[edzer]

@s-u I don't think that relying on a single package to provide proj.db will work, as the proj library checks that proj.db is in sync with it and raises an error when it isn't. Dynamic linking guarantees this, but non-synced statically built CRAN releases doesn't.

[edzer]

Another solution that may come soon is that proj.db is inlined in the code in its entirety; IIRC this is possible (and requires) C23.

[s-u]

Sure, fair enough - like I said, that was just an idea to share it, but if an individual copy is required, that's fine with me.

[rsbivand]

The copying of share/proj and share/gdal into packages at installation is and has always been applicable only to installable binaries where we know with some certainty that those resources cannot be located by R contributed packages at install time. This applies to Windows and macOS CRAN binaries, and probably to off-CRAN binaries. @edzer has explained about the run-time version check applied to proj.db by the proj library.

When the mechanism was introduced, there were only text files, needed for PROJ (and GDAL) to work. From GDAL 3 and PROJ 6, proj.db - an SQLite database - was introduced to avoid transformation from a source CRS to geographical coordinates in (some version of) WGS84, then transformation onward to the target CRS, with two sets of unmeasured transformation errors. This was known as using a transformation hub. proj.db contains extensive lists of transformations directly from source to target CRS, ordered by their accuracy, and including the names of transformation grids they use.

From PROJ 7, https://cdn.proj.org/ was launched to provide content download on demand for high-precision grids used for specified parts of thw world, or even of countries or parts of countries, to be cached locally. Typically a local appdata directory (for me .local/share/proj) contains an SQLite DB called cache.db with chunks of transformation grids. Before being used, the chunks are checked for freshness, and downloaded again if stale (my cache.db is about 10MB).

Consequently, bundlers adding all the grids to PROJ package dependencies (OSGeo4W is one such) only makes sense if you know that all users are often offline when transforming data, and that they certainly need grids covering the whole world. The old proj-datumgrid bundle that used to be all there was, and was OK to copy before PROJ 7, has a zipped size of 6MB. The new proj-datum bundle is 745MB compressed, uncompressed 790MB. Bundlers have typically mistaken the old setup for the new, and ignored the CDN, which is efficient and parsimonious. Both sf and terra can acces the CDN, other packages should do the same, and only ensure that software has, at run time, access to the proj.db created at the same time as the PROJ library (together with the other files in share/proj, such as proj.ini.

It might even make sense to hotpatch proj.ini:

[general]
; Lines starting by ; are commented lines.
;

; Network capabilities disabled by default.
; Can be overridden with the PROJ_NETWORK=ON environment variable.
; Cf https://proj.org/en/latest/usage/network.html
; Valid values = on, off
network = off

; Endpoint of the Content Delivery Network where remote resources might
; be accessed. Only used if network access is allowed (cf above "network"
; option)
; Can be overridden with the PROJ_NETWORK_ENDPOINT environment variable.
cdn_endpoint = https://cdn.proj.org

; Whether to enable a cache of remote resources that are accessed, on the
; local file system
; Valid values = on, off
cache_enabled = on

; Size of the cache in megabytes
cache_size_MB = 300

; Time-to-live delay in seconds before already accessed remote resources are
; acessed again to check if they have been updated.
cache_ttl_sec = 86400

; Can be set to on so that by default the lack of a known resource files needed
; for the best transformation PROJ would normally use causes an error, or off
; to accept missing resource files without errors or warnings.
; This default value itself is overriden by the PROJ_ONLY_BEST_DEFAULT environment
; variable if set, and then by the ONLY_BEST setting that can be
; passed to the proj_create_crs_to_crs() method, or with the --only-best
; option of the cs2cs program.
; (added in PROJ 9.2)
; Valid values = on, off
only_best_default = off

; Filename of the Certificate Authority (CA) bundle.
; Can be overriden with the PROJ_CURL_CA_BUNDLE / CURL_CA_BUNDLE environment variable.
; (added in PROJ 9.0)
; ca_bundle_path = /path/to/cabundle.pem

; Transverse Mercator (and UTM)  default algorithm: auto, evenden_snyder or poder_engsager
; * evenden_snyder is the fastest, but less accurate far from central meridian
; * poder_engsager is slower, but more accurate far from central meridian
; * default will auto-select between the two above depending on the coordinate
;   to transform and will use evenden_snyder if the error in doing so is below
;   0.1 mm (for an ellipsoid of the size of Earth)
tmerc_default_algo = poder_engsager

to turn the CDN on by default in the first variable - the first 5 variables apply to the CDN.

For all other than CRAN static binary package builds (possibly PROJ_GDAL_DATA_COPY and PROJ_LIB set in r-universe builds hopefully only for Windows/macOS), we should rely on the dependences having been set correctly by the bundler/packager, so that the PROJ/GDAL binary libraries know where to find installed files such as proj.db.

For those who relish complications, running R with spatial packages with proj.db and other installed software also using their own proj.db, problems may ensue - see closed issues at https://github.com/OSGeo/rgrass/issues.