Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support building on Windows #293

Draft
wants to merge 38 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
3a1ced3
Add Windows CI test using MinGW
nightlark Nov 8, 2024
5e35451
Use mingw32-make
nightlark Nov 8, 2024
2a3ea9c
Add Windows mmap implementation
nightlark Nov 8, 2024
2da3bd4
Add workaround for getrlimit not being available on Windows
nightlark Nov 8, 2024
d0815c3
Adjust mman.h to more closely mirror mman-win32 code it came from
nightlark Nov 8, 2024
d24d415
Disable shell filters on Windows
nightlark Nov 8, 2024
733f64e
Fix syntax error with printing error using filters on Windows
nightlark Nov 8, 2024
6af07b2
Add win32 hacks to several files
nightlark Nov 8, 2024
bd84c2d
Make sure M_PI is defined on Windows
nightlark Nov 8, 2024
354009e
Handle getting free disk space on Windows
nightlark Nov 8, 2024
0e2630a
Block include of sys/statfs.h on Windows
nightlark Nov 8, 2024
cf89df4
Fix issues with getting free disk space and tmpdir on Windows
nightlark Nov 8, 2024
f3ba70c
Add Windows equivalents for sysconf getting system information
nightlark Nov 8, 2024
ebeefe4
No madvise equivalent on Windows
nightlark Nov 8, 2024
b19e73f
No Windows equivalent of ignoring SIGPIPE signal
nightlark Nov 8, 2024
5d2d103
Fix a bugs related to pages available calculation and Windows temp paths
nightlark Nov 8, 2024
e6a3fa7
Add Windows equivalent of mkstemp
nightlark Nov 8, 2024
3067fdd
Call _mkdir on Windows with no permission bits
nightlark Nov 8, 2024
618ba29
Add include for _mkdir on Windows
nightlark Nov 8, 2024
3068134
Block the backtrace debugging code on Windows
nightlark Nov 9, 2024
49ac729
Mark mman.h functions as inline to avoid multiple definition errors
nightlark Nov 9, 2024
f63b3ec
Address sysconf to get number of CPUS in tile-join on Windows
nightlark Nov 9, 2024
d720ad0
Open 'NUL' instead of /dev/null on Windows
nightlark Nov 9, 2024
d57df2e
Use 'NUL' instead of /dev/null in a few other places on Windows builds
nightlark Nov 9, 2024
5dfff25
Add correct null device detection based on OS to the Makefile
nightlark Nov 9, 2024
08109a4
Try using the same mkstemp function on Windows
nightlark Nov 9, 2024
4111348
Use lowercase nul instead of NUL on Windows
nightlark Nov 9, 2024
10967da
Try using a 'file' within nul folder on Windows in Makefile
nightlark Nov 9, 2024
4f364a3
Change back to using /dev/null in Makefile
nightlark Nov 9, 2024
2ed1293
Set a low max number of file handles on Windows
nightlark Nov 9, 2024
c3806c9
Try 200 max files on Windows
nightlark Nov 9, 2024
b016ba3
Statically link tippecanoe.exe and upload it as an artifact
nightlark Nov 11, 2024
26daf08
Use upload artifacts action v3
nightlark Nov 11, 2024
50e4a84
Update github actions used
nightlark Nov 11, 2024
f9b4dee
Upload the artifact if the build succeeds, before tests
nightlark Nov 11, 2024
429ef6a
Remove tests.yml in root of repository
nightlark Nov 11, 2024
247b4d0
Disable static linking mingw for virustotal scanning comparison point
nightlark Nov 13, 2024
8c9c2a5
Merge branch 'main' into add-windows-ci
nightlark Dec 7, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 25 additions & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
os: [ubuntu-latest, macos-latest]
version: ['Release', 'Debug']
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Install dependencies (Ubuntu)
if: runner.os == 'Linux'
run: sudo apt-get install libsqlite3-dev
Expand All @@ -19,3 +19,27 @@ jobs:
run: brew install sqlite3
- run: uname -a; BUILDTYPE=${{ matrix.version }} make
- run: make test

test-windows:
runs-on: windows-latest
strategy:
matrix:
version: ['Release', 'Debug']
steps:
- uses: actions/checkout@v4
- uses: msys2/setup-msys2@v2
with:
update: true
install: >-
mingw-w64-x86_64-toolchain
mingw-w64-x86_64-sqlite3
- shell: msys2 {0}
#run: uname -a; BUILDTYPE=${{ matrix.version }} mingw32-make CXXFLAGS="-static-libgcc -static-libstdc++"
run: uname -a; BUILDTYPE=${{ matrix.version }} mingw32-make
- name: Upload binary artifact
uses: actions/upload-artifact@v4
with:
name: tippecanoe-binary
path: ./tippecanoe.exe
- shell: msys2 {0}
run: mingw32-make test
16 changes: 12 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,14 @@ RELEASE_FLAGS := -O3 -DNDEBUG
DEBUG_FLAGS := -O0 -DDEBUG -fno-inline-functions -fno-omit-frame-pointer


# Determine the null device based on the operating system
ifeq ($(OS),Windows_NT)
NULL_DEVICE := /dev/null
else
NULL_DEVICE := /dev/null
endif


OS := $(shell uname -s)
ifeq ($(OS),FreeBSD)
ADVSHELL = /usr/local/bin/bash
Expand Down Expand Up @@ -121,7 +129,7 @@ testargs = \
$(subst _, ,$(1))))))))

%.json.check:
./tippecanoe -q -a@ -f -o [email protected] $(call testargs,$(patsubst %.json.check,%,$(word 4,$(subst /, ,$@)))) $(foreach suffix,$(suffixes),$(sort $(wildcard $(subst $(SPACE),/,$(wordlist 1,2,$(subst /, ,$@)))/*.$(suffix)))) < /dev/null
./tippecanoe -q -a@ -f -o [email protected] $(call testargs,$(patsubst %.json.check,%,$(word 4,$(subst /, ,$@)))) $(foreach suffix,$(suffixes),$(sort $(wildcard $(subst $(SPACE),/,$(wordlist 1,2,$(subst /, ,$@)))/*.$(suffix)))) < $(NULL_DEVICE)
./tippecanoe-decode -x generator [email protected] > [email protected]
cmp [email protected] $(patsubst %.check,%,$@)
rm [email protected] [email protected]
Expand All @@ -138,7 +146,7 @@ fewer-tests: tippecanoe tippecanoe-decode geobuf-test raw-tiles-test parallel-te
%.json.checkbuf:
for i in $(wildcard $(subst $(SPACE),/,$(wordlist 1,2,$(subst /, ,$@)))/*.json); do ./tippecanoe-json-tool -w $$i | ./node_modules/geobuf/bin/json2geobuf > $$i.geobuf; done
for i in $(wildcard $(subst $(SPACE),/,$(wordlist 1,2,$(subst /, ,$@)))/*.json.gz); do gzip -dc $$i | ./tippecanoe-json-tool -w | ./node_modules/geobuf/bin/json2geobuf > $$i.geobuf; done
./tippecanoe -q -a@ -f -o [email protected] $(call testargs,$(patsubst %.json.checkbuf,%,$(word 4,$(subst /, ,$@)))) $(foreach suffix,$(suffixes),$(addsuffix .geobuf,$(sort $(wildcard $(subst $(SPACE),/,$(wordlist 1,2,$(subst /, ,$@)))/*.$(suffix))))) < /dev/null
./tippecanoe -q -a@ -f -o [email protected] $(call testargs,$(patsubst %.json.checkbuf,%,$(word 4,$(subst /, ,$@)))) $(foreach suffix,$(suffixes),$(addsuffix .geobuf,$(sort $(wildcard $(subst $(SPACE),/,$(wordlist 1,2,$(subst /, ,$@)))/*.$(suffix))))) < $(NULL_DEVICE)
./tippecanoe-decode -x generator [email protected] | sed 's/checkbuf/check/g' | sed 's/\.geobuf//g' > [email protected]
cmp [email protected] $(patsubst %.checkbuf,%,$@)
rm [email protected] [email protected]
Expand All @@ -156,7 +164,7 @@ parallel-test: $(eval SHELL:=$(ADVSHELL))
cat tests/parallel/in[1234].json | ./tippecanoe -q -z5 -f -pi -l test -n test -o tests/parallel/linear-pipe.mbtiles
cat tests/parallel/in[1234].json | ./tippecanoe -q -z5 -f -pi -l test -n test -P -o tests/parallel/parallel-pipe.mbtiles
cat tests/parallel/in[1234].json | sed 's/^/@/' | tr '@' '\036' | ./tippecanoe -q -z5 -f -pi -l test -n test -o tests/parallel/implicit-pipe.mbtiles
./tippecanoe -q -z5 -f -pi -l test -n test -P -o tests/parallel/parallel-pipes.mbtiles <(cat tests/parallel/in1.json) <(cat tests/parallel/empty1.json) <(cat tests/parallel/empty2.json) <(cat tests/parallel/in2.json) /dev/null <(cat tests/parallel/in3.json) <(cat tests/parallel/in4.json)
./tippecanoe -q -z5 -f -pi -l test -n test -P -o tests/parallel/parallel-pipes.mbtiles <(cat tests/parallel/in1.json) <(cat tests/parallel/empty1.json) <(cat tests/parallel/empty2.json) <(cat tests/parallel/in2.json) $(NULL_DEVICE) <(cat tests/parallel/in3.json) <(cat tests/parallel/in4.json)
./tippecanoe-decode -x generator -x generator_options tests/parallel/linear-file.mbtiles > tests/parallel/linear-file.json
./tippecanoe-decode -x generator -x generator_options tests/parallel/parallel-file.mbtiles > tests/parallel/parallel-file.json
./tippecanoe-decode -x generator -x generator_options tests/parallel/linear-pipe.mbtiles > tests/parallel/linear-pipe.json
Expand Down Expand Up @@ -303,7 +311,7 @@ overzoom-test: tippecanoe-overzoom
rm tests/pbf/13-1310-3166-8-30.pbf tests/pbf/13-1310-3166-8-30.pbf.json.check
# No features in child tile
./tippecanoe-overzoom -o tests/pbf/14-2616-6331.pbf tests/pbf/11-327-791.pbf 11/327/791 14/2616/6331
cmp tests/pbf/14-2616-6331.pbf /dev/null
cmp tests/pbf/14-2616-6331.pbf $(NULL_DEVICE)
rm tests/pbf/14-2616-6331.pbf
# Thinning
# 243 features in the source tile tests/pbf/0-0-0-pop.pbf
Expand Down
8 changes: 8 additions & 0 deletions decode.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,11 @@
#include <fcntl.h>
#include <dirent.h>
#include <sys/stat.h>
#ifdef _WIN32
#include "mman.h"
#else
#include <sys/mman.h>
#endif
#include <protozero/pbf_reader.hpp>
#include "mvt.hpp"
#include "projection.hpp"
Expand All @@ -25,6 +29,10 @@
#include "pmtiles_file.hpp"
#include "errors.hpp"

#ifdef _WIN32
#include "win32_hacks.hpp"
#endif

int minzoom = 0;
int maxzoom = 32;
bool force = false;
Expand Down
19 changes: 19 additions & 0 deletions dirtiles.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@
#include <limits.h>
#include <sys/stat.h>
#include <sqlite3.h>
#ifdef _WIN32
#include <direct.h>
#endif
#include "jsonpull/jsonpull.h"
#include "mbtiles.hpp"
#include "dirtiles.hpp"
Expand All @@ -28,16 +31,28 @@ std::string dir_read_tile(std::string base, struct zxy tile) {
void dir_write_tile(const char *outdir, int z, int tx, int ty, std::string const &pbf) {
// Don't check mkdir error returns, since most of these calls to
// mkdir will be creating directories that already exist.
#ifndef _WIN32
mkdir(outdir, S_IRWXU | S_IRWXG | S_IRWXO);
#else
_mkdir(outdir);
#endif

std::string curdir(outdir);
std::string slash("/");

std::string newdir = curdir + slash + std::to_string(z);
#ifndef _WIN32
mkdir(newdir.c_str(), S_IRWXU | S_IRWXG | S_IRWXO);
#else
_mkdir(newdir.c_str());
#endif

newdir = newdir + "/" + std::to_string(tx);
#ifndef _WIN32
mkdir(newdir.c_str(), S_IRWXU | S_IRWXG | S_IRWXO);
#else
_mkdir(newdir.c_str());
#endif

newdir = newdir + "/" + std::to_string(ty) + ".pbf";

Expand Down Expand Up @@ -87,7 +102,11 @@ static bool pbfname(const char *s) {
void check_dir(const char *dir, char **argv, bool force, bool forcetable) {
struct stat st;

#ifndef _WIN32
mkdir(dir, S_IRWXU | S_IRWXG | S_IRWXO);
#else
_mkdir(dir);
#endif
std::string meta = std::string(dir) + "/" + "metadata.json";
if (force) {
unlink(meta.c_str()); // error OK since it may not exist;
Expand Down
6 changes: 6 additions & 0 deletions geojson.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,20 @@
#include <unistd.h>
#include <sys/stat.h>
#include <sys/types.h>
#ifdef _WIN32
#include "mman.h"
#else
#include <sys/mman.h>
#endif
#include <fcntl.h>
#include <ctype.h>
#include <errno.h>
#include <limits.h>
#include <sqlite3.h>
#include <stdarg.h>
#ifndef _WIN32
#include <sys/resource.h>
#endif
#include <pthread.h>
#include <vector>
#include <algorithm>
Expand Down
3 changes: 3 additions & 0 deletions geometry.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@
#include <algorithm>
#include <cstdio>
#include <unistd.h>
#ifdef _WIN32
#define _USE_MATH_DEFINES
#endif
#include <cmath>
#include <limits.h>
#include <sqlite3.h>
Expand Down
Loading