CR: Adding custom CA for disconnected networks with outbound HTTPS intercepted by a proxy. Support for hostname.

[bradleyharden]

Hi,

I'm using the current production branch (2.7.0), but I can't seem to get file syncing to work. I can sync entries in my library, but not files. In particular, I get a series of errors like this in the Zotero client:

[JavaScript Error: "S3 returned 0 (1/3FCKK2HV) -- retrying upload"]

The client also never stops trying to sync. I can see repeated POST requests to the items path in the dataserver logs, but when I open the MinIO browser, the zotero and zotero-fulltext buckets are completely empty.

Interestingly, like @aitianshi, I also noticed that the .env file S3HOST points to 10.5.5.1 instead of 10.5.5.7. However, I tried switching the IP to 10.5.5.7 as well as minio, but neither seems to affect the behavior.

Finally, if I change the S3HOST to my-vm-url:9000 and expose port 9000 of the MinIO container, I get a different behavior. With that setup, I do actually see files in the MinIO bucket. But I am only ever able to sync exactly two files: no more, no less. After that, the dataserver starts issuing a series of curl error 7: couldn't connect to server errors, and I get 500 errors in the Zotero client.

Am I doing something wrong? Should file syncing work? Or is it not supported yet?

[uniuuu]

Hi @bradleyharden

[JavaScript Error: "S3 returned 0 (1/3FCKK2HV) -- retrying upload"]

Finally, if I change the S3HOST to my-vm-url:9000

That's mean you're using VM setup where your server is on computer/instance different from client.
Ensure that during build of both client and server you're using configuration for VM and not Localhost.

Interestingly, like @aitianshi, I also noticed that the .env file S3HOST points to 10.5.5.1 instead of 10.5.5.7. However, I tried switching the IP to 10.5.5.7 as well as minio, but neither seems to affect the behavior.

This is correct configuration 10.5.5.1 an not 10.5.5.7.

May I know do you use hostname for your VM as my-vm-url ? If you use hostname then it may not work if one of your internal DNS resolvers isn't resolving it.
The setup will work if you use VM IP address for both server and client configuration unless again you are using localhost setup scenario. Refer to https://github.com/uniuuu/zotprime#localhost-and-vm-installation
URL's are supported for kubernetes setup currently.

Please let me know which one you use localhost or VM? And then try the above approach and if you still have issue then I'll request logs and more details about how the setup done.

[bradleyharden]

@uniuuu, sorry, I'll try to be more explicit.

That's mean you're using VM setup where your server is on computer/instance different from client.
Ensure that during build of both client and server you're using configuration for VM and not Localhost.

Yes, I am running in a VM setup. And yes, I made sure to build both the client and the server using the VM hostname.

This is correct configuration 10.5.5.1 an not 10.5.5.7.

Ok. Thanks for the clarification. I thought it must be correct, because I looked at the Git blame for that line, and the change looked intentional.

May I know do you use hostname for your VM as my-vm-url? If you use hostname then it may not work if one of your internal DNS resolvers isn't resolving it.

Yes, I am using the hostname. The VM is running on an internal, private network. I usually don't have any problem with DNS over that network. For example, I can use my laptop to access the MinIO console at my-vm-url:9001. Unless there is something subtle going on that I don't understand, I don't think DNS is the problem here. The Zotero client is able to connect to both the dataserver and the stream-server. I can see evidence of both connections in the logs. However, just to be sure, I will try using the IP of the VM instead.

Please let me know which one you use localhost or VM? And then try the above approach and if you still have issue then I'll request logs and more details about how the setup done.

I think I've done everything correctly. I tried to follow the instructions as closely as possible. Thank you in advance for any additional help you can provide.

Oh, I should add a few more details, since they might be relevant. I am trying to run this on an internal network, and we have our own root certificate. I added that certificate into the dataserver, stream-server, tinymce-clean-server and the zotero-client Dockerfiles. I used the docker-compose-dev.yml file, so all the containers were built locally. Is that enough? Do any of the other Dockerfiles need to be modified?

For those 4 Dockerfiles, I had to use two slightly different approaches, depending on whether I could use wget and https without installing any additional packages. For the zotero-client and the dataserver, I added this:

RUN mkdir -p /usr/local/share/ca-certificates/ \
 && wget https://internal-url/cert.cer -O /usr/local/share/ca-certificates/cert.crt \
 && cat /usr/local/share/ca-certificates/cert.crt >> /etc/ssl/certs/ca-certificates.crt

And for the stream-server and tinymce-clean-server , I added this

COPY cert.crt /usr/local/share/ca-certificates/cert.crt
RUN mkdir -p /etc/ssl/certs \
 && touch /etc/ssl/certs/ca-certificates.crt \
 && cat /usr/local/share/ca-certificates/cert.crt >> /etc/ssl/certs/ca-certificates.crt

I also add

RUN npm config set cafile /etc/ssl/certs/ca-certificates.crt

to any Dockerfiles running npm.

[uniuuu]

Hi @bradleyharden
Please try to switch to IP address and not hostname (or URL).
Hostname doesn't work because DNS resolvers do not resolve it inside docker container. At this point it's not tested and not supported to use hostname.
Moreover if there is a need to add hostname support for docker compose setup then the best approach would be at add reverse proxy container to be able handle both URL's or/and hostnames plus a bonus will be SSL termination.

For URL's I would recommend to use on-premise Kubernetes (Microk8s) instead of docker compose. I'll write a guide for it and let you know when it's added.

Yes, I am running in a VM setup. And yes, I made sure to build both the client and the server using the VM hostname.

Yes, I am using the hostname. The VM is running on an internal, private network. I usually don't have any problem with DNS over that network. For example, I can use my laptop to access the MinIO console at my-vm-url:9001. Unless there is something subtle going on that I don't understand, I don't think DNS is the problem here. The Zotero client is able to connect to both the dataserver and the stream-server. I can see evidence of both connections in the logs. However, just to be sure, I will try using the IP of the VM instead.

[bradleyharden]

Hi @uniuuu,

I still see the same behavior when using an IP address.

To eliminate any possibility that the internal network or Dockerfile changes have any effect, I tried to reproduce the problem using the localhost setup a local laptop connected to the open internet.

Here's what I did:

• Cloned a fresh copy of the repo
• Built the client with DOCKER_BUILDKIT=1 sudo docker build --progress=plain --file client.Dockerfile --output build .
  • I verified that the default values for the ARGs were for the localhost configuration on Linux, so I didn't supply any --build-args.
• Copied .env_example to .env and docker-compose-dev.yml to docker-compose.yml
  • Once again, I checked that all the default values were for the localhost configuration.
• Built the images using sudo docker compose build
• Ran sudo docker compose up

At this point, I found that the minio/mc container was giving me errors with curl. I guessed that something had changed in the past few days, so I pinned the miniomc.Dockerfile to minio/mc:RELEASE.2023-10-14T01-57-03Z. Then I re-ran sudo docker compose build.

Then, I tried again:

• I brought the containers down with sudo docker compose down
• I deleted all volumes on the system with sudo docker volume rm ...
• I brought the containers back up with sudo docker compose up
• I ran the bin/init.sh script
• I opened the Zotero client and imported my existing library from an RDF file
• I logged in to sync with the default admin admin login.
• I changed the setting to sync manually
• I pressed the sync button
• I opened the error console

At this point, I still see the same behavior. I get S3 returned 0 errors in the console. I see POST requests to the dataserver. But when I log in to MinIO through localhost:9001, I don't see any files in either of the two buckets.

Am I doing something wrong here?

[aitianshi]

Except for the preprint issue we have found @uniuuu I still have the issue @bradleyharden is explaining for other docs if I don't change Minio IP. There's something been missing with all default values using the whole Zotero in local.

[uniuuu]

To eliminate any possibility that the internal network or Dockerfile changes have any effect, I tried to reproduce the problem using the localhost setup a local laptop connected to the open internet.

* Copied `.env_example` to `.env` and `docker-compose-dev.yml` to `docker-compose.yml`

Hi @bradleyharden @aitianshi
You're good to go with IP address, no need to use localhost unless your client and server on the same single machine without VM. If you're still having the error after below described step then we'll troubleshoot it with IP address installation first.
Please try to use prod yaml file and not dev as per the guide in readme. And there is no build step for server too but instead it will pull prod images from dockerhub. VM/Localhost installation
$ cp docker-compose-prod.yml docker-compose.yml

* Cloned a fresh copy of the repo

* Built the client with `DOCKER_BUILDKIT=1 sudo docker build --progress=plain --file client.Dockerfile --output build .`

Another step you may miss is to switch to production branch as per guide.

$ git clone --recursive https://github.com/uniuuu/zotprime.git
$ git checkout production   

[bradleyharden]

@uniuuu,

• Yes, I am already using the production branch, which points to v2.7.0.
• I would prefer, if possible, to build all the images myself. That's why I chose docker-compose-dev.yml. Is that not a valid choice? If -prod works but -dev does not, isn't that a bug?
• It is easier for me to reproduce your work if I use the localhost configuration, because I can do that on a single laptop connected to the open internet. My VM is on an internal network and all traffic is re-signed with our root certificate. If I want to test with the VM, I need to modify the images to include our root certificate.

Either way, I will try using the -prod file.

[aitianshi]

@bradleyharden If you can import some files from the outside to your internal network, you can try the solution there: #12. I had the same issue but I could build everything from a public network then import all images to my private network, suggested by @uniuuu .

[bradleyharden]

@aitianshi, yes, I saw that suggestion in your issue. It could work for me, but it's also pretty easy for me to modify the Dockerfiles and rebuild. I would prefer that, if possible, so that I have a record and a way to reproduce things without manual intervention.

[uniuuu]

@uniuuu,

* Yes, I am already using the `production` branch, which points to `v2.7.0`.

* I would prefer, if possible, to build all the images myself. That's why I chose `docker-compose-dev.yml`. Is that not a valid choice? If `-prod` works but `-dev` does not, isn't that a bug?

* It is easier for me to reproduce your work if I use the `localhost` configuration, because I can do that on a single laptop connected to the open internet. My VM is on an internal network and all traffic is re-signed with our root certificate. If I want to test with the VM, I need to modify the images to include our root certificate.

Either way, I will try using the -prod file.

Hi @bradleyharden

The pointed out issue that dev on your side doesn't work - is still okay as it's meant for development purpose and not production level deployment.

That's understood that you want to build images. But the images you'll build won't guarantee their consistency unless at your side all is working, and no issues with getting dependencies. So you have to keep control over your build and ensure it wasn't impacted by your environment setup or limits. Anyway it's best practice to use prebuild and verified images. So i have made changes to production branch to avoid confusion by having dev docker-compose there.

I am open to troubleshoot the image build failure. But for this you have provide details what do you modify in images exactly? And then also do provide output saved in file of the progress build of the images which has to be done by next command:
BUILDKIT_PROGRESS=plain docker compose build

Also I have added microk8s guide microk8s-installation and I encourage to switch from docker to k8s for better server handling experience.

[bradleyharden]

@uniuuu, I won't be able to try anything until later today, but I wanted to clarify one thing now.

In my last attempt, using the localhost configuration, I didn't make any changes to the Dockerfiles, docker-compose.yml or .env file. The only exception was to pin the version of the minio/mc image to RELEASE.2023-10-14T01-57-03Z, because the latest version no long comes with curl installed.

I will try with the pre-built images, but I think I've done that before and seen the same errors. I will try again to be certain.

[uniuuu]

Hi @bradleyharden @aitianshi
I couldn't reproduce the issue on my side.
Could you please write on which Linux do you run server ? What's docker version?

[bradleyharden]

@uniuuu, sorry, I'm just very busy at the moment. I still want to figure this out. I will try to get back to you soon. Thanks for your help.

[bradleyharden]

@uniuuu, I did as you asked and tried again using the pre-built containers. I just cloned a fresh copy, switched to the most recent production branch (v2.7.2 now), built the Zotero client, ran docker compose up, and then ran ./bin/init.sh. I get the same behavior as before.

To reiterate, I'm running on an Ubuntu 22.04 laptop connected to the open internet. I'm not behind an internal network. I'm also running in the localhost configuration.

Here is the output of docker info:

Client: Docker Engine - Community
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 10
  Running: 10
  Paused: 0
  Stopped: 0
 Images: 10
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc version: v1.1.9-0-gccaecfc
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.2.0-34-generic
 Operating System: Ubuntu 22.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.29GiB
 Name: hardebj1-ll3
 ID: f1dea25b-c36a-46f2-8cae-6581dd32a3d4
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[bradleyharden]

I can also confirm that the suggestion by @aitianshi to change the minio IP in .env does change the behavior. With 10.5.5.7, I can now see files stored in MinIO, but once again, it is only 2 files, even though I would expect more.

[uniuuu]

Hi @bradleyharden

Thank you for the information provided.
The issue has been fixed in commit aef5cda
Please try re-clone the production branch and test it to verify that it's no more recurring.

[bradleyharden]

@uniuuu, exposing port 9000 of the MinIO container is something I had tried on my own once before. That change removes the S3 returned 0 errors, but it replaces them with errors of the form:

[JavaScript Error: "HTTP POST http://localhost:8080/users/1/items/3UYBIXN9/file failed with status code 500:

An error occurred"]

I just confirmed that I still see this behavior. I updated my repo to v2.7.3, did a docker system prune and rebuilt everything from scratch. I get the same 500 response I had gotten before. And once again, I see exactly 2 files in the zotero bucket from the MinIO console.

[bradleyharden]

I will copy some lines from the logs that I think might be relevant:

zotprime-zotprime-dataserver-1     | [Sat Nov 11 12:44:16.642020 2023] [php:notice] [pid 738] [client 10.5.5.1:42568] Deadlock found when trying t
o get lock; try restarting transaction\n\nShard: 0\n\nQuery:\nINSERT INTO storageFiles (hash, filename, size, zip) VALUES (?,?,?,?)\n\nParams:\nAr
ray\n(\n    [0] => 5b859a4546d1c4c432a19362c3e91495\n    [1] => 3UYBIXN9.zip\n    [2] => 1008678\n    [3] => 1\n)\n\n\n in /var/www/zotero/include
/DB.inc.php:1303 (POST /users/1/items/3UYBIXN9/file) (05effef2c8)

zotprime-zotprime-dataserver-1     | [Sat Nov 11 12:44:54.364149 2023] [php:warn] [pid 735] [client 10.5.5.1:54624] PHP Warning:  iconv(): Wrong e
ncoding, conversion from "UTF-8" to "ASCII//IGNORE" is not allowed in /var/www/zotero/model/Item.inc.php on line 3159

zotprime-zotprime-dataserver-1     | [Sat Nov 11 12:44:54.365647 2023] [php:notice] [pid 735] [client 10.5.5.1:54624] You have an error in your SQ
L syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'AS new\n\t\t\t\tON DUPLICATE KEY UPDA
TE storageFileID=new.storageFileID, mtime=new...' at line 1\n\nShard: 1\n\nQuery:\nINSERT INTO storageFileItems (storageFileID, itemID, mtime, siz
e) VALUES (?,?,?,?) AS new\n\t\t\t\tON DUPLICATE KEY UPDATE storageFileID=new.storageFileID, mtime=new.mtime, size=new.size\n\nParams:\nArray\n(\n
)\n\n\n\n\nShard: 1\n\nQuery:\nINSERT INTO storageFileItems (storageFileID, itemID, mtime, size) VALUES (?,?,?,?) AS new\n\t\t\t\tON DUPLICATE KEY
 UPDATE storageFileID=new.storageFileID, mtime=new.mtime, size=new.size\n\nParams:\nArray\n(\n    [0] => 9\n    [1] => 77\n    [2] => 169972462300
0\n    [3] => 1008678\n)\n\n\n in /var/www/zotero/include/DB.inc.php:1303 (POST /users/1/items/3UYBIXN9/file) (a64a43cae6)

[bradleyharden]

As a separate question, I noticed something else recently. The stream server repeatedly prints this line:

zotprime-zotprime-streamserver-1   | [11/Nov/2023:12:56:05 -0500] [0 connections, 0 subscriptions]

When the Zotero client was open, I used to see non-zero values for both connections and subscriptions. But lately, those values are always zero, whether the client is open or not. Is that the expected behavior? Is it relevant to this problem at all?

[uniuuu]

Hi @bradleyharden

zotprime-zotprime-streamserver-1   | [11/Nov/2023:12:56:05 -0500] [0 connections, 0 subscriptions]

You may have changes with your IP and now the client cannot connect to the stream server. So try to rebuild client and ensure IP address set the same as VM's IP address. See below next my comment.

• It is easier for me to reproduce your work if I use the localhost configuration, because I can do that on a single laptop connected to the open internet. My VM is on an internal network and all traffic is re-signed with our root certificate. If I want to test with the VM, I need to modify the images to include our root certificate.

If you use VM you cannot use localhost setup. This is probably the root cause of the issue as it's confusing past in README.md.
The localhost setup - means client and server are sitting in one single host. Because localhost resolves to IP loopback address 127.0.0.1.
So if you have VM then there are already two hosts - one host is VM and it has it's own IP address, second host is your laptop and it's has it's own IP address. And if you run client from laptop then localhost will point to laptop's loopback address. But your server is in VM and localhost points to VM's loopback address. So the client cannot communicate with server in VM if you use localhost setup.
For VM setup you must use IP address of the VM. If your VM's network is bridged then it will be VM's IP address. If VM's network behind NAT then it will be IP that set as VM's gateway and you have to do port-forwarding for ZotPrime ports.
If this is correct and it was the issue, may I ask how you see what improvements can be added into README to make this clear regarding selection of setup? In worse case scenario if this will keep to confuse then best is to consider removing locahost setup option from description. I doubt localhost setup is common setup now because it's more like for testing purposes or for quick run or when you don't use VM in laptop. Here we have the server part and the client part - server should have it's own host when in production. The localhost setup is inherited from original ZotPrime repository I do not know intention behind it except my assumption that it was just very quick to make that time and for VM setup it was additional effort to add-on. Anyway I saw in posts some that people who run original ZotPrime have done VM setup themselves that time by modifying configurations.

My VM is on an internal network and all traffic is re-signed with our root certificate. If I want to test with the VM, I need to modify the images to include our root certificate.

Hi @bradleyharden Please try to switch to IP address and not hostname (or URL). Hostname doesn't work because DNS resolvers do not resolve it inside docker container. At this point it's not tested and not supported to use hostname. Moreover if there is a need to add hostname support for docker compose setup then the best approach would be at add reverse proxy container to be able handle both URL's or/and hostnames plus a bonus will be SSL termination.

For URL's I would recommend to use on-premise Kubernetes (Microk8s) instead of docker compose. I'll write a guide for it and let you know when it's added.

If I got you right what you are trying to achieve is to add HTTPS (which will require SSL/TLS termination on all ports used).
I confirm the current ZotPrime docker compose doesn't support it.
Modifying images probably will lead to issues as it's never been tested yet.
The best approach for HTTPS support is to add proxy service which will allow to add your certificate and do SSL/TLS termination.
I find docker compose is not best solution for production. The docker swarm and kubernetes are proven production grade solutions for that.
If Microk8s setup is something that most people will find hard to setup then switching from docker compose to docker swarm with adding proxy service (Caddy/Envoy/Traefik) can be considered.

RUN mkdir -p /usr/local/share/ca-certificates/ \
 && wget https://internal-url/cert.cer -O /usr/local/share/ca-certificates/cert.crt \
 && cat /usr/local/share/ca-certificates/cert.crt >> /etc/ssl/certs/ca-certificates.crt

Installing CA for each OS is not yet tested/supported. But I'll study this question what would be the best approach to add support for it.

[JavaScript Error: "HTTP POST http://localhost:8080/users/1/items/3UYBIXN9/file failed with status code 500:

An error occurred"]

zotprime-zotprime-dataserver-1     | [Sat Nov 11 12:44:16.642020 2023] [php:notice] [pid 738] [client 10.5.5.1:42568] Deadlock found when trying t
o get lock; try restarting transaction\n\nShard: 0\n\nQuery:\nINSERT INTO storageFiles (hash, filename, size, zip) VALUES (?,?,?,?)\n\nParams:\nAr
ray\n(\n    [0] => 5b859a4546d1c4c432a19362c3e91495\n    [1] => 3UYBIXN9.zip\n    [2] => 1008678\n    [3] => 1\n)\n\n\n in /var/www/zotero/include
/DB.inc.php:1303 (POST /users/1/items/3UYBIXN9/file) (05effef2c8)

zotprime-zotprime-dataserver-1     | [Sat Nov 11 12:44:54.364149 2023] [php:warn] [pid 735] [client 10.5.5.1:54624] PHP Warning:  iconv(): Wrong e
ncoding, conversion from "UTF-8" to "ASCII//IGNORE" is not allowed in /var/www/zotero/model/Item.inc.php on line 3159

zotprime-zotprime-dataserver-1     | [Sat Nov 11 12:44:54.365647 2023] [php:notice] [pid 735] [client 10.5.5.1:54624] You have an error in your SQ
L syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'AS new\n\t\t\t\tON DUPLICATE KEY UPDA
TE storageFileID=new.storageFileID, mtime=new...' at line 1\n\nShard: 1\n\nQuery:\nINSERT INTO storageFileItems (storageFileID, itemID, mtime, siz
e) VALUES (?,?,?,?) AS new\n\t\t\t\tON DUPLICATE KEY UPDATE storageFileID=new.storageFileID, mtime=new.mtime, size=new.size\n\nParams:\nArray\n(\n
)\n\n\n\n\nShard: 1\n\nQuery:\nINSERT INTO storageFileItems (storageFileID, itemID, mtime, size) VALUES (?,?,?,?) AS new\n\t\t\t\tON DUPLICATE KEY
 UPDATE storageFileID=new.storageFileID, mtime=new.mtime, size=new.size\n\nParams:\nArray\n(\n    [0] => 9\n    [1] => 77\n    [2] => 169972462300
0\n    [3] => 1008678\n)\n\n\n in /var/www/zotero/include/DB.inc.php:1303 (POST /users/1/items/3UYBIXN9/file) (a64a43cae6)

When you do reinstall ZotPrime ensure the volumes all are deleted.
Do next command that removes all containers and deletes all associated volumes:
docker compose down -v
Just in case some issue possible due to file format or content please share also a file sample.

A quintessential for this case would be that before you'll try to implement it your target production environment, first you need to try to setup in testing environment without doing any changes and do not use any restricted network or with policies that require to have TLS . And go with VM setup. Build client with setting IP address of VM (VM with bridged network).

Attached is 3 min video showing steps deleting client's data, deleting server, creating a new server, initialization, connecting client.

ZotPrime.mp4

[uniuuu]

Hi @bradleyharden
Good day.
Please let me know if it helped to resolve the issue.
Since there is no update I'll close this issue and convert it to discussion.
Feel free to post your reply in the discussion or reopen this issue if you find that the reported error is still not solved.

[uniuuu]

Hi @bradleyharden
Feel free to post your reply in here in the discussion.

[bradleyharden]

@uniuuu, sorry for the delay. I have a young baby, so I'm short on free time. It's a long weekend here in the US, so I have a little extra time now.

First, I just want to emphasize that I really appreciate your work on Zotprime. And thank you for taking the time to help me figure out this issue.

That being said, I'm a little frustrated. Although I am not a web developer, I do have some software development experience, and I would consider myself pretty technically proficient. I have read through the documentation several times. Many of the explanations and suggestions you have given me I already understand or have already tried. I keep trying to convince you of that, but you don't seem to believe me.

If you use VM you cannot use localhost setup. This is probably the root cause of the issue as it's confusing past in README.md.
The localhost setup - means client and server are sitting in one single host. Because localhost resolves to IP loopback address 127.0.0.1.

Yes, I understand what localhost means. Let me try to be crystal clear. I am not using a VM or changing the IP address at all. In fact, I'm not going to even try the VM setup until I can prove that the localhost configuration works.

I am intentionally using the localhost configuration, because it is the simplest configuration and requires essentially no changes to the cloned repository. I am trying to exactly reproduce something you have done.

In worse case scenario if this will keep to confuse then best is to consider removing locahost setup option from description. I doubt localhost setup is common setup now because it's more like for testing purposes or for quick run or when you don't use VM in laptop.

Please don't remove the localhost option. I agree that it is not useful beyond testing, but testing is an important use case. That's exactly why I'm using it here.

Modifying images probably will lead to issues as it's never been tested yet.

I said this before, but I'll reiterate. I am no longer trying to build or modify your images. I am using the images pulled from DockerHub.

I find docker compose is not best solution for production. The docker swarm and kubernetes are proven production grade solutions for that.

That sounds fine to me. But I will not try anything more advanced than the localhost configuration until I can prove that works.

Installing CA for each OS is not yet tested/supported. But I'll study this question what would be the best approach to add support for it.

Again, I'm no longer doing this and won't do this until I can get the simpler solution to work.

When you do reinstall ZotPrime ensure the volumes all are deleted.

I thought I mentioned this, but maybe I didn't. Yes, I have been deleting all volumes every time I try a new configuration.

Do next command that removes all containers and deletes all associated volumes: docker compose down -v

I was not aware of the -v option. That's useful. I was manually deleting volumes, which is a pain.

A quintessential for this case would be that before you'll try to implement it your target production environment, first you need to try to setup in testing environment without doing any changes and do not use any restricted network or with policies that require to have TLS .

Yes, that's exactly what I've been trying to do with the localhost configuration.

And go with VM setup. Build client with setting IP address of VM (VM with bridged network).

This is where I disagree. I don't really have another machine to test a VM configuration. If I wanted to do that, it would have to be on a company internal network, which would require some level of modification to the project. That's why I'm using the localhost configuration.

Let me ask a question of my own. Have you actually tried to reproduce this behavior exactly as described in the the documentation? Is there any chance that your development environment somehow gives different results? For example, maybe you're working from images that haven't yet been pushed to DockerHub?

In the next post (which may not be immediate), I will try to provide an exact sequence of shell commands to reproduce the issue. Hopefully that will get both of us on the same page.

[bradleyharden]

@uniuuu, here is a complete accounting of my actions. Please let me know if you see any problem with what I've done. As far as I can tell, I am completely clearing any state from previous attempts, and I am working from your unmodified code.

# Show that there is no residual Zotero data
$ rm -r Zotero/ .zotero/
rm: remove 2 arguments recursively? y
rm: cannot remove 'Zotero/': No such file or directory
rm: cannot remove '.zotero/': No such file or directory

# Show that there is no residual Docker data
$ sudo docker image list
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE

$ sudo docker container list -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

$ sudo docker volume list
DRIVER    VOLUME NAME

$ sudo docker system prune
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all dangling images
  - all dangling build cache

Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B

# Freshly clone the repo
$ git clone --recursive https://github.com/uniuuu/zotprime.git

# ...

$ cd zotprime/

$ git switch production 
M       client/zotero-client
M       stack/dataserver/dataserver
Branch 'production' set up to track remote branch 'production' from 'origin'.
Switched to a new branch 'production'

# We need to update the submodules after switching branches
$ git submodule update --recursive
warning: unable to rmdir 'pdf-reader': Directory not empty
Submodule path 'client/zotero-client': checked out '7e82cc5ffd82c695b1556b2ee25e86ddfbcb4e15'
Submodule path 'client/zotero-client/chrome/content/zotero/locale/csl': checked out 'a7c6f63e25323ac2f375943417d7f778f875f11c'
Submodule path 'client/zotero-client/chrome/content/zotero/xpcom/translate': checked out 'de2e270a37b9182c0b3017e54096b9b2d506926d'
Submodule path 'client/zotero-client/chrome/content/zotero/xpcom/translate/modules/utilities': checked out 'cccf1235a318c259345fc623d5e9d6770ba19df7'
Submodule path 'client/zotero-client/chrome/content/zotero/xpcom/translate/modules/utilities/resource/schema/global': checked out '1b12272d44134a652519e9192e5a936ac9fcd707'
Submodule path 'client/zotero-client/chrome/content/zotero/xpcom/utilities': checked out '9c89b23153ce621ed0f1d581a5e32248704c6fb7'
Submodule path 'client/zotero-client/note-editor': checked out 'e2e3009bbce0070488989c8678bb2da3e22d7514'
Submodule path 'client/zotero-client/pdf-worker': checked out 'a30d5cc8f5e0606e48433dfd02b4558881b667ab'
Submodule path 'client/zotero-client/pdf-worker/pdf.js': checked out '159a1d5612b803ca1d8b1f5d7498d5e84a58c284'
Submodule path 'client/zotero-client/pdf-worker/pdf.js/test/ttx/fonttools-code': checked out 'd8170131a3458ffbc19089cf33249777bde390e7'
Submodule path 'client/zotero-client/reader': checked out '79a1cd348d09e7f650e24074ff04bcc2fb8565a2'
Submodule path 'client/zotero-client/resource/SingleFile': checked out '0bca0227851348ef9bbaec780e88deb32b1cc03d'
Submodule path 'client/zotero-client/resource/schema/global': checked out '1f5331e902ed2b765a379b9586c65d1a192c151d'
Submodule path 'client/zotero-client/styles': checked out 'a791f0f2188a31dc9a025c6b8b404b06a1455165'
Submodule path 'client/zotero-client/translators': checked out 'f88a29b3f03136d3e42020b8579726dbf9b17652'
Submodule path 'stack/dataserver/dataserver': checked out '6a05549921cf18c9334439a2d0c9178bdae2c39d'

# Looks like that leaves behind some untracked files
$ git status
On branch production
Your branch is up to date with 'origin/production'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)
        modified:   client/zotero-client (untracked content)

no changes added to commit (use "git add" and/or "git commit -a")

# Let's delete them
$ rm -r client/zotero-client/pdf-reader/
rm: remove 1 argument recursively? y

# Now everything is clean
$ git status
On branch production
Your branch is up to date with 'origin/production'.

nothing to commit, working tree clean

# Check the Docker version, which is important for the next step
$ docker --version
Docker version 24.0.7, build afdd53b

# DOCKER_BUILDKIT=1 is not necessary for me
# From the Docker documentation: "If you are running a version of Docker Engine
# version earlier than 23.0, you can enable BuildKit either by setting an
# environment variable"
# The build_args aren't necessary either, since I am using the default values
# specified in client.Dockerfile
$ sudo docker build --progress=plain --file client.Dockerfile --output build .

# ...

# The files were built as root, so make them mine
$ sudo chown -R hardebj1:hardebj1 build

# Back to setting up the server
$ cp .env_example .env

# I like to see the logs, so I don't detach. Instead, I use tmux for another shell
$ sudo docker compose up

# After all images have been pulled and containers started, open a separate
# shell in tmux
$ ./bin/init.sh

# ...

# Start the client
$ ./build/Zotero_linux-x86_64/zotero

At this point, I do things in the Zotero client GUI

• Cancel installation of plugin
• File -> Import -> Import my previously exported library
• The library has something like 50 citations, 1 PDF and many HTML documents
• Edit -> Preferences -> Sync
• Enter admin:admin
• Close preferences
• Press sync button
• Tools -> Developer -> Error console

[JavaScript Error: "HTTP POST http://localhost:8080/users/1/items/3RCHNC2H/file failed with status code 500:

An error occurred"]

Now to the browser

• Go to localhost:9001/
• Enter username & password
• Observe exactly 2 objects in the zotero bucket
• Download both through the MinIO interface.
• Both are HTML documents. Neither is the PDF

Finally, note that the Zotero client sync arrow never stopped spinning, even after the time I spent creating and editing this post.

As a follow-up, I will show that the documents don't sync if I clear the data and try to pull from the server

• Close the Zotero client
• Reopen it
• Edit -> Preferences -> Sync
• Click "Unlink account"
• Check "Remove my Zotero data from this computer"
• Confirm
• The client is now empty
• Close and reopen
• Edit -> Preferences -> Sync
• Enter admin:admin
• Click "Set up syncing"
• Close the preferences
• Click the sync button
• All of my citation sync
• Try to open one of the HTML documents. Get:

The attached file could not be found at the following path:

/home/hardebj1/Zotero/storage/8CT2TEJN/6786372.html

It may have been moved or deleted outside of Zotero, or, if the file was added on another computer, it may not yet have been synced to zotero.org.

• There's no PDF icon, so it did not sync

[uniuuu]

Hi @bradleyharden
Good day.
Before we'll troubleshoot further, please do next below.
Since switching to production branch failing

$ git switch production
M client/zotero-client
M stack/dataserver/dataserver
Branch 'production' set up to track remote branch 'production' from 'origin'.
Switched to a new branch 'production'

$ git submodule update --recursive
warning: unable to rmdir 'pdf-reader': Directory not empty

Try next command git clone -b production --recursive --single-branch https://github.com/uniuuu/zotprime.git to clone production branch straight.
Note git switch is different from git checkout you may have no untracked content, modified content and other failed git submodule's structure if to use second one. Try both these approaches and see if can have working production branch.

[uniuuu]

Hi @bradleyharden
Good day.
Wanted to check, did it work for you that time?

[bradleyharden]

@uniuuu, no, I gave up trying.

Note git switch is different from git checkout you may have no untracked content, modified content and other failed git submodule's structure if to use second one

To my knowledge, this statement isn't accurate. I've been using git switch for years now, and I've never observed any difference between switch and checkout for this particular use case. In fact, switch and restore were added to split the overloaded behavior of checkout, which can be used to both switch branches and restore files in the working tree. That means switch and checkout are intentionally redundant.

I also routinely use git submodules, and git always tells me when there are modified or untracked changes in a submodule. As I said above, after clearing the untracked files, git status gave me nothing to commit, working tree clean. I don't see any reason why using git checkout or git clone would make any difference here. As long as I'm on the correct commit with a clean working tree, everything should be identical.

As far as I could tell, I was able to reproduce the issue with your unmodified repo. Did you ever try reproducing it yourself, with the steps I gave above?

Debugging the issue didn't seem to be going anywhere, so I gave up.

[uniuuu]

@bradleyharden Please try with latest subrelease. Included mentioned command in README already. So to ensure the only branch pulled is the production one so no need to switch between branches.

Try next command git clone -b production --recursive --single-branch https://github.com/uniuuu/zotprime.git to clone production branch straight.

In your steps you mentioned

$ git switch production
M client/zotero-client
M stack/dataserver/dataserver
Branch 'production' set up to track remote branch 'production' from 'origin'.
Switched to a new branch 'production'

Because of the above output you cannot do already next step smoothly

We need to update the submodules after switching branches
$ git submodule update --recursive

And you mentioned:

Let's delete them
$ rm -r client/zotero-client/pdf-reader/
rm: remove 1 argument recursively? y

Because of the above step the client won't build correctly.

It was working with checkout previously but after CI flow keeps pushing new commits to submodules it started to fail. To save time and minimize effort the above mention command has been introduced.

[uniuuu]

Hi @bradleyharden

The issue:

[JavaScript Error: "S3 returned 0 (1/3FCKK2HV) -- retrying upload"]

Has been fixed in v2.8.3

[uniuuu]

Hi @bradleyharden

The issue:

[JavaScript Error: "S3 returned 0 (1/3FCKK2HV) -- retrying upload"]

Has been fixed in v2.8.3

Hi @bradleyharden
Good day.

Issue was resolved previously and I hope it is working on your side too.
Just an additional note - I saw such errors also occurring when initially client with the user account was used to upload files and then client was flushed i.e. rm -rf ~/Zotero/ ~/.zotero/, reinstalled and then connected with the same user to dataserver, that made synchronization between local (client) and remote (dataserver).

Fresh reinstall of the server and client resolved issue.

I'll monitor this more to find all such edge cases.

[uniuuu]

Oh, I should add a few more details, since they might be relevant. I am trying to run this on an internal network, and we have our own root certificate. I added that certificate into the dataserver, stream-server, tinymce-clean-server and the zotero-client Dockerfiles. I used the docker-compose-dev.yml file, so all the containers were built locally. Is that enough? Do any of the other Dockerfiles need to be modified?

For those 4 Dockerfiles, I had to use two slightly different approaches, depending on whether I could use wget and https without installing any additional packages. For the zotero-client and the dataserver, I added this:

RUN mkdir -p /usr/local/share/ca-certificates/ \
 && wget https://internal-url/cert.cer -O /usr/local/share/ca-certificates/cert.crt \
 && cat /usr/local/share/ca-certificates/cert.crt >> /etc/ssl/certs/ca-certificates.crt

And for the stream-server and tinymce-clean-server , I added this

COPY cert.crt /usr/local/share/ca-certificates/cert.crt
RUN mkdir -p /etc/ssl/certs \
 && touch /etc/ssl/certs/ca-certificates.crt \
 && cat /usr/local/share/ca-certificates/cert.crt >> /etc/ssl/certs/ca-certificates.crt

I also add

RUN npm config set cafile /etc/ssl/certs/ca-certificates.crt

to any Dockerfiles running npm.

@bradleyharden
Custom CA support has been added too custom-ca-certificate.md
dataserver, stream-server, tinymce-clean-server, zotero-client and phphmyadmin images are enough for it.

[haecks]

Hello @uniuuu ! Thanks a lot for your hard work!

I've been struggeling quite a while now on trying to enable TLS-encrypted connection from the build client to zotprime dataserver and streamserver through an nginx-proxy (as i mentioned, that just patching the datserver's apache with openssl itself is not nearly sufficient). My nginx-configuration pipes SSL-Ports 8443 to the http://zotprime-dataserver:8080 and 8481 to the ws://zotprime-streamserver:8081, which it supposedly does correctly as checked.

Even though i was able to provide the proper server's certificate matching the keys configured in nginx with somewhat like DOCKER_BUILDKIT=1 docker build --progress=plain --no-cache --file client.Dockerfile --build-arg HOST_DS=https://<SERVER_IP>:8443/ --build-arg HOST_ST=wss://<SERVER_IP>:8481/ --build-arg MLW=l --secret id=custom_ca,src=./custom_ca.crt --output build . for building the client, no connection through the nginx-proxy is succesful through that client (checked the build for success).
Keeping HOST_DS as an https://-connection just leads to the client error:
TypeError: can't access property 1, ZOTERO_CONFIG.PROXY_AUTH_URL.match(...) is null http.js:1414:69
keeping it on the SSL-Port but using http instead of https is of course rejected by nginx as improper non encrypted context.
Every option for the websocket though other than sticking with the plain 8080-Port for standard ws:// only leads to
The connection was refused when attempting to contact wss://<SERVER_IP>:8481/. streamer.js:153:20.

So i like to ask, what your example given in custom-ca-certificate.md tries to accomplish, as you still used unencrypted procols (http://, ws://) and ports despite providing the custom_ca-option for building the client.
My assumption was, that this certificate should verify the server.

Maybe i am just limited to the client's modules trying to check my "insecure" self-signed - neither Let' Encrypt or otherwise registered - certificate. I m aware that many common javascript SSL-engines do not respect already anchored system-allocated trust on such certificates. Maybe that applies to the modules used by the Zotero-client as well ...

Thanks everybody for enlightening me here!

[uniuuu]

Hi @haecks

The custom-ca-certificate.md is specifically purposed for organizations on disconnected/air-gapped or corporate networks where all outbound HTTPS is intercepted by a proxy (common in enterprises, government, etc.).

What I understood you're doing a different setup:

• to have a nginx proxy terminating SSL on ports 8443/8481
• to build the client with https:// and wss:// URLs

You no need custom CA in that case ignore it.

[uniuuu]

@haecks I've never included https setup into docker compose becasue it's not robust enough for production.
If you're building production - my advice, do not use docker compose.
Instead setup Microk8s. If you wish I can help you with this setup we can create a separate discussion and if you getting any issue I'll check on it.
What k8s (Microk8s) gives you:

• HA
• Out of box nginx ingress with TLS termination.
• Let's Encrypt or BYO certificate
• Immediate container (pod) recovery
• Rolling updates with zero downtime
• Secrets
  and other advantages.

[haecks]

Hello @uniuuu ! Thank you very much for that clarifying immediate reply and my apology for this late answer!
Sadly i am neither familiar with kubernetes-environments nor able to investigate currently that much further into it in my current busy state :-(
I have recently switched to Docker's SWARM-capabilites when setting up own projects relying on different services, at most for keeping the credentials in it's proper secrets cache and because of other benefits to networking.
As this service is currently only being tested around in one office at only one location, i will stick to the plain http-connection for now. If this is widely accepted by the staff and the custom-compiled mobile-client (s. #31 (comment)) may become available, i will get back to it then.
So far a lot of thanks for your great work!