CR: Support for hostname, adding CA, https in dockert compose setup.

[bradleyharden]

Hi,

I'm using the current production branch (2.7.0), but I can't seem to get file syncing to work. I can sync entries in my library, but not files. In particular, I get a series of errors like this in the Zotero client:

[JavaScript Error: "S3 returned 0 (1/3FCKK2HV) -- retrying upload"]

The client also never stops trying to sync. I can see repeated POST requests to the items path in the dataserver logs, but when I open the MinIO browser, the zotero and zotero-fulltext buckets are completely empty.

Interestingly, like @aitianshi, I also noticed that the .env file S3HOST points to 10.5.5.1 instead of 10.5.5.7. However, I tried switching the IP to 10.5.5.7 as well as minio, but neither seems to affect the behavior.

Finally, if I change the S3HOST to my-vm-url:9000 and expose port 9000 of the MinIO container, I get a different behavior. With that setup, I do actually see files in the MinIO bucket. But I am only ever able to sync exactly two files: no more, no less. After that, the dataserver starts issuing a series of curl error 7: couldn't connect to server errors, and I get 500 errors in the Zotero client.

Am I doing something wrong? Should file syncing work? Or is it not supported yet?

[uniuuu]

Hi @bradleyharden

[JavaScript Error: "S3 returned 0 (1/3FCKK2HV) -- retrying upload"]

Finally, if I change the S3HOST to my-vm-url:9000

That's mean you're using VM setup where your server is on computer/instance different from client.
Ensure that during build of both client and server you're using configuration for VM and not Localhost.

Interestingly, like @aitianshi, I also noticed that the .env file S3HOST points to 10.5.5.1 instead of 10.5.5.7. However, I tried switching the IP to 10.5.5.7 as well as minio, but neither seems to affect the behavior.

This is correct configuration 10.5.5.1 an not 10.5.5.7.

May I know do you use hostname for your VM as my-vm-url ? If you use hostname then it may not work if one of your internal DNS resolvers isn't resolving it.
The setup will work if you use VM IP address for both server and client configuration unless again you are using localhost setup scenario. Refer to https://github.com/uniuuu/zotprime#localhost-and-vm-installation
URL's are supported for kubernetes setup currently.

Please let me know which one you use localhost or VM? And then try the above approach and if you still have issue then I'll request logs and more details about how the setup done.

[bradleyharden]

@uniuuu, sorry, I'll try to be more explicit.

That's mean you're using VM setup where your server is on computer/instance different from client.
Ensure that during build of both client and server you're using configuration for VM and not Localhost.

Yes, I am running in a VM setup. And yes, I made sure to build both the client and the server using the VM hostname.

This is correct configuration 10.5.5.1 an not 10.5.5.7.

Ok. Thanks for the clarification. I thought it must be correct, because I looked at the Git blame for that line, and the change looked intentional.

May I know do you use hostname for your VM as my-vm-url? If you use hostname then it may not work if one of your internal DNS resolvers isn't resolving it.

Yes, I am using the hostname. The VM is running on an internal, private network. I usually don't have any problem with DNS over that network. For example, I can use my laptop to access the MinIO console at my-vm-url:9001. Unless there is something subtle going on that I don't understand, I don't think DNS is the problem here. The Zotero client is able to connect to both the dataserver and the stream-server. I can see evidence of both connections in the logs. However, just to be sure, I will try using the IP of the VM instead.

Please let me know which one you use localhost or VM? And then try the above approach and if you still have issue then I'll request logs and more details about how the setup done.

I think I've done everything correctly. I tried to follow the instructions as closely as possible. Thank you in advance for any additional help you can provide.

Oh, I should add a few more details, since they might be relevant. I am trying to run this on an internal network, and we have our own root certificate. I added that certificate into the dataserver, stream-server, tinymce-clean-server and the zotero-client Dockerfiles. I used the docker-compose-dev.yml file, so all the containers were built locally. Is that enough? Do any of the other Dockerfiles need to be modified?

For those 4 Dockerfiles, I had to use two slightly different approaches, depending on whether I could use wget and https without installing any additional packages. For the zotero-client and the dataserver, I added this:

RUN mkdir -p /usr/local/share/ca-certificates/ \
 && wget https://internal-url/cert.cer -O /usr/local/share/ca-certificates/cert.crt \
 && cat /usr/local/share/ca-certificates/cert.crt >> /etc/ssl/certs/ca-certificates.crt

And for the stream-server and tinymce-clean-server , I added this

COPY cert.crt /usr/local/share/ca-certificates/cert.crt
RUN mkdir -p /etc/ssl/certs \
 && touch /etc/ssl/certs/ca-certificates.crt \
 && cat /usr/local/share/ca-certificates/cert.crt >> /etc/ssl/certs/ca-certificates.crt

I also add

RUN npm config set cafile /etc/ssl/certs/ca-certificates.crt

to any Dockerfiles running npm.

[uniuuu]

Hi @bradleyharden
Please try to switch to IP address and not hostname (or URL).
Hostname doesn't work because DNS resolvers do not resolve it inside docker container. At this point it's not tested and not supported to use hostname.
Moreover if there is a need to add hostname support for docker compose setup then the best approach would be at add reverse proxy container to be able handle both URL's or/and hostnames plus a bonus will be SSL termination.

For URL's I would recommend to use on-premise Kubernetes (Microk8s) instead of docker compose. I'll write a guide for it and let you know when it's added.

Yes, I am running in a VM setup. And yes, I made sure to build both the client and the server using the VM hostname.

Yes, I am using the hostname. The VM is running on an internal, private network. I usually don't have any problem with DNS over that network. For example, I can use my laptop to access the MinIO console at my-vm-url:9001. Unless there is something subtle going on that I don't understand, I don't think DNS is the problem here. The Zotero client is able to connect to both the dataserver and the stream-server. I can see evidence of both connections in the logs. However, just to be sure, I will try using the IP of the VM instead.

[bradleyharden]

Hi @uniuuu,

I still see the same behavior when using an IP address.

To eliminate any possibility that the internal network or Dockerfile changes have any effect, I tried to reproduce the problem using the localhost setup a local laptop connected to the open internet.

Here's what I did:

• Cloned a fresh copy of the repo
• Built the client with DOCKER_BUILDKIT=1 sudo docker build --progress=plain --file client.Dockerfile --output build .
  • I verified that the default values for the ARGs were for the localhost configuration on Linux, so I didn't supply any --build-args.
• Copied .env_example to .env and docker-compose-dev.yml to docker-compose.yml
  • Once again, I checked that all the default values were for the localhost configuration.
• Built the images using sudo docker compose build
• Ran sudo docker compose up

At this point, I found that the minio/mc container was giving me errors with curl. I guessed that something had changed in the past few days, so I pinned the miniomc.Dockerfile to minio/mc:RELEASE.2023-10-14T01-57-03Z. Then I re-ran sudo docker compose build.

Then, I tried again:

• I brought the containers down with sudo docker compose down
• I deleted all volumes on the system with sudo docker volume rm ...
• I brought the containers back up with sudo docker compose up
• I ran the bin/init.sh script
• I opened the Zotero client and imported my existing library from an RDF file
• I logged in to sync with the default admin admin login.
• I changed the setting to sync manually
• I pressed the sync button
• I opened the error console

At this point, I still see the same behavior. I get S3 returned 0 errors in the console. I see POST requests to the dataserver. But when I log in to MinIO through localhost:9001, I don't see any files in either of the two buckets.

Am I doing something wrong here?

[aitianshi]

Except for the preprint issue we have found @uniuuu I still have the issue @bradleyharden is explaining for other docs if I don't change Minio IP. There's something been missing with all default values using the whole Zotero in local.

[uniuuu]

To eliminate any possibility that the internal network or Dockerfile changes have any effect, I tried to reproduce the problem using the localhost setup a local laptop connected to the open internet.

* Copied `.env_example` to `.env` and `docker-compose-dev.yml` to `docker-compose.yml`

Hi @bradleyharden @aitianshi
You're good to go with IP address, no need to use localhost unless your client and server on the same single machine without VM. If you're still having the error after below described step then we'll troubleshoot it with IP address installation first.
Please try to use prod yaml file and not dev as per the guide in readme. And there is no build step for server too but instead it will pull prod images from dockerhub. VM/Localhost installation
$ cp docker-compose-prod.yml docker-compose.yml

* Cloned a fresh copy of the repo

* Built the client with `DOCKER_BUILDKIT=1 sudo docker build --progress=plain --file client.Dockerfile --output build .`

Another step you may miss is to switch to production branch as per guide.

$ git clone --recursive https://github.com/uniuuu/zotprime.git
$ git checkout production   

[bradleyharden]

@uniuuu,

• Yes, I am already using the production branch, which points to v2.7.0.
• I would prefer, if possible, to build all the images myself. That's why I chose docker-compose-dev.yml. Is that not a valid choice? If -prod works but -dev does not, isn't that a bug?
• It is easier for me to reproduce your work if I use the localhost configuration, because I can do that on a single laptop connected to the open internet. My VM is on an internal network and all traffic is re-signed with our root certificate. If I want to test with the VM, I need to modify the images to include our root certificate.

Either way, I will try using the -prod file.

[aitianshi]

@bradleyharden If you can import some files from the outside to your internal network, you can try the solution there: #12. I had the same issue but I could build everything from a public network then import all images to my private network, suggested by @uniuuu .

[bradleyharden]

@aitianshi, yes, I saw that suggestion in your issue. It could work for me, but it's also pretty easy for me to modify the Dockerfiles and rebuild. I would prefer that, if possible, so that I have a record and a way to reproduce things without manual intervention.

[uniuuu]

@uniuuu,

* Yes, I am already using the `production` branch, which points to `v2.7.0`.

* I would prefer, if possible, to build all the images myself. That's why I chose `docker-compose-dev.yml`. Is that not a valid choice? If `-prod` works but `-dev` does not, isn't that a bug?

* It is easier for me to reproduce your work if I use the `localhost` configuration, because I can do that on a single laptop connected to the open internet. My VM is on an internal network and all traffic is re-signed with our root certificate. If I want to test with the VM, I need to modify the images to include our root certificate.

Either way, I will try using the -prod file.

Hi @bradleyharden

The pointed out issue that dev on your side doesn't work - is still okay as it's meant for development purpose and not production level deployment.

That's understood that you want to build images. But the images you'll build won't guarantee their consistency unless at your side all is working, and no issues with getting dependencies. So you have to keep control over your build and ensure it wasn't impacted by your environment setup or limits. Anyway it's best practice to use prebuild and verified images. So i have made changes to production branch to avoid confusion by having dev docker-compose there.

I am open to troubleshoot the image build failure. But for this you have provide details what do you modify in images exactly? And then also do provide output saved in file of the progress build of the images which has to be done by next command:
BUILDKIT_PROGRESS=plain docker compose build

Also I have added microk8s guide microk8s-installation and I encourage to switch from docker to k8s for better server handling experience.