TLS verification issue with remote Docker Engine when pulling an image

I have set up a private local Docker engine to run on an Ubuntu VM. It is loaded with the certificates needed to enable TLS. From what I can tell it is working perfectly.

This is the status output of the docker engine running on the VM:

ubuntu@worker-docker-node:~$ sudo systemctl status docker● docker.service - Docker Application Container Engine     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)    Drop-In: /etc/systemd/system/docker.service.d└─override.conf     Active: active (running) since Tue 2024-03-05 08:20:49 SAST; 1 day 3h agoTriggeredBy: ● docker.socket       Docs: https://docs.docker.com   Main PID: 24598 (dockerd)      Tasks: 9     Memory: 238.2M        CPU: 12.925s     CGroup: /system.slice/docker.service└─24598 /usr/bin/dockerd -D -H unix:///var/run/docker.sock --tlsverify --tlscert=/home/ubuntu/.docker/server-cert.pem --tlscacert=/home/ubuntu/.docker/ca.pem --tlskey=/home/ubuntu/.docker/server-key.pem -H tcp://0.0.0.0:2376

Using the client certificates, I can connect to it using openssl using the following command:

openssl s_client -connect worker-docker-node:2376 -cert cert.pem -key key.pem -CAfile ca.pem

When making use of the client certificates and the docker commands in the CLI, I can connect, list the images, and even create a new image successfully when using the remote Docker Engine.

The problem comes in when trying to pull images, it just refuses to.The server (Docker Engine) complains about not receiving the client certificate during the TLS handshakeThis happens when trying to pull on both my Ubuntu VM and Windows Machine.

Docker Engine logs after trying to pull:

Mar 06 11:53:23 worker-docker-node dockerd[24598]: time="2024-03-06T11:53:23.541141069+02:00" level=debug msg="Calling HEAD /_ping"Mar 06 11:53:23 worker-docker-node dockerd[24598]: time="2024-03-06T11:53:23.542081072+02:00" level=debug msg="Calling POST /v1.42/images/create?fromImage=worker-docker-node%3A2376%2Ftestimage&tag=2.4.0.1001"Mar 06 11:53:23 worker-docker-node dockerd[24598]: time="2024-03-06T11:53:23.544068477+02:00" level=debug msg="Trying to pull worker-docker-node:2376/testimage from https://worker-docker-node:2376"Mar 06 11:53:23 worker-docker-node dockerd[24598]: 2024/03/06 11:53:23 http: TLS handshake error from 127.0.0.1:33794: tls: client didn't provide a certificateMar 06 11:53:23 worker-docker-node dockerd[24598]: time="2024-03-06T11:53:23.546664284+02:00" level=warning msg="Error getting v2 registry: Get \"https://worker-docker-node:2376/v2/\": remote error: tls: bad certificate"Mar 06 11:53:23 worker-docker-node dockerd[24598]: time="2024-03-06T11:53:23.546676984+02:00" level=info msg="Attempting next endpoint for pull after error: Get \"https://worker-docker-node:2376/v2/\": remoteerror: tls: bad certificate"Mar 06 11:53:23 worker-docker-node dockerd[24598]: time="2024-03-06T11:53:23.546684584+02:00" level=debug msg="Trying to pull worker-docker-node:2376/testimage from http://worker-docker-node:2376"Mar 06 11:53:23 worker-docker-node dockerd[24598]: time="2024-03-06T11:53:23.548224588+02:00" level=info msg="Attempting next endpoint for pull after error: read tcp 127.0.0.1:33818->127.0.1.1:2376: read: connection reset by peer"Mar 06 11:53:23 worker-docker-node dockerd[24598]: time="2024-03-06T11:53:23.549488492+02:00" level=error msg="Handler for POST /v1.42/images/create returned error: read tcp 127.0.0.1:33818->127.0.1.1:2376: read: connection reset by peer"

The pull cli command I am using is:

docker --tlsverify --tlscacert=ca.pem --tlscert=cert.pem --tlskey=key.pem -H=worker-docker-node:2376 image pull worker-docker-node:2376/testimage:2.4.0.1001

I have also tried just placing the certificates into the .docker/ directory and not specifying them within the cli command. It finds the certificates but it still results in the same error.

I have also set up a few Java tests that use the Java Docker Client SDK to perform a few actions to just test the connection. When making use of the client certificates from the Java code, I can list the images out and even create a new image successfully, but is also can't pull.

My first thought would be that there is something wrong with the certificates, but they seem to work when connecting using docker and openssl, so it can't be that. Right?

In the background, I guess Docker is handling these TLS connection verifications differently depending on the call.

Can someone please assist me with this, I can't think of anything else to try.