Jenkins SSH failing to authenticate with private key after server upgrade

I have an instance of Jenkins running on docker on an Ubuntu server. This Jenkins instance runs jobs using ssh scripts ("Execute shell script on remote houst using ssh") connecting to the same Ubuntu server. The authentication is done via private key and it has been working for over a year without issues.

Recently, I upgraded my Ubuntu version (do-release-upgrade) - after which all the scripts involving the server started failing authentication.

Error message:

[SSH] executing...[SSH] Exception:Auth failcom.jcraft.jsch.JSchException: Auth fail

I am not certain why, but I have been doing some troubleshooting:

1. There are a few other SSH scripts connecting to other machines and those are still working fine. This is not related to SSH plugin or Jenkins itself, but the server to which it connects.

2. I had two users configured with private keys for this server (say <user> and root) and both stopped working. As such, it doesn't seem like it has something to do with the specific user, but with the server.

3. I tried with a simple echo "hello" and it's the same issue. This confirms that the scripts are not the issue - and they shouldn't as they have not been changed.

4. If I change the auth method from private key to username and password, the system connects fine. As such, it doesn't feel like either machine, or proxies are blocking it. And again, no changes here on any set up other than upgrading Ubuntu. This should discard many basics like making sure the IP/port is correct, etc.

5. I tried recreating the keys, and the result is the same. I've used a few variants to create them (see below "Key variants" for more details). When I do this, I am copying the contents of the private key to Jenkins; and I have tried generating the keys with and without passphrase. Do note that I followed the same process that I followed last time when I created the keys and it has been working for all this time (I documented it so I wouldn't forget).

6. To be safe, I tried to completely remove the ~/.ssh/ folder and recreating it too just in case there was some issues.

7. If I try to login to the server using the private key from my Macbook, the connection works fine (ssh -i key.pem <user>@<ip> -p <port>) where key.pem is the same contents that I am pasting to the Jenkins credentials. This makes me think that the key is properly generated.

8. One thing I had to do on my Macbook after the upgrade was removing the old reference to the server from known_hosts. However, I was not able to find the same on the Jenkins config anywhere - and the fact that it was able to connect with username and password makes me think that there's nothing preventing the connection, but that Jenkins is not able to use the private key properly.

9. I know some of the files on the server changed after the upgrade, including /etc/ssh/sshd_config. I had to revert some of the settings, including changing PasswordAuthentication no to PasswordAuthentication yes - which I believe was the setup I had before. (See below contents of my sshd_config with some parts masked).

10. I have also tried updating Jenkins and its plugins, you never know.

While using username and password works, I do want to make sure this was with the private key as it feels like a safer approach and easier to manager different user access and access level. Any thoughts on how to solve it or what to try next? :)

Thanks in advance!

Appendix

Key variants

Generic

# Generate keysmkdir ~/.ssh; cd ~/.ssh/ && ssh-keygen -t rsa -m PEM -C "Jenkins agent key" -f "jenkinsAgent_rsa"# Authorize keyscat jenkinsAgent_rsa.pub >> ~/.ssh/authorized_keyschmod 700 ~/.sshchmod 600 ~/.ssh/authorized_keys ~/.ssh/jenkinsAgent_rsa# Get private key for Jenkinscat ~/.ssh/jenkinsAgent_rsa

With root access

# Run before the other commandssudo su $AGENT_USER

or

# Run before the other commandssudo su

Different keygen

ssh-keygen -t rsa -b 4096 -m PEM -C "Jenkins agent key" -f "jenkinsAgent_rsa"

ssh-keygen -t rsa -C "Jenkins agent key" -f "jenkinsAgent_rsa"

Generated key (format, content stripped)

This is what I copy over to Jenkins.

-----BEGIN RSA PRIVATE KEY-----Proc-Type: 4,ENCRYPTEDDEK-Info: AES-128-CBC,518C612F99A868<somecharsremoved>09C9A6047E20sDqB0yOas0sf4dJRLvb8IxyOvoIwr8Ls3uMbugwTOJ/(...)MuZyUzWXnhUaWgnXJlTLhnmXiJ9XgUClYftQpP4RZw9Ult/dje5XD81RmwwhuxUV-----END RSA PRIVATE KEY-----

sshd_config (removing all comments)

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/gamesInclude /etc/ssh/sshd_config.d/*.confPort <port># This used to be `ChallengeResponseAuthentication no` but I assume no issues.KbdInteractiveAuthentication noUsePAM yesX11Forwarding yesPrintMotd noAcceptEnv LANG LC_*Subsystem       sftp    /usr/lib/openssh/sftp-serverPasswordAuthentication yes