A May 24th 2023 change of GitHub’s RSA SSH host key just hit me on an older, less used, system doing a git clone. The change was prompted by an unintended exposure of the private part of the host key by GitHub itself.

GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository

Github blog-post, “We updated our RSA SSH host key”, 2023-03-23

I’m getting even more reluctant to trust a large organization where such a thing can happen. Additionally they didn’t communicate about this in a way that caught my attention. And now that it has my attention, they do not want to share the specifics about the incident to enable assessment of impact (like exposure duration, popularity of repository etc).

Below is the manual recovery with explanations of what happens and why. I don’t like the way the official blog-post instructs users to retrieving the validated host key in a non-visible way, without human eye balls doing the validation. This is the only chance in the process for injecting human trust into the validation of the host key, don’t skip it!

Consequence #1: fatal MitM warning
When an SSH server’s host key is different to a connecting client (system) compared to what the system has previously seen and acknowledged by the user to be a valid key from the intended counterpart (you do check those host key fingerprint strings, right? If not; PHComp help, WinSCP help), it obviously should and do warn the user about this. Whether the change is of malicious origin or not, is up to the user to investigate. So with OpenSSH this happens;

$ git clone git@github.com:mikini/name-suggestion-index
Cloning into 'name-suggestion-index'...
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s.
Please contact your system administrator.
Add correct host key in <~>/.ssh/known_hosts to get rid of this message.
Offending RSA key in <~>/.ssh/known_hosts:17
  remove with:
  ssh-keygen -f "<~>/.ssh/known_hosts" -R "github.com"
RSA host key for github.com has changed and you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
$ 

Remedy #1: remove known host keys for domain (pr. OpenSSH instruction)
Remove host keys associated with github.com domain using -R option to ssh-keygen (default for -f is the UserKnownHostsFile config keyword with default ~/.ssh/known_hosts, so no need to specify that):

$ ssh-keygen -R github.com
# Host github.com found: line 17
# Host github.com found: line 18
# Host github.com found: line 19
<~>/.ssh/known_hosts updated.
Original contents retained as <~>/.ssh/known_hosts.old
$

Consequence #2: non-fatal domain and IP mismatch
Now, because host key references are cached for both the domain and IP there are still mismatches caused by the key exchange choosing ECDSA host key and the key associated with the IP address is the old RSA key. This is prompted during each connection attempt:

$ git clone git@github.com:mikini/name-suggestion-index
Cloning into 'name-suggestion-index'...
Warning: the ECDSA host key for 'github.com' differs from the key for the IP address '140.82.121.3'
Offending key for IP in <~>/.ssh/known_hosts:63
Matching host key in <~>/.ssh/known_hosts:72
Are you sure you want to continue connecting (yes/no)? yes
remote: Enumerating objects: 153255, done.
remote: Counting objects: 100% (177/177), done.
remote: Compressing objects: 100% (103/103), done.
remote: Total 153255 (delta 103), reused 121 (delta 74), pack-reused 153078
Receiving objects: 100% (153255/153255), 550.17 MiB | 2.39 MiB/s, done.
Resolving deltas: 100% (111423/111423), done.
$

Remedy #2: remove known host keys for IPs
To get rid of the mismatch warning, the known host keys associated with IP addresses needs to be removed also. Potentially any of the listed endpoint IP addresses used with an RSA host key on the system earlier will now conflict with the ECDSA from server, requiring removal:

$ ssh-keygen -R 140.82.121.3
# Host 140.82.121.3 found: line 63
<~>/.ssh/known_hosts updated.
Original contents retained as <~>/.ssh/known_hosts.old
$

$ ssh-keygen -R 140.82.121.4
# Host 140.82.121.4 found: line 161
<~>/.ssh/known_hosts updated.
Original contents retained as <~>/.ssh/known_hosts.old
$

After remedy
Now, the SSH client will see an unknown host key and prompt for validation. Be sure to verify the presented host key with the officially communicated one (from doc. article or API):

$ git clone git@github.com:mikini/name-suggestion-index
Cloning into 'name-suggestion-index'...
The authenticity of host 'github.com (140.82.121.4)' can't be established.
ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'github.com,140.82.121.4' (ECDSA) to the list of known hosts.
remote: Enumerating objects: 153152, done.
remote: Counting objects: 100% (183/183), done.
remote: Compressing objects: 100% (83/83), done.
Receiving objects: 100% (153152/153152), 550.22 MiB | 1.42 MiB/s, done.
remote: Total 153152 (delta 109), reused 158 (delta 100), pack-reused 152969
Resolving deltas: 100% (111352/111352), done.
$