GitHub to be trusted their own SSH host key?
Jun 11th, 2023 by miki

A May 24th 2023 change of GitHub’s RSA SSH host key just hit me on an older, less used, system doing a git clone. The change was prompted by an unintended exposure of the private part of the host key by GitHub itself.’s RSA SSH private key was briefly exposed in a public GitHub repository

Github blog-post, “We updated our RSA SSH host key”, 2023-03-23

I’m getting even more reluctant to trust a large organization where such a thing can happen. Additionally they didn’t communicate about this in a way that caught my attention. And now that it has my attention, they do not want to share the specifics about the incident to enable assessment of impact (like exposure duration, popularity of repository etc).

Below is the manual recovery with explanations of what happens and why. I don’t like the way the official blog-post instructs users to retrieving the validated host key in a non-visible way, without human eye balls doing the validation. This is the only chance in the process for injecting human trust into the validation of the host key, don’t skip it!

Consequence #1: fatal MitM warning
When an SSH server’s host key is different to a connecting client (system) compared to what the system has previously seen and acknowledged by the user to be a valid key from the intended counterpart (you do check those host key fingerprint strings, right? If not; PHComp help, WinSCP help), it obviously should and do warn the user about this. Whether the change is of malicious origin or not, is up to the user to investigate. So with OpenSSH this happens;

$ git clone
Cloning into 'name-suggestion-index'...
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
Please contact your system administrator.
Add correct host key in <~>/.ssh/known_hosts to get rid of this message.
Offending RSA key in <~>/.ssh/known_hosts:17
  remove with:
  ssh-keygen -f "<~>/.ssh/known_hosts" -R ""
RSA host key for has changed and you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Remedy #1: remove known host keys for domain (pr. OpenSSH instruction)
Remove host keys associated with domain using -R option to ssh-keygen (default for -f is the UserKnownHostsFile config keyword with default ~/.ssh/known_hosts, so no need to specify that):

$ ssh-keygen -R
# Host found: line 17
# Host found: line 18
# Host found: line 19
<~>/.ssh/known_hosts updated.
Original contents retained as <~>/.ssh/known_hosts.old

Consequence #2: non-fatal domain and IP mismatch
Now, because host key references are cached for both the domain and IP there are still mismatches caused by the key exchange choosing ECDSA host key and the key associated with the IP address is the old RSA key. This is prompted during each connection attempt:

$ git clone
Cloning into 'name-suggestion-index'...
Warning: the ECDSA host key for '' differs from the key for the IP address ''
Offending key for IP in <~>/.ssh/known_hosts:63
Matching host key in <~>/.ssh/known_hosts:72
Are you sure you want to continue connecting (yes/no)? yes
remote: Enumerating objects: 153255, done.
remote: Counting objects: 100% (177/177), done.
remote: Compressing objects: 100% (103/103), done.
remote: Total 153255 (delta 103), reused 121 (delta 74), pack-reused 153078
Receiving objects: 100% (153255/153255), 550.17 MiB | 2.39 MiB/s, done.
Resolving deltas: 100% (111423/111423), done.

Remedy #2: remove known host keys for IPs
To get rid of the mismatch warning, the known host keys associated with IP addresses needs to be removed also. Potentially any of the listed endpoint IP addresses used with an RSA host key on the system earlier will now conflict with the ECDSA from server, requiring removal:

$ ssh-keygen -R
# Host found: line 63
<~>/.ssh/known_hosts updated.
Original contents retained as <~>/.ssh/known_hosts.old

$ ssh-keygen -R
# Host found: line 161
<~>/.ssh/known_hosts updated.
Original contents retained as <~>/.ssh/known_hosts.old

After remedy
Now, the SSH client will see an unknown host key and prompt for validation. Be sure to verify the presented host key with the officially communicated one (from doc. article or API):
$ git clone
Cloning into 'name-suggestion-index'...
The authenticity of host ' (' can't be established.
ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added ',' (ECDSA) to the list of known hosts.
remote: Enumerating objects: 153152, done.
remote: Counting objects: 100% (183/183), done.
remote: Compressing objects: 100% (83/83), done.
Receiving objects: 100% (153152/153152), 550.22 MiB | 1.42 MiB/s, done.
remote: Total 153152 (delta 109), reused 158 (delta 100), pack-reused 152969
Resolving deltas: 100% (111352/111352), done.
Silent browser redirect on invalid certificate
Nov 15th, 2018 by miki

Hit an odd browser behaviour today.

Trawling through some of those nice and dandy terms for a service (no I won’t tell) I followed a link and suddenly hit an “insecure connection” message from Firefox.

Examining the certificate using the usual “openssl s_client” and “openssl x509” tools surely enough revealed that the served certificate didn’t include the second-level but only on the third-level www sub-domain. Strangely enough I discovered that when entering the same URL directly into the address bar of Firefox the connection was somehow redirected to the www sub-domain and loaded fine without any complaints from Firefox.

Looking into what was happening on the wire using wget directed to not check the certificate (–no-check-certificate) and displaying server responses (–server-response/-S) revealed that the server behind the misconfigured certificate was aiming to issue a HTTP 301 redirect to the valid www sub-domain of the site (actual site replaced by and localhost ip);

$ wget –server-response –no-check-certificate –output-document=/dev/null
–2018-11-15 18:59:14–
Resolving (…
Connecting to (||:443… connected.
WARNING: no certificate subject alternative name matches
requested host name ‘’.
HTTP request sent, awaiting response…
HTTP/1.1 301 Moved Permanently
Content-Type: text/html; charset=UTF-8
Date: Thu, 15 Nov 2018 17:59:13 GMT
Content-Length: 152
Location: [following]

Apparently Firefox’s behaviour in this scenario differs depending on the method in which the user is supplying the URL, in some cases silently ignoring the TLS/SSL warning. A little more experimentation revealed that it was actually not the source of the URL, link followed versus one entered in address bar, that made any difference. Instead it is the presence of a trailing slash on the URL that triggers the silent suppression of the serious red flag that the second-level domain asked to be fetched is not present in the served certificate. I was fooled by the fact that when entering URLs in the address bar Firefox suggests ending the URL on a slash (‘/’) if it is not already present. Manually removing this when editing in the address bar also makes Firefox display the security warning. Alas, the link I originally followed was also missing the trailing slash and behaved as expected by throwing me into a security warning.

The whole “feature” of silently ignoring a security issue seemed very odd to me, but a bit of searching revealed that this was apparently championed by Google Chrome a couple of years ago. It is described in this servertastic post which also directs to a discussion on Twitter, with some key points replicated below, about its presence in Chrome and confirmation from a Chrome team member that the behaviour is intended.

The thread ends with a guy who had dug up the source code of the feature in Chromium (on which Chrome is based) known as “SSLCommonNameMismatchHandling” in file browser/ssl/ (see all mentions across code base).

This has obviously also trickled down into Firefox, however, not much mention of that is to be found. Not even traces of it by some quick searches of the mozilla-central codebase. Some day I promise (really!) to dig through all source of Mozilla and hunt down the implementation, but for now I’ll revert to a bit of practical experimentation showing the behaviour in the different browsers and operating systems I happen to have access to at the moment;

  • Ubuntu 16.04
    • Firefox 62.0.3: warning, with trailing ‘/’: No warning
    • Chromium 69.0.3497.81: no warning, with trailing ‘/’: no warning
  • Windows Server 2016
    • Firefox 62.0.2: warning, with trailing ‘/’: no warning
    • Google Chrome 70.3538.102: no warning, with trailing ‘/’: no warning
    • Internet Explorer 11.2580.14393.0: warning, with trailing ‘/’: warning
  • Windows Server 2008
    • Firefox 59.0.3: warning, with trailing ‘/’: warning
    • Google Chrome 70.3538.102: no warning, with trailing ‘/’: no warning
    • Internet Explorer 11.0.9600.19080: warning, with trailing ‘/’: warning

So somewhere between 59.0.3 and 62.0.2 Firefox also implemented a policy of silently accepting invalid certificates when certain non-obvious criteria is met (is the redirect actually followed and certs checked, or is “www.” just prefixed?), but this happens only when the URL ends on a slash. Go figure…

»  Substance:WordPress   »  Style:Ahren Ahimsa
© 2023 Mikkel Kirkgaard Nielsen, contents CC BY-SA 4.0