GitHub to be trusted their own SSH host key?
Jun 11th, 2023 by miki

A May 24th 2023 change of GitHub’s RSA SSH host key just hit me on an older, less used, system doing a git clone. The change was prompted by an unintended exposure of the private part of the host key by GitHub itself.’s RSA SSH private key was briefly exposed in a public GitHub repository

Github blog-post, “We updated our RSA SSH host key”, 2023-03-23

I’m getting even more reluctant to trust a large organization where such a thing can happen. Additionally they didn’t communicate about this in a way that caught my attention. And now that it has my attention, they do not want to share the specifics about the incident to enable assessment of impact (like exposure duration, popularity of repository etc).

Below is the manual recovery with explanations of what happens and why. I don’t like the way the official blog-post instructs users to retrieving the validated host key in a non-visible way, without human eye balls doing the validation. This is the only chance in the process for injecting human trust into the validation of the host key, don’t skip it!

Consequence #1: fatal MitM warning
When an SSH server’s host key is different to a connecting client (system) compared to what the system has previously seen and acknowledged by the user to be a valid key from the intended counterpart (you do check those host key fingerprint strings, right? If not; PHComp help, WinSCP help), it obviously should and do warn the user about this. Whether the change is of malicious origin or not, is up to the user to investigate. So with OpenSSH this happens;

$ git clone
Cloning into 'name-suggestion-index'...
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
Please contact your system administrator.
Add correct host key in <~>/.ssh/known_hosts to get rid of this message.
Offending RSA key in <~>/.ssh/known_hosts:17
  remove with:
  ssh-keygen -f "<~>/.ssh/known_hosts" -R ""
RSA host key for has changed and you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Remedy #1: remove known host keys for domain (pr. OpenSSH instruction)
Remove host keys associated with domain using -R option to ssh-keygen (default for -f is the UserKnownHostsFile config keyword with default ~/.ssh/known_hosts, so no need to specify that):

$ ssh-keygen -R
# Host found: line 17
# Host found: line 18
# Host found: line 19
<~>/.ssh/known_hosts updated.
Original contents retained as <~>/.ssh/known_hosts.old

Consequence #2: non-fatal domain and IP mismatch
Now, because host key references are cached for both the domain and IP there are still mismatches caused by the key exchange choosing ECDSA host key and the key associated with the IP address is the old RSA key. This is prompted during each connection attempt:

$ git clone
Cloning into 'name-suggestion-index'...
Warning: the ECDSA host key for '' differs from the key for the IP address ''
Offending key for IP in <~>/.ssh/known_hosts:63
Matching host key in <~>/.ssh/known_hosts:72
Are you sure you want to continue connecting (yes/no)? yes
remote: Enumerating objects: 153255, done.
remote: Counting objects: 100% (177/177), done.
remote: Compressing objects: 100% (103/103), done.
remote: Total 153255 (delta 103), reused 121 (delta 74), pack-reused 153078
Receiving objects: 100% (153255/153255), 550.17 MiB | 2.39 MiB/s, done.
Resolving deltas: 100% (111423/111423), done.

Remedy #2: remove known host keys for IPs
To get rid of the mismatch warning, the known host keys associated with IP addresses needs to be removed also. Potentially any of the listed endpoint IP addresses used with an RSA host key on the system earlier will now conflict with the ECDSA from server, requiring removal:

$ ssh-keygen -R
# Host found: line 63
<~>/.ssh/known_hosts updated.
Original contents retained as <~>/.ssh/known_hosts.old

$ ssh-keygen -R
# Host found: line 161
<~>/.ssh/known_hosts updated.
Original contents retained as <~>/.ssh/known_hosts.old

After remedy
Now, the SSH client will see an unknown host key and prompt for validation. Be sure to verify the presented host key with the officially communicated one (from doc. article or API):
$ git clone
Cloning into 'name-suggestion-index'...
The authenticity of host ' (' can't be established.
ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added ',' (ECDSA) to the list of known hosts.
remote: Enumerating objects: 153152, done.
remote: Counting objects: 100% (183/183), done.
remote: Compressing objects: 100% (83/83), done.
Receiving objects: 100% (153152/153152), 550.22 MiB | 1.42 MiB/s, done.
remote: Total 153152 (delta 109), reused 158 (delta 100), pack-reused 152969
Resolving deltas: 100% (111352/111352), done.
GitHub CLI on Ubuntu LTS (16.04 & 18.04)
Mar 4th, 2020 by miki

Inspired by a John Sullivan (of FSF) tweet asking about opinions on the new and shiny MS(TM) GitHub(TM) CLI(TM) tool named “gh”(TM) I wanted to try it out on one of my Ubuntu LTS systems (16.04).

I’ve always disliked the proprietary and centralised monoculture of github, especially after that thing with MS, so I’ve mostly avoided using the service for code I produce myself. If interested in independence and decentralisation maybe you should read up on that long fabled subject? Thus I’ve never even tried the predecessor “hub”(TM) tool (main page), and exactly how it differs from this new “CLI” thing is not obviously apparent from the communications.

Nevertheless, the “gh” tool, officially dubbed  GitHub CLI, being distributed under MIT license and written in go(lang) (afterwards, found out that this is indeed also applicable to the predecessing “hub” tool) made me curious enough to go see what they’re up to. Maybe its not that sinister a plot and they really just want to improve the independent and freely available mechanisms for source code storage, distribution and maintenance (I wish)?


A fairly recent (2019-09) version of go, >=1.13, is required so some gymnastics are needed to make stuff work on older distros like Ubuntu’s LTS (with 5 years support from Canonical). Anyway, this is what I did to build a useable “gh” ELF executable (20 MiB!, se below), this is from source, no precompiled stuff, no snaps. It was done on Ubuntu 16.04 (xenial) but I’ve been somewhat careful to inspect that things ought to be good for 18.04 (bionic) too. Non-essential output from the commands is largely discarded here. See Gory Details section below for the full monty.

Add go PPA & install packages

$ sudo add-apt-repository ppa:longsleep/golang-backports
$ sudo apt update
$ sudo apt install golang
$ go version
go version go1.13.4 linux/amd64

Clone repo & build

$ cd
$ git clone .githubcli
$ cd .githubcli
$ make

Setup PATH & test

$ echo 'export PATH="$HOME/.githubcli/bin:$PATH"' >> ~/.bash_profile
$ bash --login
$ gh
Work seamlessly with GitHub from the command line.

GitHub CLI is in early stages of development, and we'd love to hear your
feedback at <>

  gh [command]

Available Commands:
  help        Help about any command
  issue       Create and view issues
  pr          Create, view, and checkout pull requests
  repo        Create, clone, fork, and view repositories

      --help              Show help for command
  -R, --repo OWNER/REPO   Select another repository using the OWNER/REPO format
      --version           Show gh version

Use "gh [command] --help" for more information about a command.
subcommand is required

Trying it out

Randomly picked one of my few github repos, it contains some js experiments on parsing NMEA formatted coordinates, rather uninterestingly empty.


user@host:~$ gh repo clone mikini/coordinates 
Notice: authentication required
Press Enter to open in your browser... 

<auth session in browser>

Authentication complete. Press Enter to continue...

Cloning into 'coordinates'...
remote: Enumerating objects: 10, done.
remote: Total 10 (delta 0), reused 0 (delta 0), pack-reused 10
Unpacking objects: 100% (10/10), done.
Checking connectivity... done.

An ordinary OAuth request was spawned for authentication on The javascript executed in the browser seemed to interact with the gh tool through a local TCP connection on localhost:45454 which was the redirection target on authorisation (URL parameter: “redirect_uri=http://localhost:45454/callback”). The socket was rightfully not bound after the authentication had succeeded but this is definitely an attack vector, albeit hopefully only for locally running processes.

Listing issues & prs

user@host:~$ cd coordinates/
user@host:~$:~/coordinates$ gh issue list

Issues for mikini/coordinates

There are no open issues
miki@vcas-miki2:~/coordinates$ gh pr view
no open pull requests found for branch "master"

Creating issues

miki@vcas-miki2:~/coordinates$ gh issue create

Creating issue in mikini/coordinates

? Title Testing gh
? Body <Received>
? What's next? Preview in browser
Opening in your browser.

miki@vcas-miki2:~/coordinates$ gh issue create

Creating issue in mikini/coordinates

? Title Testing gh tool
? Body <Received>
? What's next? [Use arrows to move, type to filter]
> Preview in browser

On the first issue creation interaction I selected “preview in browser” which opened a draft issue in the browser, and dumped me back into the shell on the command line. The second interaction actually resulted in #1.

Gory Details

PPA options

The golang-backports PPA is maintained independently by Simon Eisenmann (credits: ~longsleep, @github, and carries packages up to:

There’s also an official PPA from the golang project (~gophers) which however only carries packages up to 1.11 for both.

Full (mostly) installation dump

Add go PPA, install packages

user@host:~$ sudo add-apt-repository ppa:longsleep/golang-backports
Golang 1.8, 1.9, 1.10, 1.11, 1.12, 1.13 and 1.14 PPA for Ubuntu
 More info:
Press [ENTER] to continue or ctrl-c to cancel adding it

gpg: keyring `/tmp/tmpvnuym97a/secring.gpg' created
gpg: keyring `/tmp/tmpvnuym97a/pubring.gpg' created
gpg: requesting key 56A3D45E from hkp server
gpg: /tmp/tmpvnuym97a/trustdb.gpg: trustdb created
gpg: key 56A3D45E: public key "Launchpad PPA for Simon Eisenmann" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

user@host:~$ sudo apt update
Hit:1 xenial InRelease
Get:2 xenial-updates InRelease [109 kB] 
Get:37 xenial-security/universe DEP-11 64x64 Icons [194 kB]
Get:38 xenial-security/multiverse amd64 DEP-11 Metadata [2.468 B]
Fetched 7.614 kB in 3s (2.346 kB/s) 
Reading package lists... Done
Building dependency tree 
Reading state information... Done
419 packages can be upgraded. Run 'apt list --upgradable' to see them.

user@host:~$ apt list --upgradeable golang -a
Listing... Done
golang/xenial,xenial 2:1.13~1longsleep1+xenial all [upgradable from: 2:1.6-1ubuntu4]
golang/xenial,xenial,now 2:1.6-1ubuntu4 all [installed,upgradable to: 2:1.13~1longsleep1+xenial]


user@host:~$ sudo apt install golang
Reading package lists... Done
Building dependency tree 
Reading state information... Done
The following packages were automatically installed and are no longer required:
golang-1.6 golang-1.6-doc golang-1.6-go golang-1.6-src
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
golang-1.13 golang-1.13-doc golang-1.13-go golang-1.13-src golang-doc golang-go golang-src
Recommended packages:
The following NEW packages will be installed
golang-1.13 golang-1.13-doc golang-1.13-go golang-1.13-src
The following packages will be upgraded:
golang golang-doc golang-go golang-src
4 to upgrade, 4 to newly install, 0 to remove and 413 not to upgrade.
Need to get 60,2 MB of archives.
After this operation, 322 MB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 xenial/main amd64 golang-1.13-src amd64 1.13.4-1longsleep1+xenial [12,7 MB]
Get:2 xenial/main amd64 golang-1.13-go amd64 1.13.4-1longsleep1+xenial [44,9 MB]
Get:3 xenial/main amd64 golang-1.13-doc all 1.13.4-1longsleep1+xenial [2.542 kB]
Get:4 xenial/main amd64 golang-1.13 all 1.13.4-1longsleep1+xenial [25,2 kB]
Get:5 xenial/main amd64 golang-src amd64 2:1.13~1longsleep1+xenial [3.838 B]
Get:6 xenial/main amd64 golang-go amd64 2:1.13~1longsleep1+xenial [23,0 kB]
Get:7 xenial/main amd64 golang-doc all 2:1.13~1longsleep1+xenial [3.880 B]
Get:8 xenial/main amd64 golang all 2:1.13~1longsleep1+xenial [3.826 B]
Fetched 60,2 MB in 5s (10,8 MB/s) 
Selecting previously unselected package golang-1.13-src.
(Reading database ... 460737 files and directories currently installed.)
Preparing to unpack .../golang-1.13-src_1.13.4-1longsleep1+xenial_amd64.deb ...
Unpacking golang-1.13-src (1.13.4-1longsleep1+xenial) ...
Selecting previously unselected package golang-1.13-go.
Preparing to unpack .../golang-1.13-go_1.13.4-1longsleep1+xenial_amd64.deb ...
Unpacking golang-1.13-go (1.13.4-1longsleep1+xenial) ...
Selecting previously unselected package golang-1.13-doc.
Preparing to unpack .../golang-1.13-doc_1.13.4-1longsleep1+xenial_all.deb ...
Unpacking golang-1.13-doc (1.13.4-1longsleep1+xenial) ...
Selecting previously unselected package golang-1.13.
Preparing to unpack .../golang-1.13_1.13.4-1longsleep1+xenial_all.deb ...
Unpacking golang-1.13 (1.13.4-1longsleep1+xenial) ...
Preparing to unpack .../golang-src_2%3a1.13~1longsleep1+xenial_amd64.deb ...
Unpacking golang-src (2:1.13~1longsleep1+xenial) over (2:1.6-1ubuntu4) ...
Preparing to unpack .../golang-go_2%3a1.13~1longsleep1+xenial_amd64.deb ...
Unpacking golang-go (2:1.13~1longsleep1+xenial) over (2:1.6-1ubuntu4) ...
Preparing to unpack .../golang-doc_2%3a1.13~1longsleep1+xenial_all.deb ...
Unpacking golang-doc (2:1.13~1longsleep1+xenial) over (2:1.6-1ubuntu4) ...
Preparing to unpack .../golang_2%3a1.13~1longsleep1+xenial_all.deb ...
Unpacking golang (2:1.13~1longsleep1+xenial) over (2:1.6-1ubuntu4) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up golang-1.13-src (1.13.4-1longsleep1+xenial) ...
Setting up golang-1.13-go (1.13.4-1longsleep1+xenial) ...
Setting up golang-1.13-doc (1.13.4-1longsleep1+xenial) ...
Setting up golang-1.13 (1.13.4-1longsleep1+xenial) ...
Setting up golang-src (2:1.13~1longsleep1+xenial) ...
Setting up golang-go (2:1.13~1longsleep1+xenial) ...
Setting up golang-doc (2:1.13~1longsleep1+xenial) ...
Setting up golang (2:1.13~1longsleep1+xenial) ...

user@host:~$ go version
go version go1.13.4 linux/amd64

Clone repo & build

user@host:~$ git clone .githubcli
Cloning into '.githubcli'...
remote: Enumerating objects: 168, done.
remote: Counting objects: 100% (168/168), done.
remote: Compressing objects: 100% (100/100), done.
remote: Total 5577 (delta 93), reused 122 (delta 67), pack-reused 5409
Receiving objects: 100% (5577/5577), 7.53 MiB | 2.61 MiB/s, done.
Resolving deltas: 100% (3269/3269), done.
Checking connectivity... done.

user@host:~$ cd .githubcli
user@host:~/.githubcli (master=)$ make
go: downloading v0.0.0-20191202100458-e7afc7fbc510
go: downloading v1.1.0
go: downloading v0.1.6
go: downloading v0.0.6
go: downloading v3.0.0-20200121175148-a6ecf24a6d71
go: downloading v0.0.12
go: downloading v0.3.2
go: extracting v1.1.0
go: extracting v0.1.6
go: extracting v0.0.0-20191202100458-e7afc7fbc510
go: extracting v0.0.6
go: extracting v0.0.12
go: downloading v0.0.0-20200223170610-d5e6a3e2c0ae
go: downloading v0.0.0-20200219234226-1ad67e1f0ef4
go: downloading v0.0.0-20170206155736-9520e82c474b
go: downloading v1.9.0
go: extracting v3.0.0-20200121175148-a6ecf24a6d71
go: downloading v1.2.0
go: downloading v1.0.5
go: downloading v2.0.7
go: downloading v0.0.0-20180428030007-95032a82bc51
go: downloading v0.1.1-0.20200304134224-7e5c90143acc
go: extracting v0.0.0-20170206155736-9520e82c474b
go: extracting v1.0.5
go: extracting v1.2.0
go: downloading v2.2.8
go: downloading v2.0.0
go: downloading v0.0.4
go: extracting v0.0.0-20180428030007-95032a82bc51
go: extracting v1.9.0
go: extracting v2.0.7
go: downloading v1.7.0
go: extracting v0.1.1-0.20200304134224-7e5c90143acc
go: downloading v1.1.24
go: downloading v1.0.2
go: extracting v2.2.8
go: extracting v0.0.4
go: downloading v0.0.4
go: downloading v0.1.0
go: extracting v1.7.0
go: extracting v1.0.2
go: extracting v2.0.0
go: extracting v0.0.0-20200223170610-d5e6a3e2c0ae
go: extracting v0.1.0
go: extracting v0.0.4
go: downloading v0.0.0-20200219183655-46282727080f
go: extracting v1.1.24
go: downloading v0.7.2-0.20200304075647-34d9c7143bf5
go: downloading v0.0.8
go: downloading v0.4.0
go: downloading v2.0.1
go: extracting v0.0.0-20200219234226-1ad67e1f0ef4
go: extracting v0.0.8
go: extracting v2.0.1
go: downloading v1.0.0
go: extracting v0.7.2-0.20200304075647-34d9c7143bf5
go: extracting v1.0.0
go: extracting v0.4.0
go: downloading v1.2.0
go: downloading v1.0.3
go: downloading v0.0.0-20190703233501-fc88cf888a3f
go: downloading v0.0.0-20160403171240-cbb64ac3d964
go: extracting v0.3.2
go: extracting v0.0.0-20190703233501-fc88cf888a3f
go: extracting v0.0.0-20160403171240-cbb64ac3d964
go: extracting v0.0.0-20200219183655-46282727080f
go: extracting v1.0.3
go: extracting v1.2.0
go: finding v0.0.6
go: finding v2.0.7
go: finding v0.0.4
go: finding v0.0.0-20191202100458-e7afc7fbc510
go: finding v0.0.0-20170206155736-9520e82c474b
go: finding v3.0.0-20200121175148-a6ecf24a6d71
go: finding v1.0.5
go: finding v1.2.0
go: finding v1.1.0
go: finding v1.9.0
go: finding v0.3.2
go: finding v2.0.0
go: finding v0.1.1-0.20200304134224-7e5c90143acc
go: finding v0.0.0-20180428030007-95032a82bc51
go: finding v0.1.6
go: finding v2.2.8
go: finding v2.0.1
go: finding v0.0.12
go: finding v1.7.0
go: finding v0.0.0-20200219234226-1ad67e1f0ef4
go: finding v0.4.0
go: finding v0.0.0-20200223170610-d5e6a3e2c0ae
go: finding v1.0.0
go: finding v1.1.24
go: finding v0.7.2-0.20200304075647-34d9c7143bf5
go: finding v0.0.0-20190703233501-fc88cf888a3f
go: finding v1.2.0
go: finding v1.0.3
go: finding v0.1.0
go: finding v1.0.2
go: finding v0.0.0-20160403171240-cbb64ac3d964
go: finding v0.0.4
go: finding v0.0.8
go: finding v0.0.0-20200219183655-46282727080f
user@host:~/.githubcli (master=)$ 

Cursory inspection of exec

user@host:~/.githubcli (master=)$ ls -l bin/gh
-rwxrwxr-x 1 user user 21097487 Mar  4 18:03 bin/gh
user@host:~/.githubcli (master=)$ 

user@host:~/.githubcli (master=)$ file bin/gh
bin/gh: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, not stripped
user@host:~/.githubcli (master=)$ 

user@host:~/.githubcli (master=)$ ldd bin/gh =>  (0x00007ffdc4329000) => /lib/x86_64-linux-gnu/ (0x00007fb1018c2000) => /lib/x86_64-linux-gnu/ (0x00007fb1014f8000)
	/lib64/ (0x00007fb101adf000)
user@host:~/.githubcli (master=)$ 

user@host:~/.githubcli (master=)$ nm -D bin/gh
                 U abort
00000000004fd140 T _cgo_panic
000000000045bf10 T _cgo_topofstack
00000000004fd190 T crosscall2
                 U __errno_location
                 U __fprintf_chk
                 U fputc
                 U free
                 U freeaddrinfo
                 U fwrite
                 U gai_strerror
                 U getaddrinfo
                 U getnameinfo
                 U malloc
                 U mmap
                 U munmap
                 U nanosleep
                 U pthread_attr_destroy
                 U pthread_attr_getstacksize
                 U pthread_attr_init
                 U pthread_cond_broadcast
                 U pthread_cond_wait
                 U pthread_create
                 U pthread_detach
                 U pthread_mutex_lock
                 U pthread_mutex_unlock
                 U pthread_sigmask
                 U setenv
                 U sigaction
                 U sigaddset
                 U sigemptyset
                 U sigfillset
                 U sigismember
                 U __stack_chk_fail
                 U stderr
                 U strerror
                 U unsetenv
                 U __vfprintf_chk
user@host:~/.githubcli (master=)$ 

Setup PATH & test

user@host:~/.githubcli (master=)$ cd
user@host:~$ echo 'export PATH="$HOME/.githubcli/bin:$PATH"' >> ~/.bash_profile
user@host:~$ bash --login
user@host:~$ gh
Work seamlessly with GitHub from the command line.

GitHub CLI is in early stages of development, and we'd love to hear your
feedback at <>

  gh [command]

Available Commands:
  help        Help about any command
  issue       Create and view issues
  pr          Create, view, and checkout pull requests
  repo        Create, clone, fork, and view repositories

      --help              Show help for command
  -R, --repo OWNER/REPO   Select another repository using the OWNER/REPO format
      --version           Show gh version

Use "gh [command] --help" for more information about a command.
subcommand is required
»  Substance:WordPress   »  Style:Ahren Ahimsa
© 2023 Mikkel Kirkgaard Nielsen, contents CC BY-SA 4.0