A May 24th 2023 change of GitHub’s RSA SSH host key just hit me on an older, less used, system doing a git clone. The change was prompted by an unintended exposure of the private part of the host key by GitHub itself.
GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository Github blog-post, “We updated our RSA SSH host key”, 2023-03-23
GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository
I’m getting even more reluctant to trust a large organization where such a thing can happen. Additionally they didn’t communicate about this in a way that caught my attention. And now that it has my attention, they do not want to share the specifics about the incident to enable assessment of impact (like exposure duration, popularity of repository etc).
Below is the manual recovery with explanations of what happens and why. I don’t like the way the official blog-post instructs users to retrieving the validated host key in a non-visible way, without human eye balls doing the validation. This is the only chance in the process for injecting human trust into the validation of the host key, don’t skip it!
Consequence #1: fatal MitM warningWhen an SSH server’s host key is different to a connecting client (system) compared to what the system has previously seen and acknowledged by the user to be a valid key from the intended counterpart (you do check those host key fingerprint strings, right? If not; PHComp help, WinSCP help), it obviously should and do warn the user about this. Whether the change is of malicious origin or not, is up to the user to investigate. So with OpenSSH this happens;
$ git clone git@github.com:mikini/name-suggestion-index Cloning into 'name-suggestion-index'... @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the RSA key sent by the remote host is SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s. Please contact your system administrator. Add correct host key in <~>/.ssh/known_hosts to get rid of this message. Offending RSA key in <~>/.ssh/known_hosts:17 remove with: ssh-keygen -f "<~>/.ssh/known_hosts" -R "github.com" RSA host key for github.com has changed and you have requested strict checking. Host key verification failed. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. $
Remedy #1: remove known host keys for domain (pr. OpenSSH instruction)Remove host keys associated with github.com domain using -R option to ssh-keygen (default for -f is the UserKnownHostsFile config keyword with default ~/.ssh/known_hosts, so no need to specify that):
$ ssh-keygen -R github.com# Host github.com found: line 17# Host github.com found: line 18# Host github.com found: line 19<~>/.ssh/known_hosts updated.Original contents retained as <~>/.ssh/known_hosts.old$Consequence #2: non-fatal domain and IP mismatchNow, because host key references are cached for both the domain and IP there are still mismatches caused by the key exchange choosing ECDSA host key and the key associated with the IP address is the old RSA key. This is prompted during each connection attempt:
$ git clone git@github.com:mikini/name-suggestion-index Cloning into 'name-suggestion-index'... Warning: the ECDSA host key for 'github.com' differs from the key for the IP address '140.82.121.3' Offending key for IP in <~>/.ssh/known_hosts:63 Matching host key in <~>/.ssh/known_hosts:72 Are you sure you want to continue connecting (yes/no)? yes remote: Enumerating objects: 153255, done. remote: Counting objects: 100% (177/177), done. remote: Compressing objects: 100% (103/103), done. remote: Total 153255 (delta 103), reused 121 (delta 74), pack-reused 153078 Receiving objects: 100% (153255/153255), 550.17 MiB | 2.39 MiB/s, done. Resolving deltas: 100% (111423/111423), done. $
Remedy #2: remove known host keys for IPsTo get rid of the mismatch warning, the known host keys associated with IP addresses needs to be removed also. Potentially any of the listed endpoint IP addresses used with an RSA host key on the system earlier will now conflict with the ECDSA from server, requiring removal:
$ ssh-keygen -R 140.82.121.3 # Host 140.82.121.3 found: line 63 <~>/.ssh/known_hosts updated. Original contents retained as <~>/.ssh/known_hosts.old $ $ ssh-keygen -R 140.82.121.4 # Host 140.82.121.4 found: line 161 <~>/.ssh/known_hosts updated. Original contents retained as <~>/.ssh/known_hosts.old $ After remedy Now, the SSH client will see an unknown host key and prompt for validation. Be sure to verify the presented host key with the officially communicated one (from doc. article or API):
$ git clone git@github.com:mikini/name-suggestion-index Cloning into 'name-suggestion-index'... The authenticity of host 'github.com (140.82.121.4)' can't be established. ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added 'github.com,140.82.121.4' (ECDSA) to the list of known hosts. remote: Enumerating objects: 153152, done. remote: Counting objects: 100% (183/183), done. remote: Compressing objects: 100% (83/83), done. Receiving objects: 100% (153152/153152), 550.22 MiB | 1.42 MiB/s, done. remote: Total 153152 (delta 109), reused 158 (delta 100), pack-reused 152969 Resolving deltas: 100% (111352/111352), done. $
I took the chance on a summer day last year, 2019-06-20, to take a peek at the construction site of the Norwegian Bulk Infrastructure data center DK01 Campus being built in Kjersing, Esbjerg, Denmark. The pictures were stowed away until now but I think they deserve to be set free, so here goes.
The data center is a part of Bulk Infrastructure’s involvement in the Havfrue/AEC-2 subsea cable system (link to a previous blog post with details), built in cooperation with Google and Facebook, which is going to land on the Western shore of Jutland in the near future (ready for service expected in 2020-Q3). Bulk Infrastructure is going to build and operate an extension of the main cable trunk (with reduced capacity) to Norway and its datacenters present there.
It seems the DK01 Campus data center is going to act as an exchange point between other fiber networks Bulk is involved in and also landing in Esbjerg;
Havfrue
Havhingsten
Havsil
The location in Esbjerg is indicated by the orange area outline on the map below, courtesy of OpenStreetMap.
Arriving to the area from the highway driving along the Kjersing Ringvej the site is partly visible at your left hand.
Taking the 3rd exit in the roundabout onto Guldborgsundvej and turning the first left corner the site is just in front of you on the right.
Getting close the inner construction work is visible through the still open facade.
Stepping out and taking a snapshot closer to the fence.
Walking around the end of the building. Small compartments are visible.
At the other side there’s some foundation extending from the tall white wall barely visible. It is probably going to have lighter walls erected. Could be administration offices, where the high ceiling room with walls already standing is the main data center hall.
A lot of temporary arrangements on site for the construction period and site protection.
For the guests, like me, there is even a nice information board with outline map showing some details. As anticipated, offices on left side of the data center hall (right side of the building in the yellow marking, map is facing North, most pictures taken South-West). And also smaller rooms in the hall itself in the Northerne end of the building that we saw above. This is probably to be able to segment co-located equipment for restricting access.
Came by Esbjerg Harbour on September 24th 2019 and saw what was obviously a cable ship docked at the quay. A giant ship and I immediately thought that the mermaid might be closing in on Jutland. Some quick drive-by pictures and vessel details below:
New sighting on 2019-11-03:
JV article about ship and ongoing upgrades causing noise.
Tænk hvor meget trafik, processerings- og ventetid der ville være sparet hvis nogen gad at overveje hvad billeder på en hjemmeside anvendes til og tilpasse de transmitterede data derefter!
Her et enkelt sølle eksempel af mange observerede (ingen særlig grund til at klandre den specifikke afsender her, blev bare trigget af at klikke på et link i et nyhedsbrev just modtaget) fra SDFE‘s nyhedsoversigtsside, hvor det er så åbenlyst tydeligt på billedindlæsningshastigheden at billederne er alt for meget større end nødvendigt.
Inspicering i browseren (f.eks. med Firefox dev-tools: F12->Network) viser da også at billedindholdet fylder 14,85 MiB ud af sidens samlede størrelse på 16,23 MiB (91,5%).
SDFE.dk nyhedsoversigtsside, billederoverførsel
Hvad værre er at billederne på denne side kun anvendes som meget små illustrationer til en række artikler, men hentes som enorme filer i opløsningen 1920×1080. Inspektion af det første billede i browserens DOM-inspektør (Firefox: F12->Inspector) afslører at browseren kun viser billedet i en opløsning på 100×56! Der går altså 1920*1080-100*56=2’068’000 (99,73%) pixels til fuldstændig spilde.
SDFE.dk digebillede, vist størrelse vs. datastørrelse
Et hurtigt forsøg med at nedskalere et enkelt billede (terminal-foo!), uden at gå ind i så avancerede emner som JPG-kompressionskvalitet eller alternative billedformater (for ikke at tale om reel billedoptimering), kan give en ide om hvor meget data der egentlig overføres unødigt ved visning af denne side:
$ wget –quiet https://sdfe.dk/media/2919562/dige-1920×1080.jpg $ gm convert -resize 100×56 dige-1920×1080.jpg dige-100×56.jpg $ ls -l dige-* -rw-rw-r– 1 miki miki 17420 Apr 30 15:54 dige-100×56.jpg -rw-rw-r– 1 miki miki 1568308 Apr 24 14:54 dige-1920×1080.jpg $
Billedets størrelse reduceres her fra 1’568’308 bytes (1,60 MiB) til 17’420 bytes (17,01 KiB). Det er relativt en besparelse i overførselsmængden på dette billede på 8903%, vel at mærke med nøjagtigt det samme visuelle udbytte! For ikke at tale om den besparelse i hukommelsesforbrug og processeringstid der også opås ved at sidebeskuerens browser skal forholde sig til en tilsvarende mindre mængde data.
Under antagelse af at tilsvarende reduktioner kan opnås på det samlede sæt af billeder på denne side, vil overførselsbehovet kunne reduceres fra 15’206,40 KiB til 170,80 KiB. Tænk lige over det næste gang, webmaster!
2020-11-25 add news item about cable extension to Copenhagen, add Bulk data center blog link 2019-06-04 add details of Bulk data center in Esbjerg and infrastructure, add local news items about construction start 2019-05-08 add system summary from FCC application, elaborate on landing point discrepancies between FCC/cablemap, link to docs describing seg. 5 cable lay schedule 2019-03-06 fix links to submarinecablemap.com and some press, add info from TE Subcom experience doc., some general touch ups 2019-01-22 change “Danish Press Coverage” to “National Press”, add “International Press”, add some National about datacenter prospects & International Press items about contractors choosen 2018-10-05 initial commit
Europe, Denmark and my local neighbourhood of Western Jutland is going to get its connectivity boosted by the Havfrue transatlantic cable system being built by a consortium consisting of Google, Facebook, Aqua Comms and Bulk Infrastructure. To quote the announcement done by Google;
To increase capacity and resiliency in our North Atlantic systems, we’re working with Facebook, Aqua Comms and Bulk Infrastructure to build a direct submarine cable system connecting the U.S. to Denmark and Ireland. This cable, called Havfrue (Danish for “mermaid”), will be built by TE SubCom and is expected to come online by the end of 2019. Google blog post, 2018-01-16
Digging into the details first reveals the projected trench as illustrated in below by some of the stakeholders;
Projected trench of the Havfrue cable as illustrated by cloud.google.com.
Projected trench of the Havfrue cable as illustrated by TE SubCom.
Projected trench of the Havfrue cable as illustrated by submarinecablemap.com.
EDIT 2020-11-25: Additionally in 2019-06-21 Interxion announced a direct connection between the AEC2 landing site in Blaabjerg to its two datacenters in Ballerup/Copenhagen.
More digging into the Danish parts reveals that most sources mention Blåbjerg (Blaabjerg) as the Danish landing point for Havfrue (just as TAT-14), although ComputerWorld DK (see National Press below) relays the information that it will land at Endrup (where COBRAcable is terminated). However, a FCC application dated 2018-05-25 SCL-00214S (pdf) refers to it as the “Havfrue system” and specifically states that a new cable landing station will be constructed in Blaabjerg (as well as in Leckanvy, Ireland and Kristiansand, Norway);
The Havfrue system will consist of three segments. (1) The Main Trunk will connect the existing cable landing station at Wall, New Jersey with a new cable landing station to be constructed at Blaabjerg, Denmark. (2) The Ireland Branch will connect a new cable landing station to be constructed at Old Head Beach, Leckanvy, Ireland with a branching unit on the Main Trunk. (3) The Norway Branch will connect a new cable landing station at Kristiansand, Norway with a branching unit on the Main Trunk.
Google is currently also projecting its own private subsea cables, some of the rationale behind their mixed private/consortium/lease approach are disclosed in blog post from 2018-07-17 announcing the Dunant cable, which is the first Google private transatlantic subsea cable projected to connect Virginia Beach and France.
EDIT 2020-11-25: see blog post detailing my visit to the construction site in June 2019
Bulk has announced that the Esbjerg data center location will be referred to as DK01 Campus which is described on the about page (EDIT 2020-11-25: now has its own page with different wording) as follows:
Bulk’s DK01 Campus, Esbjerg, southwest Denmark, will be a scalable Carrier Neutral Colocation data center ready for customers Q4 2019. Esbjerg is becoming a highly strategic data center location with several subsea fiber systems terminating within or nearby. These include Havfrue (US, Ireland, Norway, Denmark), Havhingsten (Ireland, Denmark), Cobra (Holland, Denmark), Skagerrak 4 (Norway Denmark), DANICE (Iceland, Denmark) and TAT-14 (United Kingdom, France, the Netherlands, Germany, and Denmark). Combined with excellent terrestrial connectivity, this will make Esbjerg the main international entry point to the Nordics and enable the Bulk DK01 campus to be the natural traffic exchange point.
An article (translated) in the local newspaper JydskeVestkysten first revealed the exact location of the center and renderings of its visual appearence and construction. The location is in Kjersing industrial area North of Esbjerg.
A further map of the Bulk connections between Norway, Denmark and Ireland has been revealed in an article of Capacitymedia and on Bulk’s own fiber networks page. Also a partnership with Amazon about delivering both connectivity and datacenter infrastructure for AWS has been announced.
Taking a deeper look into the meta data of the document containing the Environmental Assessment (Danish: “miljøvurdering” shortened “MV”) and Environmental Impact Assessment (Danish: “miljøkonsekvensvurdering” or “vurdering af virkningerne på miljøet” shortened “VVM”) of the announced data center in Esbjerg reveals an interesting embedded title of the document which has not been carried out into other publicly used references.
The embedded PDF title of the document uses the “Project Ember” term which has not been indicated by other sources than articles in the JydskeVestkysten newspaper. The paper cite municipal sources but the municipality has not used the name directly in any of their communications.
The report authored by consultants COWI contains the following naming:
Below a dump of the full meta data:
$ pdfinfo MV-VVM_afgr%c3%a6nsning.pdf Title: Microsoft Word – Project_Ember_MV-VVM_afgrænsning_v.4.0.docx Author: lojo Creator: PScript5.dll Version 5.2.2 Producer: Acrobat Distiller 15.0 (Windows) CreationDate: Fri May 25 15:49:44 2018 ModDate: Fri May 25 15:49:44 2018 Tagged: no UserProperties: no Suspects: no Form: none JavaScript: no Pages: 25 Encrypted: no Page size: 595.22 x 842 pts (A4) Page rot: 0 File size: 234728 bytes Optimized: yes PDF version: 1.5
2019-06-03 add (local|national) press items about bulk data center (follow this in post about Havfrue, no further updates here), minor text fixes 2019-03-07 add local and national press items announcing cancellation of project 2019-02-27 add local press item about property value, environmentalist opposition and local educational initiatives 2019-02-21 add local press item about unsatisfied land owners 2019-01-22 add official approval of plans, fix original chronology of Official Documentation items, add (local|national|international) press items about a.o. announcement of Bulk Infrastructure datacenter 2018-12-19 add documentation and local press items about postponed permit decision from municipality 2018-11-30 add a bunch of local press items, and archaeological section to documentation 2018-10-04 add local and national press item about Amsterdam trip and announcing Facebook as the developer 2018-09-06 add local press item about downscaling and older national press, reorder press items (top=latest) 2018-08-19 add local press item and Official Documentation section about housing abandonment 2018-08-01 add local press item with letter to editor 2018-06-13 updated with 1 new local + 1 new national press, rewrite first paragraphs, mention project name, mention DDI trade association, mention investindk & havfrue cable 2018-06-12 initial commit
The local media of Western Jutland, JydskeVestkysten, has spearheaded the coverage of an interesting technology related story over the last weeks. The Esbjerg municipality planning departments has started to reveal details of the preparations for the development of an industrial site on a large swath of land just outside of Esbjerg seemingly for the purpose of a hyperscale data center of the proportions employed by FANG sized (Facebook, Amazon, Netflix, Google) organizations. According to the media the project is by some municipal sources referred to as “Project Ember“. I have been unable to confirm this name from official documentation yet released or any other sources.
Neither the newly formed trade association named Danish Data Center Industry (DDI/DanishDCI) (in Danish: “Datacenter Industrien“) or the state’s Invest in Denmark office has brought any more light to the issue. The former has, however, tweeted a couple of times about it when it hit the national media and the latter has brought forward a vague hint that Western Denmark is an “attractive data centre hub“. I’m not in any doubt that this is partly driven by the announcement of the “HAVFRUE consortium“, which includes Facebook, that they intend to install a 108 Tb/s transatlantic cable crossing from New Jersey to Ireland and Esbjerg, as also announced by Invest in Denmark in January.
Below is an outline of the area in question (on an OpenStreetMap based map using the umap project) that I have drawn from the only geographical details yet leaked which is contained in the meeting agenda mentioned below. See also a visualisation of the area on a photo taken by local photographer Christer Holte.
I have collected links to all official documentation I have been able to locate and to press coverage below, and intend to keep updating this post as details is being revealed.
<br />
See full screen
Edit 2023-04-23: on present day Ring is known instead as GNU Jami. Contact me using ring:f20607f4f974714ba91c664b153496fb931020e5 on the Ring distributed communication platform: ring.cx
