SIDEBAR
»
S
I
D
E
B
A
R
«
GitHub to be trusted their own SSH host key?
Jun 11th, 2023 by miki

A May 24th 2023 change of GitHub’s RSA SSH host key just hit me on an older, less used, system doing a git clone. The change was prompted by an unintended exposure of the private part of the host key by GitHub itself.

GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository

Github blog-post, “We updated our RSA SSH host key”, 2023-03-23

I’m getting even more reluctant to trust a large organization where such a thing can happen. Additionally they didn’t communicate about this in a way that caught my attention. And now that it has my attention, they do not want to share the specifics about the incident to enable assessment of impact (like exposure duration, popularity of repository etc).

Below is the manual recovery with explanations of what happens and why. I don’t like the way the official blog-post instructs users to retrieving the validated host key in a non-visible way, without human eye balls doing the validation. This is the only chance in the process for injecting human trust into the validation of the host key, don’t skip it!

Consequence #1: fatal MitM warning
When an SSH server’s host key is different to a connecting client (system) compared to what the system has previously seen and acknowledged by the user to be a valid key from the intended counterpart (you do check those host key fingerprint strings, right? If not; PHComp help, WinSCP help), it obviously should and do warn the user about this. Whether the change is of malicious origin or not, is up to the user to investigate. So with OpenSSH this happens;

$ git clone git@github.com:mikini/name-suggestion-index
Cloning into 'name-suggestion-index'...
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s.
Please contact your system administrator.
Add correct host key in <~>/.ssh/known_hosts to get rid of this message.
Offending RSA key in <~>/.ssh/known_hosts:17
  remove with:
  ssh-keygen -f "<~>/.ssh/known_hosts" -R "github.com"
RSA host key for github.com has changed and you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
$

Remedy #1: remove known host keys for domain (pr. OpenSSH instruction)
Remove host keys associated with github.com domain using -R option to ssh-keygen (default for -f is the UserKnownHostsFile config keyword with default ~/.ssh/known_hosts, so no need to specify that):

$ ssh-keygen -R github.com
# Host github.com found: line 17
# Host github.com found: line 18
# Host github.com found: line 19
<~>/.ssh/known_hosts updated.
Original contents retained as <~>/.ssh/known_hosts.old
$

Consequence #2: non-fatal domain and IP mismatch
Now, because host key references are cached for both the domain and IP there are still mismatches caused by the key exchange choosing ECDSA host key and the key associated with the IP address is the old RSA key. This is prompted during each connection attempt: 

$ git clone git@github.com:mikini/name-suggestion-index
Cloning into 'name-suggestion-index'...
Warning: the ECDSA host key for 'github.com' differs from the key for the IP address '140.82.121.3'
Offending key for IP in <~>/.ssh/known_hosts:63
Matching host key in <~>/.ssh/known_hosts:72
Are you sure you want to continue connecting (yes/no)? yes
remote: Enumerating objects: 153255, done.
remote: Counting objects: 100% (177/177), done.
remote: Compressing objects: 100% (103/103), done.
remote: Total 153255 (delta 103), reused 121 (delta 74), pack-reused 153078
Receiving objects: 100% (153255/153255), 550.17 MiB | 2.39 MiB/s, done.
Resolving deltas: 100% (111423/111423), done.
$

Remedy #2: remove known host keys for IPs
To get rid of the mismatch warning, the known host keys associated with IP addresses needs to be removed also. Potentially any of the listed endpoint IP addresses used with an RSA host key on the system earlier will now conflict with the ECDSA from server, requiring removal:

$ ssh-keygen -R 140.82.121.3
# Host 140.82.121.3 found: line 63
<~>/.ssh/known_hosts updated.
Original contents retained as <~>/.ssh/known_hosts.old
$

$ ssh-keygen -R 140.82.121.4
# Host 140.82.121.4 found: line 161
<~>/.ssh/known_hosts updated.
Original contents retained as <~>/.ssh/known_hosts.old
$

After remedy
Now, the SSH client will see an unknown host key and prompt for validation. Be sure to verify the presented host key with the officially communicated one (from doc. article or API):
$ git clone git@github.com:mikini/name-suggestion-index
Cloning into 'name-suggestion-index'...
The authenticity of host 'github.com (140.82.121.4)' can't be established.
ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'github.com,140.82.121.4' (ECDSA) to the list of known hosts.
remote: Enumerating objects: 153152, done.
remote: Counting objects: 100% (183/183), done.
remote: Compressing objects: 100% (83/83), done.
Receiving objects: 100% (153152/153152), 550.22 MiB | 1.42 MiB/s, done.
remote: Total 153152 (delta 109), reused 158 (delta 100), pack-reused 152969
Resolving deltas: 100% (111352/111352), done.
$
Comments Off on GitHub to be trusted their own SSH host key?
Project Ember: name of a data center
Aug 20th, 2018 by miki

Taking a deeper look into the meta data of the document containing the Environmental Assessment (Danish: “miljøvurdering” shortened “MV”) and Environmental Impact Assessment (Danish: “miljøkonsekvensvurdering” or “vurdering af virkningerne på miljøet” shortened “VVM”) of the announced data center in Esbjerg reveals an interesting embedded title of the document which has not been carried out into other publicly used references.

The embedded PDF title of the document uses the “Project Ember” term which has not been indicated by other sources than articles in the JydskeVestkysten newspaper. The paper cite municipal sources but the municipality has not used the name directly in any of their communications.

The report authored by consultants COWI contains the following naming:

  • Official in text: “MILJØVURDERING (MV) OG MILJØKONSEKVENSVURDERING (VVM) – ERHVERVSOMRÅDE VED ESBJERG, AFGRÆNSNINGSRAPPORT“
  • Filename of distributed PDF: “MV-VVM_afgrænsning.pdf “
  • PDF meta data title: “Microsoft Word – Project_Ember_MV-VVM_afgrænsning_v.4.0.docx“

Below a dump of the full meta data:

$ pdfinfo MV-VVM_afgr%c3%a6nsning.pdf
Title:          Microsoft Word – Project_Ember_MV-VVM_afgrænsning_v.4.0.docx
Author:         lojo
Creator:        PScript5.dll Version 5.2.2
Producer:       Acrobat Distiller 15.0 (Windows)
CreationDate:   Fri May 25 15:49:44 2018
ModDate:        Fri May 25 15:49:44 2018
Tagged:         no
UserProperties: no
Suspects:       no
Form:           none
JavaScript:     no
Pages:          25
Encrypted:      no
Page size:      595.22 x 842 pts (A4)
Page rot:       0
File size:      234728 bytes
Optimized:      yes
PDF version:    1.5

Comments Off on Project Ember: name of a data center
Hyperscale data center coming to Esbjerg
Jun 13th, 2018 by miki

2019-06-03 add (local|national) press items about bulk data center (follow this in post about Havfrue, no further updates here), minor text fixes
2019-03-07 add local and national press items announcing cancellation of project
2019-02-27 add local press item about property value, environmentalist opposition and local educational initiatives
2019-02-21 add local press item about unsatisfied land owners
2019-01-22 add official approval of plans, fix original chronology of Official Documentation items, add (local|national|international) press items about a.o. announcement of Bulk Infrastructure datacenter
2018-12-19 add documentation and local press items about postponed permit decision from municipality
2018-11-30 add a bunch of local press items, and archaeological section to documentation
2018-10-04 add local and national press item about Amsterdam trip and announcing Facebook as the developer
2018-09-06 add local press item about downscaling and older national press, reorder press items (top=latest)
2018-08-19 add local press item and Official Documentation section about housing abandonment
2018-08-01 add local press item with letter to editor
2018-06-13 updated with 1 new local + 1 new national press, rewrite first paragraphs, mention project name, mention DDI trade association, mention investindk & havfrue cable
2018-06-12 initial commit

Project Ember?

The local media of Western Jutland, JydskeVestkysten, has spearheaded the coverage of an interesting technology related story over the last weeks. The Esbjerg municipality planning departments has started to reveal details of the preparations for the development of an industrial site on a large swath of land just outside of Esbjerg seemingly for the purpose of a hyperscale data center of the proportions employed by FANG sized (Facebook, Amazon, Netflix, Google) organizations. According to the media the project is by some municipal sources referred to as “Project Ember“. I have been unable to confirm this name from official documentation yet released or any other sources.

Neither the newly formed trade association named Danish Data Center Industry (DDI/DanishDCI) (in Danish: “Datacenter Industrien“) or the state’s Invest in Denmark office has brought any more light to the issue. The former has, however, tweeted a couple of times about it when it hit the national media and the latter has brought forward a vague hint that Western Denmark is an “attractive data centre hub“. I’m not in any doubt that this is partly driven by the announcement of the “HAVFRUE consortium“, which includes Facebook, that they intend to install a 108 Tb/s transatlantic cable crossing from New Jersey to Ireland and Esbjerg, as also announced by Invest in Denmark in January.

Below is an outline of the area in question (on an OpenStreetMap based map using the umap project) that I have drawn from the only geographical details yet leaked which is contained in the meeting agenda mentioned below. See also a visualisation of the area on a photo taken by local photographer Christer Holte.

I have collected links to all official documentation I have been able to locate and to press coverage below, and intend to keep updating this post as details is being revealed.

See full screen

Official Documentation

  • 2019-01-21: Notitia Networks ApS was assigned a permit for construction by city council as documented in minutes of the city council meeting at 2019-01-21, item 11, p. 28-35
    • Changes to municipal plan approved with minor changes
      • Addition of passage detailing that an ecological corridor planned in the area should be moved accordingly
      • Explicitly state that land reserved for E20 (European highway) is being removed from plan (I understand this as being used for this project instead of E20, but not quite sure)
      • Corrections to various erroneous references, quoted noise limits and uniformity of maps
    • Area planning
      • National Road Department (Danish: Vejdirektoratet) asked for clarification that signs and other marketing in the open land is prohibited, and that supervision authority for signs that are erected near roads is awarded to the department
      • Clarification of unclear maps showing road outline
      • By citizen request a visualization of the visual consequences from the viewpoint of Nørregårdsparken has been produced and amended to the appendixes
      • Allowed area of buildings for the security facilities at entrance/exit increased from 50 m2 to 75 m2
      • Various references and minor clarifications added
    • Plan & Environment Committee: approved on meeting 2019-01-08 (item 5, p. 11-17)
    • Financial Committe: approved the recommendation from Plan & Environment Comittee on meeting 2019-01-14 (item 13, p. 27-33)
    • Administration recommendation to city council: approve
    • City Council: approved the recommendation from Plan & Environment Comittee and Financial Committe
  • 2018-12-18: An extensive trove of documents totalling 12 appendixes to “case 04” (“sag 04”) pertaining to the municipality hearing has been released as a part of the agenda for 2018-12-18 meeting in Municipal Planning & Environment committee (Plan & Miljø-udvalg), the final decision postponed for January meeting
    • minutes: “Resolution Plan & Environment Committee on 18-12-2018: Postponed for meeting on 8 January 2019 for further investigation.“
    • appendix 03: complete environmental assessment
    • appendix 04: supplemental visualisations
    • appendix 06: updated map of area
    • appendix 12: 158 pages of citizen comments and the municipal department’s comments to those
  • 2018-10-11: Sydvestjyske Museer (Museum of Southwest Jutland) has released an article about their preliminary findings (Google Translate’d) of the excavations done in the area.
    • Large parts is old heath without archaeological interest
    • In the eastern part remnants from Stone Age, Iron Age and WWII has been found
    • A complete predecessor of Andrup from around 0-200 AC has been found, parts of a later settlement from 200-800 AC has also been found, these are pending further investigations in 2019
  • 2018-08-07: Area mentioned in agenda/minutes for 2018-08-07 meeting in Municipal Planning & Environment committee (Plan & Miljø-udvalg)
    • Details in item 9 “Demolition of housing – Nordre Tovrupvej 21 and 26, Esbjerg” (Danish: “Nedlæggelse af boliger – Nordre Tovrupvej 21 og 26, Esbjerg”), p. 21-22 (case referred to as “Dok.nr.: 11768”, “Sagsid.: 18/20401”)
    • Requests that the committee approve liquidation and demolition of two current municipality owned rental houses in the area for the possible sale of the area for commercial purposes
    • Current inhabitants are willing to agree to voluntarily leave the rentals, but such formal agreements have not yet been established
    • Technical & Construction Committee assesses that the liquidation and subsequent sale of the aree will have a positive impact and is not outside current statutes
    • Approved by the committee
  • 2018-06-01: Public hearing announced (Google Translate’d) (original) about changed use of the area
    • Hearing closes 2018-06-15 (14 day period)
    • Accompanying report about environmental impact (VVM) discloses even more details
      • Area referred to as used for “establishing of extraordinary space consuming commercial entity near Esbjerg in the form of a data center” (ch. 2, p. 8)
      • Total area: 250 ha = 2’500’000 m2 (1 hectare = 10’000 m2) (ch. 2.2, p. 8)
      • Building area: “Current project entails approx. 250’000 m2 under roof with 200’000 m2 data warehouses and 50’000 m2 administration, logistics and service buildings, in addition to one or two 150 kV high voltage substations, each of approx. 30’000 m2 and diesel emergency power facilities of 6’500 m2” (ch. 2.3, p. 9)
      • Heat surplus: “Planning will leave open the possibility of reusing surplus heat produced at the facility, however no such plan exist at the moment” (ch. 2.3, p. 9)
  • 2018-05-28: Area mentioned in agenda/minutes for 2018-06-01 meeting in Municipal Technical & Construction committee (Teknik & Bygge-udvalg)
    • Details in item 7 “Closure of public and private roads in Andrup” (Danish: “Nedlæggelse af offentlige og private fællesveje ved Andrup”), p. 14-16 (case referred to as “Dok.nr.: 11186”, “Sagsid.: 18/12587”)
    • Area is referred to as “a contiguous area laid out for commercial purposes“
    • Includes map with outline of area
    • Suggests public roads being closed for cars, new cycling paths being constructed passing North of area
    • Approved by the committee

Local Press

National Press

International Press

Comments Off on Hyperscale data center coming to Esbjerg
»  Substance:WordPress   »  Style:Ahren Ahimsa
© 2023 Mikkel Kirkgaard Nielsen, contents CC BY-SA 4.0