SIDEBAR
»
S
I
D
E
B
A
R
«
GeoNames.org database id becoming a meteorological location reference
Aug 20th, 2020 by miki

Looking at weather forecasts in my native Denmark I have always noticed a location reference used by DMI, Danmarks Meteorologiske Institut (en:Danish Meteorological Institute). Somehow I managed to not look into the details of that before now.

But this morning while scouting for rain showers and trying out the forecast on the new website of Yr (the long time open data weather service from Norwegian Broadcasting Corporation (NRK) and the Norwegian Meteorological institute (MET)), as kindly suggested by the YR main site which uses the old site by default, I saw the same reference included in the URL.

A closer look at different service’s forecast URLs for my usual whereabouts in Tjæreborg, Denmark, Europe, Earth reveals (se summary below) that they all share this reference and that it stems from the GeoNames.org (wikipedia article) database’s integer reference to the location:

This particular piece of data is described as a field in the “geoname” table of the GeoNames.org main database, and referred to as “geonameid” in the documentation:

geonameid         : integer id of record in geonames database

The contents of the GeoNames.org database is licensed under the Creative Commons Attribution 4.0 International (aka. “CC BY 4.0”, SPDX ID:”CC-BY-4.0″) and available both for your own download and perusal (documentation here) or using web services on geonames.org.

Note, however, that GeoNames.org accumulates a wealth of data sources that according to the OpenStreetMap project might contain copyrighted data. Together with the attribution requirements of its CC BY license this causes OpenStreetMap to not accept data from GeoNames.org into the project’s, ODbL licensed, database.

Fun fact: in Norwegian “yr” actually means drizzle (da:støvregn)
Practical hint: OpenSearch entry for GeoNames.org: http://www.geonames.org/opensearch-description.xml (to add in current browser: go here->find “Plugins”->click “opensearch plugin”)

The Weather Services

2611610 on GeoNames.org

“Get”ting from web service

$ wget -q -O- "https://secure.geonames.org/get?geonameId=2611610&username=demo"
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<geoname>
    <toponymName>Tjæreborg</toponymName>
    <name>Tjæreborg</name>
    <lat>55.46457</lat>
    <lng>8.57968</lng>
    <geonameId>2611610</geonameId>
    <countryCode>DK</countryCode>
    <countryName>Denmark</countryName>
    <fcl>P</fcl>
    <fcode>PPL</fcode>
    <fclName>city, village,...</fclName>
    <fcodeName>populated place</fcodeName>
    <population>2146</population>
    <adminCode1 ISO3166-2="83">21</adminCode1>
    <adminName1>South Denmark</adminName1>
    <asciiName>Tjaereborg</asciiName>
    <alternateNames>Tjaereborg,Tjæreborg</alternateNames>
    <elevation/>
    <srtm3>12</srtm3>
    <astergdem>11</astergdem>
    <continentCode>EU</continentCode>
    <adminCode2>561</adminCode2>
    <adminName2>Esbjerg Kommune</adminName2>
    <alternateName lang="post">6731</alternateName>
    <alternateName lang="unlc">DKTJA</alternateName>
    <alternateName lang="link">https://en.wikipedia.org/wiki/Tj%C3%A6reborg%2C_Denmark</alternateName>
    <alternateName>Tjæreborg</alternateName>
    <timezone dstOffset="2.0" gmtOffset="1.0">Europe/Copenhagen</timezone>
    <bbox>
        <west>8.57175</west>
        <north>55.46907</north>
        <east>8.58762</east>
        <south>55.46008</south>
        <accuracyLevel>0</accuracyLevel>
    </bbox>
</geoname>
$

Extracting from “Gazetteer” exports

$ wget -q https://download.geonames.org/export/dump/DK.zip
$ unzip DK.zip 
Archive:  DK.zip
  inflating: readme.txt              
  inflating: DK.txt                  
$ grep ^2611610 DK.txt
2611610	Tjæreborg	Tjaereborg	Tjaereborg,Tjæreborg	55.46457	8.57968	P	PPL	DK\
		21	561			2146		12	Europe/Copenhagen	2017-10-18
$

 

Amending an open OSM changeset on command line (by hand)
Jul 25th, 2020 by miki

WARNING: as OSM user “mmd” wisely points out (in comment to OSM diary for this post) the sort of thing described here is dangerous to do by hand, and should not be done on the main production instances (there are testing instances for playing around with the API, a little documentation here).
He also points out that the feature packed JOSM editor actually supports continuing a changeset regardless of where it has been initiated. So if you just need to continue working on a changeset (but remember the 1 hour idle timeout) be sure to check out the JSOM upload documentation. Thanks mmd, for being the sane voice ;).

During an editing session the Android OpenStreetMap editor Vespucci crashed on me, which made the mapping UI unusable (objects greyed out and unable to select or edit, had to purge all data to recover functionality). Luckly, I could still navigate the menus, upload changes and opt to not close the changeset. Now, I had long wondered whether the OSM API allowed to continue amending changes to an open changeset from some other editor, so the quest began.

I had the intention of adding a tag representing the name of a building in Esbjerg known as “ISC Huset” (ISC is an engineering consultancy, see wikipedia, and more about the construction), which houses a number of healtcare clinics. The building’s address is Borgergade 70, 6700 Esbjerg (current address node).

This blog post willl attempt to actually add the tag by hand on the command line.

References to OSM objects used:

Option summary for GNU wget used to do HTTP requests below:

  • -nv (or –no-verbose): only report errors and basic information
  • -O- (or –output-document=-): write output to standard out (stdout)
  • –user=<user name>: authenticate as <user name> using HTTP Basic authentication
  • –ask-password: ask on command line for HTTP Basic authentication password
  • –post-data=”<data>”: initiate a HTTP POST request with <data> as payload
  • –server-response (or -S): show full HTTP headers of server response

NOTE: I’ve broken some long output lines replicated below to make it fit the blog, but inserted an escape character (\) before the inserted newline to help copy’n’pasting.

Retrieve changeset metadata

Lets start by looking at the changeset’s metadata.

This can be done by issuing an unauthenticated GET request to the “/api/0.6/changeset/<changeset id>” endpoint.

Note that the ‘changeset’ element has the attribute ‘open=”true”‘ required to be able to modify the changeset. The editor used to create the changeset needs to have done this without explicitly closing it (Vespucci & JOSM closes by default but can be configured not to, iD always closes).

$ wget -nv -O- http://api.openstreetmap.org/api/0.6/changeset/88490797
<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="CGImap 0.8.3 (802 thorn-02.openstreetmap.org)" copyright="OpenStreetMap and contributors"\
 attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/">
 <changeset id="88490797" created_at="2020-07-25T09:51:30Z" open="true" user="mikini" uid="2051"\
 min_lat="55.4654072" min_lon="8.4378053" max_lat="55.5258532" max_lon="8.4639026" comments_count="0" changes_count="68">
  <tag k="source" v="survey;research"/>
  <tag k="locale" v="da-DK"/>
  <tag k="created_by" v="Vespucci 14.1.4.0"/>
  <tag k="comment" v="Details at Klevestien &amp; Borgergade in Esbjerg and Tarp."/>
 </changeset>
</osm>
2020-07-25 14:26:52 URL:http://api.openstreetmap.org/api/0.6/changeset/88490797 [709] -> "-" [1]
$

Pinging the changeset with an empty osmChange structure

To test that the changeset is open, and that we can authenticate correctly, lets try amending it with an empty osmChange structure.

This can be done by issuing an authenticated POST request to the “/api/0.6/changeset/<changeset id>/upload” endpoint.

This also seem to reset the 60 minute timer used for auto-closing the changeset (see mention of “idle timeout” in changeset wiki article).

$ wget -nv -O- --user=mikini --ask-password https://api.openstreetmap.org/api/0.6/changeset/88490797/upload --post-data="<osmChange></osmChange>"
Password for user ‘mikini’: 
Authentication selected: Basic realm="Web Password", charset="UTF-8"
<?xml version="1.0" encoding="UTF-8"?>
<diffResult version="0.6" generator="CGImap 0.8.3 (8531 thorn-03.openstreetmap.org)" copyright="OpenStreetMap and contributors"\
 attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/"/>
2020-07-25 14:27:20 URL:https://api.openstreetmap.org/api/0.6/changeset/88490797/upload [278] -> "-" [1]
$

Procedure to change the building

The API’s “changeset/<changeset id>/upload” method supports only modifications encoded in the osmChange format which requires changes to be described as complete way/node/relation objects. That is, you can not ask the API to “add this tag to this way”, you need instead to describe the modified way completely saying “this way now looks like this”, including anything (like node references or existing tags) that was not modified. So to make a modificatino to the building’s way we need to retrieve the current version, modify it as needed and upload the complete new way.

Thus the procedure contains these steps;

  1. Retrieve current building (GET /way/*)
  2. Modify building data (locally)
  3. Amend changeset (POST /changeset/*/upload)

Retrieving current building (GET /way/*)

This can be done by issuing an unauthenticated GET request to the “/api/0.6/way/<way id>” endpoint.

The stdin splitter ‘tee’ is used here to both show the result in terminal and put it into file 185369466_v3.osc that we can use for amending the way with the wanted modifications.

$ wget -nv -O- http://api.openstreetmap.org/api/0.6/way/185369466|tee 185369466_v3.osc
<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="CGImap 0.8.3 (28697 thorn-01.openstreetmap.org)" copyright="OpenStreetMap and contributors"\
 attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/">
 <way id="185369466" visible="true" version="3" changeset="84400254" timestamp="2020-04-30T09:35:21Z" user="mikini" uid="2051">
  <nd ref="1959614623"/>
  <nd ref="1959614727"/>
  <nd ref="6299449794"/>
  <nd ref="1959614650"/>
  <nd ref="1959614630"/>
  <nd ref="6299449793"/>
  <nd ref="1959614765"/>
  <nd ref="7466482063"/>
  <nd ref="7466482064"/>
  <nd ref="7466482065"/>
  <nd ref="7466482062"/>
  <nd ref="1959614729"/>
  <nd ref="1959614623"/>
  <tag k="building" v="yes"/>
 </way>
</osm>
2020-07-25 14:54:30 URL:http://api.openstreetmap.org/api/0.6/way/185369466 [769] -> "-" [1]
$

Modify building data (locally)

Now we need to build an osmChange file out of the existing <way>…</way> element from the output above describing the wanted building. This involves;

  1. removing the <xml> tag (maybe not needed, but it is not mentioned at all in the osmChange documentation)
  2. replacing <osm> with <osmChange>
  3. adding a <modify> element to the <osmChange> element, containing the existing <way>
  4. update the “changeset” attribute to the changeset we’re amending
  5. amending the contents of <way> with the wanted <tag>s

Use your favorite editor (emacs would be my preference) to load the 185369466_v3.osm file, make the modifications and save it as 185369466_v4.osc. OSM tags are a XML empty-element tags containing the OSM tag’s key and value in the “k” and “v” attributes, thus the “name” tag of the building I needed to add would be ‘<tag k=”name” v=”ISC Huset”/>’, I also added some other related tags (“source:name” and “website”).

The finished .osc file now looks like this;

$ cat 185369466_v4.osc
<osmChange>
  <modify>
    <way id="185369466" visible="true" version="3" changeset="88490797" timestamp="2020-04-30T09:35:21Z" user="mikini" uid="2051">
      <nd ref="1959614623"/>
      <nd ref="1959614727"/>
      <nd ref="6299449794"/>
      <nd ref="1959614650"/>
      <nd ref="1959614630"/>
      <nd ref="6299449793"/>
      <nd ref="1959614765"/>
      <nd ref="7466482063"/>
      <nd ref="7466482064"/>
      <nd ref="7466482065"/>
      <nd ref="7466482062"/>
      <nd ref="1959614729"/>
      <nd ref="1959614623"/>
      <tag k="building" v="yes"/>
      <tag k="name" v="ISC Huset"/>
      <tag k="source:name" v="sign;website"/>
      <tag k="website" v="https://www.isc.dk/isc-huset-esbjerg/"/>
    </way>
  </modify>
</osmChange>
$

Wdiff’ing against the .osm source shows exactly what changed (additions between “{+” & “+}”, removals between “[-” & “-]”);

$ wdiff 185369466_v3.osm 185369466_v4.osc
[-<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="CGImap 0.8.3 (28697 thorn-01.openstreetmap.org)" copyright="OpenStreetMap and contributors"\
 attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/">-]{+<osmChange>
  <modify>+}
    <way id="185369466" visible="true" version="3" [-changeset="84400254"-] {+changeset="88490797"+} timestamp="2020-04-30T09:35:21Z" user="mikini" uid="2051">
      <nd ref="1959614623"/>
      <nd ref="1959614727"/>
      <nd ref="6299449794"/>
      <nd ref="1959614650"/>
      <nd ref="1959614630"/>
      <nd ref="6299449793"/>
      <nd ref="1959614765"/>
      <nd ref="7466482063"/>
      <nd ref="7466482064"/>
      <nd ref="7466482065"/>
      <nd ref="7466482062"/>
      <nd ref="1959614729"/>
      <nd ref="1959614623"/>
      <tag k="building" v="yes"/>
      {+<tag k="name" v="ISC Huset"/>
      <tag k="source:name" v="sign;website"/>
      <tag k="website" v="https://www.isc.dk/isc-huset-esbjerg/"/>+}
    </way>
[-</osm>-]
  {+</modify>
</osmChange>+}
$

Amend changeset (POST /changeset/*/upload)

Now, we’ll again use the changeset upload method but this time supplying our actual osmChange elemet in the .osc file.

The output is a bit elaborate, because I had enabled full output from wget while debugging what changes to the <way> element was needed for the server to accept the upload (only the “changeset” attribute needs to match the open changeset as outlined in the “Modify building data” above). I’ve highligted the actual server response telling that the changes were accepted and way #185369466 is now at v4.

$ wget -S -O- --user=mikini --ask-password https://api.openstreetmap.org/api/0.6/changeset/88490797/upload --post-file=185369466_v4.osc
Password for user ‘mikini’: 
--2020-07-25 15:44:41--  https://api.openstreetmap.org/api/0.6/changeset/88490797/upload
Resolving api.openstreetmap.org (api.openstreetmap.org)... 130.117.76.12, 130.117.76.13, 130.117.76.11, ...
Connecting to api.openstreetmap.org (api.openstreetmap.org)|130.117.76.12|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 401 Unauthorized
  Date: Sat, 25 Jul 2020 13:44:41 GMT
  Server: Apache/2.4.29 (Ubuntu)
  Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
  Expect-CT: max-age=0, report-uri="https://openstreetmap.report-uri.com/r/d/ct/reportOnly"
  WWW-Authenticate: Basic realm="Web Password", charset="UTF-8"
  Cache-Control: no-cache
  Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
  Expect-CT: max-age=0, report-uri="https://openstreetmap.report-uri.com/r/d/ct/reportOnly"
  Content-Length: 22
  Content-Type: text/plain; charset=utf-8
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
Authentication selected: Basic realm="Web Password", charset="UTF-8"
Reusing existing connection to api.openstreetmap.org:443.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Sat, 25 Jul 2020 13:44:42 GMT
  Server: Apache/2.4.29 (Ubuntu)
  Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
  Expect-CT: max-age=0, report-uri="https://openstreetmap.report-uri.com/r/d/ct/reportOnly"
  Content-Encoding: identity
  Cache-Control: private, max-age=0, must-revalidate
  Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
  Expect-CT: max-age=0, report-uri="https://openstreetmap.report-uri.com/r/d/ct/reportOnly"
  Vary: Accept-Encoding
  Content-Type: application/xml; charset=utf-8
  Keep-Alive: timeout=5, max=99
  Connection: Keep-Alive
  Transfer-Encoding: chunked
Length: unspecified [application/xml]
Saving to: ‘STDOUT’
- [<=> ] 0 --.-KB/s
<?xml version="1.0" encoding="UTF-8"?>
<diffResult version="0.6" generator="CGImap 0.8.3 (8537 thorn-03.openstreetmap.org)" copyright="OpenStreetMap and contributors"\
 attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/">
 <way old_id="185369466" new_id="185369466" new_version="4"/>
</diffResult>
- [ <=> ] 353 --.-KB/s in 0s

2020-07-25 15:44:42 (20,3 MB/s) - written to stdout [353]

$

Result

That was it, the building is now named in OSM, in a changset amended by hand.

Take a look at https://www.openstreetmap.org/way/185369466/.

GitHub CLI on Ubuntu LTS (16.04 & 18.04)
Mar 4th, 2020 by miki

Inspired by a John Sullivan (of FSF) tweet asking about opinions on the new and shiny MS(TM) GitHub(TM) CLI(TM) tool named “gh”(TM) I wanted to try it out on one of my Ubuntu LTS systems (16.04).

I’ve always disliked the proprietary and centralised monoculture of github, especially after that thing with MS, so I’ve mostly avoided using the service for code I produce myself. If interested in independence and decentralisation maybe you should read up on that long fabled subject? Thus I’ve never even tried the predecessor “hub”(TM) tool (main page), and exactly how it differs from this new “CLI” thing is not obviously apparent from the communications.

Nevertheless, the “gh” tool, officially dubbed  GitHub CLI, being distributed under MIT license and written in go(lang) (afterwards, found out that this is indeed also applicable to the predecessing “hub” tool) made me curious enough to go see what they’re up to. Maybe its not that sinister a plot and they really just want to improve the independent and freely available mechanisms for source code storage, distribution and maintenance (I wish)?

TL:DR

A fairly recent (2019-09) version of go, >=1.13, is required so some gymnastics are needed to make stuff work on older distros like Ubuntu’s LTS (with 5 years support from Canonical). Anyway, this is what I did to build a useable “gh” ELF executable (20 MiB!, se below), this is from source, no precompiled stuff, no snaps. It was done on Ubuntu 16.04 (xenial) but I’ve been somewhat careful to inspect that things ought to be good for 18.04 (bionic) too. Non-essential output from the commands is largely discarded here. See Gory Details section below for the full monty.

Add go PPA & install packages

$ sudo add-apt-repository ppa:longsleep/golang-backports
$ sudo apt update
$ sudo apt install golang
$ go version
go version go1.13.4 linux/amd64

Clone repo & build

$ cd
$ git clone https://github.com/cli/cli .githubcli
$ cd .githubcli
$ make

Setup PATH & test

$ echo 'export PATH="$HOME/.githubcli/bin:$PATH"' >> ~/.bash_profile
$ bash --login
$ gh
Work seamlessly with GitHub from the command line.

GitHub CLI is in early stages of development, and we'd love to hear your
feedback at <https://forms.gle/umxd3h31c7aMQFKG7>

Usage:
  gh [command]

Available Commands:
  help        Help about any command
  issue       Create and view issues
  pr          Create, view, and checkout pull requests
  repo        Create, clone, fork, and view repositories

Flags:
      --help              Show help for command
  -R, --repo OWNER/REPO   Select another repository using the OWNER/REPO format
      --version           Show gh version

Use "gh [command] --help" for more information about a command.
subcommand is required
$

Trying it out

Randomly picked one of my few github repos, it contains some js experiments on parsing NMEA formatted coordinates, rather uninterestingly empty.

Clone

user@host:~$ gh repo clone mikini/coordinates 
Notice: authentication required
Press Enter to open github.com in your browser... 

<auth session in browser>

Authentication complete. Press Enter to continue...

Cloning into 'coordinates'...
remote: Enumerating objects: 10, done.
remote: Total 10 (delta 0), reused 0 (delta 0), pack-reused 10
Unpacking objects: 100% (10/10), done.
Checking connectivity... done.
user@host:~$

An ordinary OAuth request was spawned for authentication on https://github.com/login/oauth/authorize. The javascript executed in the browser seemed to interact with the gh tool through a local TCP connection on localhost:45454 which was the redirection target on authorisation (URL parameter: “redirect_uri=http://localhost:45454/callback”). The socket was rightfully not bound after the authentication had succeeded but this is definitely an attack vector, albeit hopefully only for locally running processes.

Listing issues & prs

user@host:~$ cd coordinates/
user@host:~$:~/coordinates$ gh issue list

Issues for mikini/coordinates

There are no open issues
miki@vcas-miki2:~/coordinates$ gh pr view
no open pull requests found for branch "master"
miki@vcas-miki2:~/coordinates$

Creating issues

miki@vcas-miki2:~/coordinates$ gh issue create

Creating issue in mikini/coordinates

? Title Testing gh
? Body <Received>
? What's next? Preview in browser
Opening github.com/mikini/coordinates/issues/new/ in your browser.
miki@vcas-miki2:~/coordinates$

miki@vcas-miki2:~/coordinates$ gh issue create

Creating issue in mikini/coordinates

? Title Testing gh tool
? Body <Received>
? What's next? [Use arrows to move, type to filter]
> Preview in browser
Submit
Cancel

https://github.com/mikini/coordinates/issues/1
miki@vcas-miki2:~/coordinates$

On the first issue creation interaction I selected “preview in browser” which opened a draft issue in the browser, and dumped me back into the shell on the command line. The second interaction actually resulted in #1.

Gory Details

PPA options

The golang-backports PPA is maintained independently by Simon Eisenmann (credits: ~longsleep, @github, stdin.xyz) and carries packages up to:

There’s also an official PPA from the golang project (~gophers) which however only carries packages up to 1.11 for both.

Full (mostly) installation dump

Add go PPA, install packages

user@host:~$ sudo add-apt-repository ppa:longsleep/golang-backports
Golang 1.8, 1.9, 1.10, 1.11, 1.12, 1.13 and 1.14 PPA for Ubuntu
 More info: https://launchpad.net/~longsleep/+archive/ubuntu/golang-backports
Press [ENTER] to continue or ctrl-c to cancel adding it

gpg: keyring `/tmp/tmpvnuym97a/secring.gpg' created
gpg: keyring `/tmp/tmpvnuym97a/pubring.gpg' created
gpg: requesting key 56A3D45E from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpvnuym97a/trustdb.gpg: trustdb created
gpg: key 56A3D45E: public key "Launchpad PPA for Simon Eisenmann" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK
user@host:~$

user@host:~$ sudo apt update
Hit:1 http://dk.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://dk.archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB] 
...
...
Get:37 http://security.ubuntu.com/ubuntu xenial-security/universe DEP-11 64x64 Icons [194 kB]
Get:38 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 DEP-11 Metadata [2.468 B]
Fetched 7.614 kB in 3s (2.346 kB/s) 
Reading package lists... Done
Building dependency tree 
Reading state information... Done
419 packages can be upgraded. Run 'apt list --upgradable' to see them.
user@host:~$

user@host:~$ apt list --upgradeable golang -a
Listing... Done
golang/xenial,xenial 2:1.13~1longsleep1+xenial all [upgradable from: 2:1.6-1ubuntu4]
golang/xenial,xenial,now 2:1.6-1ubuntu4 all [installed,upgradable to: 2:1.13~1longsleep1+xenial]

user@host:~$

user@host:~$ sudo apt install golang
Reading package lists... Done
Building dependency tree 
Reading state information... Done
The following packages were automatically installed and are no longer required:
golang-1.6 golang-1.6-doc golang-1.6-go golang-1.6-src
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
golang-1.13 golang-1.13-doc golang-1.13-go golang-1.13-src golang-doc golang-go golang-src
Recommended packages:
golang-1.13-race-detector-runtime
The following NEW packages will be installed
golang-1.13 golang-1.13-doc golang-1.13-go golang-1.13-src
The following packages will be upgraded:
golang golang-doc golang-go golang-src
4 to upgrade, 4 to newly install, 0 to remove and 413 not to upgrade.
Need to get 60,2 MB of archives.
After this operation, 322 MB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://ppa.launchpad.net/longsleep/golang-backports/ubuntu xenial/main amd64 golang-1.13-src amd64 1.13.4-1longsleep1+xenial [12,7 MB]
Get:2 http://ppa.launchpad.net/longsleep/golang-backports/ubuntu xenial/main amd64 golang-1.13-go amd64 1.13.4-1longsleep1+xenial [44,9 MB]
Get:3 http://ppa.launchpad.net/longsleep/golang-backports/ubuntu xenial/main amd64 golang-1.13-doc all 1.13.4-1longsleep1+xenial [2.542 kB]
Get:4 http://ppa.launchpad.net/longsleep/golang-backports/ubuntu xenial/main amd64 golang-1.13 all 1.13.4-1longsleep1+xenial [25,2 kB]
Get:5 http://ppa.launchpad.net/longsleep/golang-backports/ubuntu xenial/main amd64 golang-src amd64 2:1.13~1longsleep1+xenial [3.838 B]
Get:6 http://ppa.launchpad.net/longsleep/golang-backports/ubuntu xenial/main amd64 golang-go amd64 2:1.13~1longsleep1+xenial [23,0 kB]
Get:7 http://ppa.launchpad.net/longsleep/golang-backports/ubuntu xenial/main amd64 golang-doc all 2:1.13~1longsleep1+xenial [3.880 B]
Get:8 http://ppa.launchpad.net/longsleep/golang-backports/ubuntu xenial/main amd64 golang all 2:1.13~1longsleep1+xenial [3.826 B]
Fetched 60,2 MB in 5s (10,8 MB/s) 
Selecting previously unselected package golang-1.13-src.
(Reading database ... 460737 files and directories currently installed.)
Preparing to unpack .../golang-1.13-src_1.13.4-1longsleep1+xenial_amd64.deb ...
Unpacking golang-1.13-src (1.13.4-1longsleep1+xenial) ...
Selecting previously unselected package golang-1.13-go.
Preparing to unpack .../golang-1.13-go_1.13.4-1longsleep1+xenial_amd64.deb ...
Unpacking golang-1.13-go (1.13.4-1longsleep1+xenial) ...
Selecting previously unselected package golang-1.13-doc.
Preparing to unpack .../golang-1.13-doc_1.13.4-1longsleep1+xenial_all.deb ...
Unpacking golang-1.13-doc (1.13.4-1longsleep1+xenial) ...
Selecting previously unselected package golang-1.13.
Preparing to unpack .../golang-1.13_1.13.4-1longsleep1+xenial_all.deb ...
Unpacking golang-1.13 (1.13.4-1longsleep1+xenial) ...
Preparing to unpack .../golang-src_2%3a1.13~1longsleep1+xenial_amd64.deb ...
Unpacking golang-src (2:1.13~1longsleep1+xenial) over (2:1.6-1ubuntu4) ...
Preparing to unpack .../golang-go_2%3a1.13~1longsleep1+xenial_amd64.deb ...
Unpacking golang-go (2:1.13~1longsleep1+xenial) over (2:1.6-1ubuntu4) ...
Preparing to unpack .../golang-doc_2%3a1.13~1longsleep1+xenial_all.deb ...
Unpacking golang-doc (2:1.13~1longsleep1+xenial) over (2:1.6-1ubuntu4) ...
Preparing to unpack .../golang_2%3a1.13~1longsleep1+xenial_all.deb ...
Unpacking golang (2:1.13~1longsleep1+xenial) over (2:1.6-1ubuntu4) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up golang-1.13-src (1.13.4-1longsleep1+xenial) ...
Setting up golang-1.13-go (1.13.4-1longsleep1+xenial) ...
Setting up golang-1.13-doc (1.13.4-1longsleep1+xenial) ...
Setting up golang-1.13 (1.13.4-1longsleep1+xenial) ...
Setting up golang-src (2:1.13~1longsleep1+xenial) ...
Setting up golang-go (2:1.13~1longsleep1+xenial) ...
Setting up golang-doc (2:1.13~1longsleep1+xenial) ...
Setting up golang (2:1.13~1longsleep1+xenial) ...
user@host:~$ 

user@host:~$ go version
go version go1.13.4 linux/amd64
user@host:~$ 

Clone repo & build

user@host:~$ git clone https://github.com/cli/cli .githubcli
Cloning into '.githubcli'...
remote: Enumerating objects: 168, done.
remote: Counting objects: 100% (168/168), done.
remote: Compressing objects: 100% (100/100), done.
remote: Total 5577 (delta 93), reused 122 (delta 67), pack-reused 5409
Receiving objects: 100% (5577/5577), 7.53 MiB | 2.61 MiB/s, done.
Resolving deltas: 100% (3269/3269), done.
Checking connectivity... done.
user@host:~$ 

user@host:~$ cd .githubcli
user@host:~/.githubcli (master=)$ make
go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/mattn/go-colorable v0.1.6
go: downloading github.com/spf13/cobra v0.0.6
go: downloading gopkg.in/yaml.v3 v3.0.0-20200121175148-a6ecf24a6d71
go: downloading github.com/mattn/go-isatty v0.0.12
go: downloading golang.org/x/text v0.3.2
go: extracting github.com/mitchellh/go-homedir v1.1.0
go: extracting github.com/mattn/go-colorable v0.1.6
go: extracting github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
go: extracting github.com/spf13/cobra v0.0.6
go: extracting github.com/mattn/go-isatty v0.0.12
go: downloading golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae
go: downloading golang.org/x/crypto v0.0.0-20200219234226-1ad67e1f0ef4
go: downloading github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b
go: downloading github.com/briandowns/spinner v1.9.0
go: extracting gopkg.in/yaml.v3 v3.0.0-20200121175148-a6ecf24a6d71
go: downloading github.com/hashicorp/go-version v1.2.0
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/AlecAivazis/survey/v2 v2.0.7
go: downloading github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51
go: downloading github.com/charmbracelet/glamour v0.1.1-0.20200304134224-7e5c90143acc
go: extracting github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b
go: extracting github.com/spf13/pflag v1.0.5
go: extracting github.com/hashicorp/go-version v1.2.0
go: downloading gopkg.in/yaml.v2 v2.2.8
go: downloading github.com/cpuguy83/go-md2man/v2 v2.0.0
go: downloading github.com/henvic/httpretty v0.0.4
go: extracting github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51
go: extracting github.com/briandowns/spinner v1.9.0
go: extracting github.com/AlecAivazis/survey/v2 v2.0.7
go: downloading github.com/fatih/color v1.7.0
go: extracting github.com/charmbracelet/glamour v0.1.1-0.20200304134224-7e5c90143acc
go: downloading github.com/yuin/goldmark v1.1.24
go: downloading github.com/microcosm-cc/bluemonday v1.0.2
go: extracting gopkg.in/yaml.v2 v2.2.8
go: extracting github.com/henvic/httpretty v0.0.4
go: downloading github.com/olekukonko/tablewriter v0.0.4
go: downloading github.com/muesli/reflow v0.1.0
go: extracting github.com/fatih/color v1.7.0
go: extracting github.com/microcosm-cc/bluemonday v1.0.2
go: extracting github.com/cpuguy83/go-md2man/v2 v2.0.0
go: extracting golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae
go: extracting github.com/muesli/reflow v0.1.0
go: extracting github.com/olekukonko/tablewriter v0.0.4
go: downloading golang.org/x/net v0.0.0-20200219183655-46282727080f
go: extracting github.com/yuin/goldmark v1.1.24
go: downloading github.com/alecthomas/chroma v0.7.2-0.20200304075647-34d9c7143bf5
go: downloading github.com/mattn/go-runewidth v0.0.8
go: downloading github.com/muesli/termenv v0.4.0
go: downloading github.com/russross/blackfriday/v2 v2.0.1
go: extracting golang.org/x/crypto v0.0.0-20200219234226-1ad67e1f0ef4
go: extracting github.com/mattn/go-runewidth v0.0.8
go: extracting github.com/russross/blackfriday/v2 v2.0.1
go: downloading github.com/shurcooL/sanitized_anchor_name v1.0.0
go: extracting github.com/alecthomas/chroma v0.7.2-0.20200304075647-34d9c7143bf5
go: extracting github.com/shurcooL/sanitized_anchor_name v1.0.0
go: extracting github.com/muesli/termenv v0.4.0
go: downloading github.com/dlclark/regexp2 v1.2.0
go: downloading github.com/lucasb-eyer/go-colorful v1.0.3
go: downloading github.com/google/goterm v0.0.0-20190703233501-fc88cf888a3f
go: downloading github.com/danwakefield/fnmatch v0.0.0-20160403171240-cbb64ac3d964
go: extracting golang.org/x/text v0.3.2
go: extracting github.com/google/goterm v0.0.0-20190703233501-fc88cf888a3f
go: extracting github.com/danwakefield/fnmatch v0.0.0-20160403171240-cbb64ac3d964
go: extracting golang.org/x/net v0.0.0-20200219183655-46282727080f
go: extracting github.com/lucasb-eyer/go-colorful v1.0.3
go: extracting github.com/dlclark/regexp2 v1.2.0
go: finding github.com/spf13/cobra v0.0.6
go: finding github.com/AlecAivazis/survey/v2 v2.0.7
go: finding github.com/henvic/httpretty v0.0.4
go: finding github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
go: finding github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b
go: finding gopkg.in/yaml.v3 v3.0.0-20200121175148-a6ecf24a6d71
go: finding github.com/spf13/pflag v1.0.5
go: finding github.com/hashicorp/go-version v1.2.0
go: finding github.com/mitchellh/go-homedir v1.1.0
go: finding github.com/briandowns/spinner v1.9.0
go: finding golang.org/x/text v0.3.2
go: finding github.com/cpuguy83/go-md2man/v2 v2.0.0
go: finding github.com/charmbracelet/glamour v0.1.1-0.20200304134224-7e5c90143acc
go: finding github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51
go: finding github.com/mattn/go-colorable v0.1.6
go: finding gopkg.in/yaml.v2 v2.2.8
go: finding github.com/russross/blackfriday/v2 v2.0.1
go: finding github.com/mattn/go-isatty v0.0.12
go: finding github.com/fatih/color v1.7.0
go: finding golang.org/x/crypto v0.0.0-20200219234226-1ad67e1f0ef4
go: finding github.com/muesli/termenv v0.4.0
go: finding golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae
go: finding github.com/shurcooL/sanitized_anchor_name v1.0.0
go: finding github.com/yuin/goldmark v1.1.24
go: finding github.com/alecthomas/chroma v0.7.2-0.20200304075647-34d9c7143bf5
go: finding github.com/google/goterm v0.0.0-20190703233501-fc88cf888a3f
go: finding github.com/dlclark/regexp2 v1.2.0
go: finding github.com/lucasb-eyer/go-colorful v1.0.3
go: finding github.com/muesli/reflow v0.1.0
go: finding github.com/microcosm-cc/bluemonday v1.0.2
go: finding github.com/danwakefield/fnmatch v0.0.0-20160403171240-cbb64ac3d964
go: finding github.com/olekukonko/tablewriter v0.0.4
go: finding github.com/mattn/go-runewidth v0.0.8
go: finding golang.org/x/net v0.0.0-20200219183655-46282727080f
user@host:~/.githubcli (master=)$ 

Cursory inspection of exec

user@host:~/.githubcli (master=)$ ls -l bin/gh
-rwxrwxr-x 1 user user 21097487 Mar  4 18:03 bin/gh
user@host:~/.githubcli (master=)$ 

user@host:~/.githubcli (master=)$ file bin/gh
bin/gh: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, not stripped
user@host:~/.githubcli (master=)$ 

user@host:~/.githubcli (master=)$ ldd bin/gh
	linux-vdso.so.1 =>  (0x00007ffdc4329000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb1018c2000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb1014f8000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fb101adf000)
user@host:~/.githubcli (master=)$ 

user@host:~/.githubcli (master=)$ nm -D bin/gh
                 U abort
00000000004fd140 T _cgo_panic
000000000045bf10 T _cgo_topofstack
00000000004fd190 T crosscall2
                 U __errno_location
                 U __fprintf_chk
                 U fputc
                 U free
                 U freeaddrinfo
                 U fwrite
                 U gai_strerror
                 U getaddrinfo
                 U getnameinfo
                 U malloc
                 U mmap
                 U munmap
                 U nanosleep
                 U pthread_attr_destroy
                 U pthread_attr_getstacksize
                 U pthread_attr_init
                 U pthread_cond_broadcast
                 U pthread_cond_wait
                 U pthread_create
                 U pthread_detach
                 U pthread_mutex_lock
                 U pthread_mutex_unlock
                 U pthread_sigmask
                 U setenv
                 U sigaction
                 U sigaddset
                 U sigemptyset
                 U sigfillset
                 U sigismember
                 U __stack_chk_fail
                 U stderr
                 U strerror
                 U unsetenv
                 U __vfprintf_chk
user@host:~/.githubcli (master=)$ 

Setup PATH & test

user@host:~/.githubcli (master=)$ cd
user@host:~$ echo 'export PATH="$HOME/.githubcli/bin:$PATH"' >> ~/.bash_profile
user@host:~$ bash --login
user@host:~$ gh
Work seamlessly with GitHub from the command line.

GitHub CLI is in early stages of development, and we'd love to hear your
feedback at <https://forms.gle/umxd3h31c7aMQFKG7>

Usage:
  gh [command]

Available Commands:
  help        Help about any command
  issue       Create and view issues
  pr          Create, view, and checkout pull requests
  repo        Create, clone, fork, and view repositories

Flags:
      --help              Show help for command
  -R, --repo OWNER/REPO   Select another repository using the OWNER/REPO format
      --version           Show gh version

Use "gh [command] --help" for more information about a command.
subcommand is required
user@host:~$
Automating wav to ogg conversion
Jun 6th, 2019 by miki

Needed some space on the disk.

Had some music, some of it were wav files.

Threw together this script to transmogify it into ogg using ffmpeg.

Statistics say that 24 wav files occupying 1,7 GiB was made into 24 ogg files occupying 105 MiB, a reduction to ~6% (du -h calculates 2^10 numbers but uses SI 10^3 prefixes..).

miki@filly:~/Music$ find . -iname "*.wav" -print0 | sed s/.wav/.ogg/g |du --files0-from=- -h -c|tail -1
105M	total
miki@filly:~/Music$ find . -iname "*.wav" -print0 | sed s/.wav/.wav/g |du --files0-from=- -h -c|tail -1
1,7G total
miki@filly:~/Music$ find . -iname "*.wav" -print0 | sed s/.wav/.ogg/g |du --files0-from=- -h |wc -l
24
miki@filly:~/Music$

Script

Install prerequisite ffmpeg and ffprobe utilities on apt distributions by doing “apt install ffmpeg”.

BEWARE: script deletes any wav files in current directory and below without warning if conversion to ogg is fine according to ffmpeg return code and stdout/err.

Manual confirmation of all deletions can be enabled by uncommenting line 8 (CONFIRM=..).

It could (very) easily be amended for conversion between any container or compression format supported by ffmpeg by replacing “wav” and “ogg” in glob pattern and OGG variable definition (for completeness also in echo output…).

miki@filly:~/wavtest$ cat ~/hometools/wav2ogg.sh
#! /bin/bash
#
# Make .ogg out of all .wav in cwd and below, remove .wav!
#
# When converting remove only if conversion is ok.
# ffprobe any present .ogg and check if playtime is same as wav, remove only if all is good.

#CONFIRM=-i # confirm deletion?

REQS="ffmpeg ffprobe"
which $REQS  >/dev/null
if [ ! $? -eq 0 ]; then
    echo We need $REQS to do stuff!
    exit 1
fi

shopt -s globstar nullglob nocaseglob # enable 2-star magic
for f in **/*.wav; do
    OGG=$(dirname "$f")/$(basename "$f" wav)ogg
    echo -n "$f: "
    if [ ! -f "$OGG" ]; then
        echo -en "\e[33mno ogg\e[0m"
        CONVERT=1
    else
        echo -en "\e[32malready there\e[0m"
        if [ ! "$(ffprobe "$f" 2>&1|grep Duration|cut -d, -f1)" = "$(ffprobe "$OGG" 2>&1|grep Duration|cut -d, -f1)" ]; then
            echo -en " - \e[31mogg corrupt\e[0m"
            CONVERT=1
            rm $CONFIRM "$OGG"
        else
            OGGOK=1
        fi
    fi
    if [ $CONVERT ]; then
        echo -en " - \e[33mconverting\e[0m"
        OUT=$(ffmpeg -loglevel error -i "$f" "$OGG" 2>&1)
        if [ ! $? -eq 0 ]; then
            echo -e " \e[31mfatal error!\e[0m\nffmpeg output was:\n\n$OUT\n"
            rm $CONFIRM "$OGG"
        elif [ ! -z "$OUT" ]; then
            echo -e " \e[31mnon-fatal error!\e[0m\nffmpeg output was:\n\n$OUT\n"
        else
            OGGOK=1
        fi
    fi
    if [ $OGGOK ]; then
        echo -e " - all good removing wav"
        rm $CONFIRM "$f"
    fi
done
miki@filly:~/wavtest$ 

Usage and Output

Before

miki@filly:~/wavtest$ ls -lR
.:
total 59964
drwxrwxr-x 3 miki miki     4096 Jun  6 22:33 one
-rw-rw-r-- 1 miki miki 61390864 Jun  6 22:32 Sordid Affair (Man Without Country Version).wav

./one:
total 30720
drwxrwxr-x 3 miki miki     4096 Jun  6 22:34 two
-rw-rw-r-- 1 miki miki 31451304 Jun  6 22:30 Vision Man MASTER.wav

./one/two:
total 55352
-rw-rw-r-- 1 miki miki 56675660 Jun  6 22:34 Bye Barat - Going For Broke (UNKWON´s Autumn Remix) LIM_02.wav
drwxrwxr-x 2 miki miki     4096 Jun  6 22:34 three

./one/two/three:
total 375112
-rw-rw-r-- 1 miki miki 384108056 Jun  6 22:30 Rigoremix - Pornographe - Final2.wav
miki@filly:~/wavtest$ du -hs .
509M	.
miki@filly:~/wavtest$

During

In real life, also in technicolor!

miki@filly:~/wavtest$ ~/hometools/wav2ogg.sh 
one/two/Bye Barat - Going For Broke (UNKWON´s Autumn Remix) LIM_02.wav: no ogg - converting - all good removing wav
one/two/three/Rigoremix - Pornographe - Final2.wav: no ogg - converting - all good removing wav
one/Vision Man MASTER.wav: no ogg - converting - all good removing wav
Sordid Affair (Man Without Country Version).wav: no ogg - converting - all good removing wav
miki@filly:~/wavtest$

After

miki@filly:~/wavtest$ ls -lR
.:
total 4952
drwxrwxr-x 3 miki miki    4096 Jun  6 22:35 one
-rw-rw-r-- 1 miki miki 5066460 Jun  6 22:35 Sordid Affair (Man Without Country Version).ogg

./one:
total 2528
drwxrwxr-x 3 miki miki    4096 Jun  6 22:34 two
-rw-rw-r-- 1 miki miki 2580752 Jun  6 22:35 Vision Man MASTER.ogg

./one/two:
total 4680
-rw-rw-r-- 1 miki miki 4785594 Jun  6 22:34 Bye Barat - Going For Broke (UNKWON´s Autumn Remix) LIM_02.ogg
drwxrwxr-x 2 miki miki    4096 Jun  6 22:35 three

./one/two/three:
total 19456
-rw-rw-r-- 1 miki miki 19922936 Jun  6 22:35 Rigoremix - Pornographe - Final2.ogg
miki@filly:~/wavtest$ du -hs .
31M .
miki@filly:~/wavtest$

Metadata

ffmpeg does some nice metadata conversions out of the box;

miki@filly:~/Music$ ffprobe ~/"wavbak/Delayscape  A Short While Rigolin Edit Audaccity.wav" 2>&1| grep Input -A 1000
Input #0, wav, from '/home/miki/wavbak/Delayscape  A Short While Rigolin Edit Audaccity.wav':
  Metadata:
    title           : Delayscape - A Short While (Rigolin Edit)
    artist          : rigolin
    comment         : Love Melody
    date            : 2018
  Duration: 00:06:33.97, bitrate: 3072 kb/s
    Stream #0:0: Audio: pcm_f32le ([3][0][0][0] / 0x0003), 48000 Hz, 2 channels, flt, 3072 kb/s
miki@filly:~/Music$ ffprobe "./Delayscape - Heinz Beauvaix/Soundcloud/Delayscape  A Short While Rigolin Edit Audaccity.ogg" 2>&1| grep Input -A 1000
Input #0, ogg, from './Delayscape - Heinz Beauvaix/Soundcloud/Delayscape  A Short While Rigolin Edit Audaccity.ogg':
  Duration: 00:06:33.97, start: 0.000000, bitrate: 99 kb/s
    Stream #0:0: Audio: vorbis, 48000 Hz, stereo, fltp, 112 kb/s
    Metadata:
      ENCODER         : Lavc56.60.100 libvorbis
      TITLE           : Delayscape - A Short While (Rigolin Edit)
      ARTIST          : rigolin
      comment         : Love Melody
      DATE            : 2018
miki@filly:~/Music$

Source Material

Silent browser redirect on invalid certificate
Nov 15th, 2018 by miki

Hit an odd browser behaviour today.

Trawling through some of those nice and dandy terms for a service (no I won’t tell) I followed a link and suddenly hit an “insecure connection” message from Firefox.

Examining the certificate using the usual “openssl s_client” and “openssl x509” tools surely enough revealed that the served certificate didn’t include the second-level but only on the third-level www sub-domain. Strangely enough I discovered that when entering the same URL directly into the address bar of Firefox the connection was somehow redirected to the www sub-domain and loaded fine without any complaints from Firefox.

Looking into what was happening on the wire using wget directed to not check the certificate (–no-check-certificate) and displaying server responses (–server-response/-S) revealed that the server behind the misconfigured certificate was aiming to issue a HTTP 301 redirect to the valid www sub-domain of the site (actual site replaced by foo.bar and localhost ip);

$ wget –server-response –no-check-certificate –output-document=/dev/null https://foo.bar
–2018-11-15 18:59:14– https://foo.bar/
Resolving foo.bar (foo.bar)… 127.0.0.1
Connecting to foo.bar (foo.bar)|127.0.0.1|:443… connected.
WARNING: no certificate subject alternative name matches
requested host name ‘foo.bar’.
HTTP request sent, awaiting response…
HTTP/1.1 301 Moved Permanently
Content-Type: text/html; charset=UTF-8
Location: https://www.foo.bar/
Date: Thu, 15 Nov 2018 17:59:13 GMT
Content-Length: 152
Location: https://www.foo.bar/ [following]

Apparently Firefox’s behaviour in this scenario differs depending on the method in which the user is supplying the URL, in some cases silently ignoring the TLS/SSL warning. A little more experimentation revealed that it was actually not the source of the URL, link followed versus one entered in address bar, that made any difference. Instead it is the presence of a trailing slash on the URL that triggers the silent suppression of the serious red flag that the second-level domain asked to be fetched is not present in the served certificate. I was fooled by the fact that when entering URLs in the address bar Firefox suggests ending the URL on a slash (‘/’) if it is not already present. Manually removing this when editing in the address bar also makes Firefox display the security warning. Alas, the link I originally followed was also missing the trailing slash and behaved as expected by throwing me into a security warning.

The whole “feature” of silently ignoring a security issue seemed very odd to me, but a bit of searching revealed that this was apparently championed by Google Chrome a couple of years ago. It is described in this servertastic post which also directs to a discussion on Twitter, with some key points replicated below, about its presence in Chrome and confirmation from a Chrome team member that the behaviour is intended.

The thread ends with a guy who had dug up the source code of the feature in Chromium (on which Chrome is based) known as “SSLCommonNameMismatchHandling” in file browser/ssl/common_name_mismatch_handler.cc (see all mentions across code base).

This has obviously also trickled down into Firefox, however, not much mention of that is to be found. Not even traces of it by some quick searches of the mozilla-central codebase. Some day I promise (really!) to dig through all source of Mozilla and hunt down the implementation, but for now I’ll revert to a bit of practical experimentation showing the behaviour in the different browsers and operating systems I happen to have access to at the moment;

  • Ubuntu 16.04
    • Firefox 62.0.3: warning, with trailing ‘/’: No warning
    • Chromium 69.0.3497.81: no warning, with trailing ‘/’: no warning
  • Windows Server 2016
    • Firefox 62.0.2: warning, with trailing ‘/’: no warning
    • Google Chrome 70.3538.102: no warning, with trailing ‘/’: no warning
    • Internet Explorer 11.2580.14393.0: warning, with trailing ‘/’: warning
  • Windows Server 2008
    • Firefox 59.0.3: warning, with trailing ‘/’: warning
    • Google Chrome 70.3538.102: no warning, with trailing ‘/’: no warning
    • Internet Explorer 11.0.9600.19080: warning, with trailing ‘/’: warning

So somewhere between 59.0.3 and 62.0.2 Firefox also implemented a policy of silently accepting invalid certificates when certain non-obvious criteria is met (is the redirect actually followed and certs checked, or is “www.” just prefixed?), but this happens only when the URL ends on a slash. Go figure…

Raw HTTP session with telnet
Jul 21st, 2010 by miki

Once in a while it is useful to dismiss abstractions and layers that makes daily routines easier and take the raw approach. Like when debugging a software problem that doesn’t make sense, it is nice to see the underlying basic stuff is behaving nicely, to better be able to locate where the unexpected occurs.

In the IP world of the internet, the swiss army knife for debugging interprocess communications in a totally protocol agnostic way is called ‘telnet’. Telnet opens up a communication channel between your local computer and a daemon/server on a specific port on a specific IP address. Then it gets out of the way for you to talk directly to the daemon in clear text.

Knowledge of how to interact using a specific protocol can be very useful to check server availability and functionality. All common protocols in use on the internet (like DNS, HTTP, SMTP, POP3, IMAP, XMPP etc.) can be debugged like this, because all of them transfers data in clear text (or at least initiates other transfer types from a clear text session). Full specifications for the HTTP protocol can be found in IETF RFC2616. I keep forgetting this, and end up digging around for it when needed, therefore this blog post.

Interactively using Telnet

Below is a basic HTTP session to my web server www.mikini.dk using telnet on the command line.

Red text is local text input by me. Blue text is local text by telnet application. Green text is server response.

$ telnet www.mikini.dk 80
Trying 92.61.152.47…
Connected to 92-61-152-47.static.servage.net.
Escape character is ‘^]’.
GET /index.php/2010/06 HTTP/1.1
Host: www.mikini.dk

HTTP/1.1 200 OK
Date: Wed, 21 Jul 2010 09:19:13 GMT
Server: Apache
X-Pingback:
http://www.mikini.dk/xmlrpc.php
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8

2bd2
<!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Transitional//EN” “
http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd“>

<… remainder of HTML document dropped …>

We are asking the server at www.mikini.dk port 80 (default http port) for the document at path /index.php/2010/06 (“GET /index.php/2010/06”). Notice the two CRLF characters after the Host header field, this indicates to the server that the request header is done, and that it should begin parsing header and send its response. Also notice that even though you tell the telnet program on the commandline that you want to access www.mikini.dk, you have to tell it again to the server in th HTTP Host field. Thats because telnet is only concerned about the IP address of the server, it resolves www.mikini.dk to the IP 92.61.152.47 through DNS and forgets about it. From the servers point of view, it needs to know which of its virtual hosts you want to talk to, cause one server application on one port on one ip can potentially host thousands of separate websites (virtual hosts).

One-liner using netcat (nc)

Below is the same session using nc (netcat) as a one-liner (wow, old PHP at Hostinger, better get that move to a self-administered box going).

$ echo -ne "GET /index.php/2010/06 HTTP/1.1\r\nHost:www.mikini.dk\r\n\r\n"|nc www.mikini.dk 80 | head -10
HTTP/1.1 200 OK
Date: Thu, 05 Jul 2018 16:30:07 GMT
Server: Apache
X-Powered-By: PHP/5.5.35
Link: <http://www.mikini.dk/wp-json/>; rel="https://api.w.org/"
Content-Length: 32356
Content-Type: text/html; charset=UTF-8

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

$

2018-07-05: nc command line added

»  Substance:WordPress   »  Style:Ahren Ahimsa
© 2020 Mikkel Kirkgaard Nielsen, contents CC BY-SA 4.0