SIDEBAR
»
S
I
D
E
B
A
R
«
GeoNames.org database id becoming a meteorological location reference
Aug 20th, 2020 by miki

Looking at weather forecasts in my native Denmark I have always noticed a location reference used by DMI, Danmarks Meteorologiske Institut (en:Danish Meteorological Institute). Somehow I managed to not look into the details of that before now.

But this morning while scouting for rain showers and trying out the forecast on the new website of Yr (the long time open data weather service from Norwegian Broadcasting Corporation (NRK) and the Norwegian Meteorological institute (MET)), as kindly suggested by the YR main site which uses the old site by default, I saw the same reference included in the URL.

A closer look at different service’s forecast URLs for my usual whereabouts in Tjæreborg, Denmark, Europe, Earth reveals (se summary below) that they all share this reference and that it stems from the GeoNames.org (wikipedia article) database’s integer reference to the location:

This particular piece of data is described as a field in the “geoname” table of the GeoNames.org main database, and referred to as “geonameid” in the documentation:

geonameid         : integer id of record in geonames database

The contents of the GeoNames.org database is licensed under the Creative Commons Attribution 4.0 International (aka. “CC BY 4.0”, SPDX ID:”CC-BY-4.0″) and available both for your own download and perusal (documentation here) or using web services on geonames.org.

Note, however, that GeoNames.org accumulates a wealth of data sources that according to the OpenStreetMap project might contain copyrighted data. Together with the attribution requirements of its CC BY license this causes OpenStreetMap to not accept data from GeoNames.org into the project’s, ODbL licensed, database.

Fun fact: in Norwegian “yr” actually means drizzle (da:støvregn)
Practical hint: OpenSearch entry for GeoNames.org: http://www.geonames.org/opensearch-description.xml (to add in current browser: go here->find “Plugins”->click “opensearch plugin”)

The Weather Services

2611610 on GeoNames.org

“Get”ting from web service

$ wget -q -O- "https://secure.geonames.org/get?geonameId=2611610&username=demo"
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<geoname>
    <toponymName>Tjæreborg</toponymName>
    <name>Tjæreborg</name>
    <lat>55.46457</lat>
    <lng>8.57968</lng>
    <geonameId>2611610</geonameId>
    <countryCode>DK</countryCode>
    <countryName>Denmark</countryName>
    <fcl>P</fcl>
    <fcode>PPL</fcode>
    <fclName>city, village,...</fclName>
    <fcodeName>populated place</fcodeName>
    <population>2146</population>
    <adminCode1 ISO3166-2="83">21</adminCode1>
    <adminName1>South Denmark</adminName1>
    <asciiName>Tjaereborg</asciiName>
    <alternateNames>Tjaereborg,Tjæreborg</alternateNames>
    <elevation/>
    <srtm3>12</srtm3>
    <astergdem>11</astergdem>
    <continentCode>EU</continentCode>
    <adminCode2>561</adminCode2>
    <adminName2>Esbjerg Kommune</adminName2>
    <alternateName lang="post">6731</alternateName>
    <alternateName lang="unlc">DKTJA</alternateName>
    <alternateName lang="link">https://en.wikipedia.org/wiki/Tj%C3%A6reborg%2C_Denmark</alternateName>
    <alternateName>Tjæreborg</alternateName>
    <timezone dstOffset="2.0" gmtOffset="1.0">Europe/Copenhagen</timezone>
    <bbox>
        <west>8.57175</west>
        <north>55.46907</north>
        <east>8.58762</east>
        <south>55.46008</south>
        <accuracyLevel>0</accuracyLevel>
    </bbox>
</geoname>
$

Extracting from “Gazetteer” exports

$ wget -q https://download.geonames.org/export/dump/DK.zip
$ unzip DK.zip 
Archive:  DK.zip
  inflating: readme.txt              
  inflating: DK.txt                  
$ grep ^2611610 DK.txt
2611610	Tjæreborg	Tjaereborg	Tjaereborg,Tjæreborg	55.46457	8.57968	P	PPL	DK\
		21	561			2146		12	Europe/Copenhagen	2017-10-18
$

 

Amending an open OSM changeset on command line (by hand)
Jul 25th, 2020 by miki

WARNING: as OSM user “mmd” wisely points out (in comment to OSM diary for this post) the sort of thing described here is dangerous to do by hand, and should not be done on the main production instances (there are testing instances for playing around with the API, a little documentation here).
He also points out that the feature packed JOSM editor actually supports continuing a changeset regardless of where it has been initiated. So if you just need to continue working on a changeset (but remember the 1 hour idle timeout) be sure to check out the JSOM upload documentation. Thanks mmd, for being the sane voice ;).

During an editing session the Android OpenStreetMap editor Vespucci crashed on me, which made the mapping UI unusable (objects greyed out and unable to select or edit, had to purge all data to recover functionality). Luckly, I could still navigate the menus, upload changes and opt to not close the changeset. Now, I had long wondered whether the OSM API allowed to continue amending changes to an open changeset from some other editor, so the quest began.

I had the intention of adding a tag representing the name of a building in Esbjerg known as “ISC Huset” (ISC is an engineering consultancy, see wikipedia, and more about the construction), which houses a number of healtcare clinics. The building’s address is Borgergade 70, 6700 Esbjerg (current address node).

This blog post willl attempt to actually add the tag by hand on the command line.

References to OSM objects used:

Option summary for GNU wget used to do HTTP requests below:

  • -nv (or –no-verbose): only report errors and basic information
  • -O- (or –output-document=-): write output to standard out (stdout)
  • –user=<user name>: authenticate as <user name> using HTTP Basic authentication
  • –ask-password: ask on command line for HTTP Basic authentication password
  • –post-data=”<data>”: initiate a HTTP POST request with <data> as payload
  • –server-response (or -S): show full HTTP headers of server response

NOTE: I’ve broken some long output lines replicated below to make it fit the blog, but inserted an escape character (\) before the inserted newline to help copy’n’pasting.

Retrieve changeset metadata

Lets start by looking at the changeset’s metadata.

This can be done by issuing an unauthenticated GET request to the “/api/0.6/changeset/<changeset id>” endpoint.

Note that the ‘changeset’ element has the attribute ‘open=”true”‘ required to be able to modify the changeset. The editor used to create the changeset needs to have done this without explicitly closing it (Vespucci & JOSM closes by default but can be configured not to, iD always closes).

$ wget -nv -O- http://api.openstreetmap.org/api/0.6/changeset/88490797
<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="CGImap 0.8.3 (802 thorn-02.openstreetmap.org)" copyright="OpenStreetMap and contributors"\
 attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/">
 <changeset id="88490797" created_at="2020-07-25T09:51:30Z" open="true" user="mikini" uid="2051"\
 min_lat="55.4654072" min_lon="8.4378053" max_lat="55.5258532" max_lon="8.4639026" comments_count="0" changes_count="68">
  <tag k="source" v="survey;research"/>
  <tag k="locale" v="da-DK"/>
  <tag k="created_by" v="Vespucci 14.1.4.0"/>
  <tag k="comment" v="Details at Klevestien &amp; Borgergade in Esbjerg and Tarp."/>
 </changeset>
</osm>
2020-07-25 14:26:52 URL:http://api.openstreetmap.org/api/0.6/changeset/88490797 [709] -> "-" [1]
$

Pinging the changeset with an empty osmChange structure

To test that the changeset is open, and that we can authenticate correctly, lets try amending it with an empty osmChange structure.

This can be done by issuing an authenticated POST request to the “/api/0.6/changeset/<changeset id>/upload” endpoint.

This also seem to reset the 60 minute timer used for auto-closing the changeset (see mention of “idle timeout” in changeset wiki article).

$ wget -nv -O- --user=mikini --ask-password https://api.openstreetmap.org/api/0.6/changeset/88490797/upload --post-data="<osmChange></osmChange>"
Password for user ‘mikini’: 
Authentication selected: Basic realm="Web Password", charset="UTF-8"
<?xml version="1.0" encoding="UTF-8"?>
<diffResult version="0.6" generator="CGImap 0.8.3 (8531 thorn-03.openstreetmap.org)" copyright="OpenStreetMap and contributors"\
 attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/"/>
2020-07-25 14:27:20 URL:https://api.openstreetmap.org/api/0.6/changeset/88490797/upload [278] -> "-" [1]
$

Procedure to change the building

The API’s “changeset/<changeset id>/upload” method supports only modifications encoded in the osmChange format which requires changes to be described as complete way/node/relation objects. That is, you can not ask the API to “add this tag to this way”, you need instead to describe the modified way completely saying “this way now looks like this”, including anything (like node references or existing tags) that was not modified. So to make a modificatino to the building’s way we need to retrieve the current version, modify it as needed and upload the complete new way.

Thus the procedure contains these steps;

  1. Retrieve current building (GET /way/*)
  2. Modify building data (locally)
  3. Amend changeset (POST /changeset/*/upload)

Retrieving current building (GET /way/*)

This can be done by issuing an unauthenticated GET request to the “/api/0.6/way/<way id>” endpoint.

The stdin splitter ‘tee’ is used here to both show the result in terminal and put it into file 185369466_v3.osc that we can use for amending the way with the wanted modifications.

$ wget -nv -O- http://api.openstreetmap.org/api/0.6/way/185369466|tee 185369466_v3.osc
<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="CGImap 0.8.3 (28697 thorn-01.openstreetmap.org)" copyright="OpenStreetMap and contributors"\
 attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/">
 <way id="185369466" visible="true" version="3" changeset="84400254" timestamp="2020-04-30T09:35:21Z" user="mikini" uid="2051">
  <nd ref="1959614623"/>
  <nd ref="1959614727"/>
  <nd ref="6299449794"/>
  <nd ref="1959614650"/>
  <nd ref="1959614630"/>
  <nd ref="6299449793"/>
  <nd ref="1959614765"/>
  <nd ref="7466482063"/>
  <nd ref="7466482064"/>
  <nd ref="7466482065"/>
  <nd ref="7466482062"/>
  <nd ref="1959614729"/>
  <nd ref="1959614623"/>
  <tag k="building" v="yes"/>
 </way>
</osm>
2020-07-25 14:54:30 URL:http://api.openstreetmap.org/api/0.6/way/185369466 [769] -> "-" [1]
$

Modify building data (locally)

Now we need to build an osmChange file out of the existing <way>…</way> element from the output above describing the wanted building. This involves;

  1. removing the <xml> tag (maybe not needed, but it is not mentioned at all in the osmChange documentation)
  2. replacing <osm> with <osmChange>
  3. adding a <modify> element to the <osmChange> element, containing the existing <way>
  4. update the “changeset” attribute to the changeset we’re amending
  5. amending the contents of <way> with the wanted <tag>s

Use your favorite editor (emacs would be my preference) to load the 185369466_v3.osm file, make the modifications and save it as 185369466_v4.osc. OSM tags are a XML empty-element tags containing the OSM tag’s key and value in the “k” and “v” attributes, thus the “name” tag of the building I needed to add would be ‘<tag k=”name” v=”ISC Huset”/>’, I also added some other related tags (“source:name” and “website”).

The finished .osc file now looks like this;

$ cat 185369466_v4.osc
<osmChange>
  <modify>
    <way id="185369466" visible="true" version="3" changeset="88490797" timestamp="2020-04-30T09:35:21Z" user="mikini" uid="2051">
      <nd ref="1959614623"/>
      <nd ref="1959614727"/>
      <nd ref="6299449794"/>
      <nd ref="1959614650"/>
      <nd ref="1959614630"/>
      <nd ref="6299449793"/>
      <nd ref="1959614765"/>
      <nd ref="7466482063"/>
      <nd ref="7466482064"/>
      <nd ref="7466482065"/>
      <nd ref="7466482062"/>
      <nd ref="1959614729"/>
      <nd ref="1959614623"/>
      <tag k="building" v="yes"/>
      <tag k="name" v="ISC Huset"/>
      <tag k="source:name" v="sign;website"/>
      <tag k="website" v="https://www.isc.dk/isc-huset-esbjerg/"/>
    </way>
  </modify>
</osmChange>
$

Wdiff’ing against the .osm source shows exactly what changed (additions between “{+” & “+}”, removals between “[-” & “-]”);

$ wdiff 185369466_v3.osm 185369466_v4.osc
[-<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="CGImap 0.8.3 (28697 thorn-01.openstreetmap.org)" copyright="OpenStreetMap and contributors"\
 attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/">-]{+<osmChange>
  <modify>+}
    <way id="185369466" visible="true" version="3" [-changeset="84400254"-] {+changeset="88490797"+} timestamp="2020-04-30T09:35:21Z" user="mikini" uid="2051">
      <nd ref="1959614623"/>
      <nd ref="1959614727"/>
      <nd ref="6299449794"/>
      <nd ref="1959614650"/>
      <nd ref="1959614630"/>
      <nd ref="6299449793"/>
      <nd ref="1959614765"/>
      <nd ref="7466482063"/>
      <nd ref="7466482064"/>
      <nd ref="7466482065"/>
      <nd ref="7466482062"/>
      <nd ref="1959614729"/>
      <nd ref="1959614623"/>
      <tag k="building" v="yes"/>
      {+<tag k="name" v="ISC Huset"/>
      <tag k="source:name" v="sign;website"/>
      <tag k="website" v="https://www.isc.dk/isc-huset-esbjerg/"/>+}
    </way>
[-</osm>-]
  {+</modify>
</osmChange>+}
$

Amend changeset (POST /changeset/*/upload)

Now, we’ll again use the changeset upload method but this time supplying our actual osmChange elemet in the .osc file.

The output is a bit elaborate, because I had enabled full output from wget while debugging what changes to the <way> element was needed for the server to accept the upload (only the “changeset” attribute needs to match the open changeset as outlined in the “Modify building data” above). I’ve highligted the actual server response telling that the changes were accepted and way #185369466 is now at v4.

$ wget -S -O- --user=mikini --ask-password https://api.openstreetmap.org/api/0.6/changeset/88490797/upload --post-file=185369466_v4.osc
Password for user ‘mikini’: 
--2020-07-25 15:44:41--  https://api.openstreetmap.org/api/0.6/changeset/88490797/upload
Resolving api.openstreetmap.org (api.openstreetmap.org)... 130.117.76.12, 130.117.76.13, 130.117.76.11, ...
Connecting to api.openstreetmap.org (api.openstreetmap.org)|130.117.76.12|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 401 Unauthorized
  Date: Sat, 25 Jul 2020 13:44:41 GMT
  Server: Apache/2.4.29 (Ubuntu)
  Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
  Expect-CT: max-age=0, report-uri="https://openstreetmap.report-uri.com/r/d/ct/reportOnly"
  WWW-Authenticate: Basic realm="Web Password", charset="UTF-8"
  Cache-Control: no-cache
  Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
  Expect-CT: max-age=0, report-uri="https://openstreetmap.report-uri.com/r/d/ct/reportOnly"
  Content-Length: 22
  Content-Type: text/plain; charset=utf-8
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
Authentication selected: Basic realm="Web Password", charset="UTF-8"
Reusing existing connection to api.openstreetmap.org:443.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Sat, 25 Jul 2020 13:44:42 GMT
  Server: Apache/2.4.29 (Ubuntu)
  Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
  Expect-CT: max-age=0, report-uri="https://openstreetmap.report-uri.com/r/d/ct/reportOnly"
  Content-Encoding: identity
  Cache-Control: private, max-age=0, must-revalidate
  Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
  Expect-CT: max-age=0, report-uri="https://openstreetmap.report-uri.com/r/d/ct/reportOnly"
  Vary: Accept-Encoding
  Content-Type: application/xml; charset=utf-8
  Keep-Alive: timeout=5, max=99
  Connection: Keep-Alive
  Transfer-Encoding: chunked
Length: unspecified [application/xml]
Saving to: ‘STDOUT’
- [<=> ] 0 --.-KB/s
<?xml version="1.0" encoding="UTF-8"?>
<diffResult version="0.6" generator="CGImap 0.8.3 (8537 thorn-03.openstreetmap.org)" copyright="OpenStreetMap and contributors"\
 attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/">
 <way old_id="185369466" new_id="185369466" new_version="4"/>
</diffResult>
- [ <=> ] 353 --.-KB/s in 0s

2020-07-25 15:44:42 (20,3 MB/s) - written to stdout [353]

$

Result

That was it, the building is now named in OSM, in a changset amended by hand.

Take a look at https://www.openstreetmap.org/way/185369466/.

OSM Use & Attribute: Bisnode Navne & Numre Erhverv website
Mar 28th, 2019 by miki

OSM Use & Attribute?

I regularly stumble upon maps which I cane recognize as derived from OpenStreetMap. Usually by map style, the level of detail or by seeking out details I know is only present in OSM. I usually inspect the attribution and license accompanying the material to assess whether it is prepared by an organisation who understands the needs to and cares about having its chain of copyright somewhat traceable. I’ll attempt to do a small concise writeup of the findings going along this in the future. Maybe I’ll be able to produce a series of “OSM Use & Attribute” posts?

Quite randomly the first one became the Danish company Bisnode.dk using OSM tiles from Mapquest (the ones who fled from an open approach in 2015) to indicate locations in the commercial and subscription-required yellow pages product named Bisnode Navne & Numre® Erhverv (aka. erhverv.nnmarkedsdata.dk).

Attribution Compliance Analysis

For reference: OSM’s Copyright and License page

Below is a screen-shot from within the walled garden, and the actual link leading to the page (accessible to me because I’m the owner of the company in question).

Bisnode obviously uses Mapquest as supplier of their maps. Mapquest do make sure to display a nice and prominent copyright attribution line in the map with text linking to respectively Mapquest’s Mapbox attribution page (“MAPBOX” text), OpenStreetMap’s about page (“OPENSTREETMAP” text) and Mapquest’s TOS (“TERMS” text).

The OSM attribution doesn’t comply exactly with the words of the OSM attribution instructions; to use the text “© OpenStreetMap contributors” and make it clear that data is distributed under the ODbL license (suggested done by linking to the OSM copyright page which only happens indirectly through the about page). This is almost fulfilled on the 3rd Party Notices and Licences which is, however, buried behind the long read of the Mapquest TOS, and where the attribution strangely becomes the rather twisted “© Open Street Map and Contributors”. Who are OSM beyond its contributors? And why doesn’t Mapquest know the correct name of the project? Is it to prevent search engine hits of the more unique concatenation of the three words? That’s a guess.

Strangely an almost similar text but also with a reference to the ODbL is to be found in the “Mapbox Streets” section of the Mapbox attribution page.

Tile Server Compliance Analysis

For reference: OSM/OSMF’s Tile Usage Policy

The tiles are obviously rendered using the Mapquest cartography and thus can be served only by Mapquest.

 

Bookmarklet to view OSM from coordinates parsed from URL
Aug 12th, 2016 by miki

Here’s a quick Javascript bookmarklet I threw together for opening a new browser window showing an area in OpenStreetMap that is defined from extracting the current window’s URL and searching after useable values for latitude, longitude and zoom.

It grew out of an annoyance over Mapillary’s rendering of the Danish endpoint of the under construction HVDC Cobra Cable(more about it from 4C, even more from EnergiNet and the EU) in Endrup nearby where I live.

In Mapillary’s tiles rendering it is just a bunch of roads and a single POI indication. Whereas the Mapnik rendering shows the existing power infrastructure and the area under construction.

Cobra HVDC, Endrup, Mapillary vs. OpenStreetMap

Well, I have been subjecting my oldstyle C brain to some Javascript lately so I decided to use that haemorrhage for attempting to put together a bookmarklet extracting coordinates from the current window’s URL and opening a new with the same approximate location in OSM.

It ended up like the below code block, and should also be usable on any other sites which receives locations via URL (both using HTTP GET notation with ‘?’ and locally in the page using anchors with ‘#’) and identifying them with key-value pairs using common names ({z,zoom},{lat},{lng,lon}). Note that this doesn’t include OpenStreetMap itself neither Google Maps as they only use the lat/lon values.

javascript:(function (){params={};kvs=document.location.href.split('&');kvs.forEach(function(kv){if(kv.indexOf('?'))kv=kv.substr(kv.indexOf('?')+1);if(kv.indexOf('#'))kv=kv.substr(kv.indexOf('#')+1);skv=kv.split('=');params[skv[0]]=skv[1];});window.open('http://openstreetmap.org/#map='+(params.z?params.z:13)+'/'+(params.lat?params.lat:55.5)+'/'+(params.lng?params.lng:(params.lon?params.lon:8.5)));console.log(params);})();

Copy and paste the above into the “Location” or “URL” of a bookmark and you’ll be able to click it to open a new OSM window on, at least for Mapillary maps pages, the same location as the original site. If nothing is found it will default to coordinates of my hometown of Esbjerg at 55.5/.8.5.

Here’s a prettified edition of the code:

params={};
kvs=document.location.href.split('&');
kvs.forEach(function(kv){
  if(kv.indexOf('?'))
    kv=kv.substr(kv.indexOf('?')+1);
  if(kv.indexOf('#'))
    kv=kv.substr(kv.indexOf('#')+1);
  skv=kv.split('=');
  params[skv[0]]=skv[1];
});
window.open( 'http://openstreetmap.org/#map='
            +(params.z?params.z:(params.zoom?params.zoom:13))+'/'
            +(params.lat?params.lat:55.5)+'/'
            +(params.lng?params.lng:(params.lon?params.lon:8.5))
           );
console.log(params);

Possible TODOs

  • get rid of default location, warn instead
  • support and test more sites (gmaps/osm!)
  • scan page contents for other geo markers
Street View Googling Denmark again
Aug 19th, 2010 by miki

Just saw the Google Street View car driving by at my daily whereabouts in Esbjerg, Denmark. Did get a little exhilerated and spilled some coffee in excitement and attempt to wave at the cameras (Hi mom!!) . Here’s a shot of the car taken by itself when passing my home at it’s last visit in 2009.

Apparently Google Streetview has started collecting data again here i Denmark, as they did in Norway, Sweden, Ireland and South Africa in July. Now with the Wi-Fi privacy issue resolved.

The following netfrenzy brought me to the official schedule at http://www.google.dk/help/maps/streetview/where-is-street-view.html which shows some mid to large sized cities in Denmark beeing up for an update, including Esbjerg. Strangely I haven’t noticed any lack of imagery in my local area. I know that most of Denmark was beeing photographed during July and August 2009, so something must be missing, or of inferior quality since they are coming back.

I also found a fun Street View Partner site; Legoland in California has allowed Street View to take a tour through the park. Fun for me, because the Lego Group HQ and original Legoland Billund is close to Esbjerg, and I have been going there regularly wiht the kids. Going to be fun to take them for a virtual walk in the park in California ;). The Billund park doesn’t seem to have been mapped internally, though.

Also a hilarious joke on Google Streetview from an interesting group of people; The Free Art and Technology (F.A.T.) Lab. Will be digging these nutty people out for sure.

»  Substance:WordPress   »  Style:Ahren Ahimsa
© 2020 Mikkel Kirkgaard Nielsen, contents CC BY-SA 4.0