WARNING: as OSM user “mmd” wisely points out (in comment to OSM diary for this post) the sort of thing described here is dangerous to do by hand, and should not be done on the main production instances (there are testing instances for playing around with the API, a little documentation here). He also points out that the feature packed JOSM editor actually supports continuing a changeset regardless of where it has been initiated. So if you just need to continue working on a changeset (but remember the 1 hour idle timeout) be sure to check out the JSOM upload documentation. Thanks mmd, for being the sane voice ;).
During an editing session the Android OpenStreetMap editor Vespucci crashed on me, which made the mapping UI unusable (objects greyed out and unable to select or edit, had to purge all data to recover functionality). Luckly, I could still navigate the menus, upload changes and opt to not close the changeset. Now, I had long wondered whether the OSM API allowed to continue amending changes to an open changeset from some other editor, so the quest began.
I had the intention of adding a tag representing the name of a building in Esbjerg known as “ISC Huset” (ISC is an engineering consultancy, see wikipedia, and more about the construction), which houses a number of healtcare clinics. The building’s address is Borgergade 70, 6700 Esbjerg (current address node).
This blog post willl attempt to actually add the tag by hand on the command line.
References to OSM objects used:
Option summary for GNU wget used to do HTTP requests below:
NOTE: I’ve broken some long output lines replicated below to make it fit the blog, but inserted an escape character (\) before the inserted newline to help copy’n’pasting.
Lets start by looking at the changeset’s metadata.
This can be done by issuing an unauthenticated GET request to the “/api/0.6/changeset/<changeset id>” endpoint.
Note that the ‘changeset’ element has the attribute ‘open=”true”‘ required to be able to modify the changeset. The editor used to create the changeset needs to have done this without explicitly closing it (Vespucci & JOSM closes by default but can be configured not to, iD always closes).
$ wget -nv -O- http://api.openstreetmap.org/api/0.6/changeset/88490797 <?xml version="1.0" encoding="UTF-8"?> <osm version="0.6" generator="CGImap 0.8.3 (802 thorn-02.openstreetmap.org)" copyright="OpenStreetMap and contributors"\ attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/"> <changeset id="88490797" created_at="2020-07-25T09:51:30Z" open="true" user="mikini" uid="2051"\ min_lat="55.4654072" min_lon="8.4378053" max_lat="55.5258532" max_lon="8.4639026" comments_count="0" changes_count="68"> <tag k="source" v="survey;research"/> <tag k="locale" v="da-DK"/> <tag k="created_by" v="Vespucci 14.1.4.0"/> <tag k="comment" v="Details at Klevestien & Borgergade in Esbjerg and Tarp."/> </changeset> </osm> 2020-07-25 14:26:52 URL:http://api.openstreetmap.org/api/0.6/changeset/88490797 [709] -> "-" [1] $
To test that the changeset is open, and that we can authenticate correctly, lets try amending it with an empty osmChange structure.
This can be done by issuing an authenticated POST request to the “/api/0.6/changeset/<changeset id>/upload” endpoint.
This also seem to reset the 60 minute timer used for auto-closing the changeset (see mention of “idle timeout” in changeset wiki article).
$ wget -nv -O- --user=mikini --ask-password https://api.openstreetmap.org/api/0.6/changeset/88490797/upload --post-data="<osmChange></osmChange>" Password for user ‘mikini’: Authentication selected: Basic realm="Web Password", charset="UTF-8" <?xml version="1.0" encoding="UTF-8"?> <diffResult version="0.6" generator="CGImap 0.8.3 (8531 thorn-03.openstreetmap.org)" copyright="OpenStreetMap and contributors"\ attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/"/> 2020-07-25 14:27:20 URL:https://api.openstreetmap.org/api/0.6/changeset/88490797/upload [278] -> "-" [1] $
The API’s “changeset/<changeset id>/upload” method supports only modifications encoded in the osmChange format which requires changes to be described as complete way/node/relation objects. That is, you can not ask the API to “add this tag to this way”, you need instead to describe the modified way completely saying “this way now looks like this”, including anything (like node references or existing tags) that was not modified. So to make a modificatino to the building’s way we need to retrieve the current version, modify it as needed and upload the complete new way.
Thus the procedure contains these steps;
This can be done by issuing an unauthenticated GET request to the “/api/0.6/way/<way id>” endpoint.
The stdin splitter ‘tee’ is used here to both show the result in terminal and put it into file 185369466_v3.osc that we can use for amending the way with the wanted modifications.
$ wget -nv -O- http://api.openstreetmap.org/api/0.6/way/185369466|tee 185369466_v3.osc <?xml version="1.0" encoding="UTF-8"?> <osm version="0.6" generator="CGImap 0.8.3 (28697 thorn-01.openstreetmap.org)" copyright="OpenStreetMap and contributors"\ attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/"> <way id="185369466" visible="true" version="3" changeset="84400254" timestamp="2020-04-30T09:35:21Z" user="mikini" uid="2051"> <nd ref="1959614623"/> <nd ref="1959614727"/> <nd ref="6299449794"/> <nd ref="1959614650"/> <nd ref="1959614630"/> <nd ref="6299449793"/> <nd ref="1959614765"/> <nd ref="7466482063"/> <nd ref="7466482064"/> <nd ref="7466482065"/> <nd ref="7466482062"/> <nd ref="1959614729"/> <nd ref="1959614623"/> <tag k="building" v="yes"/> </way> </osm> 2020-07-25 14:54:30 URL:http://api.openstreetmap.org/api/0.6/way/185369466 [769] -> "-" [1] $
Now we need to build an osmChange file out of the existing <way>…</way> element from the output above describing the wanted building. This involves;
Use your favorite editor (emacs would be my preference) to load the 185369466_v3.osm file, make the modifications and save it as 185369466_v4.osc. OSM tags are a XML empty-element tags containing the OSM tag’s key and value in the “k” and “v” attributes, thus the “name” tag of the building I needed to add would be ‘<tag k=”name” v=”ISC Huset”/>’, I also added some other related tags (“source:name” and “website”).
The finished .osc file now looks like this;
$ cat 185369466_v4.osc <osmChange> <modify> <way id="185369466" visible="true" version="3" changeset="88490797" timestamp="2020-04-30T09:35:21Z" user="mikini" uid="2051"> <nd ref="1959614623"/> <nd ref="1959614727"/> <nd ref="6299449794"/> <nd ref="1959614650"/> <nd ref="1959614630"/> <nd ref="6299449793"/> <nd ref="1959614765"/> <nd ref="7466482063"/> <nd ref="7466482064"/> <nd ref="7466482065"/> <nd ref="7466482062"/> <nd ref="1959614729"/> <nd ref="1959614623"/> <tag k="building" v="yes"/> <tag k="name" v="ISC Huset"/> <tag k="source:name" v="sign;website"/> <tag k="website" v="https://www.isc.dk/isc-huset-esbjerg/"/> </way> </modify> </osmChange> $
Wdiff’ing against the .osm source shows exactly what changed (additions between “{+” & “+}”, removals between “[-” & “-]”);
$ wdiff 185369466_v3.osm 185369466_v4.osc [-<?xml version="1.0" encoding="UTF-8"?> <osm version="0.6" generator="CGImap 0.8.3 (28697 thorn-01.openstreetmap.org)" copyright="OpenStreetMap and contributors"\ attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/">-]{+<osmChange> <modify>+} <way id="185369466" visible="true" version="3" [-changeset="84400254"-] {+changeset="88490797"+} timestamp="2020-04-30T09:35:21Z" user="mikini" uid="2051"> <nd ref="1959614623"/> <nd ref="1959614727"/> <nd ref="6299449794"/> <nd ref="1959614650"/> <nd ref="1959614630"/> <nd ref="6299449793"/> <nd ref="1959614765"/> <nd ref="7466482063"/> <nd ref="7466482064"/> <nd ref="7466482065"/> <nd ref="7466482062"/> <nd ref="1959614729"/> <nd ref="1959614623"/> <tag k="building" v="yes"/> {+<tag k="name" v="ISC Huset"/> <tag k="source:name" v="sign;website"/> <tag k="website" v="https://www.isc.dk/isc-huset-esbjerg/"/>+} </way> [-</osm>-] {+</modify> </osmChange>+} $
Now, we’ll again use the changeset upload method but this time supplying our actual osmChange elemet in the .osc file.
The output is a bit elaborate, because I had enabled full output from wget while debugging what changes to the <way> element was needed for the server to accept the upload (only the “changeset” attribute needs to match the open changeset as outlined in the “Modify building data” above). I’ve highligted the actual server response telling that the changes were accepted and way #185369466 is now at v4.
$ wget -S -O- --user=mikini --ask-password https://api.openstreetmap.org/api/0.6/changeset/88490797/upload --post-file=185369466_v4.osc Password for user ‘mikini’: --2020-07-25 15:44:41-- https://api.openstreetmap.org/api/0.6/changeset/88490797/upload Resolving api.openstreetmap.org (api.openstreetmap.org)... 130.117.76.12, 130.117.76.13, 130.117.76.11, ... Connecting to api.openstreetmap.org (api.openstreetmap.org)|130.117.76.12|:443... connected. HTTP request sent, awaiting response... HTTP/1.1 401 Unauthorized Date: Sat, 25 Jul 2020 13:44:41 GMT Server: Apache/2.4.29 (Ubuntu) Strict-Transport-Security: max-age=31536000; includeSubDomains; preload Expect-CT: max-age=0, report-uri="https://openstreetmap.report-uri.com/r/d/ct/reportOnly" WWW-Authenticate: Basic realm="Web Password", charset="UTF-8" Cache-Control: no-cache Strict-Transport-Security: max-age=31536000; includeSubDomains; preload Expect-CT: max-age=0, report-uri="https://openstreetmap.report-uri.com/r/d/ct/reportOnly" Content-Length: 22 Content-Type: text/plain; charset=utf-8 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Authentication selected: Basic realm="Web Password", charset="UTF-8" Reusing existing connection to api.openstreetmap.org:443. HTTP request sent, awaiting response... HTTP/1.1 200 OK Date: Sat, 25 Jul 2020 13:44:42 GMT Server: Apache/2.4.29 (Ubuntu) Strict-Transport-Security: max-age=31536000; includeSubDomains; preload Expect-CT: max-age=0, report-uri="https://openstreetmap.report-uri.com/r/d/ct/reportOnly" Content-Encoding: identity Cache-Control: private, max-age=0, must-revalidate Strict-Transport-Security: max-age=31536000; includeSubDomains; preload Expect-CT: max-age=0, report-uri="https://openstreetmap.report-uri.com/r/d/ct/reportOnly" Vary: Accept-Encoding Content-Type: application/xml; charset=utf-8 Keep-Alive: timeout=5, max=99 Connection: Keep-Alive Transfer-Encoding: chunked Length: unspecified [application/xml] Saving to: ‘STDOUT’ - [<=> ] 0 --.-KB/s <?xml version="1.0" encoding="UTF-8"?> <diffResult version="0.6" generator="CGImap 0.8.3 (8537 thorn-03.openstreetmap.org)" copyright="OpenStreetMap and contributors"\ attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/"> <way old_id="185369466" new_id="185369466" new_version="4"/> </diffResult> - [ <=> ] 353 --.-KB/s in 0s 2020-07-25 15:44:42 (20,3 MB/s) - written to stdout [353] $
That was it, the building is now named in OSM, in a changset amended by hand.
Take a look at https://www.openstreetmap.org/way/185369466/.
Here’s a preliminary HOWTO from my recent external flashing of the BIOS ROM on a Lenovo X200 thinkpad (Wikipedia). The particular firmware flashed was a Libreboot build for this machine (instructions for X200 and details of the external flashing procedure) but anything goes (but anything may not be useful, though).
I’ll amend this HOWTO with more detailed instructions and pictures in the following days (warning: a prediction) to hopefully make it more complete and useful for the vary inhabitants of LibreBootLand.
A programmer kit was bought on AliExpress for $4.20 containing a USB programmer board and an SOIC-8 clip which ended up not being used as the particular X200 had a SOIC-16 chip so a separately ordered SOIC-16 chip ($3.11) was used.
Buy: https://www.aliexpress.com/item/32898599200.html @ $4.20
The programmer is based on the WCH CH341a chip which is an USB <-> seriel/parallel/uart interface. The manufacturer WCH being WinChipHead aka. WCH (Nanjing QinHeng Electronics Co.,Ltd) (maybe also aka. WCH-IC (Jiangsu Qinheng Co., Ltd)). There are lots of options for buying board varieties based on the CH341a chip, to get you started here is a BangGood search and an AliExpress search.
Boards like this has also been described by others including a deduced schematic, EEVblog critique of the I/O pin power on similar boards (not yet confirmed whether that is true for this programmer too, I guess so, but at least one flashing done without damage) and a mention on hackaday of other board types.
There are a bunch of downloads from the WCH site regarding the chip including a Chinese datasheet, no English language documentation seems to be available from the manufacturer however. There are some English editions of the datasheet to be found, of unknown origin. They seem plausible enough to use, though. Somebody has attempted to collect documentation about the chip in a Git repository.
Buy: https://www.aliexpress.com/item/32869145935.html @ $3.11
To attach physically to the Macronix MX25L6405D flash memory chip in a SOIC-16 package present on the X200 in question (words are that this is the norm although the board can be populated with a SOIC-8 too) a clip that matches the pins of the SOIC-16 package is needed. I bought the one mentioned above for $3.11 at random from AliExpress and this worked fine. In the pictures the wiring is hooked up correctly to the programmer to allow for flashing as described below.
6405 <-> CH341a
MISO<->MIOS (label error, should be MISO)
MOSI<->MOSI
CLK <-> CLK
CS <-> SS
GND <-> GND
First tried driving the the flash chip from VDD on ch341a but this was unsuccessful, no chip could be found, so the 6405 was hooked up to external 3.3v power supply with supply GND connected to GND on CH341a to align the ground potential between ch341a I/O supply and 6405 supply (important!).
To get the latest ECP (Embedded Controller Program) from Lenovo (no free alternative exists) containing software for the MCU controlling low level hardware like battery charging/keyboard/backlight stuff you need to update the BIOS which also updates the ECP. Most recent version for X200 is “BIOS: 3.22 / ECP: 1.07“. This is not needed if you already have these versions on the machine, check current versions by pressing ThinkVantage during boot and choosing “Enter Setup”. If your system has a Windows installation download and run the “BIOS Update Utility“executeable. Else you’ll need to get the “BIOS Update Bootable CD” and somehow get it on a CD and find a CD-ROM drive. Alternatively on a Linux system the CD file system can be extracted and added to Grub to be directly bootable. Below was done on an Ubuntu 16.04 system: $ sudo apt install genisoimage syslinux $ wget -q https://download.lenovo.com/ibmdl/pub/pc/pccbbs/mobiles/6duj48us.iso $ geteltorito 6duj48us.iso > 6duj48us.img Booting catalog starts at sector: 20 Manufacturer of CD: NERO BURNING ROM Image architecture: x86 Boot media type is: harddisk El Torito image starts at sector 27 and has 75776 sector(s) of 512 Bytes Image has been written to stdout …. $ sudo cp 6duj48us.im /boot $ sudo cp /usr/lib/syslinux/memdisk /boot $ sudo nano /etc/grub.d/40_custom <add lines below to the end of file, preserve the “exec tail…” line> menuentry “BIOS Update” { linux16 /memdisk initrd16 /6duj48us.im } $ sudo update-grub Reboot, press <left shift> key while booting to access Grub, choose BIOS Update menu entry and follow the Lenovo update procedure. To start flashing it requires both a connected power supply and also a working, non-exhausted battery (!) mounted in the machine. This is tiresome for owners of worn out batteries… Some notes about the flashing process can be found in the documentation of a patch set for the Lenovo BIOS.
To get the latest ECP (Embedded Controller Program) from Lenovo (no free alternative exists) containing software for the MCU controlling low level hardware like battery charging/keyboard/backlight stuff you need to update the BIOS which also updates the ECP. Most recent version for X200 is “BIOS: 3.22 / ECP: 1.07“. This is not needed if you already have these versions on the machine, check current versions by pressing ThinkVantage during boot and choosing “Enter Setup”.
If your system has a Windows installation download and run the “BIOS Update Utility“executeable. Else you’ll need to get the “BIOS Update Bootable CD” and somehow get it on a CD and find a CD-ROM drive. Alternatively on a Linux system the CD file system can be extracted and added to Grub to be directly bootable. Below was done on an Ubuntu 16.04 system:
$ sudo apt install genisoimage syslinux $ wget -q https://download.lenovo.com/ibmdl/pub/pc/pccbbs/mobiles/6duj48us.iso $ geteltorito 6duj48us.iso > 6duj48us.img Booting catalog starts at sector: 20 Manufacturer of CD: NERO BURNING ROM Image architecture: x86 Boot media type is: harddisk El Torito image starts at sector 27 and has 75776 sector(s) of 512 Bytes Image has been written to stdout …. $ sudo cp 6duj48us.im /boot $ sudo cp /usr/lib/syslinux/memdisk /boot $ sudo nano /etc/grub.d/40_custom <add lines below to the end of file, preserve the “exec tail…” line> menuentry “BIOS Update” { linux16 /memdisk initrd16 /6duj48us.im } $ sudo update-grub Reboot, press <left shift> key while booting to access Grub, choose BIOS Update menu entry and follow the Lenovo update procedure. To start flashing it requires both a connected power supply and also a working, non-exhausted battery (!) mounted in the machine. This is tiresome for owners of worn out batteries… Some notes about the flashing process can be found in the documentation of a patch set for the Lenovo BIOS.
$ sudo apt install genisoimage syslinux $ wget -q https://download.lenovo.com/ibmdl/pub/pc/pccbbs/mobiles/6duj48us.iso $ geteltorito 6duj48us.iso > 6duj48us.img Booting catalog starts at sector: 20 Manufacturer of CD: NERO BURNING ROM Image architecture: x86 Boot media type is: harddisk El Torito image starts at sector 27 and has 75776 sector(s) of 512 Bytes Image has been written to stdout …. $ sudo cp 6duj48us.im /boot $ sudo cp /usr/lib/syslinux/memdisk /boot $ sudo nano /etc/grub.d/40_custom <add lines below to the end of file, preserve the “exec tail…” line> menuentry “BIOS Update” { linux16 /memdisk initrd16 /6duj48us.im } $ sudo update-grub
Reboot, press <left shift> key while booting to access Grub, choose BIOS Update menu entry and follow the Lenovo update procedure. To start flashing it requires both a connected power supply and also a working, non-exhausted battery (!) mounted in the machine. This is tiresome for owners of worn out batteries…
Some notes about the flashing process can be found in the documentation of a patch set for the Lenovo BIOS.
sudo apt install flashrom
ch341a support in flashrom
Download the stable LibreBoot firmware: https://www.mirrorservice.org/sites/libreboot.org/release/stable/20160907/rom/grub/libreboot_r20160907_grub_x200_8mb.tar.xz
The brave will of course want to compile it themselves.
$ cd $ wget https://www.mirrorservice.org/sites/libreboot.org/release/stable/20160907/rom/grub/libreboot_r20160907_grub_x200_8mb.tar.xz --2019-06-07 07:35:21-- https://www.mirrorservice.org/sites/libreboot.org/release/stable/20160907/rom/grub/libreboot_r20160907_grub_x200_8mb.tar.xz Resolving www.mirrorservice.org (www.mirrorservice.org)... 212.219.56.184, 2001:630:341:12::184 Connecting to www.mirrorservice.org (www.mirrorservice.org)|212.219.56.184|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1632800 (1,6M) [application/x-xz] Saving to: ‘libreboot_r20160907_grub_x200_8mb.tar.xz’ libreboot_r20160907_grub_x200_8mb.tar.xz 100%[================================================================================>] 1,56M --.-KB/s in 0,1s 2019-06-07 07:35:21 (13,9 MB/s) - ‘libreboot_r20160907_grub_x200_8mb.tar.xz’ saved [1632800/1632800] $ tar tf libreboot_r20160907_grub_x200_8mb.tar.xz libreboot_r20160907_grub_x200_8mb/ libreboot_r20160907_grub_x200_8mb/x200_8mb_deqwertz_txtmode.rom libreboot_r20160907_grub_x200_8mb/x200_8mb_esqwerty_txtmode.rom libreboot_r20160907_grub_x200_8mb/x200_8mb_frazerty_txtmode.rom libreboot_r20160907_grub_x200_8mb/x200_8mb_frdvbepo_txtmode.rom libreboot_r20160907_grub_x200_8mb/x200_8mb_itqwerty_txtmode.rom libreboot_r20160907_grub_x200_8mb/x200_8mb_svenska_txtmode.rom libreboot_r20160907_grub_x200_8mb/x200_8mb_ukdvorak_txtmode.rom libreboot_r20160907_grub_x200_8mb/x200_8mb_ukqwerty_txtmode.rom libreboot_r20160907_grub_x200_8mb/x200_8mb_usdvorak_txtmode.rom libreboot_r20160907_grub_x200_8mb/x200_8mb_usqwerty_txtmode.rom libreboot_r20160907_grub_x200_8mb/x200_8mb_deqwertz_vesafb.rom libreboot_r20160907_grub_x200_8mb/x200_8mb_esqwerty_vesafb.rom libreboot_r20160907_grub_x200_8mb/x200_8mb_frazerty_vesafb.rom libreboot_r20160907_grub_x200_8mb/x200_8mb_frdvbepo_vesafb.rom libreboot_r20160907_grub_x200_8mb/x200_8mb_itqwerty_vesafb.rom libreboot_r20160907_grub_x200_8mb/x200_8mb_svenska_vesafb.rom libreboot_r20160907_grub_x200_8mb/x200_8mb_ukdvorak_vesafb.rom libreboot_r20160907_grub_x200_8mb/x200_8mb_ukqwerty_vesafb.rom libreboot_r20160907_grub_x200_8mb/x200_8mb_usdvorak_vesafb.rom libreboot_r20160907_grub_x200_8mb/x200_8mb_usqwerty_vesafb.rom libreboot_r20160907_grub_x200_8mb/ChangeLog libreboot_r20160907_grub_x200_8mb/NEWS libreboot_r20160907_grub_x200_8mb/version libreboot_r20160907_grub_x200_8mb/versiondate $
As the MAC address of the ethnernet PHY is stored in the flash, yo have your X200 ethernet MAC address correspond to the sticker on the back of the machine, and also avoid a potential but improbable DHCP/ARP conflict, the MAC address from the label/ifconfig from the existing system must be embedded into the flash file that we are going to program into the flash chip.
For this a tool called ich9gen is needed, this is a part of the libreboot repository and we need to build it ourselves.
$ git clone https://notabug.org/libreboot/libreboot Cloning into 'libreboot'... remote: Counting objects: 29080, done. remote: Compressing objects: 100% (9855/9855), done. remote: Total 29080 (delta 18748), reused 27899 (delta 18057) Receiving objects: 100% (29080/29080), 63.90 MiB | 11.13 MiB/s, done. Resolving deltas: 100% (18748/18748), done. Checking connectivity... done. $ cd libreboot/projects/ich9gen/sources $ make gcc -I. -Wall -Wextra -g -std=c99 -c src/ich9deblob.c -o obj/ich9deblob.o gcc -I. -Wall -Wextra -g -std=c99 -c src/common/descriptor_gbe.c -o obj/common/descriptor_gbe.o gcc -I. -Wall -Wextra -g -std=c99 -c src/descriptor/descriptor.c -o obj/descriptor/descriptor.o gcc -I. -Wall -Wextra -g -std=c99 -c src/gbe/gbe.c -o obj/gbe/gbe.o gcc -I. -Wall -Wextra -g -std=c99 -c src/common/x86compatibility.c -o obj/common/x86compatibility.o gcc -I. -Wall -Wextra -g -std=c99 obj/ich9deblob.o obj/common/descriptor_gbe.o \ obj/common/x86compatibility.o obj/descriptor/descriptor.o obj/gbe/gbe.o \ -o ich9deblob gcc -I. -Wall -Wextra -g -std=c99 -c src/ich9gen.c -o obj/ich9gen.o gcc -I. -Wall -Wextra -g -std=c99 -c src/ich9gen/mkdescriptor.c -o obj/ich9gen/mkdescriptor.o gcc -I. -Wall -Wextra -g -std=c99 -c src/ich9gen/mkgbe.c -o obj/ich9gen/mkgbe.o gcc -I. -Wall -Wextra -g -std=c99 obj/ich9gen.o obj/ich9gen/mkdescriptor.o obj/ich9gen/mkgbe.o \ obj/common/descriptor_gbe.o \ obj/common/x86compatibility.o obj/descriptor/descriptor.o obj/gbe/gbe.o \ -o ich9gen gcc -I. -Wall -Wextra -g -std=c99 -c src/demefactory.c -o obj/demefactory.o gcc -I. -Wall -Wextra -g -std=c99 obj/demefactory.o obj/common/descriptor_gbe.o \ obj/common/x86compatibility.o obj/descriptor/descriptor.o obj/gbe/gbe.o \ -o demefactory $
Running ich9gen itself generates the flash descriptor (fd) header including possible configuration section where the MAC address is stored for the gigabit ethernet (gbe) PHY onboard the ICH9 chipset. When run six 12 KiB files for respectively 4, 8 and 16 MiB binary images and chipsets including (gbe) and excluding (nogbe) gigabit ethernet PHY are generated.
“aa:bb:cc:dd:ee:ff” in the commandline should be replaced with the actual 12 hex digits from the label on the machine or by running ifconfig on the machine using the existing Lenovo BIOS.
$ cd ~/libreboot_r20160907_grub_x200_8mb/ $ ~/libreboot/projects/ich9gen/sources/ich9gen --macaddress aa:bb:cc:dd:ee:ff You selected to change the MAC address in the Gbe section. This has been done. The modified gbe region has also been dumped as src files: mkgbe.c, mkgbe.h To use these in ich9gen, place them in src/ich9gen/ and re-build ich9gen. descriptor and gbe successfully written to the file: ich9fdgbe_4m.bin Now do: dd if=ich9fdgbe_4m.bin of=libreboot.rom bs=1 count=12k conv=notrunc (in other words, add the modified descriptor+gbe to your ROM image) descriptor and gbe successfully written to the file: ich9fdgbe_8m.bin Now do: dd if=ich9fdgbe_8m.bin of=libreboot.rom bs=1 count=12k conv=notrunc (in other words, add the modified descriptor+gbe to your ROM image) descriptor and gbe successfully written to the file: ich9fdgbe_16m.bin Now do: dd if=ich9fdgbe_16m.bin of=libreboot.rom bs=1 count=12k conv=notrunc (in other words, add the modified descriptor+gbe to your ROM image) descriptor successfully written to the file: ich9fdnogbe_4m.bin Now do: dd if=ich9fdnogbe_4m.bin of=yourrom.rom bs=1 count=4k conv=notrunc (in other words, add the modified descriptor to your ROM image) descriptor successfully written to the file: ich9fdnogbe_8m.bin Now do: dd if=ich9fdnogbe_8m.bin of=yourrom.rom bs=1 count=4k conv=notrunc (in other words, add the modified descriptor to your ROM image) descriptor successfully written to the file: ich9fdnogbe_16m.bin Now do: dd if=ich9fdnogbe_16m.bin of=yourrom.rom bs=1 count=4k conv=notrunc (in other words, add the modified descriptor to your ROM image)
$ cd ~/libreboot_r20160907_grub_x200_8mb/ $ cp -v x200_8mb_usqwerty_vesafb{,_customised}.rom 'x200_8mb_usqwerty_vesafb.rom' -> 'x200_8mb_usqwerty_vesafb_customised.rom' $ dd if=ich9fdgbe_8m.bin of=x200_8mb_usqwerty_vesafb_customised.rom bs=1 count=12k conv=notrunc 12288+0 records in 12288+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0,0299453 s, 410 kB/s $
flashrom -p ch341a_spi -c MX25L6405D -r rom1.bin flashrom -p ch341a_spi -c MX25L6405D -r rom2.bin flashrom -p ch341a_spi -c MX25L6405D -r rom3.bin flashrom -p ch341a_spi -c MX25L6405D -r rom4.bin flashrom -p ch341a_spi -c MX25L6405D -r rom5.bin cmp rom{1,2}.bin cmp rom{1,3}.bin cmp rom{1,4}.bin cmp rom{1,5}.bin
flashrom -p ch341a_spi -c MX25L6405D -r rom1.bin
flashrom -p ch341a_spi -c MX25L6405D -r rom2.bin
flashrom -p ch341a_spi -c MX25L6405D -r rom3.bin
flashrom -p ch341a_spi -c MX25L6405D -r rom4.bin
flashrom -p ch341a_spi -c MX25L6405D -r rom5.bin
cmp rom{1,2}.bin
cmp rom{1,3}.bin
cmp rom{1,4}.bin
cmp rom{1,5}.bin
$ sudo flashrom -p ch341a_spi -c MX25L6405D -w ~/libreboot_r20160907_grub_x200_8mb/x200_8mb_usqwerty_vesafb_customised.rom $
Thanks to the fine people at the libimobiledevice project, who bothers to reverse engineer Apple products, I recently succeeded in resurrecting a relative’s iPad stuck in a boot loop (something with jailbreaking, running Sydia, missing an iOS update and attempted Sydia removal) without any use of proprietary tools.
This is a brief recipe of the procedure done using Ubuntu 16.04.
As the required tool from libimobiledevice, idevicerestore, is not packaged in the Ubuntu libimobiledevice package we need to build this from scratch from the sources.
iPad in recovery mode during firmware download using libimobiledevice
sudo apt install libusbmuxd-dev libplist-dev libplist++contents under-dev libzip-dev
cd git clone https://git.libimobiledevice.org/libimobiledevice.git cd libimobiledevice/ ./autogen.sh make
cd git clone https://git.libimobiledevice.org/libirecovery.git cd libirecovery ./autogen.sh make
cd git clone https://git.libimobiledevice.org/idevicerestore.git cd idevicerestore CFLAGS="-I$HOME/libirecovery/include -I$HOME/libimobiledevice/include" LDFLAGS="-L$HOME/libirecovery/src/.libs \ -L$HOME/libimobiledevice/src/.libs" PKG_CONFIG_PATH=~/libirecovery:~/libimobiledevice/src ./autogen.sh make
sudo $HOME/idevicerestore/src/idevicerestore --latest NOTE: using cached version data Found device in Recovery mode Identified device as j71ap, iPad4,1 Latest firmware is iPad_64bit_11.2_15C114_Restore.ipsw Verifying 'iPad_64bit_11.2_15C114_Restore.ipsw'... Checksum matches. Extracting BuildManifest from IPSW Product Version: 11.2 Product Build: 15C114 Major: 15 INFO: device serial number is DMPM4V3SFK15 Device supports Image4: true Variant: Customer Upgrade Install (IPSW) This restore will update your device without losing data. Using cached filesystem from 'iPad_64bit_11.2_15C114_Restore/058-86080-124.dmg' Found ECID 6653578882512 Getting ApNonce in recovery mode... 03 6b cc ac 57 8a b4 29 29 c1 a9 fe e4 97 54 3b a8 36 59 5a Trying to fetch new SHSH blob Getting SepNonce in recovery mode... df 5c ad 67 48 bd 38 b4 6f 72 0a 5c b0 81 87 c3 95 37 4a da WARNING: Unable to find BbChipID node WARNING: Unable to find BbSkeyId node Request URL set to https://gs.apple.com/TSS/controller?action=2 Sending TSS request attempt 1... response successfully received Received SHSH blobs Extracting iBEC.ipad4.RELEASE.im4p... Personalizing IMG4 component iBEC... Sending iBEC (710360 bytes)... Recovery Mode Environment: iBoot build-version=iBoot-4076.30.43 iBoot build-style=RELEASE Sending AppleLogo... Extracting applelogo@2x~ipad.im4p... Personalizing IMG4 component AppleLogo... Sending AppleLogo (22709 bytes)... ramdisk-size=0x10000000 Extracting 058-85997-124.dmg... Personalizing IMG4 component RestoreRamDisk... Sending RestoreRamDisk (59978774 bytes)... Extracting DeviceTree.j71ap.im4p... Personalizing IMG4 component RestoreDeviceTree... Sending RestoreDeviceTree (101420 bytes)... Extracting kernelcache.release.ipad4... Personalizing IMG4 component RestoreKernelCache... Sending RestoreKernelCache (13226783 bytes)... About to restore device... Waiting for device... Device 3fb0f5cc97b83c61c85d4b8333796d9e536a4c83 is now connected in restore mode... Connecting now... Connected to com.apple.mobile.restored, version 15 Device 3fb0f5cc97b83c61c85d4b8333796d9e536a4c83 has successfully entered restore mode Hardware Information: BoardID: 16 ChipID: 35168 UniqueChipID: 6653578882512 ProductionMode: true Starting FDR listener thread About to send NORData... Found firmware path Firmware/all_flash Getting firmware manifest from build identity Extracting LLB.ipad4.RELEASE.im4p... Personalizing IMG4 component LLB... Extracting applelogo@2x~ipad.im4p... Personalizing IMG4 component AppleLogo... Extracting batterycharging0@2x~ipad.im4p... Personalizing IMG4 component BatteryCharging0... Extracting batterycharging1@2x~ipad.im4p... Personalizing IMG4 component BatteryCharging1... Extracting batteryfull@2x~ipad.im4p... Personalizing IMG4 component BatteryFull... Extracting batterylow0@2x~ipad.im4p... Personalizing IMG4 component BatteryLow0... Extracting batterylow1@2x~ipad.im4p... Personalizing IMG4 component BatteryLow1... Extracting glyphplugin@2x~ipad-lightning.im4p... Personalizing IMG4 component BatteryPlugin... Extracting DeviceTree.j71ap.im4p... Personalizing IMG4 component DeviceTree... Extracting recoverymode@2x~ipad-lightning.im4p... Personalizing IMG4 component RecoveryMode... Extracting iBoot.ipad4.RELEASE.im4p... Personalizing IMG4 component iBoot... Extracting sep-firmware.j71.RELEASE.im4p... Personalizing IMG4 component RestoreSEP... Extracting sep-firmware.j71.RELEASE.im4p... Personalizing IMG4 component SEP... Sending NORData now... Done sending NORData About to send RootTicket... Sending RootTicket now... Done sending RootTicket Waiting for NAND (28) Checking filesystems (15) Checking filesystems (15) Unmounting filesystems (29) Unmounting filesystems (29) Creating filesystem (12) About to send filesystem... Connected to ASR Validating the filesystem Filesystem validated Sending filesystem now... [==================================================] 100.0% Done sending filesystem Verifying restore (14) [==================================================] 100.0% Checking filesystems (15) Checking filesystems (15) Mounting filesystems (16) Mounting filesystems (16) About to send KernelCache... Extracting kernelcache.release.ipad4... Personalizing IMG4 component KernelCache... Sending KernelCache now... Done sending KernelCache Installing kernelcache (27) About to send DeviceTree... Extracting DeviceTree.j71ap.im4p... Personalizing IMG4 component DeviceTree... Sending DeviceTree now... Done sending DeviceTree Certifying Savage (61) Flashing firmware (18) [==================================================] 100.0% Updating gas gauge software (47) Updating gas gauge software (47) Updating Stockholm (55) About to send FUD data... Sending FUD data now... Done sending FUD data About to send FUD data... Sending FUD data now... Done sending FUD data Fixing up /var (17) Modifying persistent boot-args (25) Unmounting filesystems (29) Unmounting filesystems (29) Got status message Status: Restore Finished Cleaning up... DONE
iPad during firmware flashing using libimobiledevice
If you want to interact with iDevices from within Ubuntu during ordinary use, you could also install some utils and plugins for that. Below will fx. add a context menu in nautilus with info about the iDevice and install the ideviceinstaller command line utility which can be used to administer installed applications on the device.
sudo apt install libimobiledevice-utils nautilus-ideviceinfo ideviceinstaller
Mit svar på et spørgsmål i Facebook-gruppen Danske Arduino Entusiaster omkring Arduino ROM/Flash, PROGMEM og system-inklude-filer.
Spørgsmål
Hej er der en der ved hvor jeg kan hente dett lib. <avr/pgmspace.h> jeg skal bruge denne funktion PROGMEM så jeg kan gemme et billede i Arduino uden SD kort det kan være der er en der kender en anden måde at gøre det på.
Svar
pgmspace.h er en inklude-fil som er en del af c-biblioteket til AVR-arkitekturen (avr-libc). C-bibliotekets inklude-filer vil normalt ligge i kompilerens “system include”-sti (se GCC options -I og -isystem). Dermed kan den inkluderes blot med “#include <avr/pgmspace.h>”. Se evt. også Arduino-referencen på https://www.arduino.cc/en/Reference/PROGMEM. Bemærk at PROGMEM ikke er en funktion, men en storage modifier (lager-modifikator) som fortæller kompileren at den kan placere en en given variabel i ikke-skrivbar lager (ROM/Flash). Der skal efterfølgende anvendes specielle funktioner til at læse data fra en sådan variabel (se referencen). Arduino-frameworket har dog lavet en nem måde at placere konstant-strenge i Flash på (normalt lagres de i SRAM!), nemlig funktionen F() som kan anvendes direkte i f.eks. printf/write/print (Serial.print(F(“Waiting for connection”));) Hvis du vil inspicere indholdet af pgmspace.h, kan du finde filen i Arduino IDE’ets installations-mappe under hardware/tools/avr/avr/include/avr/pgmspace.h. Det er ikke en man kan/skal redigere manuelt i, da den er tæt koblet med den binære kode i selve biblioteket. Der findes også EEPROM-lager du sikkert vil kunne bruge til samme formål; https://www.arduino.cc/en/Reference/EEPROM
Se svaret på Facebook.
Den videre færd med F()
Da jeg ikke kunne finde en uddybende forklaring på F()-funktionen (som egentlig er en makro) i Arduino-dokumentationen (brugen nævnes meget kort i PROGMEM , Memory og Print), gravede jeg efterfølgende lidt rundt for at lære mere. I de sparsomme Arduino-eksempler er den anvendt udelukkende med konstante strenge, hvilket også viser sig at være et krav (eller i hvert fald noget der kan castes til const char *).
Makroen er defineret af Arduino-frameworket i filen hardware/arduino/avr/cores/arduino/WString.h (referencerne er ifht. min lokale installation af Arduino 1.6.9, pt. er nyeste 1.6.13) således:
#define F(string_literal) (reinterpret_cast<const __FlashStringHelper *>(PSTR(string_literal)))
Altså parametren til F() bruges som parameter til PSTR() (progmem string, er mit bud på navn) som er en makro defineret i pgmspace.h fra avr-libc.
Dens funktion er at caste parametrens type til konstant streng-pointer med PROGMEM modifier;
#define PSTR(s) ((const PROGMEM char *)(s))
Skal vi se på hvad PROGMEM rent faktisk er, så finder vi endnu et sæt makroer der ender med at blive udviddet til kompiler-attributten __progmem__, igen definieret i pgmspace.h (hardware/tools/avr/avr/include/avr/pgmspace.h):
#define PROGMEM __ATTR_PROGMEM__ #define __ATTR_PROGMEM__ __attribute__((__progmem__))
#define PROGMEM __ATTR_PROGMEM__
#define __ATTR_PROGMEM__ __attribute__((__progmem__))
__progmem__ attributten er en instruks til kompileren (GCC) og linkeren om ved programmering/flashing af programmet at placere disse data i en sektion af hukommelsen der hedder “.progmem“. Se evt. mere om dette i GCC-kompilerens dokumentation. For hver AVR-chip kompileren understøtter er der eksakte definitioner af hvilke hukommelsesadresser .progmem ligger på for netop denne chip.
Dvs. når man i sin kode skriver F(“test”) får man i virkeligheden:
(reinterpret_cast<const __FlashStringHelper *>(((const __attribute__((__progmem__)) char *)(“test”)))
Altså en konstant streng der lagres i AVR-processorens progmem-sektion, og som returværdi får en pointer til en konstant instans af en klasse kaldet “__FlashStringHelper“. Denne klasse må være lavet sådan at den anvender de korrekte mekanismer til at læse fra progmem-området (måske mere om dette i en senere artikel). Arduinos funktion-bibliotek (Serial.print() mm.) er lavet således at de direkte kan tage en parameter af denne type som erstatning for en konstant-streng (og det er netop her Arduino-frameworket viser sin værdi ved at abstrahere sådanne kompleksiteter væk fra programmøren).
Below is a handy shell one-liner for comparing the decompressed contents of a compressed file against a plain file. The purpose here is to test whether the compressed file is actually derived from compressing the plain file. This particular example is from a real life situation where log rotation by cron.daily on a Ubuntu server had begun failing. The situation is thought to be the result of an interrupted logrotate execution leaving a compressed (but non-rotated) intermediate file in the file system that prevented further log rotation on subsequent executions.
The command constructs a pipe between two process, specified using the | symbol (vertical line), to send the decompressed contents from gunzip stdout to stdin of the compare tool. The example uses diff as compare tool which is suitable for textual contents. You could use f.x. cmp instead if the contents is binary. Both diff and cmp interprets the file name “-” as meaning that input should be read from stdin.
$ gunzip --stdout /var/log/syslog.1.gz | diff --report-identical-files - /var/log/syslog.1 Files - and /var/log/syslog.1 are identical
The above uses long options for clarity, in a real life situation you would probably be using short options instead. That means -c instead of –stdout and –s instead of –report-identical-files.
$ gunzip -c /var/log/syslog.1.gz | diff -s - /var/log/syslog.1 Files - and /var/log/syslog.1 are identical
Fra en tråd i Facebook-gruppen “Linux for begyndere“.
Spørgsmål Hvordan bliver man root bruger i linux mint ?? i grafisk brugerflade ??
Svar For fremtidig reference: Hvis man ønsker at køre en X-klient (et vilkårligt grafisk/GUI program) som root-brugeren, men vise dets vinduer på en X-server (typisk dit desktop environment/DE, som f.eks Gnome/KDE/Unity/lxde/xfce m.f.) der eksekveres af en ikke-privilegeret bruger kan man gøre som følger:
1) som X-server-brugeren kør kommandoen ‘xhost +’ i en grafisk konsol/terminal. Dette tillader at alle brugere og maskiner må vise vinduer på X-serveren (ja, X er en netværksprotokol). Tilladelsen bevares indtil X-serveren genstartes, eller den trækkes tilbage med ‘xhost -‘.
2) som root (su/sudo) eksekver det ønskede GUI-program, med specifikation af hvilken X-server og hvilket display dets vinduer ønskes vist på i DISPLAY environment-variablen (ja, det er muligt at køre flere separate X-servere/displays på samme maskine). Simpleste form med visning på display 0 på den lokale maskine vil være (med xterm-konsollen som eksempel) ‘DISPLAY=:0 xterm’. Vil man vise vinduet på en anden maskines X-server skal IP-adressen blot angives før ‘:’, som f.eks.: ‘DISPLAY=192.168.1.10:0 xterm’ (det er stadig en xterm der afvikles på den lokale maskine, vinduet vises blot på en ekstern X-server (ja, det kan være farligt, pas på).
Bemærk at man med sudo skal passe på at sætte env-vars i den rigtige shell. F.eks. vil sudo direkte foran ovenstående ikke virke da sudo afskærmer env af sikkerhedsgrunde. I stedet vil man kunne benytte følgende trick: sudo bash -c ‘DISPLAY=:0 xterm’ (enkelt citationstegn til -c er vigtigt).
Until your device is sufficiently patched against the Stagefright vulnerabilty I recommend disabling automatic MMS retrieval on any Android phones from 2.2 and up (which is hopefully all in current use) to prevent unattended triggering.
Howtos for Google and Samsung devices are here.
Below are screenshots of how to do it on HTC One M7 using the stock (HTC Sense) messaging application called “SMS”. The procedure is likely to be very similar on most HTC devices using Sense. The UI shown is in Danish locale, the English menus will be something like SMS->Settings->Multi Media Messages (MMS)->Automatic Retrieval.
Schneier discusses details here and this seems to be the commit in CyanogenMod for the underlying problem in the media library. Check aælso the issue’s review page
A very annoying feature of the Ångström image that is shipped with the BeagleBone Black, is that a display connected to the HDMI output of the board will by default be blanked when powering up, and is first woken when any pointer activity occur (touch/mouse).
This seems to originate from the fbdev that is used for displaying graphics, and it took me some time to figure out how to cirumvent it. The normal X commands for controlling blanking of “xset -dpms” or “xset s off” did nothing, and neither did the terminal options of “setterm powersave off” or “setterm powerdown 0”. I went all the way back to old ANSI escape sequences trying “echo -e ‘\033[9;X]'” without success.
Luckily I fell by at Armadeus.com’s framebuffer tips, which listed the sys-fs node named /sys/class/graphics/fb0/blank that controls blanking of the low level framebuffer, thus executing (as root)
echo 0 > /sys/class/graphics/fb0/blank
disables blanking and wakes up the BBB HDMI output.
To do this at every boot (really login) you can use the Gnome Startup Applications Preferences (gnome-session-properties) to execute this at Gnome autologin, or add it to whatever startup script you see fit.
Beware that you might need to delay the execution when using the gnome-session-properties, I had to put in a sleep, but that probably depends on what other stuff is starting up from it.
Once in a while it is useful to dismiss abstractions and layers that makes daily routines easier and take the raw approach. Like when debugging a software problem that doesn’t make sense, it is nice to see the underlying basic stuff is behaving nicely, to better be able to locate where the unexpected occurs.
In the IP world of the internet, the swiss army knife for debugging interprocess communications in a totally protocol agnostic way is called ‘telnet’. Telnet opens up a communication channel between your local computer and a daemon/server on a specific port on a specific IP address. Then it gets out of the way for you to talk directly to the daemon in clear text.
Knowledge of how to interact using a specific protocol can be very useful to check server availability and functionality. All common protocols in use on the internet (like DNS, HTTP, SMTP, POP3, IMAP, XMPP etc.) can be debugged like this, because all of them transfers data in clear text (or at least initiates other transfer types from a clear text session). Full specifications for the HTTP protocol can be found in IETF RFC2616. I keep forgetting this, and end up digging around for it when needed, therefore this blog post.
Below is a basic HTTP session to my web server www.mikini.dk using telnet on the command line.
Red text is local text input by me. Blue text is local text by telnet application. Green text is server response.
$ telnet www.mikini.dk 80 Trying 92.61.152.47… Connected to 92-61-152-47.static.servage.net. Escape character is ‘^]’. GET /index.php/2010/06 HTTP/1.1 Host: www.mikini.dk
HTTP/1.1 200 OK Date: Wed, 21 Jul 2010 09:19:13 GMT Server: Apache X-Pingback: http://www.mikini.dk/xmlrpc.php Transfer-Encoding: chunked Content-Type: text/html; charset=UTF-8
2bd2 <!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Transitional//EN” “http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd“>
<… remainder of HTML document dropped …>
We are asking the server at www.mikini.dk port 80 (default http port) for the document at path /index.php/2010/06 (“GET /index.php/2010/06”). Notice the two CRLF characters after the Host header field, this indicates to the server that the request header is done, and that it should begin parsing header and send its response. Also notice that even though you tell the telnet program on the commandline that you want to access www.mikini.dk, you have to tell it again to the server in th HTTP Host field. Thats because telnet is only concerned about the IP address of the server, it resolves www.mikini.dk to the IP 92.61.152.47 through DNS and forgets about it. From the servers point of view, it needs to know which of its virtual hosts you want to talk to, cause one server application on one port on one ip can potentially host thousands of separate websites (virtual hosts).
Below is the same session using nc (netcat) as a one-liner (wow, old PHP at Hostinger, better get that move to a self-administered box going).
$ echo -ne "GET /index.php/2010/06 HTTP/1.1\r\nHost:www.mikini.dk\r\n\r\n"|nc www.mikini.dk 80 | head -10 HTTP/1.1 200 OK Date: Thu, 05 Jul 2018 16:30:07 GMT Server: Apache X-Powered-By: PHP/5.5.35 Link: <http://www.mikini.dk/wp-json/>; rel="https://api.w.org/" Content-Length: 32356 Content-Type: text/html; charset=UTF-8 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> $
2018-07-05: nc command line added
