I’ve been hit by the “periodic boot failure” issue of the Beaglebone Black (aka BBB) reported by quite a few on the net. For most users this is an inconvenient annoyance, but for people, like me, using the platform in embedded applications, this issue causes a serious stability issue of the whole system, when 100% reliable boot is not achievable.

After having hunches about instability caused by the intermittent experiences during development, where the board was seen failing boot on power on, and not getting much more help from the net than “try the recommended power supply” (which btw. I can’t use because I live in a country where main sockets are non-US, even a bit non-European standard) I decided to make a systematic test to get the basic facts straight.

I settled on trying to establish some reasonable statistics about the error’s frequency on plain BBBs to have a reference against testing whether a theory put forward on the mailinglist (here and here) about the uboot bootloader being confused by noise being interpreted as valid data on UART0_RXD (pin E15) of the AM3358x  (see near U15 page 4, of the BBB REV B schematics) as the cause of the failure.

Results

This is the results (test report detailed below), I’m posting a writeup in the Beagleboard mailinglist (Edit: my post here), so hopefully you’ll find further discussion about this issue there soon.

Failure rates

Plain BBB (DUT#2+3), Element14 & mbest branded

• DUT#2: Element14 branded: 4 fails/ 120 boots = 1/30 = 3.33%
  
• DUT#3: mbest branded: 2 fail / 40 boots = 1/20 = 5.00
  
• Overall: 6 fails / 160 boots = 3/80 = 3.75%
  

Modified BBB (DUT#1), CircuitCo branded

• U15 removed, pull down on UART0_RXD : 0 fails / 40 boots = 0.00%
  
• U15 removed, no pull down on UART0_RX: 0 fails / 40 boots = 0.00%
  
• Overall: 0 fails / 80 boots = 0.00%
  

Interpretation

The two differing Element14 branded BBB products I have access to, but both PCB REV B6, exhibits a somewhat varying boot failure rate. But overall the boards fail to boot in almost 4 of 100 boots.

Investigating the theory relating to noise on UART0_RXD seems to have paid off, as first removing U15 (SN74LVC2G241: Dual Buffer/Driver With 3-State Outputs) for the purpose of adding a pull down on its pad 6 (which is connected to UART0_RXD)  alleviated the problem altogether. But also the experiment of removing the pull down and redo the test, showed that the act of removing U15 itself caused the boot to always succeed.

Unfortunately, in hindsight, I was too quick to grab the soldering iron, because I should have verified and quantized the occurrence of the failure on the actual board being modified. A shame It didn’t occur to me before modification, but I’ll be more than willing to try to remove U15 on DUT#2, which has had the highest failure rate, if discussions prove that it is a reasonable theory of the root cause of the failure. That is why I continued testing it through to 120 boots, to get more samples for improving statistics in the event that I pull U15 from it later.

Effect of removing U15

The success of removing U15 could be caused by the now floating AM3388 input UART0_RXD (pin E15) which presumably has a default weak internal pull up/down (the AM335x TRM says reset value is pad-dependent (register conf_uart0_rxd in Table 9.7 p. 1366 and  Section 9.3.1.50 p. 1420), which I haven’t yet figured out the exact meaning of)  stabilizing the signal

The activity when U15 is in place is somehow exhibited on output 1Y (pin 6) , probably because it is not stable, and thus has an erratic state, during the first moments of the chips power up sequence. This erratic behaviour can in fortunate/unfortunate circumstances be interpreted as valid bits and resulting bytes by the uart rxd cirtcuitry, which also can happen to be latched into the uart fifo rx buffer, waiting for uboot to read them when its code is executed looking for a user interrupt.

I’ll put in the disclaimer on this thesis, that I haven’t yet studied U15 in detail, but it is advertised as both a level converter, ESD protection and power live-insertion/partial-power-down suggesting it does something in reaction to it’s power condition.

Also the recommendation on page 1 of its datasheet; “To ensure the high-impedance state during power up or power down, OE (active low) should be tied to VCC through a pullup resistor, and OE (active high) should be tied to GND through a pulldown resistor”, seems not to be followed in the BBB circutry, as the OEs are hardwired in to be always active (opposite of the recommendation in the power up/down condition). If this is actually a problem, I need to do further analysis to establish.

Pictures

Detailed test report

(formatted in nice emacs org-mode)

* BBB boot lockup test report

** Device under test #1:
Modified Beaglebone Black produced by CircuitCo (PCB REV B6, serial 007142901445, marked
"beaglebone"+ beagle logo and "beagleboard.org").

Modified by adding hard pulldown resistor on TI AM3358 pin E15 (uart0 rx). Specifically
U15 was removed and terminal pin 6 (1Y=UART0_RX) was shorted to J1 pin 1 through a 82k5
ohm resistor.

** Device under test #2:
Unmodified Beaglebone Black (BBB) produced by Element 14 (PCB REV B6, serial
EM-400524+XA6001961, marked "Element 14").

** Device under test #3:
Unmodified Beaglebone Black (BBB) produced by mbest (PCB REV B6, serial
EM-400441+XA3001688).

** Power supply
Jentec Technologies CF1805-E, output 5V 3A. Danish plug.
Sourced from a D-Link DUB-H7 USB 2.0 HUB.

** Test 1 procedure

Each Beaglebone was tested by consequtively applying power by inserting the 
plug into the mains socket while keeping the DC barrel connector inserted and 
verifying that the power led light up, and then noting whether boot from SD-card 
succeeded or failed. Then removing the PSU from the mains connector waiting 5 
seconds and repeat.

The power supply and SD-card used was the same for all three DUTs.

Results can be seen in section Test 1 results.

** Test 2 procedure

After a short analysis of test 1 results I decided to try to remove the resistor, 
to see if the behavious was restored.

Otherwise test procedure was identical to test 1.
Results can be seen in section Test 2 results.

** Test 1 results (pull down and reference boards)

| Boot no | DUT#1 | DUT#2   | DUT#3   | Note                                  |
|       1 | boot  | boot    | boot    |                                       |
|       2 | boot  | boot    | boot    |                                       |
|       3 | boot  | boot    | boot    |                                       |
|       4 | boot  | boot    | boot    |                                       |
|       5 | boot  | boot    | boot    |                                       |
|       6 | boot  | boot    | boot    |                                       |
|       7 | boot  | boot    | boot    |                                       |
|       8 | boot  | boot    | boot    |                                       |
|       9 | boot  | boot    | boot    |                                       |
|      10 | boot  | boot    | boot    |                                       |
|      11 | boot  | no boot | boot    | DUT#2: pwr sw=lock, rst sw=boot       |
|      12 | boot  | boot    | boot    |                                       |
|      13 | boot  | boot    | boot    |                                       |
|      14 | boot  | boot    | boot    |                                       |
|      15 | boot  | boot    | boot    |                                       |
|      16 | boot  | boot    | boot    |                                       |
|      17 | boot  | boot    | boot    |                                       |
|      18 | boot  | boot    | boot    |                                       |
|      19 | boot  | boot    | boot    |                                       |
|      20 | boot  | no boot | boot    | DUT#2:  pwr sw=no boot, rst sw=boot   |
|      21 | boot  | boot    | boot    |                                       |
|      22 | boot  | boot    | boot    |                                       |
|      23 | boot  | boot    | boot    |                                       |
|      24 | boot  | boot    | boot    |                                       |
|      25 | boot  | boot    | boot    |                                       |
|      26 | boot  | boot    | boot    |                                       |
|      27 | boot  | boot    | boot    |                                       |
|      28 | boot  | boot    | boot    |                                       |
|      29 | boot  | boot    | boot    |                                       |
|      30 | boot  | boot    | boot    | DUT#3: pause before comencing test 31 |
|      31 | boot  | boot    | no boot | DUT#3: pwr sw=no boot, rst sw=boot    |
|      32 | boot  | no boot | boot    | DUT#2: pwr sw=no boot, rst sw=boot    |
|      33 | boot  | boot    | boot    |                                       |
|      34 | boot  | boot    | boot    |                                       |
|      35 | boot  | boot    | boot    |                                       |
|      36 | boot  | boot    | no boot | DUT#3: pwr sw=no boot, rst sw=boot    |
|      37 | boot  | boot    | boot    |                                       |
|      38 | boot  | boot    | boot    |                                       |
|      39 | boot  | boot    | boot    |                                       |
|      40 | boot  | boot    | boot    |                                       |
|      41 |       | boot    |         |                                       |
|      42 |       | boot    |         |                                       |
|      43 |       | boot    |         |                                       |
|      44 |       | boot    |         |                                       |
|      45 |       | boot    |         |                                       |
|      46 |       | boot    |         |                                       |
|      47 |       | boot    |         |                                       |
|      48 |       | boot    |         |                                       |
|      49 |       | boot    |         |                                       |
|      50 |       | boot    |         |                                       |
|      51 |       | boot    |         |                                       |
|      52 |       | boot    |         |                                       |
|      53 |       | boot    |         |                                       |
|      53 |       | boot    |         |                                       |
|      54 |       | boot    |         |                                       |
|      55 |       | boot    |         |                                       |
|      56 |       | boot    |         |                                       |
|      57 |       | boot    |         |                                       |
|      58 |       | boot    |         |                                       |
|      59 |       | boot    |         |                                       |
|      60 |       | boot    |         |                                       |
|      61 |       | boot    |         |                                       |
|      62 |       | boot    |         |                                       |
|      63 |       | boot    |         |                                       |
|      64 |       | boot    |         |                                       |
|      65 |       | boot    |         |                                       |
|      66 |       | boot    |         |                                       |
|      67 |       | boot    |         |                                       |
|      68 |       | boot    |         |                                       |
|      69 |       | boot    |         |                                       |
|      70 |       | boot    |         |                                       |
|      71 |       | boot    |         |                                       |
|      72 |       | boot    |         |                                       |
|      73 |       | boot    |         |                                       |
|      74 |       | boot    |         |                                       |
|      75 |       | boot    |         |                                       |
|      76 |       | boot    |         |                                       |
|      77 |       | boot    |         |                                       |
|      78 |       | boot    |         |                                       |
|      79 |       | boot    |         |                                       |
|      80 |       | boot    |         |                                       |
|      81 |       | boot    |         |                                       |
|      82 |       | boot    |         |                                       |
|      83 |       | boot    |         |                                       |
|      84 |       | boot    |         |                                       |
|      85 |       | boot    |         |                                       |
|      86 |       | boot    |         |                                       |
|      87 |       | boot    |         |                                       |
|      88 |       | boot    |         |                                       |
|      89 |       | boot    |         |                                       |
|      90 |       | boot    |         |                                       |
|      91 |       | boot    |         |                                       |
|      92 |       | boot    |         |                                       |
|      93 |       | boot    |         |                                       |
|      94 |       | boot    |         |                                       |
|      95 |       | boot    |         |                                       |
|      96 |       | boot    |         |                                       |
|      97 |       | boot    |         |                                       |
|      98 |       | boot    |         |                                       |
|      99 |       | boot    |         |                                       |
|     100 |       | boot    |         |                                       |
|     101 |       | boot    |         |                                       |
|     102 |       | boot    |         |                                       |
|     103 |       | boot    |         |                                       |
|     104 |       | boot    |         |                                       |
|     105 |       | boot    |         |                                       |
|     106 |       | no boot |         |                                       |
|     107 |       | boot    |         |                                       |
|     108 |       | boot    |         |                                       |
|     109 |       | boot    |         |                                       |
|     110 |       | boot    |         |                                       |
|     111 |       | boot    |         |                                       |
|     112 |       | boot    |         |                                       |
|     113 |       | boot    |         |                                       |
|     114 |       | boot    |         |                                       |
|     115 |       | boot    |         |                                       |
|     116 |       | boot    |         |                                       |
|     117 |       | boot    |         |                                       |
|     118 |       | boot    |         |                                       |
|     119 |       | boot    |         |                                       |
|     120 |       | boot    |         |                                       |

General DUT#3 behaviour: slower boot, pause after power on, and visible delay while 
lighting USRLED1-3 until SD-card boots. Might be caused by a different uboot edition
than DUT#1 and DUT#2.

** Test 2 results (DUT#1 pulldown removed)

| Boot no. | DUT#1 |
|        1 | boot  |
|        2 | boot  |
|        3 | boot  |
|        4 | boot  |
|        5 | boot  |
|        6 | boot  |
|        7 | boot  |
|        8 | boot  |
|        9 | boot  |
|       10 | boot  |
|       11 | boot  |
|       12 | boot  |
|       13 | boot  |
|       14 | boot  |
|       15 | boot  |
|       16 | boot  |
|       17 | boot  |
|       18 | boot  |
|       19 | boot  |
|       20 | boot  |
|       21 | boot  |
|       22 | boot  |
|       23 | boot  |
|       24 | boot  |
|       25 | boot  |
|       26 | boot  |
|       27 | boot  |
|       28 | boot  |
|       29 | boot  |
|       30 | boot  |
|       31 | boot  |
|       32 | boot  |
|       33 | boot  |
|       34 | boot  |
|       35 | boot  |
|       36 | boot  |
|       37 | boot  |
|       38 | boot  |
|       39 | boot  |
|       40 | boot  |

I have invested quite a lot of time and energy the last weeks, on a project of mine that has had a long time building in my consciousness.

Last September I sent out a cry for participation on hackerspaces.org, for people wanting to be part of a hackerspace in my hometown of Esbjerg, Denmark. Although Esbjerg is home to quite both technical and heavy duty industries, servicing offshore drilling and wind farms in the North Sea, there really isn’t a community were technical like-minded can meet up and have fun using their skills. My hope was to find fellow tinkers/hackers/makers/creative people for joining in on having fun with technology with me and whoever wanted to be part of the party.

During the next 3/4 year I received about 6 emails from interested parties in and around the city of Esbjerg. So in late August I acted on an idea I have long had, of the hackerspace being a part of a creative workshop that is voluntary  driven in a culture and concert venue called Tobakken. After some talks, organization of interested parties and some thoughts at both sides, we are in the process of establishing what we, in lack of a better name, currently refer to as Hackerspace/Makerspace Esbjerg.

A picture from the first meeting at the location. Mikkel is talking about important stuff, as can be seen from the neck-grab position :).

You can see some more shots of the physical space here; hmse.mikini.dk/doku.php?id=gallerier

Most of the organizational efforts and communication is in the Facebook group at the moment, but I’m trying to push stuff over to a dokuwiki-site, which I think is much more appropriate.

You are more than welcome to chime in, if you happen to be in Esbjerg and miss an outlet for the creative technological cravings inside. See you in the space!