I recently bought a Hardkernel ODROID U3 single-board computer to use in one of my projects. The ODROID U3 is an ARM-based computer capable of running Linux or Android, with Ethernet, USB, microSD, and some other ports that I wasn't planning to use. My plan is to use it for controlling a music synthesizer, with the USB ports used to interface to the synth and a little LCD panel, and the Ethernet port used to talk to other computers. This posting contains some notes on getting it to work.
Although it's not directly related to the machine itself, I'd like to mention my experience with the shipper. Hardkernel sent the machine by FedEx to my home address. I don't remember whether Hardkernel was willing to ship it to an address other than my credit card billing address - many sellers won't - but even if sending it to my workplace had been an option, I didn't want to do that because of the likelihood that it would arrive with duty and tax owing, which might be a problem if it arrived at my work address. So I gave them my home address, and FedEx showed up while I was at work, and left a helpful "sorry we missed you" card in the mailbox of apartment #2.
I live in apartment #3, a fact of which the FedEx driver was evidently aware because he or she wrote my name and correct apartment number on the card before placing the card in the clearly-labelled wrong mailbox.
I walked into the lobby of the building and encountered a man in his underwear who said he lived in #2 and asked me if I knew who lived in #3. I said that was me, and he gave me the FedEx card. I had just enough time to phone FedEx customer service, get the address of their depot way out in the boonies, and make it out there on the bus to pay CAD 21.15 in fees and taxes and pick up the package before they closed. This added expense and inconvenience was significant compared to the USD 65 base price and USD 25 shipping I'd already paid to Hardkernel.
Hardkernel had assured me on their support forum that the device could be set up and used headlessly - that is, without a keyboard and monitor. I don't know if this is a translation issue, that maybe they understood my question as being about use and not including setup, or if maybe they were assuming the use of an operating system image other than the Ubuntu that they sell on preconfigured microSD cards (one of which I'd bought), but it appears to me that the claim, at least with regard to the configuration I was trying to use, is not true.
I don't have an HDMI monitor and USB keyboard to plug into the device, at least not at home. My home monitor can probably do HDMI, but I don't have an HDMI cable for it handy, much less one with the standardized, but rare, "micro" HDMI connector used by the ODROID U3. I'd considered buying from Hardkernel the special level-shifting adapter that allows its proprietary 1.8V UART port to appear as a USB serial device for a serial console, but I hadn't done that because I believed their claims about it being able to run headlessly with Ethernet control.
My experience trying to boot it with the factory OS image was that it would come up with the red light on (system power OK), the blue light would start out dim (boot loader running), then flash bright (kernel heartbeat) a couple times, then stay out. This suggests it was getting partway into its boot process and then dying. The Ethernet lights never came on, and I could see no evidence over the network that it was ever talking to the network. Unless I could find a way to hook up an HDMI monitor or a 1.8V UART serial connection, it didn't appear I could go further with that OS image. Hardkernel support recommended I test the power supply voltage.
My suspicion is that the initial configuration process for the Hardkernel-supplied Ubuntu installation simply requires console interaction. The hardcopy instruction sheet shipped with it tells you to select an "enlarge filesystem size" command and configure the screen resolution and so on, and then reboot, all using the console, before using it for the first time, and I suspect that even if (once those steps were completed) it would be able to accept SSH logins, it does not actually accept SSH logins when it first comes up out of the box. So it's not really possible to set it up headlessly, even if one might use it headlessly after it has been set up. I don't know that this is what's going on because, with neither a console nor a working Ethernet port, I had no way to see whatever error messages it may have been producing. (I suppose I could have searched the microSD card for log files afterward, but Ubuntu is systemd-based...) Another possibility is that since, as I've subsequently learned, the ODROID U3 or its commonly-used operating system installations may be unusually susceptible to ill effects from losing power without a formal shutdown, the OS image may have gotten into a bad state (such as a filesystem failing fsck) from one of my early attempts, and then the boot process was simply stopping, maybe demanding user input on the non-existant console, on all the subsequent attempts.
My next step was to try to get some other OS image, one actually designed for headless operation. I figured I'd end up building an image (probably with Slackware ARM) of my own eventually anyway, but I wanted to at least see it working with someone else's so that I could rule out a bad board, and have something to look at as a working example, before attempting to construct my own image. The first one I tried (by downloading it on my slow DSL, then dumping it onto the Hardkernel-supplied microSD card, overwriting the original contents [yes, I saved a spare copy first]) was called "Debian Jessie Particle". I didn't notice until after I tried to boot it, that that one is for the ODROID XU, not the ODROID U3. It never got past the dim-blue stage.
Next I tried the "Quiet Giant" image. That one is for ODROID U2, but Hardkernel assures me that the U3 can run all U2 software. This one produced the same results as the original Ubuntu image - dim blue, then a couple of bright blue flashes, which is how the normal boot sequence is supposed to start, but then no more blue flashes (they are supposed to repeat indefinitely) and no lights and no response on the Ethernet port.
It seemed clear that I wouldn't be able to make any more progress without a console. So (as of about noon today, Sunday) I decided to pack up the board, power supply, and so on, and take it to my office at work - where I have access to a USB keyboard and HDMI monitor, at the very least. I'd need a micro-HDMI cable, but I hoped to find that at Future Shop, which was on my way.
Neither Future Shop nor the three other places I tried along my way had a micro-HDMI cable. Hardkernel sells them for USD 5.70 but I wasn't about to spend another $45 on shipping and FedEx COD fees, and face the wait, just for that. I arrived at my office without the cable, but figuring since I was there, it couldn't hurt, I tried plugging in just the USB keyboard and Ethernet connections and trying to boot the ODROID one more time. The blue light started dim, flashed bright a couple times... and then continued flashing, one pair of two quick flashes with about a second between each pair. The Ethernet port LEDs glowed yellow and green). It seemed to have booted correctly! But without knowing what IP address, if any, my work network might have given it - and suspecting that the DHCP server, which only hands out IPs to authorized MAC addresses, probably hadn't actually given it an IP anyway - there wasn't much more I could do. So I pulled the power on it. Tried booting again, but it wouldn't. I took it home again.
At home, with some careful analysis, I determined that if the Quiet Giant image is shut down in an irregular way (for instance, by pulling the power on it), it goes into a state where it demands console-based user interaction on the next boot. Most likely, this is an unchecked filesystem triggering "single-user mode" (runlevel S, or whatever systemd calls that). It is also possible that something like this may be triggered by conditions related to an unset realtime clock, bearing in mind that without the optional battery the ODROID will not retain its realtime clock setting through a reboot. I think that demanding user intervention during boot is not EVER desirable behaviour for a server, but especially not in a headless embedded context. Anyway, the solution is, after any irregular shutdown, to pull the microSD card and use some other Linux system to fsck it. You can read for youself what Hardkernel community support had to say about that.
A further wrinkle was that if you follow the instructions to write the Quiet Giant image to the microSD card and then resize the partition and the contained filesystem to fill the card, resize2fs actually leaves the filesystem in an unchecked state. If you do not also run e2fsck on it, then congratulations, you are starting out in the filesystem-unchecked state from which Quiet Giant will not boot in an SSH-accepting way. I think it is also possible that the Quiet Giant image may contain a bad (unchecked) filesystem as distributed. I think some variation of that is what happened to me the first few times I tried. I don't know why the first attempt in my office succeeded when my previous attempts at home didn't. Maybe that time, I had rewritten the image and forgotten to resize it.
With this knowledge in hand, back at home I was able to get it to boot Quiet Giant more or less reliably. My home network DHCP gave it an address and told me what that address was, and I was able to log in via SSH. Next step: the custom OS image.
I had to take a crash course in ARM boot procedures, and I wasn't thrilled with what I discovered. Bear in mind that my background is running Linux on Intel and AMD desktop PCs. An embedded ARM system like the ODROID is a lot different. Really, an ODROID is pretty much a Samsung cell phone with the "phone" part removed and some USB ports added. It will only boot using a boot loader cryptographically signed by Samsung, which loads another boot loader cryptographically signed by Hardkernel. You do not get to have the keys to create these cryptographic signatures, so right there, the idea it's an "open" platform dies. I have strong philosophical objections to signed-code requirements, it would have significantly reduced my enthusiasm for buying this device if I had known about it before I did buy one, and it does significantly reduce my enthusiasm for buying another one or recommending this device to anyone else. That doesn't mean I absolutely won't; but, my enthusiasm is reduced. The facts that Hardkernel have indicated a willingness to sign third-party code, and that Hardkernel are probably forced to do as they do by contractual obligations to Samsung, does not make it okay. And I'm not the only person who thinks so.
Anyway, once we get past the crypto stages, a boot loader called "Das U-Boot" runs. This component is somewhat polymorphic - exactly what it does, where it lives on the boot device, and where it looks for operating systems to boot up, all depend on the particular details of the embedded system where it's running. Most of the documentation I can find for it seems to be based on undocumented assumptions about the system it will run on, that do not actually apply to the ODROID. For instance, what passes for the official user's manual is all about what to do with your "canyonlands board," whatever that is; I'm pretty sure what I have is not one. There is a Hardkernel-provided page of ODROID-specific U-Boot information, but it assumes the reader already knows the fundamental concepts of U-Boot, which I don't, and is largely devoted to how to recompile U-Boot with a cross-compiler, which would be useful only if you wanted to modify it, which in turn would be problematic because you are not allowed to have the cryptographic key required to sign a modified version so that the Samsung proprietary boot loader would chain to it in the first place. And by this point I am taking all Hardkernel-provided technical information with a fair bit of salt or possibly kimchi anyway.
It appears that at least in the Quiet Giant OS image, the binary crypto blobs and Das U-Boot occupy unpartitioned space before the start of the first partition on the microSD card. Then the first partition is a Windows FAT partition and apparently also used by U-Boot. It contains what looks like a Linux kernel and initrd, but I am not sure that those are what actually get booted because there is also a /boot directory containing a kernel inside the second partition, which is a live Linux filesystem.
Hardkernel support says you shouldn't delete the Windows partition for your own good, because you might be confused if you tried to mount the microSD card on a Windows machine. It is not clear to me whether it might possibly be okay to delete the Windows partition anyway, should you be willing to risk such confusion, or whether it would cause other problems.
U-Boot takes its orders (which kernel to boot, kernel command line, root partition identified by UUID, etc.) from a file called boot.scr in the Windows FAT partition. This file consists mostly of text, but it has a chunk of binary stuff at the start, so you can't edit it with a text editor and expect it to work. As far as I can tell, the correct way to edit the boot.scr is to have your ARM system already working, and then use a special command-line utility to add the binary stuff to a native text file. It appears to be tricky, though possible, to do this on a non-ARM Linux host.
I tried starting with the Quiet Giant image and replacing the root partition, entirely, with a Slackware ARM mini-root tree built into a brand-new ext4 filesystem that occupied all available space on the microSD card not already consumed by Windows, U-Boot, and proprietary blobs. My hope was that this root filesystem would be enough like Quiet Giant's that the untouched U-Boot installation would boot it.
It didn't work. In retrospect, I suspect that what failed was some combination of U-Boot looking for a kernel on the root filesystem (instead of in the Windows FAT filesystem) and not finding it because Slackware mini-root does not provide one; or U-Boot booting a kernel, but giving it the UUID of the now-deleted Quiet Giant root partition to load as the root.
As I was gearing up to try again, I ran into another snag: writing images to the microSD card seemed to get bogged down in USB bus resets, and proceed only at a snail's pace with a lot of errors logged to my desktop's dmesg. Aborting a write and trying again seemed to make the problem rapidly worse. It looked like the card was dying. Granted, writing an image of the entire card repeatedly is a stressful thing to do to a microSD card, but this was a brand-new card and it shouldn't be destroyed that fast. I searched the Web and found a report of somewhat similar behaviour from the Raspberry Pi community. It looks to me like the problem has to do with the on-the-fly optimizations performed by the card's built-in microcontroller. If you write massive amounts of data to the card, and especially if you abort such an operation and then try again, several times, its internal filesystem-like thing that it uses for keeping track of which blocks to reuse can become overloaded with half-written blocks in a way that doesn't necessarily kill the card, but slows it to a crawl, which in turn causes the kernel to time out trying to write, and triggers further aborts and bus resets. The solution (extracted from there and other discussions I found on the Web) is to overwrite the entire card with all zeroes. When a block is zero, it's smart enough to recognize that fact and just records "this block is zero" without really writing a block of all zeroes, and so writing an entire card's worth of zeroes lets it get back into a nice clean state instead of the mess it would be in from several consecutive whole-card writes of non-zero data.
So I rewrote the card with all zeroes, then with a fresh copy of Quiet Giant, then I went into the root filesystem and removed everything except the boot directory and the kernel modules - the stuff I think was missing from the Slackware mini-root. Then I wrote in another copy of the Slackware mini-root, and then dumped zeroes into a temporary file until the filesystem overflowed. I deleted the temporary file. The idea of that step was to zero out all the unused space in the image. I wanted to be able to save a compressed copy of this pristine image, for future experimentation, without having to compress and store all the Ubuntu stuff I had just deleted.
As of now, writing this, constructing that image and saving a copy to my desktop system has just completed. My next step will be to put the card into the ODROID and see if it boots. I hope it will!
ETA: No, it doesn't. Okay, next step is to get a micro-HDMI cable and bring it up non-headless. That's disappointing, but better than continuing to struggle with inadequate error feedback.