« FontAnvil 0.1 | Home | More ODROID adventure, or "You... »

Some notes on the ODROID U3

Sun 6 Apr 2014 by mskala Tags used: , ,

I recently bought a Hardkernel ODROID U3 single-board computer to use in one of my projects. The ODROID U3 is an ARM-based computer capable of running Linux or Android, with Ethernet, USB, microSD, and some other ports that I wasn't planning to use. My plan is to use it for controlling a music synthesizer, with the USB ports used to interface to the synth and a little LCD panel, and the Ethernet port used to talk to other computers. This posting contains some notes on getting it to work.

Although it's not directly related to the machine itself, I'd like to mention my experience with the shipper. Hardkernel sent the machine by FedEx to my home address. I don't remember whether Hardkernel was willing to ship it to an address other than my credit card billing address - many sellers won't - but even if sending it to my workplace had been an option, I didn't want to do that because of the likelihood that it would arrive with duty and tax owing, which might be a problem if it arrived at my work address. So I gave them my home address, and FedEx showed up while I was at work, and left a helpful "sorry we missed you" card in the mailbox of apartment #2.

I live in apartment #3, a fact of which the FedEx driver was evidently aware because he or she wrote my name and correct apartment number on the card before placing the card in the clearly-labelled wrong mailbox.

I walked into the lobby of the building and encountered a man in his underwear who said he lived in #2 and asked me if I knew who lived in #3. I said that was me, and he gave me the FedEx card. I had just enough time to phone FedEx customer service, get the address of their depot way out in the boonies, and make it out there on the bus to pay CAD 21.15 in fees and taxes and pick up the package before they closed. This added expense and inconvenience was significant compared to the USD 65 base price and USD 25 shipping I'd already paid to Hardkernel.

Hardkernel had assured me on their support forum that the device could be set up and used headlessly - that is, without a keyboard and monitor. I don't know if this is a translation issue, that maybe they understood my question as being about use and not including setup, or if maybe they were assuming the use of an operating system image other than the Ubuntu that they sell on preconfigured microSD cards (one of which I'd bought), but it appears to me that the claim, at least with regard to the configuration I was trying to use, is not true.

I don't have an HDMI monitor and USB keyboard to plug into the device, at least not at home. My home monitor can probably do HDMI, but I don't have an HDMI cable for it handy, much less one with the standardized, but rare, "micro" HDMI connector used by the ODROID U3. I'd considered buying from Hardkernel the special level-shifting adapter that allows its proprietary 1.8V UART port to appear as a USB serial device for a serial console, but I hadn't done that because I believed their claims about it being able to run headlessly with Ethernet control.

My experience trying to boot it with the factory OS image was that it would come up with the red light on (system power OK), the blue light would start out dim (boot loader running), then flash bright (kernel heartbeat) a couple times, then stay out. This suggests it was getting partway into its boot process and then dying. The Ethernet lights never came on, and I could see no evidence over the network that it was ever talking to the network. Unless I could find a way to hook up an HDMI monitor or a 1.8V UART serial connection, it didn't appear I could go further with that OS image. Hardkernel support recommended I test the power supply voltage.

My suspicion is that the initial configuration process for the Hardkernel-supplied Ubuntu installation simply requires console interaction. The hardcopy instruction sheet shipped with it tells you to select an "enlarge filesystem size" command and configure the screen resolution and so on, and then reboot, all using the console, before using it for the first time, and I suspect that even if (once those steps were completed) it would be able to accept SSH logins, it does not actually accept SSH logins when it first comes up out of the box. So it's not really possible to set it up headlessly, even if one might use it headlessly after it has been set up. I don't know that this is what's going on because, with neither a console nor a working Ethernet port, I had no way to see whatever error messages it may have been producing. (I suppose I could have searched the microSD card for log files afterward, but Ubuntu is systemd-based...) Another possibility is that since, as I've subsequently learned, the ODROID U3 or its commonly-used operating system installations may be unusually susceptible to ill effects from losing power without a formal shutdown, the OS image may have gotten into a bad state (such as a filesystem failing fsck) from one of my early attempts, and then the boot process was simply stopping, maybe demanding user input on the non-existant console, on all the subsequent attempts.

My next step was to try to get some other OS image, one actually designed for headless operation. I figured I'd end up building an image (probably with Slackware ARM) of my own eventually anyway, but I wanted to at least see it working with someone else's so that I could rule out a bad board, and have something to look at as a working example, before attempting to construct my own image. The first one I tried (by downloading it on my slow DSL, then dumping it onto the Hardkernel-supplied microSD card, overwriting the original contents [yes, I saved a spare copy first]) was called "Debian Jessie Particle". I didn't notice until after I tried to boot it, that that one is for the ODROID XU, not the ODROID U3. It never got past the dim-blue stage.

Next I tried the "Quiet Giant" image. That one is for ODROID U2, but Hardkernel assures me that the U3 can run all U2 software. This one produced the same results as the original Ubuntu image - dim blue, then a couple of bright blue flashes, which is how the normal boot sequence is supposed to start, but then no more blue flashes (they are supposed to repeat indefinitely) and no lights and no response on the Ethernet port.

It seemed clear that I wouldn't be able to make any more progress without a console. So (as of about noon today, Sunday) I decided to pack up the board, power supply, and so on, and take it to my office at work - where I have access to a USB keyboard and HDMI monitor, at the very least. I'd need a micro-HDMI cable, but I hoped to find that at Future Shop, which was on my way.

Neither Future Shop nor the three other places I tried along my way had a micro-HDMI cable. Hardkernel sells them for USD 5.70 but I wasn't about to spend another $45 on shipping and FedEx COD fees, and face the wait, just for that. I arrived at my office without the cable, but figuring since I was there, it couldn't hurt, I tried plugging in just the USB keyboard and Ethernet connections and trying to boot the ODROID one more time. The blue light started dim, flashed bright a couple times... and then continued flashing, one pair of two quick flashes with about a second between each pair. The Ethernet port LEDs glowed yellow and green). It seemed to have booted correctly! But without knowing what IP address, if any, my work network might have given it - and suspecting that the DHCP server, which only hands out IPs to authorized MAC addresses, probably hadn't actually given it an IP anyway - there wasn't much more I could do. So I pulled the power on it. Tried booting again, but it wouldn't. I took it home again.

At home, with some careful analysis, I determined that if the Quiet Giant image is shut down in an irregular way (for instance, by pulling the power on it), it goes into a state where it demands console-based user interaction on the next boot. Most likely, this is an unchecked filesystem triggering "single-user mode" (runlevel S, or whatever systemd calls that). It is also possible that something like this may be triggered by conditions related to an unset realtime clock, bearing in mind that without the optional battery the ODROID will not retain its realtime clock setting through a reboot. I think that demanding user intervention during boot is not EVER desirable behaviour for a server, but especially not in a headless embedded context. Anyway, the solution is, after any irregular shutdown, to pull the microSD card and use some other Linux system to fsck it. You can read for youself what Hardkernel community support had to say about that.

A further wrinkle was that if you follow the instructions to write the Quiet Giant image to the microSD card and then resize the partition and the contained filesystem to fill the card, resize2fs actually leaves the filesystem in an unchecked state. If you do not also run e2fsck on it, then congratulations, you are starting out in the filesystem-unchecked state from which Quiet Giant will not boot in an SSH-accepting way. I think it is also possible that the Quiet Giant image may contain a bad (unchecked) filesystem as distributed. I think some variation of that is what happened to me the first few times I tried. I don't know why the first attempt in my office succeeded when my previous attempts at home didn't. Maybe that time, I had rewritten the image and forgotten to resize it.

With this knowledge in hand, back at home I was able to get it to boot Quiet Giant more or less reliably. My home network DHCP gave it an address and told me what that address was, and I was able to log in via SSH. Next step: the custom OS image.

I had to take a crash course in ARM boot procedures, and I wasn't thrilled with what I discovered. Bear in mind that my background is running Linux on Intel and AMD desktop PCs. An embedded ARM system like the ODROID is a lot different. Really, an ODROID is pretty much a Samsung cell phone with the "phone" part removed and some USB ports added. It will only boot using a boot loader cryptographically signed by Samsung, which loads another boot loader cryptographically signed by Hardkernel. You do not get to have the keys to create these cryptographic signatures, so right there, the idea it's an "open" platform dies. I have strong philosophical objections to signed-code requirements, it would have significantly reduced my enthusiasm for buying this device if I had known about it before I did buy one, and it does significantly reduce my enthusiasm for buying another one or recommending this device to anyone else. That doesn't mean I absolutely won't; but, my enthusiasm is reduced. The facts that Hardkernel have indicated a willingness to sign third-party code, and that Hardkernel are probably forced to do as they do by contractual obligations to Samsung, does not make it okay. And I'm not the only person who thinks so.

Anyway, once we get past the crypto stages, a boot loader called "Das U-Boot" runs. This component is somewhat polymorphic - exactly what it does, where it lives on the boot device, and where it looks for operating systems to boot up, all depend on the particular details of the embedded system where it's running. Most of the documentation I can find for it seems to be based on undocumented assumptions about the system it will run on, that do not actually apply to the ODROID. For instance, what passes for the official user's manual is all about what to do with your "canyonlands board," whatever that is; I'm pretty sure what I have is not one. There is a Hardkernel-provided page of ODROID-specific U-Boot information, but it assumes the reader already knows the fundamental concepts of U-Boot, which I don't, and is largely devoted to how to recompile U-Boot with a cross-compiler, which would be useful only if you wanted to modify it, which in turn would be problematic because you are not allowed to have the cryptographic key required to sign a modified version so that the Samsung proprietary boot loader would chain to it in the first place. And by this point I am taking all Hardkernel-provided technical information with a fair bit of salt or possibly kimchi anyway.

It appears that at least in the Quiet Giant OS image, the binary crypto blobs and Das U-Boot occupy unpartitioned space before the start of the first partition on the microSD card. Then the first partition is a Windows FAT partition and apparently also used by U-Boot. It contains what looks like a Linux kernel and initrd, but I am not sure that those are what actually get booted because there is also a /boot directory containing a kernel inside the second partition, which is a live Linux filesystem.

Hardkernel support says you shouldn't delete the Windows partition for your own good, because you might be confused if you tried to mount the microSD card on a Windows machine. It is not clear to me whether it might possibly be okay to delete the Windows partition anyway, should you be willing to risk such confusion, or whether it would cause other problems.

U-Boot takes its orders (which kernel to boot, kernel command line, root partition identified by UUID, etc.) from a file called boot.scr in the Windows FAT partition. This file consists mostly of text, but it has a chunk of binary stuff at the start, so you can't edit it with a text editor and expect it to work. As far as I can tell, the correct way to edit the boot.scr is to have your ARM system already working, and then use a special command-line utility to add the binary stuff to a native text file. It appears to be tricky, though possible, to do this on a non-ARM Linux host.

I tried starting with the Quiet Giant image and replacing the root partition, entirely, with a Slackware ARM mini-root tree built into a brand-new ext4 filesystem that occupied all available space on the microSD card not already consumed by Windows, U-Boot, and proprietary blobs. My hope was that this root filesystem would be enough like Quiet Giant's that the untouched U-Boot installation would boot it.

It didn't work. In retrospect, I suspect that what failed was some combination of U-Boot looking for a kernel on the root filesystem (instead of in the Windows FAT filesystem) and not finding it because Slackware mini-root does not provide one; or U-Boot booting a kernel, but giving it the UUID of the now-deleted Quiet Giant root partition to load as the root.

As I was gearing up to try again, I ran into another snag: writing images to the microSD card seemed to get bogged down in USB bus resets, and proceed only at a snail's pace with a lot of errors logged to my desktop's dmesg. Aborting a write and trying again seemed to make the problem rapidly worse. It looked like the card was dying. Granted, writing an image of the entire card repeatedly is a stressful thing to do to a microSD card, but this was a brand-new card and it shouldn't be destroyed that fast. I searched the Web and found a report of somewhat similar behaviour from the Raspberry Pi community. It looks to me like the problem has to do with the on-the-fly optimizations performed by the card's built-in microcontroller. If you write massive amounts of data to the card, and especially if you abort such an operation and then try again, several times, its internal filesystem-like thing that it uses for keeping track of which blocks to reuse can become overloaded with half-written blocks in a way that doesn't necessarily kill the card, but slows it to a crawl, which in turn causes the kernel to time out trying to write, and triggers further aborts and bus resets. The solution (extracted from there and other discussions I found on the Web) is to overwrite the entire card with all zeroes. When a block is zero, it's smart enough to recognize that fact and just records "this block is zero" without really writing a block of all zeroes, and so writing an entire card's worth of zeroes lets it get back into a nice clean state instead of the mess it would be in from several consecutive whole-card writes of non-zero data.

So I rewrote the card with all zeroes, then with a fresh copy of Quiet Giant, then I went into the root filesystem and removed everything except the boot directory and the kernel modules - the stuff I think was missing from the Slackware mini-root. Then I wrote in another copy of the Slackware mini-root, and then dumped zeroes into a temporary file until the filesystem overflowed. I deleted the temporary file. The idea of that step was to zero out all the unused space in the image. I wanted to be able to save a compressed copy of this pristine image, for future experimentation, without having to compress and store all the Ubuntu stuff I had just deleted.

As of now, writing this, constructing that image and saving a copy to my desktop system has just completed. My next step will be to put the card into the ODROID and see if it boots. I hope it will!

ETA: No, it doesn't. Okay, next step is to get a micro-HDMI cable and bring it up non-headless. That's disappointing, but better than continuing to struggle with inadequate error feedback.

6 comments

Tony H.
How did you decide to buy the ODROID rather than, say, a Raspberry Pi or Adafruit's BeagleBone, or any other of the many boards with similar sounding descriptions?

Ah yes, Fedex. It could've been worse (UPS). Both are serious scammers when it comes to billing for unwanted and/or underperformed brokerage "services". Both have also been known to bill long after the fact, and don't hesitate to send it to collections if you don't pay up. We Canadians love to hate the post office, but if you can convince your shipper to use their country's post office, I find more than half the time, even on fairly high value electronics coming from Asia or the US, there is no billing from Canada Post for tax/duty/etc, no charge, and no delay. When they do manage to bill, they charge $5 plus the usually small actual duty, HST, and so on. There are web sites full of complaints about the way UPS and Fedex behave in this regard, along with various schemes to do your own brokerage.
Tony H. - 2014-04-07 17:54
Matt
Another significant advantage of the post office is that if I'm not home when the package arrives - which is *usually* the case - I can pick it up at the nearest postal outlet, which is within easy walking distance. Not a courier depot way out in the industrial zone. Unfortunately, many international shippers are not willing to negotiate about which carrier they will use.

As for choosing this board: I read some online reviews and looked for a combination of small physical size, USB ports, CPU power, price, and not too much extra stuff I don't plan to use. In retrospect, I think I underestimated how much trouble the ARM architecture would be. (Next post will be about the perils of floating point ABIs...) I might have been more satisfied holding out for something more closely resembling a PC, even if it meant trading off physical size.
Matt - 2014-04-07 19:03
Raymond Lutz
Oups... I was tempted too by the ODROID... but your experience (and that code signing thing...) me refroidit un peu. Que penser du Cubietruck? Not as beefier (CPU Wise) but more open?

Raymond Lutz - 2014-04-08 19:20
Matt
Since writing the above I've been able to use the monitor and keyboard to get the ODROID into a state where it can occasionally boot up and accept SSH connections headlessly. So far it's very unreliable, however - on 80% of attempts it seems to be shutting down a few seconds after power-on, which is not enough time to load the kernel and initialize networking. I suspect thermal issues, and am talking to Hardkernel support about it and hope this can be resolved.

Although I'm not happy about the crypto-boot thing, it's not really my top priority. After all, most PCs have closed firmware that runs before the on-disk boot loader and this isn't a whole lot different. The big issues I'd want to have settled before buying another embedded Linux device of any description would be:

1. Can I set it up headlessly - including the set-up process itself, not only normal operation? I would be most confident of this if I could buy a device that actually did not have a port for a monitor, since as soon as it has a port for a monitor, the manufacturers seem to think that means you're going to be willing to use that port.

2. Can I depend on it to boot up, and accept SSH connections, *every single time*, even after an unclean shutdown? That doesn't appear to be the case with the ODROID. The average desktop Linux box isn't perfect in this regard, but is better than the ODROID, and getting this right is a basic necessity for an embedded system.

3. Can I run a reasonably mainstream distribution of Linux on it, and not run systemd? I think this basically equates to "Can I run Slackware on it?", since most other reasonably mainstream distributions now require systemd.

If doing this over, I would put a lot less weight on the price of the board in the initial buying decision - because if I'm going to spend $150 on extra cables and hardware just to get the board to boot, I don't care whether the initial price was $50 or $80.

Overall, my impression of ARM Linux in general is negative, and I'd be inclined to look for some other hardware or software platform, even if that means I end up using an OS other than Linux and sacrificing point #3. The unclean-shutdown thing is important, and is a real source of tension between any kind of Unix (which needs to be shut down properly before power-down) and any kind of embedded application (which needs to come up working on power-up UNCONDITIONALLY). Maybe it'd be better to hold out for a real Intel/AMD Liunux host, not ARM (a friend has suggested the Intel Galileo board) where my existing Linux expertise might make it easier to deal with the shutdown issue; or else go for something that doesn't even purport to be Linux, but can better deal with the start/stop requirements of embedded use.

The Cubietruck looks like it's richer in peripherals than the ODROID, and I think it may be more solidly intended as a PC replacement than as an embedded system. For that reason I'd be hesitant about using it for my own purposes, but the same attributes may make it better for yours, depending on what you want to do. Before buying one I would want to know exactly what operating system distributions have pre-built images available for it; unlike a real PC which can boot a generic "install" image from DVD or USB drive, it appears that ARM systems *must* have customized bootable images built per device, and if the distribution you want to use does not have such an image for the particular device you want to use, you're in a world of hurt trying to cross-compile your own.
Matt - 2014-04-09 09:19
Daniel
I am kind of confused, why you had so many issues with the Odroid U3.

I just had 2 issues, which a) was a bit self-driven and b) dumb in the first row. I bought 3 ODroid U3 and was surprised no power cables were in cluded. So I bought super cheap ones off Ebay.

Funny thing was: all 3 cables had *exactly* the same issue. The Odroid booted up and freezed nearly always at the same point. i struggled with that for a good amount of months because I always thought it must be either the power supply (USB-Charger) or the Odroid's software. After I luckily had a fourth cable which I just tried because it was a tad longer than the 3 others, it worked just fine.
I exchanged all 3 USB-power cables and had no issues anymore.

The other issue was: I had 8 USB-TTL adaptors for my Cubieboard 2,Cubietruck and Raspberry boards at home, but I didn't checked the specs to see that the voltage differs to the Odroid, so I had to order the Hardkernel USB-TTL cable seperately.

I use all 3 Odroid U3 headless, so there is no additional devices connected except the obvious Ethernet cable (I would not count that as a device ;-) ). I use the microsd slot for the usual Ubuntu Installation hardkernel is offering at their website.

Cheers,

Daniel

PS: Do not except the Cubieboard/Cubietruck boards anything easier than the Odroid ... They have their own issues :-/
Daniel - 2014-10-21 07:06
woelen
I have two Odroid U3 devices, one with the 8 GByte eMMC module and the other with a 16 GByte microSD card. I had no problem setting them up through SSH. I NEVER have one of them had connected to a keyboard, mouse or monitor. I purchased the power supply from Hardkernel. It has a rather thin connector (2.2 / 0.8 mm), but the connector fits tightly and has no problems at all. I also purchased the optional active cooler with fan and replaced the standard cooler. I am not sure whether this was necessary, the fan only spins up for a few seconds, immediately after power on, but after that, it does not spin up anymore. If I start three or four threads, all using 100% CPU, then the fan spins up, speed depending on precise load and ambient temperature.

I used the ubuntu 14.04 image from the Odroid image server and had no issues. I had to run a odroid utility when I first logged in.
I used the command 'arp-scan --interface=eth0 --localnet' on my local PC, which is on the same network segment as the Odroids to find out what IP-address they obtained on booting and with ssh I could connect, using username odroid and password odroid. From there, everything went fine. I had the two little systems configured the way I liked them in just a few hours. I disabled Xorg and after booting they only use appr. 150 MByte of memory, leaving 1.85 GByte of memory for user applications. The image I used can be found here (the Ubuntu 14.04.1 images for U2/U3):
http://com.odroid.com/sigong/nf_file_board/nfile_board.php

I am quite happy with the little systems. They run stably, take very little power and were easy to setup.
woelen - 2015-01-21 14:05


(optional field)
(optional field)
Answer "bonobo" here to fight spam. ここに「bonobo」を答えてください。SPAMを退治しましょう!
I reserve the right to delete or edit comments in any way and for any reason.