Our last episode concluded with the discovery that the clip leads I was using to power the ODROID U3 had nontrivial resistance, possibly close to two Ohms for the pair. That's a lot, when one is pulling up to 2A through them from a 5V supply. I switched them for some lower-resistance ones, and the ODROID seemed to behave much better. Between that and a replacement of the Hardkernel-supplied microSD card, I was hopeful that the reliability problems would be much reduced and I could make some progress on the software side of things.
Indeed, I haven't seen it crash or fail to boot, except for verifiably software-related reasons, since I replaced the clip leads and microSD card. Since those clip leads were supplied by me and not by Hardkernel, I feel a little bit better about the quality of Hardkernel's product than I was feeling earlier. I'm still disappointed with their microSD card, though. Despite supposedly being "Class 10," it is a lot slower than the Class 6 card I bought locally to replace it, and the speed difference seems to be due to the Class 10 card randomly failing, retrying, and causing bus resets. That's not confidence-inspiring when I hope to run a whole system off the card.
I wanted to run Slackware - or at the very least, something that doesn't use systemd - on my ODROID. None of the ready-made Linux images I've successfully booted meet that criterion. There is a Slackware port for ARM, but it doesn't contain specific support for this board. There is no bootable installer for Slackware ARM because such a thing wouldn't be very useful. Most ARM boards cannot, or can only with great difficulty, boot from removeable media like a USB stick, and because the hardware usually requires a lot of kernel customization, it would be difficult or impossible to design a generic kernel for the installer to run on all hardware anyway.
Here are the basic steps I followed to get Slackware ARM running on my ODROID U3. This is the cleaned-up sequence, skipping over the failed experiments I made along the way; the idea is to provide some advice for others attempting the same thing. I'll go into more detail after the list.
- Start with the Hardkernel-supplied XUbuntu microSD card. Make a backup of the system image from it.
- Write the XUbuntu system image to the microSD card that will actually be used.
- Boot with that card, and a keyboard and monitor hooked up to the USB and micro-HDMI connections.
- Expand the root partition and otherwise configure the system far enough that it can boot headlessly and accept SSH connections.
- Shut down, reboot headless, verify the boot process, temperature sensor, and network connection work, and shut down again.
- Transfer the card to a desktop computer running Slackware (not necessarily ARM) and remove everything from the root partition except the boot stuff and /etc/fstab.
- Install Slackware ARM, using the desktop's existing Slackware scripts.
- Put the card back in the ODROID, boot it with keyboard and monitor, set up networking, edit init scripts to maximize the chances of surviving a power drop.
- Reboot headlessly.
The Hardkernel-supplied XUbuntu card is laid out something like this:
- Traditional DOS-style partition table.
- 2M unpartitioned space containing boot-loader blobs.
- Primary partition 1: 128M, containing a Windows 95 FAT filesystem, which in turn contains some U-Boot boot scripts (text with binary headers) and files called uInitrd and zImage which seem to be part of U-boot.
- Primary partition 2: roughly 4G, containing an ext2 (not ext3 or ext4) filesystem which in turn contains a live XUbuntu Linux installation. This includes a kernel and several initrd files in /boot, and also some uInitrd files, which are apparently not the same thing as initrd files. The kernel is a vmlinuz, not a zImage, and apparently not the same kernel that is in primary partition 1.
- Unpartitioned space to fill the card, roughly 4G, probably all zeroed out. (I didn't check before writing my own data into it.)
On my desktop system, the whole device is /dev/sde, and the partitions are /dev/sde1 and /dev/sde2.
To back this up, note that you must back up the unpartitioned space at the start because that's where the critical binary-blob boot software is located. You should also back up both primary partitions; it's safe to leave out the empty space at the end. You could probably fit an image of the whole works on a DVD as a straight image from /dev/sde if you compress it; or an uncompressed image of /dev/sde stretching from the start of the device to the end of partition 2, leaving out the empty space at the end.
However, a file image of an entire partitioned device like that is hard to work with: in order to mount partitions within the image (which is not absolutely necessary, but useful) you end up having to compute offsets and pass them as mount options in an inconvenient way. Instead, what I did was to image the first 2M of /dev/sde into a file, and then image each of /dev/sde1 and /dev/sde2 into its own file. That way I can mount either partition as an image file without needing to compute the offsets. The only "gotcha" is that when restoring it, I have to restore the first 2M to the entire device, reread the partition table, and then restore the two partitions individually. The set of three image files fits on a DVD.
I guess that it's because of wanting the image to fit on a DVD, that Hardkernel burns the cards this way, with a large unpartitioned space at the end. Another possibility might be that this OS image is actually meant to be able to fit on a 4G microSD card - but who uses 4G microSD cards anymore? I had a little trouble finding one even as small as 8G, last time I shopped for them.
As I mentioned, I had a lot of trouble with the Hardkernel-suppled microSD card. The pattern I observed was that when writing an image file to the card with dd, the transfer would start out okay, then slow to a crawl, and start producing USB bus reset messages in my desktop computer's dmesg at 31-second intervals. It seemed to get worse with additional attempts, eventually escalating to massive numbers of "Buffer I/O error" and "EXT4-fs warning [...] I/O error writing to inode" messages. On one occasion it did something to the desktop's kernel that looked like a prelude to a kernel panic: it started giving weird segfault-ish messages (I forget the exact wording) on any attempt to read dmesg, and on some other kernel-related operations. My system remained functional enough that I was able to execute a graceful shutdown with no apparent data loss; but it looked like, as I said, it was heading for a kernel panic. All these problems cleared up when I switched to a replacement Lexar 8G microSD card purchased locally. That was also about the time I switched to better power cables on the ODROID, but since the card writing problems were on the desktop machine anyway, I don't think they can be blamed entirely on the power cable issue.
I am not absolutely certain that it's impossible to boot and use the Hardkernel-supplied XUbuntu image without ever attaching a keyboard and monitor to the ODROID, but at the very least, it's a lot easier to get it working and debug other issues if there is a console, so I think it's best to treat the console as a necessity for building the Slackware image even if the device is planned to eventually run headless. On the first boot, it brings up a menu for expanding partition 2 to fill the rest of the card (probably a good idea) and selecting a video resolution. I think the "video resolution" choice actually alters the U-Boot configuration, making it semi-permanently change which of the boot scripts in partition 1 it will execute on subsequent boots. These scripts then specify different Linux kernel command lines.
I think I may have made a bad selection on the video resolution. I ended up with only a rectangular area covering about the upper 1/2 and left 3/4 of my screen actually working correctly. Any console text below the bottom of that zone was invisible, and text from within the zone would be echoed oddly in the area to the right. I don't know if this was a problem with the driver not matching the ODROID hardware, some issue with communication between the ODROID and my monitor (bearing in mind that there was an unusual plug adapter in the circuit), or simply that I chose the wrong option on the menu. Since I was only planning to use the video for a minimal amount of configuration work anyway, I wasn't motivated to try to debug this issue further. I used stty with "rows" and "cols" settings chosen by trial and error to force the console to only use the good part of the screen.
I also had to use a selection on the boot menu to enable SSH service for headless operation, with DHCP configuration for the IP address.
The point of rebooting again after this initial config run is to make sure that the basic boot process can bring up the machine headlessly. This has to be done headlessly (disconnect the HDMI monitor, and power the computer down, before rebooting) to avoid the "temperature stucked at 50" issue which seems to be related to hot-plugging the HDMI connection. SSH has to be enabled so that you can log in to check the temperature once it's headless.
The command to check temperature is
/sys/devices/virtual/thermal/thermal_zone0/temp . That should print
an integer, which is the temperature in degrees Centigrade multiplied by
100. Despite the precision that implies, the sensor appears to only measure
whole degrees; the last two digits are always zeros. This number should
change as the physical temperature of the heat sink changes, and in
particular, should not always be 5000 (which would indicate it's
"stucked"). I have seen it go as high as 5900, but no higher. It looks to
me like the frequency scaling is designed to keep it below 6000.
Note that the Hardkernel-supplied XUbuntu image, and mainline Slackware ARM, are both compiled for soft float. I don't think this procedure would work if they didn't match. If you wanted hard float, I think something similar could be attempted with Alien's ARM (Slackware ARM rebuilt for hard float) installed over Mani Dhillon's ARM Arch. However, I haven't tried that and am not sure if I will bother. Keeping the entire OS on a microSD card does at least make such experiments cheap - I can just put the experimental OS on another card without touching my known good configuration.
The next step is to remove everything from XUbuntu that makes it XUbuntu, leaving only the parts needed to boot that are not supplied by Slackware ARM. Transfer the card to a friendly desktop machine running Slackware and mount the XUbuntu root partition somewhere. Keep everything in /boot. Keep /etc/fstab (which tells how to mount the root filesystem by its UUID - initrd might be able to automatically guess this but I think it's safer to keep the file). Keep /lib/modules and /lib/firmware. I kept /lib/e2initrd_helper because I didn't know what it was and it looked boot-related, but if you know what it is, maybe you'll know whether you need it. I kept /lib/modprobe.d because it looked like it might be specific to the modules, but I suspect it's not actually necessary.
Blow away everything else.
It would be a good thing for the root to be mounted with noatime to reduce wear on the microSD card and I don't remember whether that's default. Now would be a good time to check /etc/fstab to make sure.
It would be a good thing to have a journal for this filesystem, and the Hardkernel image doesn't seem to come with one, so this would be a good time to run tune2fs and set that up, effectively changing the ext2 into an ext3 filesystem. (Temporarily unmount the filesystem to do this, of course.)
Don't do anything that changes the UUID of the root filesystem (for instance, deleting it entirely and starting over with a freshly built one) unless you are sure you can correctly update everywhere the UUID is recorded, which probably includes inside the not-text-editable U-Boot script files in partition 1. It should not otherwise be necessary to touch partition 1.
Slackware ARM's Web site suggests using rsync to download the precompiled packages, and that works fine. It will download a whole lot of packages that are inappropriate for headless operation, such as KDE, so if you're conservative of bandwidth you might want to do something more selective than just grabbing the entire tree.
Once you have the packages, you can install them with the desktop
machine's native Slackware installpkg script. Use commands like
installpkg --root /mnt/sdhc --menu --ask a/*.t[gx]z . The
--root option says to install to another system's root mounted other than
at the current system's root. (Don't forget this, especially not with the A
series, unless you are sure your desktop can run ARM soft-float binaries!)
The --menu --ask options direct installpkg to prompt you for which packages
to install, and to really prompt you even on packaged flagged as required.
That is important because some pretty wacky things have been given "ADD"
(required package, automatic install without asking) status in the default
tag files. For instance, "ebook-tools." The glob pattern "*.t[gx]z" is
because my copy of Slackware ARM seems to include both
txz and .tgz packages. I'm not sure it's supposed to. Of course, the
whole command line should be altered to taste.
Slackware packages contain install scripts that are supposed to run on the target system, or at least within the target system's directory space. Since our friendly Slackware desktop machine is probably a totally different architecture from the target system, it seems like running these scripts might be a problem. In practice, the only errors I saw had to do with my x86_64 "ldconfig" being unable to handle the ARM libraries, and that seemed to be harmless; I think ldconfig gets re-run during boot anyway, and running it during package installation is only necessary to make sure newly-installed packages are immediately available when a machine is installing packages to its own root.
At this point I dumped /dev/zero into a file in the root partition until it ran out of space, and then deleted the file, so as to zero out the empty space and make the backup image I was about to create more compressible. This step shouldn't be absolutely necessary.
It might be possible to get everything installed and configured sufficiently for the card to boot headlessly and accept SSH connections without a non-headless Slackware boot at all, but I didn't attempt that. So many things could go wrong, and could easily be avoided with a console, that it seemed like the only reason to insist on entirely headless configuration would be for some kind of macho challenge - and since I'd already compromised with the non-headless XUbuntu boot, it didn't seem worthwhile to keep pushing it. But feel free, if you want to make the attempt.
So I rebooted with monitor and keyboard again to do the final configuration for headless operation. I'll assume readers attempting this already know how to do that stuff for the most part. Things not to forget include:
- Set up networking in /etc/rc.d/rc.inet1.conf (likely with USE_DHCP="yes"; or run netconfig)
- Make sure sshd will start (probably by setting the execute bit on /etc/rc.d/rc.sshd)
- Set a password on the root account and create at least one non-root account.
- Turn off any servers you don't want, and make sure any that remain are secured.
- Recommended: configure sshd not to accept direct root logins.
Then, reboot and enjoy your new Slackware ODROID U3 installation. Here's an embedded Soundcloud audio thing of me trying out mine.
It remains to worry about power loss possibly screwing up the filesystem in such a way that the boot demands user input. Adding a journal to the filesystem will help a lot by making noninteractive recovery more possible, but it's not a guarantee. I was advised with Ubuntu-related distributions to set a variable called FSCKFIX to "yes" (somewhere in some init script; I wasn't paying close attention). This tells the boot-time fsck to run non-interactively. Slackware doesn't seem to have a direct equivalent. I ended up editing /etc/rc.d/rc.S to simply change the hardcoded fsck -a to fsck -y. This is a dangerous thing to do, and I wouldn't do it on a desktop system or an ordinary kind of server. The ODROID, as I'm planning to use it, is a special case because it's a headless embedded system. If it demands console interaction, that is a disaster. It means I have to disassemble the equipment of which the board is part, in order to attach the HDMI cable, and at that point, it'll be equally easy to re-image the microSD card from scratch. So anything that increases the chance it will be able to come up into a condition where I can log in and fix things without disassembling the equipment is worthwhile - even at the risk of trashing the whole OS image. "Press enter to continue" is as big a problem as "Filesystem utterly destroyed."
I still want more safety, though. I want it to be okay to pull the power on the device as a routine practice, not just as an undesirable scenario we can probably recover from. I think that means the bottom line is I have to keep the microSD card mounted read-only, except maybe for brief windows of saving any data that really needs to be saved. Keeping the mount read-only would be desirable anyway for reasons of wear minimization and to keep the OS from getting corrupted in other ways; it's really quite appropriate for an embedded system. Anything involving long-term read-write mounting of the card is insufficiently safe. So that's my next subproject: figuring out how to get my Linux environment to run acceptably with root mounted read-only. I know the Raspberry Pi people have done some work on this, so I don't have to break completely new ground.