In my last posting I described getting Slackware ARM to boot headless on the ODROID U3 single-board computer, and I said that the next step would be to try to make it handle ungraceful shutdown (power loss) better. I plan to put this board into a Eurorack synthesizer module with no easy access to the microHDMI monitor connection, and SSH over the Ethernet connection as the only access to administrative functions. If, when the power is pulled on it, it comes up on next boot in a state where it requires console interaction to do a step like checking the filesystem before it will accept SSH connections, that is a disaster; I'd have to disassemble the whole module to extract the microSDMI card and replace the OS image. To be useful, the ODROID must be guaranteed or almost guaranteed to survive a power drop and come up SSH-able on the next boot. Ideally, I want pulling the plug on it with no shutdown formalities to be the normal expected way of shutting it down too, not just an error condition from which it can recover. A good journalling filesystem can increase the chance of recovery from occasional accidental power drops, but I think the only way to make routine non-accidental power drops safe is to keep the filesystem mounted read-only - which might be desirable anyway, to reduce wear on the flash memory and prevent its being corrupted by other kinds of accidents. So this posting is about my experiences configuring Slackware ARM on the ODROID U3 to keep its root filesystem read-only.
Of course, I'm not the first to face this kind of issue. Lots of people have tried to build embedded Linux systems that will reliably come up headlessly after a power drop. Good old "live CD" Linux images face a similar constraint, too. There are a number of how-to guides on the Net. The one I found most useful was this one from Karol Guciek. During system startup a script runs (on Slackware, this is /etc/rc.d/rc.S) which starts with the root filesystem mounted read-only, checks it, and (in a normal installation) remounts it read-write. For a system that keeps its root read-only, one modifies that script to not do the remount.
However, basic operation of a Linux host requires that system services must be able to write into some directories, mostly to create temporary files. The /tmp, /run, and /var directories are in this category, and the startup script has to be modified to mount "tmpfs" filesystems (basically, RAM disks) at those locations. A few pieces of system software also try to write into /etc. Notable things written there include /etc/mtab (the list of currently mounted filesystems) and /etc/resolv.conf (configuration for the DNS resolver, which gets rewritten by dhcpcd with each new DHCP lease, on systems so configured). Guciek recommends leaving /etc read-only and working around these cases one by one, usually with symlinks; but that requires patching a few programs that do not handle symlinks correctly by default. I put a tmpfs at /etc in my own configuration - which necessitates a little extra work in the startup script to copy the microSD card's version of /etc into the RAM disk before mounting the RAM disk in its final location. Just a little of the same sort of thing must be done for /var as well: OpenSSH fails to start if /var/empty doesn't exist, so my script must create that.
I've created a repository on Github for files related to this project, and you can see the edited /etc/rc.d/rc.S there. The relevant stanza for the new tmpfs mounts looks like this:
/sbin/mount -n -t tmpfs tmpfs /mnt/rwfs/var -o mode=0755 /sbin/mount -n -t tmpfs tmpfs /mnt/rwfs/etc -o mode=0755 /usr/bin/cp -a /etc/* /mnt/rwfs/etc /usr/bin/mkdir /mnt/rwfs/var/empty /sbin/mount --move /mnt/rwfs/var /var /sbin/mount --move /mnt/rwfs/etc /etc
I put that in the script at the point where it otherwise would remount the root read-write, and commented out the logic for the read-write remount. Only the /var and /etc tmpfs mounts are shown because the existing Slackware rc.S script already mounts tmpfs filesystem on /tmp and /run.
That is the only really critical step to make Slackware ARM work with a read-only root. There are, however, some other details worth attention. I commented out a few other lines that invoke "read" to demand keyboard input, since I won't have a keyboard; in such cases it's better for it to muddle through. I'll have the ODROID connected to a Matrix Orbital LCD panel (the kind of thing people typically stick in a drive bay on an overclocked PC to display fan speeds and temperature readings); that appears in Linux as the RS-232-over-USB serial port /dev/ttyUSB0, so I added lines to the boot script to set the baud rate on that port and display progress and error messages to the LCD. That way, if it does crash in a disastrous way, I'll at least have a little more information to diagnose the problem.
Although it's not necessary with my writable tmpfs mounted on /etc, I made the modifications to replace /etc/mtab with a symlink into /proc/mounts. That meant I was able to comment out the somewhat hackish existing code for updating /etc/mtab on boot, and not have to create more in the same vein for /var and /etc.
I also changed the hardcoded "-a" (do the repair automatically when it's safe) option on the boot-time fsck to "-y" (do the repair automatically, even if it isn't safe!). I don't expect to have initial filesystem checks occur very often because the filesystem will be mounted read-only all the time; count-based checks will not happen without read-write mounting, and I have disabled time-based checks because with no realtime clock on the ODROID, there'd be no way to do them accurately. But if a check does run, and if heaven forbid it decides it needs to do unsafe repair operations, I want it to make the attempt; because if it doesn't, and demands user input, I'll be forced to do as much work as reimaging the card from scratch anyway.
A few other points are worth mentioning. The ODROID U3's Ethernet subsystem does not have a factory-assigned MAC address. When the kernel brings up the port, if it does not have a better way to choose its MAC address, it will choose one at random. That gives a low probability of collisions with other devices on the same network segment. But it screws up my home network, because the DSL modem, which serves as DHCP server, sees each new MAC address as a new device and assigns it a new IP address. Thus, the ODROID gets a new IP address on each boot, which in turn makes trouble for my local DNS, causes SSH to see it as a new device every time, and so on.
Hardkernel supplies a customized Linux kernel that will look for a file called /etc/smsc95xx_mac_addr. If that exists, its contents become the MAC address. If it does not exist, then the randomly-chosen MAC address will be written into the file for use on future reboots. But remember that we mounted /etc as a tmpfs, which will be purged and rewritten on each boot. Also, although I haven't quite debugged the circumstances of this happening, it appears that when it tries to write the file is early in the boot process, so that it may be writing to /etc on the initial RAM disk, which would be purged even if not for the read-only root modifications.
Creating the /etc/smsc95xx_mac_addr file in the microSD card's /etc (mounted read-write) resolves this problem. It's not clear to me exactly how it resolves the problem, because that directory is not available during the initrd phase when it appears the kernel looks for the file; but, never mind, it does seem to work. One can simply write the desired MAC address to the file as six hexadecimal bytes separated by colons; or (before doing the modifications to make the root mount read-only) boot one time with commands in /etc/rc.d/rc.inet1.conf to load and unload the Ethernet driver, which will cause it to rewrite the file at a point where the rewritten value will be saved. Those commands are in my posted modified version of rc.inet1.conf, ready to uncomment.
I found that inetd ran out of control - consuming 100% of one of the four CPU cores, and driving the temperature higher than I was comfortable with - after the switch to read-only root. I was not able to debug which part of the read-only root situation was causing that, but I didn't try very hard. I just disabled inetd, which I didn't really need in my configuration anyway/
Some people have set up more elaborate systems where the root is mounted with a "union" filesystem, with a read-write RAM disk overlaid on the read-only-mounted persistent root. Any changes made are stored in the RAM disk, and any unchanged files are read from the persistent storage. That way, the whole filesystem appears writable and it's not necessary for software to be smart about where it writes. One can even set up a system that, periodically or on command, briefly mounts the persistent storage read-write and dumps the accumulated changes to it. That way we get most of the benefits of both read-write and read-only mounting.
I didn't go that route largely because it appears it would have required me to rebuild the initrd image. The union-mount has to be set up at a location other than / and then installed with the pivot_root system call, which affects only the process that invokes it and that process's children. The init script inside the initrd image runs as process ID 1 and then chains to System V init, which inherits process ID 1. But /etc/rc.d/rc.S is invoked by init, and dies without leaving persistent children. That means rc.S cannot do root-pivoting that will affect the entire system; any root-pivoting, and therefore any union mounting, must be done in the initred init script before System V init runs.
My initrd image is the one that came from Hardkernel's XUbuntu, not a native Slackware initrd image. To change it I'd have to either go back to XUbuntu at least temporarily; make a Slackware initrd work on the ODROID (which would require porting a Slackware kernel to the ODROID, something that would be a lot of work I'm not eager to do); or manually unpack, modify, and repack the initrd using lower-level tools without a distribution's initrd-creation scripts. None of those is appealing, especially when I'm not absolutely sure that I even want a union-mounted root in the first place. Keeping everything really read-only, as I mentioned at the start, has other advantages beyond the ungraceful-reboot issue that made it a necessity in the first place.
I'm going to want to do something clever with hotplugging USB storage devices, and a union-mount may be the right way to handle that. Just as I want to be able to recover from a power drop, I want to be able to insert a USB drive, use files on it (mostly, but not always entirely, in a read-only way), and then pull it out without having to do an explicit unmount. Auto-mounting on insertion is easy; and maybe the right thing to do is for the auto-mount to be read-only, unified with a RAM disk. Then I can freely modify files that appear to be "on" the USB drive but are really in RAM. Pulling it out loses those files but does not trash the drive; and either a periodic process or an explicit command can commit them permanently. One problem with this approach is that my modified, unsaved files have to fit in RAM, which might be a problem in the use case I imagine of having this device record long audio files. It might actually be better to go ahead and mount USB drives read-write, with some other way of mitigating any unexpected un-plugging. Since issues related to a USB drive will not interfere with boot, however, the stakes on them are much lower.