How to convert Linux into an embedded system

Running Linux on your embedded system can be great for giving you a stable base, lots of tools, and you can often solve problems with a quick Google search.

I define embedded as any custom-built computer system, not only tiny devices. In many cases you can develop on the computer itself for ease of development, however in other cases you will need to be able to set up a cross compile environment (for example if you have a slow ARM computer and you want to compile things on your fast Intel i7 processor).

This post will give you a high level list of the steps I take when setting up my Linux systems, with links for more information is some cases.

Linux is great since you can modify just about anything, it is free, you can strip out components you do not need, and you can configure just about anything (including the scheduler). The downside is that you can configure just about anything and break something in the process. Linux also tends to be a little behind when new technologies are released; common problems have been wireless cards, SSDs, firewire and USB3. Linux is in essence just a kernel that let’s you build operating system distributions on it. Some of the common distros are Debian, Ubuntu (I know this is Debian, but this is the flavor that is very common, supported, and one that I often use), Fedora, Suse, and Wind River.

A common question is whether you need a real-time version of Linux, and the answer is that most people do not. Look at your system and determine how critical it is to have guaranteed timing that an event will happen within a specific time allotment; usually the answer is no. I heard a speaker from Space X who said they mostly just use Ubuntu and not any real-time system on their rockets. If you do need real-time Linux, there are several variants including Wind River (costs $$$) and RTEMS (FREE).

There are many people who come from the world of servers who want to setup virtual machines (VMs) and use the Linux VMs to run the robot. I generally don’t like this. I have had many reliability problems from VMs that are constantly being turned on and off. I have also had VMs randomly not recognize devices/ports that are then recognized with a reboot. You also take a large performance hit since there is a lot of overhead in running a VM. The notion that you can have a VM on the robot and then a VM on your desktop for development usually does not work since they do not stay in sync. If you are mass producing a system, then you should just create a master drive or image, and clone that to all of the other drives; a VM is not needed.

Here is a rough list of steps for how to convert your Linux system into an embedded system:

1. Choosing a filesystem. Select a volatile RAM drive for maximum reliability (and hardest development cycle). For a generic drive use journaling for reliability if the system crashes or shuts down uncleanly. In Linux this generally means EXT3 or EXT4.
2. Set the OS as read-only so you can handle system crashes. Learn more.
3. Set bootloader to not stop on failure. This is especially true for new versions of Ubuntu; if the computer does not shut down cleanly it will halt at the bootloader until you press a key. Learn More.
4. Set BIOS (I know this is not technically Linux) to autostart when power is applied (the default is to use the switch on the computer).
5. While I am in the BIOS I will often enable some of the options to speed up boot time, including reducing the length of the countdown before booting.
6. If large amounts of storage are needed, use an SSD for the OS and a spinning disk drive or second SSD for the data.
7. Change the default setting to not do a disk scan after X days or X boots. If your computer decides to do those scans you tend to stand around thinking your robot is broken, and then after it finally boots you wonder how it fixed itself.
8. Have a system watchdog. This will let you detect if your software crashes so you can restart the system. At minimum you can add a software watchdog. Even better is to purchase a computer with an integrated hardware watchdog.
9. Create a script in /etc/init.d/ for starting your process. Any paths you use should be absolute (and not relative) paths. The newer way to do this is with upstart, though I don’t have much experience with this yet.
10. Add symlinks to your /etc/init.d/ in /etc/rc*.d for the code you want to run at a given runlevel. I usually put a symlink into /etc/rc2.d/, /etc/rc3.d/, and /etc/rc5.d/. The other way to do this is to just edit the /etc/rc.local file.
11. Add /etc/udev entries for mapping devices (such as USB). This can sometimes be a tricky process that needs some experimentation with to get right. Learn More.
12. Add other drives to /etc/fstab so that the drives come up the same with proper mount points and permissions every time. One thing to watch out for is that if the drive is not available, fstab might panic and not boot fully. If you think your drive might not be available (or you just want to be cautious) you can set nofail as one of the options.
13. Disable all software updates (except maybe security updates). I have seen many times where a software update will break code that has been working for a long time.
14. Good tools to install are ssh (or openssh) server and rsync for managing and updating code.
15. Use GCC tool chains when possible. Do NOT mess with other tool chains unless you have a very good reason.
16. Use top or htop to monitor system usage after loading your code. I try to only use around 75% of CPU and memory to allow for mostly repeatable code timing.
17. If I am using a wired network (which I almost always do) I try to put my settings in /etc/networking/interfaces and not in the GUI network manager. The reason for this is that it is easier to remotely change the setting (I use SSH for everything) if you ever need to change them in the future. I rarely use wireless in an embedded application for reliability reasons. If I am using wireless I will often use the GUI out of laziness, but there are ways to still use config files for wireless (which is the better way).

There you have it – my list of things I do to stabilize and prepare a fresh Linux install to work in an embedded application.

What other steps do you take? Leave it in the comments below.

Robots for Roboticists David Kohanbash is a Robotics Engineer in Pittsburgh, PA in the United States. He loves building, playing and working with Robots.

Robots for Roboticists David Kohanbash is a Robotics Engineer in Pittsburgh, PA in the United States. He loves building, playing and working with Robots.