Software engineering principles

I have often thought about what the proper software methodology should be for the various robots that I build. My thoughts have evolved over time as I have seen these tool work. While I do not have any formal software engineering training, these are the common principles that I have seen, heard, read, etc. that I believe in (at 2am while I write this).

Disclaimer: Most of the suggestions here have come from other sources. Please check out the links attached for more information.

I will start with how to write code, since that is the basis of everything else. After that we will move into the software engineering infrastructure needed to make the software work properly.

General Programming Rules

Without having basic rules that you follow when coding, your code is doomed to fail and be unreliable. Everything else is built upon this. Many people start programming and want to use crazy template magic where everything inherits something else, however you must STOP and write simple code that is easy to understand and test.

• Keep control structures simple – Limit your branching, don’t have long nested if, case, and loop statements, no recursive magic.
• Endless loops – Do not use boundless loops unless you really want to be stuck there forever (such as the main loop). Everything else must have an upper/lower bound for exiting the loop. Think more about for loops than while loops.
• No dynamic memory – Forget that malloc exists. It can cause programs to act differently at each run time, and can also add memory leaks that can be hard to detect.
• asserts – Asserts can help you to enforce that the proper thing is helping in each function. I generally do not like the default assert functions that simply cause an exit; sometimes that is appropriate, but in many cases you should try recovering from the error instead of just exiting. You need to evaluate what is the safer approach in your application.
• Keep scope – Try to assign and initialize variables, structures, etc. in as small a scope as possible. This includes limiting your usage of global variables.
• Pointers – Limit your use of pointers when this is easily possible. You should really avoid multiple levels of pointers. They can be very confusing to follow and lead to all sorts of problems if you reference memory addresses that you should not be touching.
• Check return value from all (non void) functions – If a function can (and should be able to) return an error value, make sure to catch it and act on it. It can also be good to check the bounds of a value returned instead of blindly trusting it.
• Turn on all compiler warnings – So you catch things early. This is especially important when formatting text, working with memory, and when bit banging.
• Use parentheses to control order of operations – I have seen many people assume or forget about the proper order of operations. Using parentheses can clarify and enforce the desired result.
• Hard coded number – Please use variables for any value that might need to be changed by a programmer. If you have an even slightly non-obvious number, include a comment about the source of that number and how to derive it.
• Preprocesor – Limit the preprocessor to “simple” things, such as #define and #if. Using complex macros can lead to hard-to-understand code, poor portability, and a increased probability of bugs.
• Standards – Keep your code consistent. This includes naming, white space, and where your curly braces go. This makes it easier to follow the code. If you are editing existing code, use the format of the existing code.
• Sleep with noop’s or empty for loops – These should be avoided since they will run differently on different systems. This is primarily for embedded systems.
• Document your code – Need I say more?

Helpful links:
Rules for defensive C programming
The Power of Ten – Rules for Developing Safety Critical Code

Software Repositories

Having a code repo should be a given for every programmer and programming team. There are many free tools available, so there is no excuse to not use them.

Having a software repository lets you save your code at various points so that you can roll back to prior working versions. Repositories also let you log what was changed in each file and by whom. These tools are a great way to share code with a team – there is no need to email code back and forth. You can also put documentation in the repo so that is is always available.

There are many options to choose, from cvs, svn, mercurial, git, bitkeeper, etc. Just pick one!

Lately I have been using a hosted repo service called github.com – I like all of the tools that are packaged with git on the website. bitbucket.org is another similar option.

I generally like to create repositories with the following directories in the root directory:

• bin – A place to put all of the binaries after they are compiled. By default this directory should have nothing checked into the repository.
• config – If I have lots of config files I will put them all in this directory.
• doc – Documentation for the system (datasheets, diagrams, notes, etc..)
• externals – Source code and libraries from outside parties (ie. you did not write them)
• includes – Header files that other multiple programs want access to at compile time
• libs – A place to put all of the libraries after they are compiled. By default this directory should have nothing checked into the repository.
• src – This is where all of the code we write should live.
• tools – This is for random “tools” that do not fit anywhere else.

In addition to the directories above, I will usually put the main make file for the project and a README in the root directory.

Links:
Software repositories or just plain repo

Bug/Issue tracking

Having a central database to track bugs is very important for developing great code that runs bug free. This lets anyone who identifies a bug to report it in such a way that it does not get lost. Just telling the developer or emailing them is not sufficient. People forget things easily. Ideally, you do not want the developer to close a bug; you want somebody to report the bug, the developer to fix it, and then the original person who identified the bug verify that it was properly fixed.

A bug report needs three things:

1. What you expected to happen
2. What did happen
3. Steps to reproduce observed behavior (ie bug). Try to minimize the number of steps needed to reproduce error. Include versions of code that you are using.

Links:
Painless Bug Tracking

Automated Testing

Each of your processes/functions should have a set of tests (often called “unit tests”) performed on them to verify functionality. These tests make sure that functions work with various input extremes and combinations. Ideally you should have a test for each possible control path within a function.

For example if you are testing a “sum(number1, number2)” function, you can have the following test file:

int main () {
assert( 2 = sum(1,1));
assert( 0 = sum(-1,1));
assert( NULL = sum(1,NULL)); // or whatever you want it to be doing…
assert( 0 = sum(0,0));
assert( 1999998 = sum(999999,999999));
assert( 5 = sum(4.2,1)); // assuming integers and that floats are rounded.

RETURN (TRUE);
}

In this example assert will print an error message about where it failed and exit. (This is just my quick example, and can definitely be expanded and improved. It can also be changed to fail at compile time so that you do not need to “run” it.)

Automatic (Daily) build system

Having a build server that builds every time somebody commits code, or at minimum, every night, is valuable for identifying inadvertent bugs introduced into your system and automatically running your tests.

Often you will make a change to one module that will cause another module to fail. But as developers we get focused on our modules and tend to not try recompiling all of the code to make sure that we did not break anything. It is always better to find out you broke something earlier than later.

Certain things tend to cause other modules to break more than others. For example making a change to a widely used header file or message definition file can affect other items.

This also makes programmers only commit code that compiles to the repo and not broken code. If they commit broken code the build tools can/should send out emails to notify people that the build failed (and whose fault it was).

You can either just make a quick script to check out the repo, build it, parse the output and email the results; or you can get a proper build system such as Jenkins. Generally using somebody else’s product will be more reliable and a better use of your time.

Links:
Daily Builds Are Your Friend

Code Reviews

Code reviews are tough, but they are also one of the best ways to reduce the number of bugs in a piece of software. The traditional code review – where a person gets up and talks others in the room through the code (that the people have never seen before) – DOES NOT WORK! This might be good for getting people familiar with the program’s operation, but it is not the best way to do a code review and find problems.

To do a code review you should have the programmer send a review request to 1-3 other software engineers. The review request should have an overview of the code, a flowchart, and directions to access the code. The number of engineers and the experience level of those engineers should be based on how critical the code it. Each reviewer should:

1. Review the code (obviously) – Don’t dwell on style, you have better things to do with your time.
2. Develop a list of questions about the code
3. Write some tests and make sure they pass (these can be committed to always run in the nightly build).

Some of things reviewers should watch out for are when reviewing code before a large review:

• Does the code do what it is supposed to do (and not do what it is not supposed to do)?
• Should the code/function be written in a more efficient manner?
• Is the control flow simple/clean/good (if, else, return, etc..)?
• Do switch statements have a default catch all?
• Is the code commented adequately?
• Are variables declared at the proper scope (minimizing globals)?
• Are there memory leaks?
• What happens if things return NULL?
• Error/Exception handling
• Buffer overflows (doing raw things with char’s, int’s, etc.. memory locations)
• Pointer and pointer based math
• Array indexing
• Closing all handles (files, ports, etc..)
• Macro usage
• 64bit vs 32 bit modifications
• Multithreading (is it needed?, number of threads, race points, deadlocks, data passing, priority inversion, etc.)
• Operators (precedence and proper usage [ & vs &&, = vs == ])
• If input items from users and functions is checked before being used

(Did you notice how this list is similar to the good programming practices above?)

After doing the above steps it is now OK to hold the big meeting where people sit down and walk through and review the code; although this is not strictly necessary.

Reviewing code can be hard. Getting your programmers used to reading code and reviewing it will help strengthen their own coding. It can also be useful to have dedicated QA people who review code.

Links:
Embedded System Code Review Checklist
Microsoft: Best Practices: Code Reviews

Build System

You should be using a build tool so that you are not manually compiling each file. Probably the most common one is make.

Each make file should have at minimum the following directives:

• all – Build all of the production code
• clean – Deletes the binary and object files. This lets the next build be completely clean.
• install – Installs the binaries and libraries into the proper location. This can be a process such as:
  1. Copy old binary to <binaryName>_<archivedDateTime>
  2. Copy new binary to location with active processes
• test – Builds a set of unit/system tests that verify operation and make sure bugs were not introduced.

I believe you should be able to build, install, test, or clean your entire system with one command. As such I will normally create a make file for each process, and then create one master make file that calls all of the individual make files. Having a single make file that can build everything is good for installing a new system and makes setting up the nightly build easier.

Summary

• Write good code.
• Use a repository service (assuming it is not super sensitive data) such as github or bitbucket. That gives you the repo, the issue tracking, and the ability to easily do code reviews.
• Setup a server that builds your repository and runs your tests, at least once a day.

Do you have other favorite tools to make your software more reliable? Please leave comments below.

Main image based on sample code from wikipedia and hammer from pixabay.com/.

Robots for Roboticists David Kohanbash is a Robotics Engineer in Pittsburgh, PA in the United States. He loves building, playing and working with Robots.

Robots for Roboticists David Kohanbash is a Robotics Engineer in Pittsburgh, PA in the United States. He loves building, playing and working with Robots.