A Beginners Guide to Debugging Embedded Systems: 3 Steps to Find Bugs Quickly

Regardless of how strange your embedded C program is acting, it is technically only following orders. In this article, we'll explore a simple 3-step method called DDT (design documentation and testing) to debug embedded systems easily. Before diving into that method, let's first take a look at the general challenges facing debugging embedded code. 

 

Debugging Embedded Code Challenges

As the old saying goes, 90% of the time spent on programming is in the last 10% of the job—the dreaded debugging phase. Still, despite all the testing and debugging efforts, some big embedded systems never become completely bug-free.

In general, there are four main reasons it is so hard to debug embedded systems:

 

1. Lack Design Methodologies Training

Debugging embedded code can be a real headache, especially when you're not trained in design methodologies. Back in the day, hardware designers were the one's programming embedded systems, but they often lacked knowledge of essential design techniques, data structures, and structured programming. 

This trend continues today, with many electronic engineers handling both hardware and software development of embedded systems without getting any formal training in the design methods computer scientists swear by.

This lack of design knowledge can cause a whole bunch of problems, such as unmaintainable code that's about as organized as a pile of spaghetti (Figure 1). 

 

Figure 1. A dramatization of spaghetti on a keyboard because of a term called "spaghetti code," which has been used to describe code that is so tangled that it's nearly impossible to change or add one thing without breaking something in the code. Image courtesy of Canva Pro

 

To make matters worse, some embedded programmers often see design methodologies as unnecessary documentation procedures or crutches for beginners, which only adds fuel to the fire.

However, here's the thing; bugs in embedded code don't just pop up out of nowhere—they're created by the programmer. So, as an embedded programmer, it's crucial to learn design methodologies that help you write clean, organized, and thought-out code. 

Incorporating good design practices into your development process can boost the reliability and maintainability of your embedded systems, so it's worth taking the time to do it right.

 

"If you don’t have the time to do it right, do you have time to do it over?"–Jean-Dominique Warnier 

 

2. Pressure to Meet Product Time to Market Demands

In the book, "The Art of Programming Embedded Systems," the author reported that going over the R&D budget by 100% usually only affects the company's bottom line by 3-5% in the long run, whereas being just six months late on product delivery can result in a 30% drop in market share. It is no wonder that when building embedded devices, there is a rush to start writing their code immediately. 

Keep in mind, however, that speed can kill—literally and figuratively. Rushing in to write code without first thoroughly understanding what you have to build is one of the biggest programming mistakes you can make, regardless of the pressure. A better way to minimize time to market while still delivering a solid product is to do it in incremental releases. Start with a subset of your embedded system project that focuses on the most important functions and design it so that adding the rest of the features won't require a complete rewrite. Many embedded systems have a bunch of features, but not all are essential. It might be better to defer some of the nonessentials to version 1.1 and focus on releasing a robust yet limited version 1.0 instead.

 

3. It is Easy for Shoddy Workmanship to go Unnoticed

As embedded system programmers, we don't get judged on how pretty our work is like an artist or sculptor would. Instead, it's all about how well it does its job. The problem is that shoddy workmanship can function just as well as meticulously designed and implemented embedded code. 

When only one or two people write the whole project, it's up to us to write maintainable, well-structured code with professional honesty. Unfortunately, nobody will pat us on the back for that, and we won't get in trouble for writing spaghetti code that "works."

However, just like a poorly-built house will eventually crack, lousy code will show its deficiencies sooner or later. Here are some signs of shoddy quality when writing embedded code:

• Fixing a bug takes forever
• Adding a new feature is a total nightmare
• Changing one routine often causes a dozen unrelated bugs to crop up

 

4. Debugging Embedded Systems Can Be Tedious

Debugging is anything but easy. It takes time to find and fix a bug, especially when working with high-level languages like C. If you need to fix something, you may have to leave your debugging session, edit your code, compile it, and then download it all over again.

Debugging tools (Figure 2) can also be costly and complex, like high-speed oscilloscopes and fast emulators for 32- and 64-bit processors. 

 

Figure 2. An external debugger software inspecting an embedded system’s execution. Image used courtesy of Bart Vermeulen and NXP [click image to enlarge]

 

Also, even though simulators are useful for debugging algorithms, they're not ideal for handling real-time problems like interrupts and direct memory access (DMA), which are difficult to simulate. Not to mention, debugging embedded systems happens at the end of the project when everyone is already stressed out. There is high pressure to finish on time, and frustrations are also high. The timing itself is not friendly, making it even more challenging.

 

Introducing the Design Documentation Testing (DDT) Method

Even the best-planned embedded system projects that stay on schedule throughout development can often hit a roadblock in the final phase. Issues will arise that may be worse than expected. Tools may malfunction, prototype hardware may become unstable, and other unforeseen problems will crop up. It's important to accept these challenges and make provisions for them in your plan.

One way to do this is to use the DDT (design, documentation, testing) method. To be clear, I didn't invent this method. Credit goes to Gary McGath for introducing it in an article called "Programming Etymology." The DDT method can help you avoid writing code that's difficult to debug. Albeit debugging can't be completely avoided, this method can make it less time-consuming and less frustrating.

 

Step I: Design Embedded System Before Writing Code

The best way to stay bug-free is to write programs without bugs. The key to writing bug-free programs for your embedded systems is to design them thoroughly before starting to write code. 

First, tone up your ability to listen. Listen carefully to the customer and the specification committee. What is the system supposed to do? Who will be using it? From this, derive what you want the program to do and what data structures will be needed. From there, break the program down into a few large steps, and decide what each step will consist of in more specific terms. You can use Warnier-Orr diagrams to represent the program's structure, which will help you stay focused on the overall plan. 

The creative part of programming embedded systems lies in the design process, where you get to invent algorithms and structure the program. Writing the actual code is easy once you have a well-thought-out design. In fact, you might grow to enjoy the design process more than writing the actual code in C.

 

Step II: Documentation—Comments, Naming, and Writing Code 

Let's be honest, many embedded programmers don't like to comment on their code because they think it's a distraction that wastes time. It is wrong to think that way. Comments are the second line of defense against hard-to-debug code. They help make sure you understand your own code, and they make future maintenance a breeze. Thus, documentation shouldn’t be an afterthought. It should start with the program design—when you write what the code is going to do—and should continue with comments that explain the program's operation. However, don't just add comments for the sake of it. Good documentation is about explaining the operation of your code.

Make sure every function starts with a block that clearly describes what it does and lists its input and output parameters. Use descriptive names for functions and variables. The function name get _timer_count  is better than getdata. Also, follow a systematic naming convention.

And beware of incorrect comments. They're worse than no comments at all. So, make sure your comments are accurate and maintain their accuracy throughout the program's life cycle.  All these practices will make debugging and updating your code easier. 

As embedded systems become more complex, writing improperly documented code won’t pass. But, perhaps, writing is becoming as important a skill as coding is for embedded programmers. If you're not a fan of commenting, try to improve your writing skills. Take writing courses, practice writing, and contribute to technical writing communities like AAC.

 

Step III: Testing Embedded System for Bugs

To be sure that your testing is effective, you should make sure that your test cases are comprehensive. That means testing not only the "happy path," but also the boundary cases, edge cases, and error scenarios. 

Begin testing with the simplest version of the program after writing it with care. If the program works the first time, try it again with different data to verify its robustness. If the program doesn’t work the first time, try it again with different data using the simplest case possible. If you can't find a bug in a few minutes, use snapshot dumps or other tools to gather more information about what's going on.

Make sure that the structure of your program is consistent with the debugging tools you have (Figure 3).

 

Figure 3. Debugger tools to control and analyze the behavior of an embedded system. Image used courtesy of Bart Vermeulen and NXP

 

For instance, avoid using many interrupts if you don’t have the right tools to test real-time scenarios. 

 

Save Time and Money Using the DDT Method 

Taking a disciplined approach to embedded system development can save time and money in the long run. The DDT method, which emphasizes design, documentation, and testing, is one method that can help prevent errors and streamline the debugging process.

Starting with a well-defined project is crucial, even though it may seem pragmatic to start coding right away. A well-designed system is easier to make work and modify in the future. Documentation is essential, and accurate comments are necessary to ensure that the function's purpose and operation can be easily understood by you and your successors. Finally, when testing and debugging, you need the right tools. This cannot be better explained than when Jack Ganssle said, "Without the right tools, you'll be like a blind man hunting for a contact lens."