What is an Image Processor? Turns Out the Answer is Hazy

Embedded processors can be divided into two general categories: microcontrollers and digital signal processors (DSPs). Though the line of demarcation between microcontrollers versus DSPs is increasingly blurry, it’s usually not too difficult to place a part in one bin or the other.

A DSP, unlike a microcontroller, is optimized for the high-speed mathematical and data-transfer operations involved in applications such as real-time audio processing, software-defined radio, and artificial intelligence (AI). This article will explore processing ICs called image signal processors (ISPs). Note that it would be reasonable to include ISPs as a subgroup within the DSP category; however, as real-time image processing applications proliferate and as ISPs become increasingly powerful and versatile, I think it makes sense to give them their own category.

Overall, this article aims to showcase some examples of image, image signal, and media processors to understand the nuances between company terminology and to see how hazy the line is between those different types of products.

 

This Magic 8 Ball's answer might be right when it comes to understanding the naming of these types of processors. Hopefully, this article can help clear up some of that haze. Image [modified] used courtesy of Magic Management

 

The Need for Specialized Image Processing Hardware

General-purpose DSPs, such as the C66x devices from Texas Instruments or the Blackfin processors from Analog Devices, are two example devices capable of processing image data. However, it’s important to recognize that imaging systems place special demands on processing hardware, and ISPs will often get the job done more efficiently than general-purpose devices.

Perhaps the most important consideration is the sheer quantity of data. If we sample audio at 44.1 kHz with 16-bit resolution, a real-time DSP application will be dealing with 88,200 bytes of audio data per second. A 24-megapixel DSLR camera shooting continuously at one frame per second is operating in an entirely different realm; even if we assume one byte per pixel, we’re talking over 24 million bytes per second. Some image-processing algorithms are quite complex, and when this much data is involved, dedicated hardware modules are preferable to customized code routines executed by an overburdened central processing unit (CPU).

Furthermore, this example demonstrates why ISPs may need to be implemented with parallel processing capabilities or as a multi-core system on a chip. This is when the camera’s computational electronics must be able to support other functions—such as focus tracking or automatic exposure—while all this image data is streaming in. 

 

What is an Image Signal Processor?

That question is not as straightforward as I would like. First, let’s consider the LM98519, an “imaging signal processor” from Texas Instruments. Below in Figure 1, we can see a black diagram of this device.

 

Figure 1. A block diagram is taken from the LM98519 datasheet. Image used courtesy of Texas Instruments [click image to enlarge]

 

In this example product, this ISP is just a data converter for charge-coupled device (CCD)-based imagers. Basically, it samples and digitizes two multiplexed CCD output waveforms with the help of white-level and black-level feedback systems. However, I found the term imaging signal processor is used also to designate Intel hardware designed for complementary metal-oxide semiconductor (CMOS) sensors and an IP core for high-dynamic-range video processing. 

Here, in Figure 2, is the block diagram for another part identified as an "image signal processor" (so not quite an imaging signal processor).

 

Figure 2. The diagram is taken from the datasheet for Arm’s Mali-C55 ISP. Image used courtesy of Arm [click image to enlarge]

 

Despite almost identical terminology, this part is clearly in a different category and has far more extensive functionality. This Arm ISP performs a variety of complex mathematical operations on image data:

• Defect correction
• Noise reduction
• Demosaicing to extract RGB values from bayer-filtered pixels
• Color space conversion and more

The DaVinci processors from Texas Instruments, shown in Figure 3) are even more highly integrated than the Arm ISP:

 

Figure 3. Block diagram from the DMVA3/DMVA4 datasheet. Image used courtesy of Texas Instruments [click image to enlarge]

 

Here you see hardware for image-processing operations combined with a microprocessor and a generous selection of digital peripherals. TI calls this device a “digital media processor,” though it could be described as an image signal processor. Actually, TI uses the term image signal processor for one of the sections in DaVinci’s “imaging subsystem.”

Finally, Socionext uses both the terms image signal processor and imaging processor for its Milbeaut product line. The Milbeaut devices are multi-processor, system-on-chip (SoC) solutions for photo and video applications, and they have the distinction of serving as the basis for the extraordinarily powerful EXPEED processors found in Nikon’s DSLR and mirrorless cameras. Milbeaut's capabilities include:

• Noise reduction
• Haze removal
• Electronic image stabilization
• Face detection (which is important for advanced autofocus)
• Lens distortion correction

 

Image Processor Structure, Function, and Terminology

This review of some relevant image-processing products has, I hope, accomplished two things:

1. Conveyed a general idea of image signal processor architectures and functionalities.
2. Demonstrated that an image/imaging signal processor can apply to vastly different parts, and consequently, you have to be careful with terminology when you’re working in this field.

If someone asked me, I would suggest the following terminology conventions:

• An image signal processor denotes a device comparable to the LM98519 (since the focus is the signal coming from the image sensor).
• An image processor denotes a device comparable to the Arm Mali-C55 (since the focus is processing the entire image).
• A media processor is reserved for devices whose capabilities are significantly more sophisticated or extensive than those of an image processor.

 

Image Signal Processor Recap and Additional Resources

Image signal processors are a diverse category of digital or mixed-signal ICs that specialize in analyzing and modifying visual data. They are optimized for the many resource-intensive processing tasks involved in still-image and video applications, and they must achieve extremely high levels of sophistication and integration in order to meet the demands of cutting-edge imaging systems. ISPs enable and enhance many automatic and user-friendly functions that have revolutionized photography and videography; these include automatic exposure control, automatic focus, automatic white balance, object detection, image stabilization, and noise reduction.

If you’d like to learn more about image processing, here are some suggestions for further reading:

• Understanding Contrast, Histograms, and Standard Deviation in Digital Imagery
• Digital Image Processing: Point Operations to Adjust Brightness and Contrast
• Pixel Intensity Histogram Characteristics: Basics of Image Processing and Machine Vision
• Understanding Dynamic Range in Imaging Systems
• Image Arithmetic in DSP: Image Averaging and Image Subtraction

 

---

 

What are your thoughts on image signal processors? Do you have a different way of unpacking the different terminology conventions? Let us know in the comments down below.