Harnessing the Power of a Software-Based Image Signal Processing Approach

Article co-authored by Miki Pe’er, InnoWave.

The rapid growth of artificial intelligence (AI) is making the role of camera systems increasingly important—particularly in applications involving image and video content detection and generation. The AI era has progressed alongside substantial hardware innovations in camera systems including sensor manufacturing, silicon technology, and processing cores.

All this has led to a limitless combination of sensors, processors, GPU capabilities, and platforms. It is impossible to structure a universal camera hardware solution capable of optimally integrating all the components and fitting all applications.

Off-the-shelf hardware designs are often either excessive or insufficient. Within a camera system framework, the ISP platform is one of the key components that can be substituted with a software-based version to provide flexibility and reduce complexity in the system design.

Choosing an ISP—A Key Camera System Choice

The image signal processing (ISP) unit ranks as an essential component of a camera pipeline, responsible for processing the video stream into a consumable output format. The ISP comprises sub-blocks to process the RAW format frame coming from the sensor.

Figure1. An ISP is among the critical design choices in any camera system design.

The order and selection of ISP blocks are closely tied to the sensor types, acquisition technologies, hardware/software constraints, and data transfer rates. For example, HDR sensors or RGB-IR sensors require specific processing blocks.

Additional factors that make ISP selection difficult include hardware compatibility, frame rate, power consumption, encoding/decoding formats, and cost. Once the ISP chip is chosen and fixed into the hardware design, no further features can be added or modified. These constraints highlight the advantages of a software-based ISP.

The Case for Using a Software-Based ISP 

In theory, any camera platform equipped with one or multiple processing units, storage, and an operating system can host a software ISP core for basic image processing and signal preparation. The ISP's performance would depend on both hardware power and the complexity of the ISP itself. While this solution may not be as optimized as a dedicated hardware ISP, it provides unique advantages, including flexibility, scalability, performance, cost, and others. Now we’ll review each of these advantages.

Customization and Flexibility

A software ISP structure is designed based on specific features and requirements. The user is able to fabricate the ISP blocks by adding, removing, or modifying the algorithms depending on the application or hardware constraints. This flexibility becomes particularly important for uncommon cases such as RGB-IR 4x4, customs CFAs or neural networks with minimally-processed RAW inputs.

A possible HW solution is to develop high-end and more expensive ISPs with a comprehensive collection of ISP blocks to cover a wide range of applications, knowing not all of the ISP blocks are used optimally by the end customer. Another positive aspect of the software ISP is the ease of sharing and evolving the algorithms compared to hardware bit-accurate models; considering none of ISP vendors are willing to share the codes behind their algorithms.

Hardware Scalability and Consistency

A hardware upgrade or modification affects the nature of output data; this is an important consideration if a trained-based AI model has been built using the collected data with the current platform. Even a single HW ISP provider may introduce fundamental modifications for their next-generation platforms.

For example, a single ISP vendor might completely redesign their new ISP pipeline compared to its predecessor versions. In contrast, a software ISP remains modifiable and evolves, offering full control over the output specifications.

Performance

One of the main advantages of a software-based ISP is its capability to receive software updates, leading to iterative performance improvement and additional features over time. It also helps with relaxing the timelines and future debugging that is not possible with a HW ISP.

Cost

A software ISP can save a significant percentage of project budget and engineering costs. Typically, the hardware ISP and its compatible platforms constitute a notable portion of a camera system package.

A Software-Based Modular Solution: INNOISP

An example of software-based ISP is INNOISP. It is part of the end-to-end camera system development solution offered by InnoWave. Currently, INNOISP is available in different versions considering the customer preference: CPU-based, GPU-based Light-weight GPU. 

One unique feature of INNOISP is its modular structure. The performance of the ISP, as well as the effects of adding or removing individual ISP blocks, can be easily simulated, and visualized using the software version. For example, Figure 2 shows a rendition of a scene processed with a fully integrated HDR ISP.

Figure 2. Top—fully integrated ISP block diagram for a linear sensor, fixed focus wide FoV lens. Bottom—rendered image. (Click on image to enlarge)

The same RAW file can be processed through another ISP version that is optimized for a linear sensor by removing the expensive LTG/Contrast ISP block:

1. Signal decoding and unpacking 
2. AWB/AE
3. Black level correction
4. Lens Shading Correction
5. Demosaic
6. Color Correction
7. Gamma and YUV conversion
8. YUV Denoise/sharpening
9. Output formatter

The proposed ISP block diagram and the rendered image are shown in Figure 3.

Figure 3. Top—ISP block diagram for a linear sensor, fixed focus wide FoV lens. Bottom—rendered image. (Click on image to enlarge)

Ultimately, the ISP blocks can be further reduced for a compact low power or high-frame rate design suitable for AI-based applications:

1. Signal decoding and unpacking 
2. AWB/AF
3. Black level correction
4. Demosaic
5. Gamma and YUV conversion
6. Output formatter

The selected ISP layout and sample processed image are illustrated in Figure 4.

Figure 4. Top—ISP block diagram for a compact design with hardware limitations. Bottom—rendered image. (Click on image to enlarge)

Enabling an Optimized ISP Architecture

INNOISP offers flexibility across multiple dimensions, making it adaptable to a wide range of platforms and use cases. Its design allows it to integrate into almost any platform without a rigid system prerequisite.

The configuration process involves key hardware factors such as MPU or GPU processing cores, ISP layout, pipeline's bitrate, and the desired image quality parameters. By balancing these elements, INNOISP delivers an optimized image signal processing architecture that meets given product requirements.

All images used courtesy of InnoWave Design.