Adding an ISP and Machine Learning Acceleration to the i.MX 8M Family

Cloud computing, edge computing, and vision-based systems have taken over more and more industrial and consumer systems alike. Many modern systems, furthermore, employ machine learning and artificial intelligence to enhance the user experience. This approach often requires embedded systems designers to utilize a variety of different SoCs and specialized processors to get all these tasks done. 

NXP, however, decided to take a different approach, and they have combined an image signal processor (ISP), a neural processing unit (NPU) for machine learning acceleration, and an MPU all in one device: the advanced i.MX 8M Plus applications processor.

 

Figure 1. i.MX 8M Plus applications processor.

 

Cloud Computing vs. Edge Computing

Some context is required to understand why the i.MX 8M Plus applications processor offers powerful solutions for edge computing.

Cloud computing refers to the on-demand availability of computer system resources outside of the bounds of the user’s network. These resources could be of any kind. However, they are typically used to perform demanding tasks and for data-storage.

Often, cloud-based data centers aren’t exclusive to a single user or organization. Instead, the resources are shared between all users. Usually, public cloud services utilize a pay-as-you-go model, which means that the cost can quickly increase if unnecessary or noisy data is uploaded for processing in the cloud.

That’s where edge computing can come in handy. Edge computing puts some resources between the users and the cloud within the bounds of their network. However, typically, not all calculations are performed at the edge. Instead, the resources at the edge are utilized to filter data and discard unwanted elements before they get uploaded to the cloud and produce unnecessary costs. Oftentimes machine learning and AI are utilized to automatically categorize the results and only upload relevant information to the cloud.

However, it’s also possible to omit the cloud entirely and run all the machine learning inference locally on an edge processor — for example, the i.MX 8M Plus applications processor, which comes with a few advantages.

 

Potential Benefits of Using Edge Computing over Cloud Computing

Edge computing (such as that provided by the i.MX 8M Plus) offers a host of possible advantages over cloud computing that are worth considering more closely.

Apart from the cost factor, the reduction of data that gets uploaded to the cloud will also reduce the overall network traffic, which can speed up other applications that rely on the network. This not only increases efficiency with the tasks the processor is specifically dedicated to, but it also frees up critical resources for other, ancillary activities and priorities.

Next, running the inference locally means that the application can continue to function even when the cloud service is offline or when the network connection gets disrupted. This might be an important factor, especially for safety-critical use cases, but also for other connected applications, like home-security devices.

Furthermore, this solution provides better latency and shorter response times when compared to the utilization of a cloud service. Low latency is important, for example, when performing factory floor visual inspection and other time-critical applications.

Finally, user-privacy is another factor to consider. Every external service is a potential security risk when dealing with confidential data. Therefore, keeping important information within the bounds of the user’s network can increase the security of the entire application. This is not only important for industrial users but is also critical for private users and customers alike, when, for example, using a voice assistant or when working with personal video and image files.

 

How Much Machine Learning Performance is Needed?

With cloud computing, machine learning performance is usually not an issue or a limiting factor. When performing these operations at the edge, however, the question becomes how much power is needed. A popular way to measure machine learning performance is TOPS, which is an acronym for trillion (tera) operations per second, and it, as the name implies, refers to the number of (usually 8-bit integer multiply or accumulation) operations per second. The overall system performance, however, will depend on many more factors. TOPS is, nevertheless, still often used for quickly comparing the performance of machine learning systems.

Full voice-recognition (not only keyword spotting) at the edge turns out to require a system performance of roughly one to two TOPS. The real requirement depends heavily on the employed algorithm and on whether it's vital to understand what the user says. Object detection at 60 frames per second, as another example, takes around two to three TOPS when using an algorithm such as Yolov3.

Because processing and machine learning at the edge becomes more and more relevant, NXP added a machine learning accelerator with a performance of around 2.3 TOPS to the i.MX 8M Plus applications processor, which makes it well-equipped for various industrial tasks and many other applications without the need to employ external cloud-based services.

 

Figure 2. The i.MX 8M Plus applications processor is well-equipped for various industrial tasks that utilize external cloud-based services.

 

The i.MX 8M Plus is Ready for Embedded Vision-Based Systems

So far, this article mentioned various examples of applications that require a camera to function. One of them was a vision-based system that discards faulty parts at the end of a production line after an automated optical inspection. All such systems require visual input, for example, images from a camera. ISP functionality exists in every camera-based system. Often, the image signal processor is present in an external device, like the camera itself, and hidden away from the user. Such ISPs usually take care of a few different image optimization and filtering tasks.

In many cases, one can get away without using a dedicated ISP, for example, when using a simple USB webcam. Then, the ISP is usually already built into the camera, which will perform all the necessary conversions without the user noticing.

Such ISPs, however, have their limitations, and they typically work well for resolutions of up to two Megapixels. Furthermore, the missing control over the ISP, the added complexity, and the added power consumption can be problematic in some cases.

When a higher image resolution than 2MP (1080p) is required, the embedded systems designer can utilize an external ISP, which would, in turn, increase the overall complexity and power consumption of the system. Alternatively, the designer could use an applications processor with a built-in ISP, such as the i.MX 8M Plus. This approach offers an optimized imaging solution, particularly at 2 Megapixel and higher resolutions, without increasing the complexity of the system.

 

Summary

Edge computing has a few attractive benefits over cloud computing, and it has the potential to change the way we'll work, live, and spend leisure time in the future. Some of the benefits include increased reliability, scalability, and security, and decreased latency.

With the i.MX 8M Plus applications processor, NXP has combined two factors that enable modern applications at the edge. This new MPU will encourage the development of numerous innovative industrial devices and consumer-level products. It includes an ISP for building modern vision-based systems that require high-resolution input data.

The image information from the ISP can be directly fed into the built-in NPU to enable high-speed applications that rely on image data while keeping the CPU free for other tasks. Moreover, the i.MX 8M Plus applications processor offers a few other features that are worth a look, for example, an Ethernet controller with Time-Sensitive Networking, a DRAM controller that supports ECC, and a variety of different cores for a range of applications.

With the i.MX 8M Plus applications processor, the possibilities for high power vision-based applications at the edge are near endless, whether it be for a smart home hub, intelligent building controls, or industrial applications.

Industry Articles are a form of content that allows industry partners to share useful news, messages, and technology with All About Circuits readers in a way editorial content is not well suited to. All Industry Articles are subject to strict editorial guidelines with the intention of offering readers useful news, technical expertise, or stories. The viewpoints and opinions expressed in Industry Articles are those of the partner and not necessarily those of All About Circuits or its writers.