On-Device People and Object Detection Is on the Rise—But at the Cost of Power

June 10, 2020 by Gary Elinoff

Renesas poses a solution: a low power-consuming image signal processor and a vision-optimized AI accelerator for AI recognition accuracy.

It's only been a month since Sony announced the first "world's first intelligent vision sensors with AI processing," geared for AI-enabled cameras. Sony said that the move to include AI processing on the image sensor itself would speed up edge AI processing and extract only the essential data. This, in turn, would cut data transmission latency, protect user privacy, and slash power consumption. 


General structure of Sony's intelligent image sensor

General structure of Sony's intelligent image sensor. Screenshot used courtesy of Sony

Consumers, like those for Sony cameras, are increasingly demanding AI-based object and person recognition with applications ranging from robotics to construction security. In fact, Forbes contributor Kathleen Walch posits that such technology is transforming the construction industry. "With facial recognition and object recognition technology, AI systems are capable of detecting unsafe behavior and alerting the construction team of potential hazards," she explains. 

But despite this demand for vision AI processing, designers are still faced with a major challenge in producing this technology: its high power requirements.

One new device from Renesas illustrates how suppliers might address the common trade-off of power efficiency when it comes to real-time AI recognition in surveillance cameras, product scanners, POS terminal cameras, and similar technologies.


Taking a Note from a New ASSP for Vision AI

The new RZ/V series from Renesas may answer the challenge of power consumption by providing the processing power needed to instantly recognize both people and objects. The first member of the series, the RZ/V2M, is available in a 15 mm x 15 mm package and consumes as little as 4 W, eliminating the need for a cooling fan and heat sinks.


RZ/V series

The RZ/V series from Renesas. Image used courtesy of Renesas


The RZ/V2M is based on Renesas’ dynamically reconfigurable processor (DRP) technology, which is said to significantly accelerate image processing algorithms—approximately up to ten times faster. The on-board image signal processor (ISP) can handle 4K images at 30 fps. High dynamic range (HDR) and distortion correction functionality serve to improve AI effectiveness. 

Renesas' Hiroto Nitta (senior VP of SoC business and the IoT and Infrastructure business unit) explains, “The new RZ/V Series delivers both high performance and low power consumption, the two key issues that are keeping Vision AI processing from achieving a higher level of real-time performance."


Processing Power and Flexibility

The DRP-AI vision-optimized AI accelerator is based on an earlier DRP originally employed in a predecessor device, the RZ/A2M, which was designed for tasks such as reading 2D barcodes and iris recognition. 


DRP diagram

Diagram of Renesas’ dynamically reconfigurable processor (DRP) technology. Image used courtesy of Renesas


The DRP functionality of the RZ/V2M is combined with a multiply and accumulate (MAC) circuit, rendering a new IP core. This allows it to increase its AI processing capability by approximately a factor of ten, putting it in the 1 terra ops (TOPS) per watt class. 

Additionally, the DRP can change the configuration of its circuitry every clock cycle, giving the DRP-AI the flexibility to support constantly improving AI algorithms.


Key Specifications of the RZ/V2M

With no heat sinks or cooling fans needed, designers can expect equipment based on the RZ/V2M to be miniaturized and a reduced BoM cost.


Block diagram of RZ/V2M

Block diagram of RZ/V2M. Image used courtesy of Renesas


The unit features two Arm Cortex-A53 cores that operate at up to 1.0 GHz. DDR memory interface is 32-bit LPDDR4-3200.

Multi-stream ISP is available, and camera interfaces include two SLVS-EC and two MIPI CSI.

High-speed interfaces include:

  • Gigabit Ethernet
  • USB3.1 Gen1 host/peripheral 
  • PCIe gen 2 (1-lane)
  • NAND flash interface ONFI1.0 
  • eMMC 4.5.1
  • 2x SDIO 3.0


AI Processing at the Edge

One of the clearest directions in AI technology is the effort to move decision-making to the edge. That can mean, for example, a remote IoT node, where power availability is severely constrained. Even if power availability isn’t a critical issue, controlling the heat generation can be a problem for small embedded devices.

Renesas feels its RZ/V2M, with its high functionality and low power consumption, rises to the challenge of delivering real-time, AI-based person and object recognition.

Nitta asserts that the new series will “dramatically expand the range of AI applications in embedded devices through object recognition, for example, cameras in smart shopping cart systems that automatically calculate totals based on the cart items, robots in factories that can safely work together with humans, and medical cameras that assist doctors in making diagnoses.”



Have you worked with edge device designs requiring object recognition capabilities? What obstacles did you encounter? Share your experiences in the comments below.