ISSCC 2024: MediaTek Presents Neural Visual-Enhancement Engine for IoT

February 23, 2024 by Jake Hertz

We kick off our ISSCC coverage with MediaTek's neural visual-enhancement engine, a device that bests competitors with its energy and area efficiency—making it an appealing up-and-comer for smart devices.

This week, the IEEE is holding its 2024 International Solid-State Circuits Conference (ISSCC). Among the many research papers published for the conference, one that stood out was MediaTek’s paper titled “NVE: A 3nm 23.2TOPS/W 12b-Digital-CIM-Based Neural Engine for High-Resolution Visual-Quality Enhancement on Smart Devices.”  


MediaTek NVE

MediaTek says its NVE may extend battery life and enable more complex on-device AI processing. 

The paper outlines a new neural visual-enhancement engine (NVE) fabricated on 3 nm FinFET technology claimed to offer industry-leading area and power efficiency. 


Architectural Innovations in MediaTek's NVE

At the heart of MediaTek's new NVE are a number of architectural innovations that help boost performance and power efficiency.

One such innovation is the 18-row digital-compute-in-memory (DCIM) core, which boasts a four-cycle switch control mechanism. This core is pivotal for several reasons. Traditionally, high-resolution neural network inference, necessary for tasks like super-resolution (SR) and noise reduction (NR), has been a power-hungry affair due to computational complexity, low sparsity, and high precision requirements. 

By adopting an 18-row structure for weight storage, the need for frequent weight reloading—a significant power drain—is virtually eliminated. This design choice not only slashes power consumption but also curtails area usage, enhancing the overall efficiency of the neural engine.


Chip micrograph and summary

Chip micrograph and summary

The four-cycle switch control is another core component of the NVE's efficiency. Weight switching, an operation intrinsic to neural computations, typically incurs considerable energy expenditure. MediaTek extends the row switch cycle, thereby reducing energy consumption without ballooning the area overhead. This delicate balance between power and space efficiency is a hallmark of the NVE's design philosophy.


Optimizing the DCIM Core

The system also introduces adaptive data control and striping optimization. These mechanisms are designed to optimize the DCIM core, particularly in handling strided and transposed convolutions—operations central to many neural network models, including U-nets.


Adaptive data control for Stride2 Conv

Adaptive data control for Stride2 Conv. 

The adaptive data control mechanism intelligently manages data traversal, ensuring that only relevant pixels are processed for strided convolutions and effectively packing input channels for transposed convolutions. This optimization boosts utilization rates and minimizes idle time for the DCIM core, thereby enhancing overall efficiency.

Striping optimization works in tandem with the adaptive data control mechanism to refine data handling further. By dynamically adjusting data strip sizes and employing a compiler that optimizes the scheduling and memory allocation, this feature significantly reduces the need for external memory access. The result is a substantial decrease in DRAM usage, further contributing to the NVE's energy efficiency.


The NVE Shows Off High Compute Performance

From a raw compute perspective, MediaTek's NVE reaches a peak performance of 16.5 TOPS and a maximum operating frequency of 1,300 MHz. More important, however, is the area and energy efficiency of the system.


MediaTek reports its NVE outperforms competitors

MediaTek reports its NVE outperforms competitors in both energy efficiency and area efficiency.

With a footprint of just 1.328 mm by 1.032 mm, the NVE packs in 1,073 KB of on-chip memory and 6,336 at 12 b x 12 b of MAC, ultimately boasting an area efficiency of 12.0 TOPS/mm2. This marks a 4.51x improvement over the preceding technology. 

At the same time, the device’s energy efficiency reaches 23.2 TOPS/W when processing 4K video at 30 fps and 16.3 TOPS/W when processing 4K video at 60 fps. This marks a 7.84x improvement over the preceding technology. 


The Practical Implications of MediaTek's NVE

MediaTek's NVE, with its exceptional energy and area efficiency, is a major stride forward for smart devices. Its leap in performance translates to longer battery life and more complex on-device AI processing, allowing for real-time, high-resolution video enhancement for applications like mobile photography, video streaming, and virtual reality. Ultimately, this enables manufacturers to deliver advanced features without compromising device size or battery demands, potentially transforming user experiences with richer, cleaner visuals and paving the way for more immersive and responsive applications in consumer electronics. The paper also proves that there are still improvements to be achieved in edge computing, even at the 3 nm node.



All technical images used courtesy of MediaTek and ISSCC.