Beyond 4K Ultra HD: What’s Next for Advancing Display Technology?

The iPhone 8 is expected to be launched in September this year, and rumors suggest it will have a huge 5.8-inch OLED screen. Other rumors say problems with the screen could delay the launch. Delivering screen innovations that offer noticeably better experiences for end users is an increasingly difficult challenge for manufacturers of all types of devices, from smartwatches and smartphones to PCs, televisions and home cinema.

Increasing pixel density allows crisper images and graphics, with smoother, sharper edges, while more bits per pixel allow greater color depth to help make images more vivid and vibrant. For moving pictures, such as television, video or gaming, higher frame rates provide the key to presenting smoother, more fluid motion.

Silicon Can Stand the Pace

As far as the image or graphics processing capability to handle more and deeper pixels, at ever-increasing frame rates, Moore’s Law advances in chip performance enable successive generations of silicon to support demand for progress: rising from High Definition (HD) just a few short years ago, to the 4K UltraHD (UHD) equipment on offer for today’s living rooms, to forthcoming 8K standards.

One example can be seen in the emergence of H.265 (High-Efficiency Video Coding, or HEVC) codecs. H.265 achieves about double the compression of the preceding H.264 AVC standard, which is critical for streaming content such as 4K video at practicable bit rates over the Internet, or storing 4K movies within the capacity limit of a Blu-ray disc. While H.265 dramatically reduces the bandwidth needed to ensure high picture quality when streaming 4K content, operations such as coding, prediction and transforms are significantly more complex than for H.264 and demand much greater processor performance.

Can Viewers See the Difference?

Display technology also continues to advance in the quest to deliver more impressive experiences for end users. When Apple introduced the Retina display to its iPhones, then-CEO Steve Jobs claimed the pixel density was high enough that the human eye could not perceive individual pixels when viewing the screen from a comfortable distance of about 12 inches. Testers did indeed notice that images were crisper, and that text was sharper with more rounded edges. Since then, Apple has launched upgraded Retina displays with even greater pixel density, using terms like RetinaHD and Retina4K to differentiate them from predecessors.

Jobs’ proviso expresses the major issue for consumer electronics brands seeking to deliver better viewing experiences by increasing pixel density: the end user’s perception of image quality is heavily influenced by screen size and viewing distance. As figure 1 illustrates, the improvements possible with 8K video, for example, really come into their own when viewing from about five feet, on a screen size of about 90 inches. This effectively narrows the market for 8K consumer equipment to those homeowners prepared to give pride of place to a 90-inch screen, and who have a room large enough to be able to sit comfortably five feet or more away to view it.

 

Figure 1: Screen size and viewing distance at which increases in resolution can be perceived. (Chart printed with permission by Carlton Bale from “Does Resolution Matter?”)

 

Then, there is the question of the availability of content in the latest formats. Early buyers of HD-ready televisions would have been able to see the benefits of their investment if watching a Blu-ray disc but had to wait for HD satellite or terrestrial broadcasting – or HD set-top boxes – to arrive before they could gain the full benefits during everyday viewing. Today, a similar situation exists with 4K UHD. 4K and 4K-ready televisions are available, and prices are becoming more budget-friendly, but few channels offer a great deal of 4K content.

8K could be even slower to arrive. Today, content is often created in UHD to allow post-production to zoom, crop and digitally correct, and still ensure the visual effects and overall imagery look good in more easily consumable formats like high-definition 2K or 1080p. 8K has been rare, used only in instances where heavy post-production is required when digitizing high-quality film prints: the 2012 scan of Lawrence of Arabia is a famous example. Guardians of the Galaxy Vol. 2 completed production in 8K in June 2016, making it the first feature created in the emerging format. However, a film in 8K is unlikely to ship to theaters in that format. From the point of view of practicality, mentioned earlier, consumer demand for 8K content – such as via streaming services – is subject to doubt.

Speeding up frame rates is another area where manufacturers push boundaries and standards. Film has largely maintained a standard of 24 frames per second (fps), but directors have experimented with higher. Peter Jackson famously shot his Hobbit films at 48 fps, which reduced motion blur to the extent that some criticized the effect: apparently, movie aficionados expect to see some motion blur in their films.

Gamers: The Undisputed Winners

Video gaming, on the other hand, can deliver vastly better user experiences thanks to higher frame rates. High frame rates give a smoother feel as gamers respond to visuals on screen. For consoles, 60 fps is considered optimal, while PC games allow adjustment based on hardware performance. Frame rates of 90 fps are used by leading virtual reality sets, such as Oculus Rift and HTC Vive. Higher frame rates are needed here, because the virtual world strapped to the player’s face is perceived as real and therefore the brain expects a refresh rate that responds to movement in real time. Inconsistencies with the frame rate are thought to be behind “virtual reality sickness,” although the science is unresolved.

Faster processing means potentially faster render times and frame rates, but there’s a limit to how fast humans can perceive frame rates. One could compare perception of frame rates with game performance; the faster the figures on the screen are moving, the faster the image needs changing. In 2006, a study by Claypool, Claypool and Damaa tested players’ performances in a first-person shooter at different frame rates, and suggested that performance increases little between 30 and 60 fps. Gamers may disagree with the spirit of the study, on the basis that the experience is about “feel” as well as performance. In any case, modern games with more complicated graphics may affect the results of a study today. 

Although 60 fps seems like a good baseline for player performance and feel, the ultimate limitation is the speed at which human eyes can perceive a frame rate. Although each individual person is different, it seems that casual gamers typically experience diminishing returns above 60 fps.

Finally, the effects of increasing screen refresh rate have an important effect on perceived video quality or gaming experiences. The refresh rate – as distinct from the source video frame rate – describes the speed at which the screen can redraw the displayed image. To manage the difference between the screen refresh rate and the frame rate of the incoming source, which may be, say, 24 fps for video or 60 fps from a game console, a television will insert intermediate frames. These may be repeat frames, black frames or interpolated frames that are constructed by calculating intermediate content based on that of preceding and following frames.

A higher refresh rate provides more scope for clever use of frame insertion to present sharp, crisp images and reduce motion blur. Some of today’s fastest screens can refresh at up to 120 Hz or 165 Hz, or faster in the case of some 1080p monitors. Here, too, however, product manufacturers could be chasing diminishing returns. Over-aggressive interpolation is known to result in unnaturally smooth motion when showing a movie. This is known as the “soap opera effect” and, like The Hobbit’s super-fast frame rate, has drawn criticism from movie buffs who feel the authentic movie-viewing experience has been lost. On the other hand, today’s selection of display technology can support incredibly smooth gameplay.

The next challenge is to make monitors that refresh at high speeds but also at higher resolution. 4K screens require much more processing power and faster graphics cards to refresh a greater number of pixels while synchronizing the incoming frame rate with the monitor refresh rate. Some of today’s 4K televisions claim extremely high refresh rates up to 240 Hz or 480 Hz. These frame rates, however, are not necessarily comparable with the native frame rates quoted for televisions of old, but also express the effects of processing techniques designed to show fast action clearly on UHD screens. Advances in software and interface hardware are also needed to achieve high refresh rates with UHD screens and beyond. DisplayPort, one of the fastest audio/visual interfaces in use today, has demonstrated refresh rates up to 30 Hz at best, with screens of 8K resolution.

Every New Beginning

Home video equipment could be nearing the limit of meaningful performance advancement, with the maturing of 4K UHD and the expected arrival of 8K products. Consumers may simply be unable to perceive a worthwhile improvement in image quality or viewing experience. Gaming, and virtual reality in particular, could be the exception, and could continue to demand increased frame rates at high screen resolutions in the future.

Technological progress is difficult to halt, however, and if there is little demand for extra raw screen performance, major brands can be expected to drive progress in other technologies to deliver new and compelling experiences. These could be based on combining television and Internet content on screen, such as simultaneous display of movie action with IMDB data.

The key to developing consumer markets, as ever, is to deliver previously unimagined innovations that quickly become indispensable. 8K viewing could be the end of the road as far as ever-increasing resolution is concerned, but the starting point for something even more exciting.

Industry Articles are a form of content that allows industry partners to share useful news, messages, and technology with All About Circuits readers in a way editorial content is not well suited to. All Industry Articles are subject to strict editorial guidelines with the intention of offering readers useful news, technical expertise, or stories. The viewpoints and opinions expressed in Industry Articles are those of the partner and not necessarily those of All About Circuits or its writers.