Project Concept

The 3D4YOU concept is to establish practical 3D television. The key success factor is 3D content. The project will define a 3D delivery format and a content creation process. Establishing practical 3D television will then be demonstrated by embedding this content creation process into a 3DTV production and delivery chain, including capture, image processing, delivery and then display in the home. The project will adapt and improve on these elements of the chain so that every part integrates into a coherent interoperable delivery system. The project's objective is to provide a 3D content format that is independent of display technology, and backward compatible with 2D broadcasting. 3D images will be commonplace in mass communication in the near future. Also, several major consumer electronics companies have made demonstrations of 3D television displays that could be on the market within two years. The public's potential interest in 3DTV can been seen for the success of 3D movies in recent years. 3D imaging is already present in many graphics applications (architecture, mechanical design, games, cartoons and special effects for TV and movie production). For those applications visualisation remains 2D. In the recent period multi view display technologies have appeared that improve dramatically the immersive experience of 3D imaging which leads to the vision that 3DTV or similar services might become a reality in the near future. In the U.S., the number of 3D enabled digital cinemas is rapidly growing. In mid 2006, about 400 theatres have been equipped with 2K digital projectors with the number increasing every month. Also in Europe, the number of 3D theatres is growing. In Germany, for example, two cinemas (Munich and Nuremberg) are using the REAL-D system for the presentation of current 3D Hollywood productions such as "Chicken little", "Nightmare Before Christmas" and "Meet the Robinsons". Two additional digital 3D cinemas opened up at the end of March 07 in Weimar and Dresden (CINEMAGNUM 3D), showing also 3D Hollywood productions as well as theme park 3D films. Availability of content drives this process: Chicken Little from Disney, with its experimental release of the film in digital 3D, increased in the number of projectors using the 2K format. Several digital 3D films will surface in 2007 and several prominent filmmakers have committed to making their next productions in stereo 3D. The movie industry creates a platform for 3D movies, but there is no established solution to bring these movies to domestic market. Therefore the next challenge is to bring these 3D productions to the living room. 2D to 3D conversion and a flexible 3D format are an important strategic area. It has been recognized that multi view video is a key technology that serves a wide variety of applications, including free-viewpoint and 3D video applications for the home entertainment and surveillance business fields. Multi view video coding and transmission systems are most likely to form the basis for next generation TV broadcasting applications and facilities. Multi view video will greatly improve the efficiency of current video coding solutions performing simulcast of independent views. This project builds on the wealth of experience of the major players in European 3DTV and intends to bring the date of the start of 3D broadcasting a step closer by combining their expertise to define a 3D delivery format and a content creation process.

Project Objectives

The key technical problems that currently hamper the introduction of 3D television to the mass market are:

1. It is difficult to capture 3D video directly using the current camera technology. At least two cameras need to operate simultaneously with an adjustable but known geometry. The offset of stereo cameras needs to be adjustable to capture depth both close by and far away.
2. Stereo video (acquired with two-cameras) is currently not sufficient s input for glasses-free, multi view autostereoscopic displays. The required processing such as disparity estimation is noise sensitive resulting in low 3D picture quality.
3. 3D post-production methods and 3D video standards are largely absenter immature. The 3D4YOU project will tackle these three problems. For instance, a clever combination of two or three high-resolution video cameras with one or two low-resolution depth range sensors may make it possible to create 3D video of good quality without the need for an excessive investment in equipment. This is in contrast to installing 100 cameras for acquisition where the expense may hamper the introduction of such a system.

Developing tools for conversion of 3D formats will stimulate content creation companies to produce 3D video content at acceptable cost. The cost at which 3D video should be produced for commercial operation is not yet known. However, currently, 3D video production requires almost per frame user-interaction in the video which is certainly unacceptable. This immediately indicates the issue that need to be solved: Currently fully automated generation of high 3D video is difficult. In the future it needs to be fully or at least to some extent automatic with an acceptable minimum of manual supervising during post-production in order to fulfill the expected demands for video content. 3D4YOU will research how to convert 3D content into a 3D broadcasting format and prove the viability of the format in production and over broadcast chains. Once 3D video production becomes commercially attractive because acquisition techniques and standards are there, then this will impact the activities of content producers, broadcasters and telecom companies. As a result we may see that these companies may adopt new techniques for video production just because the output needs to be in 3D. Also, new companies could be founded that focus on acquiring 3D video and preparing it for post-production. Here there is room for differentiation since, for instance, the acquisition of a sport event will require large baselines between cameras and real time transmission whereas the shooting of narrative stories will require both small and large baselines and allows some manual post-production for achieving optimal quality. These activities will require new equipment (or a clever combination of existing equipment) and new expertise. 3D4YOU will develop practical multi view and depth capture techniques. Currently, the stereo video format is the defacto 3D standard that is used by the cinemas. Stereo acquisition may for this reason become widespread as an acquisition technique. Cinemas operate with glasses based systems and can therefore use a theatre specific stereo format. This is not the case for the glasses-free autostereoscopic 3DTV that we foresee for the home. To allow glasses-free viewing with multiple people at home, a wide baseline is needed to cover the total range of viewing angles. The current stereo video that is intended for the cinema will need considerable post-production to be suitable for viewing on a multi view autostereoscopic display. Producing visual content will therefore become more complex and may provide new opportunities for companies currently active in (3D) movie post-production. According to the NEM Strategic Research Agenda (www.nem-initiative.org/Documents/NEM-SRA-040.pdf) multi view coding will form the basis for next generation TV broadcast applications. Multi view video has the advantage that it can serve different purposes. On the one hand the multi view input can be used for 3DTV. On the other hand it can be shown on a normal TV where the viewer can select his or her preferred viewpoint of the action. Of course, a combination is possible where the viewer selects his or preferred viewpoint on a 3DTV. However, multi view acquisition with e.g. 30 views will require 30 cameras to operate simultaneously. This requires initially a large investment. We therefore see a gradual transition from stereo capture to systems with many views. We will investigate a mixture of 3D video acquisition techniques to produce an extended centre view plus depth format (possibly with one or two extra views) that is, in principle, easier to produce, edit and distribute. The success of such a simpler format relies on the ease (read cost!) at which it can be produced. We conclude that the introduction of 3DTV to the mass market is hampered by the lack of high quality 3D video content, suitable 3D formats and appropriate format conversion techniques. The variety of new distribution media further complicates this. Hence, we can identify the following major challenges that are expected to be overcome by the project:

1. Video acquisition for 3D content

Here the practicalities of multi view and depth capture techniques are of primary importance, the challenge is to find the trade off such as number of views to be recorded, and how to optimally integrate of depth capture with multi view. A further challenge is to define which shooting styles are most appropriate.

2. Conversion of captured multi view video to a 3D broadcasting format

The captured format needs new post-production tools (like enhancement and regularisation of depth maps or editing, mixing, fading and compositing of video-plus-depth representations from different sources) and a conversion step generating a suitable transmission format that is compatible with used post-production formats before the 3D content can be broadcast and displayed.

3. Coding schemes for compression and transmission

A last challenge is to provide suitable coding schemes for compression and transmission that are based on the 3D broadcasting format under study and to demonstrate their feasibility in field trials under real
distribution conditions. By addressing these three challenges from an end-to-end system point of
view, the 3D4YOU project aims to pave the way to the definition of a 3D TV system suitable for a series of applications. Different requirements could be set depending on the application, but the basic underlying
technologies (capture, format and encoding) should maintain as much commonality as possible to favour the emergence of an industry based on those technologies.

The objectives of the project are:

1. To deliver an end-to-end system for 3D high quality media.
2. To develop practical multi view and depth capture techniques.
3. To convert captured 3D content into a 3D broadcasting format.
4. To demonstrate the viability of the format in production and over broadcast chains.
5. To show reception of 3D content on 3D displays via the delivery chains.
6. To assess the project results in terms of human factors via perception tests.
7. To produce guidelines for 3D capturing to aid in the generation of 3D media production rules.
8. To propose exploitation plans for different 3D applications.