Photos and videos captured using smartphone cameras give a flat, 2D look. Despite this, some 3D features that are easy with DSLR cameras are challenging in smartphone cameras.
Manufacturers are making advanced smartphone cameras with such effects, still, it is not yet possible to render them in videos captured using smartphones.
Advanced professional cameras shot the view with both intensity and direction of light in a scene. This is known as Light Field, which is achieved by using an array of microlenses inserted between the primary lens of the camera and the camera sensor.
Due to limited space, multiple microlenses can not be placed in mobile phones. Hence, AI and machine-learning techniques are used for such image manipulation.
Scientists from IIT Madras and Northwestern University have developed deep learning algorithms that can significantly enhance the depth perception and 3D effects in videos shot using smartphone cameras.Â
The algorithm converts the stereo images captured using a smartphone into LF images. Besides preventing images from being flat, it adds a realistic 3D feel to the image. Another fascinating fact is it eliminates the need for fancy equipment or an array of lenses to capture videos with depth.
Kaushik Mitra, assistant professor, Department of Electrical Engineering, IIT Madras, said, “The algorithm first captures two videos (called stereo pair) simultaneously using the two adjacent cameras present in many smartphones these days. These stereo pairs go through a sequence of steps involving deep learning models. The stereo pairs are converted into a 7X7 grid of images, mimicking a 7X7 array of cameras, producing the LF image.”
“In addition to providing depth, our algorithm enables us to view the same video from not just one point of view but any of the 7×7 grid of viewpoints.”
The research has been published in the ‘Proceedings of International Conference on Computer Vision (ICCV), 2021’.