{"id":2553,"date":"2014-01-20T09:51:00","date_gmt":"2014-01-20T09:51:00","guid":{"rendered":"https:\/\/mapir.isa.uma.es\/mapirwebsite_wordpress\/?p=2553"},"modified":"2022-01-20T10:01:09","modified_gmt":"2022-01-20T10:01:09","slug":"scene-flow","status":"publish","type":"post","link":"https:\/\/mapir.isa.uma.es\/mapirwebsite\/?p=2553","title":{"rendered":"Scene Flow"},"content":{"rendered":"\n<div class=\"wp-block-columns\">\n<div class=\"wp-block-column\" style=\"flex-basis:100%\">\n<div class=\"wp-block-image\"><figure class=\"alignleft size-full is-resized\"><img loading=\"lazy\" src=\"https:\/\/mapir.isa.uma.es\/mapirwebsite\/wp-content\/uploads\/2022\/01\/mapir_image_scene_flow.png\" alt=\"\" class=\"wp-image-2554\" width=\"171\" height=\"136\" srcset=\"https:\/\/mapir.isa.uma.es\/mapirwebsite\/wp-content\/uploads\/2022\/01\/mapir_image_scene_flow.png 961w, https:\/\/mapir.isa.uma.es\/mapirwebsite\/wp-content\/uploads\/2022\/01\/mapir_image_scene_flow-300x239.png 300w, https:\/\/mapir.isa.uma.es\/mapirwebsite\/wp-content\/uploads\/2022\/01\/mapir_image_scene_flow-768x611.png 768w\" sizes=\"(max-width: 171px) 100vw, 171px\" \/><\/figure><\/div>\n\n\n\n<p>Scene flow is the semi-dense or dense 3D motion field of a scene that moves completely of partially with respect to a camera.<\/p>\n<\/div>\n<\/div>\n\n\n\n<!--more-->\n\n\n\n<p>The potential applications of scene flow&nbsp;are numerous. In robotics, it can be used for autonomous navigation and\/or manipulation&nbsp;in dynamic environments where the future distribution of the surrounding objects need to be predicted. Besides, it could complement and improve state-of-the-art Visual Odometry and SLAM algorithms which typically assume to work in rigid or quasi-rigid enviroments. On the other hand, it could be employed for human-robot or human-computer interaction, as well as for virtual and augmented reality. In general, it would be useful at any case where motion capture \/ analysis were required.&nbsp;&nbsp;<\/p>\n\n\n\n<p>Traditionally, scene flow has been computed from image pairs coming from stereo cameras, where both the disparity and the motion need to be estimated. However, the new RGB-D cameras changed this trend since they provide registered intensity and depth images which can be employed to estimate 3D motions accurately (for short-medium distances &lt; 5 meters). In our work, we propose several algorithms to estimate scene flow with RGB-D cameras.&nbsp;<\/p>\n\n\n\n<div id=\"toc_container\" class=\"no_bullets\"><p class=\"toc_title\">Contents<\/p><ul class=\"toc_list\"><li><a href=\"#Real-time_RGB-D_Scene_Flow\">Real-time RGB-D Scene Flow<\/a><\/li><li><a href=\"#Segmentation_from_Motion\">Segmentation from Motion<\/a><\/li><li><a href=\"#Publications\">Publications<\/a><\/li><\/ul><\/div>\n<h2><span id=\"Real-time_RGB-D_Scene_Flow\"><strong>Real-time RGB-D Scene Flow<\/strong><\/span><\/h2>\n\n\n\n<p>We have developed the first dense real-time scene flow for RGB-D cameras. Within a variational framework, photometric and geometric consistency is imposed between consecutive RGB-D frames, as well as TV regularization of the flow. Accounting for the depth data provided by RGB-D cameras, regularization of the flow field is imposed on the 3D surface (or set of surfaces) of the observed scene instead of on the image plane, leading to more geometrically consistent results. The minimization problem is efficiently solved by a primal-dual algorithm which is implemented on a GPU. This work was developed in conjuction with the CVPR group (TUM).<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"Real Time Dense Scene Flow for RGB-D Cameras\" width=\"720\" height=\"405\" src=\"https:\/\/www.youtube.com\/embed\/vvLuHZyogow?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe>\n<\/div><figcaption>Code:\u00a0<a rel=\"noreferrer noopener\" href=\"https:\/\/github.com\/MarianoJT88\/PD-Flow\" target=\"_blank\">PD-Flow (GitHub)<\/a><\/figcaption><\/figure>\n\n\n\n<h2><span id=\"Segmentation_from_Motion\"><strong>Segmentation from Motion<\/strong><\/span><\/h2>\n\n\n\n<p>We have proposed a novel joint registration and segmentation approach to estimate scene flow from RGB-D images. Instead of assuming the scene to be composed of a number of independent rigidly-moving parts, we use non-binary labels to capture non-rigid deformations at transitions between the rigid parts of the scene. Thus, the velocity of any point can be computed as a linear combination (interpolation) of the estimated rigid motions, which provides better results than traditional sharp piecewise segmentations. Within a variational framework, the smooth segments of the scene and their corresponding rigid velocities are alternately refined until convergence. A K-means-based segmentation is employed as an initialization, and the number of regions is subsequently adapted during the optimization process to capture any arbitrary number of independently moving objects. We use a weighted quadratic regularization term to favor smooth non-binary labels, but also test our algorithm with weighted TV. This work was developed in conjuction with the CVPR group (TUM).<\/p>\n\n\n\n<p>Here you can find the segmentations obtained for some sequences of the Sintel dataset, and below we show the 3D scene flow estimated for a moving person:&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" width=\"1024\" height=\"735\" src=\"https:\/\/mapir.isa.uma.es\/mapirwebsite\/wp-content\/uploads\/2022\/01\/Sintel_segmentations_MCFlow-1024x735.png\" alt=\"\" class=\"wp-image-2555\" srcset=\"https:\/\/mapir.isa.uma.es\/mapirwebsite\/wp-content\/uploads\/2022\/01\/Sintel_segmentations_MCFlow-1024x735.png 1024w, https:\/\/mapir.isa.uma.es\/mapirwebsite\/wp-content\/uploads\/2022\/01\/Sintel_segmentations_MCFlow-300x215.png 300w, https:\/\/mapir.isa.uma.es\/mapirwebsite\/wp-content\/uploads\/2022\/01\/Sintel_segmentations_MCFlow-768x551.png 768w, https:\/\/mapir.isa.uma.es\/mapirwebsite\/wp-content\/uploads\/2022\/01\/Sintel_segmentations_MCFlow.png 1299w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" width=\"1024\" height=\"264\" src=\"https:\/\/mapir.isa.uma.es\/mapirwebsite\/wp-content\/uploads\/2022\/01\/Comparison_flows_MCFlow-1024x264.png\" alt=\"\" class=\"wp-image-2556\" srcset=\"https:\/\/mapir.isa.uma.es\/mapirwebsite\/wp-content\/uploads\/2022\/01\/Comparison_flows_MCFlow-1024x264.png 1024w, https:\/\/mapir.isa.uma.es\/mapirwebsite\/wp-content\/uploads\/2022\/01\/Comparison_flows_MCFlow-300x77.png 300w, https:\/\/mapir.isa.uma.es\/mapirwebsite\/wp-content\/uploads\/2022\/01\/Comparison_flows_MCFlow-768x198.png 768w, https:\/\/mapir.isa.uma.es\/mapirwebsite\/wp-content\/uploads\/2022\/01\/Comparison_flows_MCFlow-1536x396.png 1536w, https:\/\/mapir.isa.uma.es\/mapirwebsite\/wp-content\/uploads\/2022\/01\/Comparison_flows_MCFlow.png 1844w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2><span id=\"Publications\">Publications<\/span><\/h2>\n\n\n\n<p>Here you can find a list of my publications with pdf author&#8217;s version and bibtex references:<\/p>\n\n\n\n<iframe loading=\"lazy\" style=\"line-height: 1.3em;\" src=\"https:\/\/mapir.uma.es\/mapirpubsite\/index.php\/export\/bytopic\/66\/1\/aigaion_pubs_for_joomlawrapper.css\/none\/mapir_formatted_list\/type\/none\" width=\"100%\" height=\"600\" frameborder=\"0\"><p>The list is loaded from another server. If you see this, there has been a problem.<\/p><\/iframe>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Scene flow is the semi-dense or dense 3D motion field of a scene that moves completely of partially with respect to a camera.<\/p>\n","protected":false},"author":8,"featured_media":2554,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"hide","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false},"categories":[10],"tags":[],"_links":{"self":[{"href":"https:\/\/mapir.isa.uma.es\/mapirwebsite\/index.php?rest_route=\/wp\/v2\/posts\/2553"}],"collection":[{"href":"https:\/\/mapir.isa.uma.es\/mapirwebsite\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mapir.isa.uma.es\/mapirwebsite\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mapir.isa.uma.es\/mapirwebsite\/index.php?rest_route=\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/mapir.isa.uma.es\/mapirwebsite\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2553"}],"version-history":[{"count":4,"href":"https:\/\/mapir.isa.uma.es\/mapirwebsite\/index.php?rest_route=\/wp\/v2\/posts\/2553\/revisions"}],"predecessor-version":[{"id":2560,"href":"https:\/\/mapir.isa.uma.es\/mapirwebsite\/index.php?rest_route=\/wp\/v2\/posts\/2553\/revisions\/2560"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mapir.isa.uma.es\/mapirwebsite\/index.php?rest_route=\/wp\/v2\/media\/2554"}],"wp:attachment":[{"href":"https:\/\/mapir.isa.uma.es\/mapirwebsite\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2553"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mapir.isa.uma.es\/mapirwebsite\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2553"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mapir.isa.uma.es\/mapirwebsite\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2553"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}