# Robot@Home dataset

The Robot-at-Home dataset (Robot@Home, paper here) is a collection of raw and processed data from five domestic settings compiled by a mobile robot equipped with 4 RGB-D cameras and a 2D laser scanner. Its main purpose is to serve as a testbed for semantic mapping algorithms through the categorization of objects and/or rooms.

If you use this dataset, please cite it through:

      @ARTICLE{Ruiz-Sarmiento-IJRR-2017,
author = {Ruiz-Sarmiento, J. R. and Galindo, Cipriano and Gonz{\'{a}}lez-Jim{\'{e}}nez, Javier},
title = {Robot@Home, a Robotic Dataset for Semantic Mapping of Home Environments},
journal = {International Journal of Robotics Research},
year = {2017}
}



### 1. Summary

#### Novelty of the Robot@Home dataset

This dataset is unique in three aspects:

• The provided data were captured with a rig of 4 RGB-D sensors with an overall field of view of 180°H. and 58°V., and with a 2D laser scanner.
• It comprises diverse and numerous data: sequences of RGB-D images and laser scans from the rooms of five apartments (87,000+ observations were collected), topological information about the connectivity of these rooms, and 3D reconstructions and 2D geometric maps of the visited rooms.
• The provided ground truth is dense, including per-point annotations of the categories of the objects and rooms appearing in the reconstructed scenarios, and per-pixel annotations of each RGB-D image within the recorded sequences

During the data collection, a total of 36 rooms were completely inspected, so the dataset is rich in contextual information of objects and rooms. This is a valuable feature, missing in most of the state-of-the-art datasets, which can be exploited by, for instance, semantic mapping systems that leverage relationships like \emph{pillows are usually on beds} or \emph{ovens are not in bathrooms}. Robot@Home is accompanied with a specific software to conveniently access and manage the data, namely the Object Labeling Toolkit (OLT).

The following video shows the RGB images captured by the 4 RGB-D devices at one of the inspected apartments:

#### Ground-truth information

The ground-truth information provided by the dataset comes in two flavors. On the one hand, we provide (per-point) annotations of the categories of the main objects and rooms appearing in the reconstructed scenes. A total of ~1,900 objects belonging to 157 different categories were manually labelled from the 36 visited rooms (some of them several times). These rooms are also labelled as one of 8 possible categories. On the other hand, Robot@Home includes (per-pixel) annotations of the objects appearing in the 69,000+ gathered RGB-D images. The objects and rooms are also annotated with identifiers, so they can be individually tracked along the video sequences.

#### Dataset applications

Robot@Home is specially suitable for robotic algorithms performing semantic mapping through the categorization of objects and/or rooms. It can be also useful for the recognition of instances of objects/rooms, object segmentation, or data compression/transmission algorithms. Moreover, typical robotic tasks like 3D map building, localization, or SLAM can be tested with Robot@Home, since the robot localization can be accurately estimated from the sequence of 2D scans. Finally, a number of distinctive patterns and objects has been placed on purpose in the scenarios so the dataset can be used to test object-finding algorithms.

#### Avaiable data

The data are provided in both, different formats accessible by the open source Mobile Robot Programming Toolkit (MRPT), and (human readable) plain text files and png images. The dataset includes:

• 81 sequences of observations containing ~75min. of recorded data. The total number of observations is 87,000+ (18,000+ laser scans and 69,000+ RGB-D images), which are saved in [rawlog,plain_text+images] format.
• 72 3D reconstructed scenes in [scene,plain_text] format.
• 41 2D geometric maps saved in plain text files (36 for individual rooms, and 5 maps covering each apartment).
• 72 Labelled 3D reconstructed scenes in [scene,plain_text] format, containing ~1,900 labelled objects.
• 72 Labelled RGB-D sequences in [rawlog,plain_text+images] format.

#### Intended users

Robot@Home aims to be a testbed for researchers working on semantic mapping through object/room categorization, object/room recognition, 2D/3D mapping, 2D/3D localization, object segmentation, etc.

#### Webpage structure

The Downloads section lists the different download options for getting the dataset, which has been conveniently divided for users with different interests. The Dataset usage section shows how to handle the data provided within the dataset. Finally, the Additional information section provides extra resources.

The information contained in the dataset is available divided into sessions and types, so the users can get chunks of information depending on their needs. Each downloable unit is available in both, files with formats accessible by MRPT and OLT (rawlog or scene files) represented with the icon, and human readable files (plain text files or PNG images) represented by the icon. More information about these data types can be found here. Formated files are compressed in .rar, and human readable ones in .7z.

#### Data formats

Data are provided in two flavors:

• Formated files, accesible by MRPT and OLT, represented with . Concretely:
• rawlog files: binary files that store robotic datasets and are the input of many MRPT/OLT applications for off-line processing. More info here.
• scene files: OpenGL scenes that can be loaded and modified by MRTP/OLT applications. Here more information.
• Human readable files, represented with . They are grouped into:
• plain text files: different types of (numerical) information are stored in these files, including information about the observations within a sequence, 2D laser scans, reconstructed scenes, labelled reconstructed scenes, and labeled RGB-D images. The content of these files is explained at the files headers. For example, this is the header of a 2D laser scan:

# This file contains a 2D laser scan observation.
# Its format is:
# [aperture] [max_range] [number_of_scans]
# [vector_of_scans]
# [vector_of_valid_scans]
# The aperture of the sensor is in radians, the maximum range
# and measurements are in meters. '1' means a valid scan, '0' otherwise.

• PNG images: images are stored in files with the PNG format. There are two types of images:
• RGB images: Stored as three channels (R,G, and B) of uint8_t numbers.
• Depth images: Stored as one channel of uint16_t numbers. Depth information (in meters) is encoded in the following way:

depth_i = value_of_pixel_i*(1/6,553.5);

#### Raw data

alma-s1
anto-s1
pare-s1
rx2-s1
sarmis-s1
sarmis-s2
sarmis-s3
all

The raw data contains all the data collected in the seven sessions. Each session consists of a number of sequences of observations partially or totally inspecting a house. The gathered observations can be of two types: laser scans, or RGB-D observations. The formated sequences are provided in the Rawlog common robotics dataset format, and are raw in the sense that neither extrinsic nor intrinsic calibrations are applied to the observations.

#### Laser scans

alma-s1
anto-s1
pare-s1
rx2-s1
sarmis-s1
sarmis-s2
sarmis-s3
all

The laser scanner data contains the information gathered from the URG-04LX-UG01 device mounted on the robot. They are available the laser scans per session, which in turn are divided into sequences inspecting the different rooms. These are some examples of laser scans from (left to right) a bathroom, a living room, and a bedroom.

#### RGB-D data

alma-s1
anto-s1
pare-s1
rx2-s1
sarmis-s1
sarmis-s2
sarmis-s3
all

The RGB-D data entails the observations gathered by the four RGB-D devices. They are available the RGB-D observations per session, which in turn are divided into sequences inspecting the different rooms. A RGB-D observation comprises a RGB (intensity) image, a depth image, and a colored point cloud. Below they are shown the RGB and depth images captured by each of the RGB-D devices in a bedroom.

#### Geometric maps

2D geometric maps
2D geometric maps + logs

They have been built a 2D geometric map for each room within the 5 inspected apartments, including five more of the full house. For that, we have resorted to the icp-slam application from MRPT. Next images are: (top row) examples of 2D geometric maps during their building (from left to right: a bedroom, the living room, and the full house), and (bottom row) the resultant maps.

#### Reconstructed scenes

alma-s1
anto-s1
pare-s1
rx2-s1
sarmis-s1
sarmis-s2
sarmis-s3
all

For each sequence inspecting a room, its 3D coloured point cloud reconstruction is available. Such reconstruction is built employing the Mapping application within the Object Labeling Toolkit (OLT), which registers the RGB-D observations employing different ICP-based techniques. The following images show reconstructed examples of (from left to right) a bedroom, a corridor, and a livingroom.

#### Labeled scenes

alma-s1
anto-s1
pare-s1
rx2-s1
sarmis-s1
sarmis-s2
sarmis-s3
all

They are available the annotations made over the reconstructed places. These labeled were annotated employing the Label scene application within OLT, and can be downloaded as .scene files. Next, some figures of labeled places are shown:

#### Labeled RGB-D data

alma-s1
anto-s1
pare-s1
rx2-s1
sarmis-s1
sarmis-s2
sarmis-s3
all

These files contains the labeled RGB-D observations within the different sequences. To label them we again employ OLT, and propagate the objects' annotations into the labeled places to the intensity image, depth image, and point cloud of each RGB-D observation. The figures below are examples of automatically labeled depth images.

### 3. Dataset usage

This section shows how to manipulate the provided data in different ways. If you find someone missing, please let us know and we will add it. Enjoy!

The data provided in the dataset can be easily accessed employing MRPT, since their datatypes come from it. Moreover, OLT provides additional features to manage them.

### + Visualize a labelled reconstructed scene.

Next, they are shown some pictures of the robot while it was gathering the dataset. In them we can see the 4 RGB-D rig mounted on the robot's neck, and the laser scan placed just on its base.

### Change log

• [2016-April] All the data is also available in (human readable) plain text files or PNG images.
• [2016-March] All the raw and processed data of the seven sessions are available at the webpage.
• [2016-January] Performed data collection at alma house.
• [2016-January] Inspected anto house.
• [2016-January] Recorded session at pare house.
• [2016-January] Inspected rx2 house.
• [2015-May] Dataset's webpage created.
• [2014-July] Collected three sessions at sarmis house.
• [2014-July] Started the collection of the dataset.