Multimodal control of a robot with a Microsoft Kinect camera



In this project an application was developed that could be used to control the industrial robot KUKA LBR iiwaa 7 R800 with gestures and voice. The sensors for detecting sound and images were provided by a customary Microsoft Kinect v2. The project evaluated what functionalities can be achieved with this technology.

Furthermore the calibration of camera and robot (hand-eye-calibration) needed to be as effortless as effortless as possible.


Space of time: 09/2015 – 02/2016

Coding Language: C# and Java

Team: alone

Background: Developed as a university project

Report (German): View Report

Nine Men’s Morris on a KUKA LBR IIWA 7 R800


Today’s important topic of research is robotics. Robots are used in many different industries. In the area of robotics one of the research areas is Human Robot Interaction (HRI). HRI is important for any robotics application where a human and a robot are both actuators. It is important that the HRI of a robot application is satisfying for the human so that no frustration arises while using the application.
A good way of developing or researching in the area of HRI is to develop games, since the interaction in games is one of the key factors. By developing an application where a human is playing with or against a robot, which is acting as a second player and tries to be as human-like as possible while playing, different approaches of HRI can be researched and evaluated.
In this project an application was developed that allowed a human player play the strategy board game “Nine Men’s Morris” against a robot. To enable the robot to sense its environment a camera was used to analyze the game area.

The robot

The used robot was the KUKA LBR iiwa 7 R800, which is very light weight (23.9 Kg) robot and can be used for intelligent industrial work. The seven axis of the robot allow a flexible movement in all directions. The robot has integrated sensors to simplify the implementation of force detection.

The robot can be programmed with the programming language Java and comes with a special IDE called KUKA Sunrise Workbench. Sunrise Workbench can also be used to load applications to the robot or to debug an application.

The camera

For visual analysis a Cognex In-Sight 7000 Integrated Vision System was used. The size of camera is 75mm x 55mm x 47mm and the connectors are Industrial M12 connectors. The robust and intelligent camera can analyze an image by its own. The communication to other devices can be implemented easily, because the camera can be for example set up as a Modbus/TCP server that can be reached by any device in the same network. To create analysis jobs the software “In-Sight Explorer” can be used. The software provides all features of the camera in a easy understandable user interface.



The robot is mounted on the table. It is connected via a LAN cable to a switch. The camera is attached to a beam at a fixed position and is also connected to the same switch. This way the robot and the camera will be in the same network and can communicate with each other.

 GitHub Repository      GitHub-Mark-120px-plus