Lehrstuhl Kognitive Integrierte Sensorsysteme: 3DKM

Design and Implementation of Hard- and Software of a dedicated Micro-3D-Camera System

Subject:
The project contributes as subcontractor to the BMBF-project ''Design of a Micro-3D-Camera System for the rapid time-resolved 3D-shape Acquisition of Objects'' coordinated by iC-Haus GmbH, Bodenheim. The topic of time-of-flight CMOS imager chips and camera systems for efficient depth/range data acquisition and their application in smart vision systems has been a topic of intensive research in the last decade. Numerous commercial systems, e.g., the well-known Kinect sensor, have become available. The research interests of the ISE group have been focused in this field of sensor & system design and applications, e.g., eye-tracking, gesture recognition, and robot vision. In the scope of the subcontract, ISE delivered essential parts of the PCB-level camera system hardware and software, linking iC-Haus sensor and active illumination microchip and system design with other industrial project particpants applications.

Abstract:
In computer vision depth registration and exploitation has been of significant and constantly increasing interest and importance. For instance, in automotive and robot vision systems, depth perception has predominantly been achieved by stereo camrea systems and computationally intensive algorithms for disparity computation, e.g., block matching. The requirement of two cameras, their judicious placement and calibration, however, was not in the favor of cost-sensitive and mobile applications, such as the gaming market, human-computer-interface, or robot applications. Vibrations in mobile systems were another obstacle in the use of such systems.

Thus, techniques based on a single sensor combined with active scene illumination employing the time-of-flight principle (TOF), well known in various wave length ranges in military as well as civilian applications, in an optical embodyment and realized with main-stream CMOS technology in non-scanning approach, following in the footsteps of grey-value, color, high-dynamic-range CMOS imagers, attracted the attention of researchers in the late 1990ties until today. The time-of-flight principle is depicted in the following:

A light pulse, commonly in the IR range, is generated from a laser diode by a dedicated circuit and diffused to homogeneously illuminate the scene. Natural limitations are eye safety considerations, which lead also to range limitations. The light is reflected from the surface(s) and travels back to the detector array. From the travel time a depth profile basically can be computed. Thus, the common claim of dealing with 3D-cameras and data is not quite true, as no voxel data as in, e.g., computer tomography (CT), is recorded. Occlusion, absorption, and shadow effects limit the depth/range profile recording in practical applications. Numerous researchers and companies have developed sensors, e.g., PMD (Photonic-Mixing-Device, photgates) by Schwarte et al. and PMDTec company, the CSEM Swiss Ranger by P. Seitz, T. Oggier, N. Blanc et a l., FhG IMS Duisburg/Siemens AG by Hosticka et al. (CMOS-pn-photo diodes), or Canesta Inc. activities. More recent ones, including also avalanche based photo detectors, have entered the scene and a brief summary can be found in the below mentioned project related publications. In this project, a proprietory sensor and illumination technology of iC-Haus have been employed. The sensor technology bases on a patented drift-field-mixing concept (DRIMIX) which had been realized by iC-Haus in a sub-um-SOI technology with 60x80 pixel matrix. The following pictures illustrate this work (courtesy and copyright iC-Haus):

The following picture show the sensor prototype mounted on PCB and the lens system designed by iC-Haus (courtesy and copyright iC-Haus):

For the above summarized baseline of sensor and illumination micro chip/system, a camera system has been developed in the subcontract from scratch. A conceptual system block diagram describes the basic structure:

The very first prototype based on Altera Cyclone III and dedicated circuits for A/D-conversion and interfacing shown in the following picture:

It has been improved and turned into a version to leave the lab and do measurements in the iC-Haus optical lab facilities:

The camera system is controlled by a dedicated software adapted to the configuration and protocol needs of the iC-Haus proprietary designs. The block diagram of the system software is given in the following:

The GUI with interactive configuration, measurement process control, data visualization, assessment, postprocessing, and interfaces to, e.g., Matlab:

The final integration of the prototype camera system is given in the following pictures:

The following pictures illustrate a simple lab testing scene for the development and test of the system and one recorded depth profile at ISE:

A typical industrial scene is given in the following picture, where a robot has to recognize and grasp objects aided by depth information (Courtesy Arotec):

The overall BMBF-project, including the subcontract, has concluded with a positive evaluation (See one review here). Though numerous applications can be envisioned for this class of systems, the particular ISE interest is in the following:

1. Eye-Tracking for auto-stereoscopic display and related HCI applications

2. Gesture recognition, e.g., for smart-kitchen (Emerging lab at ISE)

3. Driver-Drowsiness-Detection for cars and commercial vehicles based on fusion of IR-grey-values, depth, and further sensory context recordings (DeCaDrive)

Projects on theses topics have been started at ISE as spin-off of the subcontract, employing the dedicated DRIMIX prototype, Kinect sensors, and the Kinetic Depth Sense sensor. One example of eye-tracking is given in the following picture (more can be found on the dedicated pages of the spin-off projects):

The excellent cooperation with iC-Haus, in particular with Mr. Paulus, Mr. Burkhard, and Mr. Hänel, is gratefully acknowledged.

	Status:		concluded, duration 03/2008-03/2012, doctoral project from this project still running.
	Contractor:		iC-Haus GmbH, Bodenheim, subcontract in BMBF-project ''Entwicklung eines Mikro-3D-Kamerasystems für die schnelle zeitaufgelöste 3D-Formerfassung von Objekten'', Konsortium: iC-Haus GmbH, Sick AG, Arotec.
	Financing:		iC-Haus GmbH, Bodenheim/BMBF
	Contact:		Prof. Dr.-Ing. Andreas König
	Contributors:		Li Li, Ben Lutgen, Yanhao Xu, and Andreas König
	Publications:

			Li Li; Yanhao Xu; König, A.; , "Robust depth camera based multi-user eye tracking for autostereoscopic displays," Systems, Signals and Devices (SSD), 2012 9th International Multi-Conference on , vol., no., pp.1-6, 20-23 March 2012 doi: 10.1109/SSD.2012.6198039