Institute of Integrated Sensor Systems

Prof. Dr.-Ing. Andreas König







Student Theses




DeCaDrive - Multi-Sensor Intelligent System for On-Line Driver State and Intention Recognition

Advanced sensing systems, sophisticated algorithms and growing computational resources provide increasing leverage for the design of intelligent systems. In particular, human-machine-interfaces and related monitoring and assistance systems, e.g., for multi-media (eye-tracking, gesture recognition, human-activity/behavior-recognition), ambient assistent living or active safety technology for vehicles, can considerably benefit from the technological advance. This project bases on our previous CMOS-chips based work on the highly integrated/embedded development of such systems, enhancing the sensing to depth or 3D-vision and driving and driver related multi-sensor information. In particular, a multi-sensor approach based on color and depth vision, vehicle driving data, and biomedical driver data with efficient sensor fusion and a learning system architectute has been pursued. The goal of this work on advanced driver assistance system is driver status monitoring, e.g. drowsiness detection, and as long-range goal driver intention recognition for cars and commercial vehicles, potentially also for aircrafts. Improvement of vehicle and road/traffic safety can be expected without compromising driving experience. The first prototype with on-line drowsiness detection is presented at IAA Nutzfahrzeuge 2014 in Hannover:


Assistance systems in automotive industry have matured in the last decade and numerous products have become available, e.g., parking or braking assistants, employing vision, ultrasonic, or radar sensing:

Table 1

For vision applications in automotive, robotics, automation etc., depth registration and exploitation has been of significant and constantly increasing interest and importance, which predominantly had to be achieved by stereo camera systems and computationally intensive algorithms for disparity computation, e.g., block matching. The requirement of two cameras, their judicious placement and calibration, however, was not in the favor of cost-sensitive and mobile applications, such as the gaming market, human-computer-interface, robot applications, or automotive assistance systems. The advent of CMOS depth or 3D-cameras working in the near infrared range (NIR) gave significant incentive and leverage to the field. This holds in particular for the 2010 introduced Microsoft Kinnect, which featured an incredible low price and, thus, made cost-sensitive applications feasible. Our research targets on the exploitation of depth camera principles for driver state and intention recognition. In addition, sensor information of driving behavior, e.g., steering wheel movement or braking behavior as well as bio-medical information, e.g., pulse, will be recorded and fused for decision making. The following figure gives the basic idea of the first DeCaDrive architecture:

DeCaDrive Architecture

In the ISE DeCADrive demonstrator, driving activity is simulated by employing on the bottom screen a video driving simulator game. All controls have been ''hacked'' in the sense, that all information from the game controls will be monitored by the DeCaDrive system on a second computer. The driver is monitored by a Kinnect, assuming a distance to driver in the order of about 80 cm, and the eye regions are detected and evaluated. Assumptions on lid movement give cues or features, which are fused with vehicle driving data and bio-medical information for fusion, decision making and warning:


From the raw sensory data streams, to achieve a lean, low-cost system, a couple of relevant features have been indentified in the research work, e.g., employing automated feature selection etc., which are computed during the driver observation and displayed on the top screen:

The Kinnect related processing is achieved based on standard SDK and propriatary C-programm developed at ISE. The features are handed over to a software system developed for sake of an open-access realization in Python and the higher level tool Orange (previously Matlab), where the other sensory channels are acquired, features computed, and the overall fusion and decision making, e.g., by hierarchical Support-Vector-Machine (SVM) classification, takes place. Experiments with currently a moderate number of probands, e.g., five different persons, and several hours of driving activity have been conducted. The condition states have been categorized to not drowsy, a little drowsy, and deeply drowsy. However, it must be mentioned here, that obtaining the ground truth of the driving persons is the most challenging part in the data acquisition with strong influence on the results. This issue will have to receive stronger attention in the work after completing hardware and information processing architecture. The following figure shows the temporal aspect of some of the regarded features:

The following picture shows the on-line processing and the view of the depth camera of the driver, inculding eye region detection followed by feature detection (This part of the DeCaDrive demonstrator is on permanent exhibition in the ISE show glass in building 11, 2.OG and the diploma student Klaudius Werber has been rewarded with the Pfalzmetallpreis 2013 for his diploma thesis work):

DeCaDrive operation

The fusion of the sensory channels from vision, driving behavior, and bio-medical information and the related features has been carried out. Numerous variations have been studied and this part of the research is still in the flux. The most successful and lean variant has been implemented in the current on-line drowsiness detection prototype, currently on display at IAA Nutzfahrzeuge 2014 in Hannover (Hall 13, booth C 35). However, it can be stated here, that this multi-sensor data system gives also strong incentive to our research-oriented teaching in the field of sensor signal processing and neurocomputing. Drowsiness estimation first off-line results based on an ANN with two different training algorithms are shown in the following figure:

The recent work based on more advanced techniques from the field of computational intelligence, e.g., optimization methods for automated constrained design of lean yet robust intelligent multi-sensory systems with optimized feature computation, dimensionality reduction, and hierarchical decision making.
Details on the work and the classification activity can be found in the publications enlisted below. Further improvement is excected from additional sensor modalities, e.g., blood oxygen saturation level, an improvement in selection of relevant features, advanced classification method application, and, in particular, a revision of the drowsiness state and ground truth determination of the driver. Here, we consider to measure in the data collection phase also EEG and other data of the driver to better assess the ground truh of probands drowsiness level and correlate this with our sensor set's acquired data. The following figure shows the effect on drowsiness level classification accuracy depending on selected features:

Currently, the DeCaDrive demonstrator is refined and extended, e.g.,  additional bio-medical data have been included, e.g., obtained from impedance spectroscopy measurements of the driver. Improved driver state and intention recognition at reasonable cost is aspired for cars, commercial vehicles, e.g., busses or trucks, aggricultural vehicles, also particular costly production machines, and last not least, airplanes.

The current on-line drowsiness detecting prototype of DeCaDrive, implemented by M.E. Kittikhun Thongpull, is on display at IAA Nutzfahrzeuge in Hannover, hall 13, booth C35, from Sept. 23 to Oct. 2. It has found positive feddback in the media already, e.g., a TV-spot in SWR-Landesschau Aktuell on Sept. 25, 21:45 issue, time 7:01-9:08, and two radio interviews broadcasted in SWR4 and SWR-Info the same day. In advance of the fair, a preview was already given by a newspaper article in the Rheinpfalz.

The following research and development activities are on our agenda:

Updates on recent research results will be provided soon.


  Status:   running, duration 10/2010-today, doctoral projects related to this project still running.
  Contractor:   -
  Financing:   self-funded
  Contact:   Prof. Dr.-Ing. Andreas König
  Contributors:   Li Li, K. Thongpull, T. Bölke, K. Werber, C. F. Calvillo, K. D. Dinh, A. Guarde, and Andreas König
Li Li; K. Werber; C. F. Calvillo; K. D. Dinh; A. Guarde; König, A.; , "Multi-Sensor Soft-Computing System for Driver Drowsiness Detection " World Conference on Soft-Computing, WSC 2012, December, 2012
      Li Li, T. Bölke, A. König, "Can Impedance Spectroscopy Serve in an Embedded Multi-Sensor System to Improve Driver Drowsiness Detection ?", In: Book of Abstracts, Int. Workshop on Impedance Spectroscopy IWIS 2013, pp. 48-49, Chemnitz, Sept. 25-27, 2013 
      L. Li, K. Thongpull,  A. König, "Optimizing the Design of a Multi-Sensor System for On-Line Driver State and Drowsiness Detection", In: Tagungsband des XXVIII Messtechnisches Symposium des Arbeitskreises der Hochschullehrer für Messtechnik e.V. (AHMT),  pp. 205-214, Saarbrücken, Sept. 18-20, 2014 
      K. Thongpull,  A. König, "DeCaDrive - Multi-Sensor-System for Driver State Monitoring and Intention Recognition", Poster presentation, hall 13, C35 , IAA Nutzfahrzeuge,  Hannover, Sept. 23 - Oct. 2nd, 2014