DeCaDrive - Multi-Sensor Intelligent System for On-Line Driver State and Intention Recognition
Advanced sensing systems, sophisticated algorithms and growing computational resources provide increasing leverage for the design of intelligent systems. In particular, human-machine-interfaces and related monitoring and assistance systems, e.g., for multi-media (eye-tracking, gesture recognition, human-activity/behavior-recognition), ambient assistent living or active safety technology for vehicles, can considerably benefit from the technological advance. This project bases on our previous CMOS-chips based work on the highly integrated/embedded development of such systems, enhancing the sensing to depth or 3D-vision and driving and driver related multi-sensor information. In particular, a multi-sensor approach based on color and depth vision, vehicle driving data, and biomedical driver data with efficient sensor fusion and a learning system architectute has been pursued. The goal of this work on advanced driver assistance system is driver status monitoring, e.g. drowsiness detection, and as long-range goal driver intention recognition for cars and commercial vehicles, potentially also for aircrafts. Improvement of vehicle and road/traffic safety can be expected without compromising driving experience. The first prototype with on-line drowsiness detection is presented at IAA Nutzfahrzeuge 2014 in Hannover:
Assistance systems in automotive industry have matured in the last decade and numerous products have become available, e.g., parking or braking assistants, employing vision, ultrasonic, or radar sensing:
For vision applications in automotive, robotics, automation etc., depth registration and exploitation has been of significant and constantly increasing interest and importance, which predominantly had to be achieved by stereo camera systems and computationally intensive algorithms for disparity computation, e.g., block matching. The requirement of two cameras, their judicious placement and calibration, however, was not in the favor of cost-sensitive and mobile applications, such as the gaming market, human-computer-interface, robot applications, or automotive assistance systems. The advent of CMOS depth or 3D-cameras working in the near infrared range (NIR) gave significant incentive and leverage to the field. This holds in particular for the 2010 introduced Microsoft Kinnect, which featured an incredible low price and, thus, made cost-sensitive applications feasible. Our research targets on the exploitation of depth camera principles for driver state and intention recognition. In addition, sensor information of driving behavior, e.g., steering wheel movement or braking behavior as well as bio-medical information, e.g., pulse, will be recorded and fused for decision making. The following figure gives the basic idea of the first DeCaDrive architecture:
In the ISE DeCADrive demonstrator, driving activity is simulated by employing on the bottom screen a video driving simulator game. All controls have been ''hacked'' in the sense, that all information from the game controls will be monitored by the DeCaDrive system on a second computer. The driver is monitored by a Kinnect, assuming a distance to driver in the order of about 80 cm, and the eye regions are detected and evaluated. Assumptions on lid movement give cues or features, which are fused with vehicle driving data and bio-medical information for fusion, decision making and warning:
The fusion of the sensory channels from vision, driving behavior, and bio-medical information and the related features has been carried out. Numerous variations have been studied and this part of the research is still in the flux. The most successful and lean variant has been implemented in the current on-line drowsiness detection prototype, currently on display at IAA Nutzfahrzeuge 2014 in Hannover (Hall 13, booth C 35). However, it can be stated here, that this multi-sensor data system gives also strong incentive to our research-oriented teaching in the field of sensor signal processing and neurocomputing. Drowsiness estimation first off-line results based on an ANN with two different training algorithms are shown in the following figure:
Currently, the DeCaDrive demonstrator is refined and extended, e.g., additional bio-medical data have been included, e.g., obtained from impedance spectroscopy measurements of the driver. Improved driver state and intention recognition at reasonable cost is aspired for cars, commercial vehicles, e.g., busses or trucks, aggricultural vehicles, also particular costly production machines, and last not least, airplanes.
The current on-line drowsiness detecting prototype of DeCaDrive,
implemented by M.E. Kittikhun Thongpull, is on display at IAA
Nutzfahrzeuge in Hannover, hall 13, booth C35, from Sept. 23 to Oct. 2.
It has found positive feddback in the media already, e.g., a TV-spot in
SWR-Landesschau Aktuell on Sept. 25, 21:45 issue, time 7:01-9:08, and two radio
interviews broadcasted in SWR4 and SWR-Info the same day. In advance of
the fair, a preview was already given by a newspaper article in the Rheinpfalz.
The following research and development activities are on our agenda:
Updates on recent research results will be provided soon.
|Status:||running, duration 10/2010-today, doctoral projects related to this project still running.|
|Contact:||Prof. Dr.-Ing. Andreas König|
|Contributors:||Li Li, K. Thongpull, T. Bölke, K. Werber, C. F. Calvillo, K. D. Dinh, A. Guarde, and Andreas König|
Li Li; K. Werber; C. F. Calvillo; K. D. Dinh; A. Guarde; König, A.; , "Multi-Sensor Soft-Computing System for Driver Drowsiness Detection " World Conference on Soft-Computing, WSC 2012, December, 2012
|Li Li, T. Bölke, A. König, "Can Impedance Spectroscopy Serve in an Embedded Multi-Sensor System to Improve Driver Drowsiness Detection ?", In: Book of Abstracts, Int. Workshop on Impedance Spectroscopy IWIS 2013, pp. 48-49, Chemnitz, Sept. 25-27, 2013|
|L. Li, K. Thongpull, A. König, "Optimizing the Design of a Multi-Sensor System for On-Line Driver State and Drowsiness Detection", In: Tagungsband des XXVIII Messtechnisches Symposium des Arbeitskreises der Hochschullehrer für Messtechnik e.V. (AHMT), pp. 205-214, Saarbrücken, Sept. 18-20, 2014|
|K. Thongpull, A. König, "DeCaDrive - Multi-Sensor-System for Driver State Monitoring and Intention Recognition", Poster presentation, hall 13, C35 , IAA Nutzfahrzeuge, Hannover, Sept. 23 - Oct. 2nd, 2014|