Institute of

Cognitive Integrated Sensor Systems

Prof. Dr.-Ing. Andreas König

News

Staff

Teaching

Research

Publications

Conferences

Student Theses

Contact

Impressum

Home

Lab-on-Spoon - Multi-Sensorial 3D-integrated Measurement System for Smart-Kitchen and AAL Applications

         

Subject: (see our most recent extension of this research, the E-Taster Assistance System, also at CeBIT 2015, hall 9, Booth D23 ! )

This research combines the increasingly popular, versatile, and powerful technique of Impedance Spectroscopy with our activities on smart-environment, e.g., in our Smart-Kitchen scenario. In this food processing or culinary scenario, as well as food safety monitoring devices related search, we pursue the creation of assistance systems for the tasks of food storage, quantity and quality assessment, as well as preparation result assessment. One key aspect is the providance of capable sensory context integrated by todays' powerful available System-in-Package (SiP) integration technologies from the More-than-Moore direction of the ITRS roadmap in devices of daily living. The first item under investigation was the design of multi-sensor system embedded in or better embodied by a spoon. Pioneering work on smart-spoons and related devices has been done in the last decade, e.g., by MIT (Counter Intelligence project, Dr. T Selker et al.). The focus in our work is on achieving a highly capable analysis system as a component of an assistance system in the form of an multi-sensor, autonomous, wireless, low-power device in the shape of everyday life device with a focus on cheap mass-market realization. From 3D-printing technology to 3D electronic packaging technologies are under investigation for this aim. In the first prototype, an embedded realization of Impedance Spectroscopy is combined with color and temperature registration and applied jointly with Computational Intelligence methods to tasks, as e.g., oil state classification. See a short video presentation of Lab-on-Spoon by EbruTV Technology Feature Update, No. 45 (Folge 45).

Abstract:

In the last decade, the research on miniturized, distributed, autonomous networked sensor systems has been pursued intensively and found numerous applications. The vision of Smart Dust from Berkeley has been one of the driving factors in the development. In particular,  distributed intelligent systems in smart-environment, ambient intelligence, and ambient assisted living (AAL)  have significantly benefitted from the advance and new application fields have merged, e.g., Industry4.0 (Smart-factories), Internet-of-Things (IoT), or Cyber-Physical-Systems (CPS). From the early conceptions to most of the state-of-the-art systems, three potentials for improvement can be indentified. Firstly, the current realizations of sensing could benefit a lot from the know-how and standards in sensor, measurement, and instrumentation community, both in accuracy and diversity of methods. Secondly, the robustness of systems and the capability to correct and maintain themselves, commonly denoted as self-x- or self-*-capability, is required and can be adopted from related activities in the community of adaptive and dynamically reconfigurable hardware. Last not least, the third issues is the exploitation of current MEMS and advanced packaging technologies, e.g., 3D-printing or 3D-electronics and systems packaging.

For this aim, the smart-kitchen and related AAL scenario has been chosen to employ, combine, and refine technologies from the three lines of research mentioned above to achieve more able, more capable, and unobtrusive systems for assisted working and living.  The topic of smart kitchens has been pioneered by MIT (Counter Intelligence project, Dr. T. Selker) and has seen numerous follow-ups in the last decade. Also, the scenario of a smart spoon has been tackled the same MIT group with remarkable sensing capability, e.g. temperature,  conductivity (DC), pH-value. New lines and capabilities come from advanced sensor and measurement approaches and their efficient integration. Impedance Spectroscopy and its translation to embedded or Integrated Impedance Spectroscopy (IIS) is one of the pursued new research directions, e.g., for a Lab-on-Spoon system to provide more powerful sensorial context in Smart-Kitchen and related scenarios. The following picture shows the Kinect depth and color image based gesture controlled central unit of the ISE-Smart-Kitchen, which includes a prototype of an electronic cookbook:

The next picture shows a close-up of the opened interactive electronic cookbook for one particular recipe:

The Lab-on-Spoon is conceived to provide for the context of each preparatory step the sensorial information or feedback on food ingredient quantity and quality, i.e., classify both the correctness of the kind of ingredient as well as its quality or state, in particular, judging the freshness to avoid rotten ingredient inclusion in the preparatory process. Also, the robust detection of contaminations can be identified both as relevant for immediate application and additional  research. Thus, users potentially impaired in their qualitative and quantitative judgement on ingredients and preparations process step results'  by either restricted skills and experience or by loss of perceptive capabilities due to accident or aging can be supported by our system. Lab-on-Spoon represents a front-end component of a corresponding assistance system in the ISE-Smart-Kitchen, which provides sensorial context to restore or improve users' perceptive abilities. The sensor and measurement part of the Lab-on-Spoon system is illustrated in the following picture:

In the current prototype, embedded impedance spectroscopy based on the AD 5933 chip, temperature sensig based on a industrial pt10k sensor from UST, and an industrial color sensor (MAZeT MCS3AS) have been integrated. First measurements with the different sensor modalities have been done based on the following simplified prototype:


The first prototype has been achieved with conservative PCB-technology tailored to the needs of a basic 3D-printed spoon shape. The MakerBot Replicator was employed, delivering a first simple thermoplast prototype with obvious limitations to withstand higher temperatures in cooking processes. But more able, and more costly, processes are freely available, e.g., 3D-printing by laser-based metal powder sintering, which will provide Lab-on-Spoon embodiments suitable for the full range of temperatures in cooking processes. The following picture shows an early assembly view of a tentative prototype with embedded IS only:

  

The analog front-end with the AD5933 currently still is a very basic version with internal clock and two-wire measurement approach. It comes in two variants for the standard impedance range of the AD5933 from 1k to several M Ohm and a buffered version for small impedances less than 1k Ohm, e.g., for highly conducting ingredient liquids of high salinity. With regard to obtaining a very power-economic design for the aspired autonomous system, an Energy-Micro microcontroller has been applied in initial bread-boarding work.  Based on the first Energy-Micro bread-board prototype measurements have been taken from first examples of the chosen Smart-Kitchen application domain. The first measurement data acquired from a still somewhat cranky setup was transfered by a Python-based interface to a proprietory ISE tool QuickCog. Problems with the first setup, i.e., lack of stable and repeatable results, lack of real spoon integration and the substantial effort of Energy-Micro uC extension to wireless operation ended the student-driven activity.

The Lab-on-Spoon prototype has been moved to PhD-level research and to the Arduino family with their convenient Xbee wireless communication facility. Also it has been augmented by color and temperature and corresponding sensor fusion in QuickCog and Orange, a Python-based tool, which together provide access to  state-of-the-art proven and advanced Computational Intelligence methods for visualization, analysis, and classification of the Lab-on-Spoon data, by now. Minituarization allowed the spoon-shape integration with the extended functionality. The new system embodiment and the reliable Lab-on-Spoon-interface in Orange,  with life acquisition and classification feature, allowed much more stable and repeatable multi-sensor measurement. So, previous tasks can be done in acceptable quality now and the application scope could be considerably enlarged.
The following picture show the first example of food ingredient categorization with one of the LoS-prototypes by a feature space plot from impedance magnitude in a 10-100 kHz sweep , e.g., recognizing and distinguishing of tub water, salty water, soy sauce, and vinegar:

   
 

This first proof-of-principle with impedance values in a typical range of 100 - 2k Ohm was followed by another one example of high practical value, i.e., the classification or grading of food ingredient quality. This was done for the relevant case of oil grading, applied to vegetable oil employed in a frying device. The objective here is to provide a quality driven point of frying oil replacement by monitoring and assessing the use or wear state. The analogy to motor oil, gear box oil, or related lubricants is obvious and is under investigation by the Lab-on-Spoon system. The following picture shows one example of fresh and used frying oil feature space plot for color and  impedance values, which are in the range of 400k to about 8 M Ohm and again 10-100 kHz sweep:


A variation of this task is the discernment of various kinds of oils, e.g., oils with different temperature endurance, by the Lab-on-Spoon system. The following picture shows an example of peanut, sunflower, and olive oil in a feature space plot impedance values again for a 10-100 kHz sweep:


Further investigations and measurements for larger data sets and new applications are in progress. The next one, with regard to CeBIT 2014 partner country UK, is the distinguishment of teas represented by six different types of tea, i.e., Ceylon, Earl Grey, Darjeeling, Jasmine, Peppermint, and Camomile, which are subject to measurement of color and impedance values by the Lab-on-Spoon prototype with settings as given above for the previous examples. Additionally, the relevant features for the  task have been selected from the color and impedance values by automated feature selection:


In the case of the teas a slight confusion between Ceylon and Earl Grey can be observed for the current hardware, which will be tackled by the sensory enhancements in progress. The next example of distinguishment of beverages regards four different types of beer, which are subject to measurement of color and impedance values by the Lab-on-Spoon prototype with settings as given above for the previous examples. Again, the relevant features for the  task have been analysed from the color and impedance values by automated feature selection, which also help to significantly reduce measurement time in future measurements for this task:


The next and key one is typical to Rhineland-Palatina as a major wine-producing state and deals with the distinguishment of seven different types of wine which are again subject to measurement of color and impedance values by the Lab-on-Spoon with settings s given above for the previous examples. Investigations of the relevant features for the  task has been conducted from the color and impedance values by automated feature selection as in the previous cases:


This application is quite complex and the current state of the hardware implementation, as for the tea tasting example, meets limitations in discerment, which we confident to overcome by our sensory extensions on the way. But the example has another interesting application, as the case of wine spoiled due to time and storage conditions or due to contamination gives rise to another challenge for Lab-on-Spoon. We let ourselves be inspired by a problem wine industry faced about two decades back, where wine was contaminated by Diethylenglycol to sweeten it. In our experiment, also employed as CeBIT demonstrator, we use a much more harmless substance glycerol, which also serves as an anti freezing agent. About 10% are added to dry white wine for the Lab-on-Spoon system to detect. The sensory data for pure and contaminated wine are recorded as in the previous cases. A plot of the feature space is given in the following:


The classification was achieved by training a suitable Support-Vector-Machine on normalized and selected measurement data in Orange. Contamination detection capability of Lab-on-Spoon could be proven in numerous life demos on CeBIT, in particular, on the occasion of a visit of the prime minister of Rhineland-Palatina, Mrs. M. Dreyer, to our booth.

The Lab-on-Spoon prototype extension to  pH-value and viscosity is under investigation. The temperature, color, and Impedance Spectroscopy based prototypes in tentative wired and XBEE-wireless version will be on display on CeBiT 2014, exhibition hall 9, booth D23 of Rhineland-Palatina (Rheinland-Pfalz), presenter M.Sc. Kittikhun Thongpull of ISE. The first proof-of-principle demonstrator of Lab-on-Spoon with temperature (UST pt10k), color (MAZeT MCS3AS), and impedance spectrum measurement (AD5933 with AFE) and Arduino pro mini based version with active LED illumination is given in its unsealed USB-variant here:

The following picture shows a sealed, wireless, and autonomous Lab-on-Spoon as demonstrated at CeBIT. It has one more functional extension of value for life assistance. The temperature of the spoon content is continuously monitored and translated into a color of the spoon illumination. Low temperatures can be coded by blue, very high temperature by red color with a seamless transition of color shade in the programmable intervals. This can be exploited as a heat warning to avoid hurting by too hot spoon contents, but is can also application-specifically be changed by the smart kitchen host, i.e., be programmed to the interval of water temperature required for the quite sensitive yeast bactery cultivation needed for, e.g., bread baking:


The following pictures from the CeBIT booth show the visit of the prime minister of Rhineland-Palatina, Mrs. M. Dreyer, where the contaminated wine detection was presented as life demo:

    

    

Lab-on-Spoon and the successful demo was also reflected in the media, e.g., in a CeBIT report in  SWR1 Landesschau aktuell (MP4, 28.9 MB) on Monday, 10th of March and the Rheinpfalz newspaper. Also, CeBiT presentation was reflected in a contribution of EbruTV Technology Feature Update, No. 45 (Folge 45), about the first 3-4 minutes, on-line available since 19.11.2014.
In the next steps of the project, the following issues will be pursued towards a low-cost mass-marketable autonomous device for food safety monitoring, smart-kitchen. and AAL application:

Sponsoring

The major color sensor manufacturer MAZeT has been attracted as sponsor to the Lab-on-Spoon project and supports us, in particular, with the most recent multispectral MMCS6CS sensor for our advanced Lab-on-Spoon version.

                          


  Status:   Running, duration 11/01/2011 - today
  Partners:   -
  Financing:   Self-funded research
  PI/Contact:   Prof. Dr.-Ing. Andreas König
  Contributors:   Prof. Dr.-Ing. Andreas König (Concept, multi-sensor Arduino Lab-on-Spoon design and implementation, measurements, and CI on QuickCog, CeBIT 2014 presentation),
M.Sc. Kittikhun Thongpull (Python drivers, CI on Orange,
CeBIT 2014 presentation),
M.Sc. Abhay Chandra Kammara (Smart-kitchen, e-cookbook and sensor context)

Students:

Thomas Gräf, Color sensor module design, student assistantship (HIWI) funded by AmI research center in 2004, reuse for Lab-on-Spoon.
Thomas Bölke, Diplomarbeit, MPT, reuse of some  Arduino code from ISE  DeCaDrive-project.

A. Renner, L. Minghan (Semester projects (Studienarbeiten), design of new Impedance Spectroscopy CMOS chip)


Students (Energy-Micro-based activity):
Z. Espejo (Master thesis, Energy Micro uC, breadboarding, end March 2013),
D. Los Arcos (Master thesis
+ student assistantship (HIWI), Energy Micro uC PCBs, measurements, 3D-prints, end Jul. 2013),
H. Pekmezci (Semester projects + student assistantship (HIWI), Python driver for
Energy Micro uC, guided measurements with basic CI, end Dec. 2013),
(Please note, that the Energy Micro based activities have been suspended and replaced by a more reliable and capable Arduino/Orange based multi-sensor Lab-on-Spoon-system)



  Publications:  
      A. König, K. Thongpull, ”Lab-on-Spoon – a 3-D integrated hand-held multi-sensor system for low-cost food quality, safety, and processing monitoring in assisted-living systems”, Journal of Sensors and Sensor Systems, 4, 63-75, doi:10.5194/jsss-4-63-2015, 2015
     
      König, A.: Design and Application of Intelligent Integrated Impedance Spectroscopy Systems with Self-x Properties. In Book of Abstracts: Workshop Gesundheitszentrum Palatina, TU Kaiserslautern, 4. Jul., Kaiserslautern, 2013
     
     
Pekmezci, H., Los Arcos, D., König, A.: Embedded Impedance Spectroscopy for Lab-on-Spoon Realization in Living Assistance Systems and Intelligent Environments. In Abstract Book: International Workshop on Impedance Spectroscopy, IWIS 2013, (Student paper) pp. 41-42, Chemnitz, 25-27 Sept., 2013.
       
       
Andreas König, “Automated and Holistic Design of Intelligent and Distributed Integrated Sensor Systems with Self-x Properties for Applications in Vision, Robotics, Smart Environments, and Culinary Assistance Systems.” In Advances in Neuro-Information Processing, 15th Int. Conf. on Neural Information Processing of the Asia-Pacific Neural Network Assembly (ICONIP'08), pp. 69-70, November 25-28, Auckland, New Zealand, 2008, Revised Selected Papers, ISBN 978-3-642-02489-4, Springer, 2009.
       

       
Andreas König, “Automated and Holistic Design of Intelligent and Distributed Integrated Sensor Systems with Self-x Properties for Applications in Vision, Robotics, Smart Environments, and Culinary Assistance Systems.” Invited Talk, Int. Conf. on Neural Information Processing of the Asia-Pacific Neural Network Assembly (ICONIP'08), Book of Abstracts, pp. 69-70, November 25-28, Auckland, New Zealand, 2008