Machine Vision and Intelligent Systems

Professor Matti Pietikäinen and Professor Juha Röning,
Information Processing Laboratory and Computer Engineering Laboratory,
Department of Electrical Engineering, University of Oulu
mkp
ee.oulu.fi, jjree.oulu.fi
http://www.ee.oulu.fi/mvis

 


Background and Mission

Machine vision is a continuously growing area of research dealing with processing and analyzing of image data. It plays a key role in the development of intelligent systems. The principal goal of research on intelligent systems is to develop machines that have the ability to perceive, reason, move, and learn from experiences.

The mission of the MVIS group is to carry out leading edge long term research on machine vision and intelligent systems technology, with an aim of bringing the results of the research close to applications.

The main areas of research are:

The research on image analysis concentrates on problems in texture analysis, color and face image analysis, document image analysis, and adaptive systems for color-texture inspection. The focus of image sequence analysis is in tracking and motion estimation, video indexing, camera calibration, and interventional MRI. The research on intelligent systems concentrates on context-aware mobile systems, intelligent service robots, neural network methodology for signal analysis, and analysis of biomedical ECG and EEG signals.

The research in the above areas provides the basis for original applied research in various areas, including industrial automation, mobile communications, multimedia, intelligent interfaces and medical engineering.

The group co-operates with many international and domestic partners. In applied research, the group has played central roles in European Esprit projects and several joint projects funded by the National Technology Agency (Tekes) and industry. As a result of our co-operation with the respected Fraunhofer IITB Institute (Karlsruhe, Germany), a one year postdoctoral visit to Fraunhofer was made by Dr. Hannu Kauppinen. The group is also active in the scientific community. For example, in 2000 the group organized an Infotech Workshop on Real-Time Image Sequence Analysis, Prof. Röning was elected a Fellow of the SPIE, and Prof. Pietikäinen was invited to the editorial boards of the IEEE Transactions on Pattern Analysis and Machine Intelligence and Pattern Recognition journals.

The activities of the MVIS group are led by professors Matti Pietikäinen (Director), Juha Röning (Associate Director), Olli Silvén and Tapio Seppänen.

Scientific Progress

Image analysis

Image texture analysis has been a topic of intensive research for about three decades. During the past few years, the group has developed theoretically and computationally very simple nonparametric approaches to texture analysis based on Local Binary Patterns (LBP) and signed gray level differences. In 2000, LBP-type operators for multiresolution gray scale and rotation invariant texture classification were further investigated and research on the theoretical foundations of the rotation-invariant LBP was begun. Combined use of color and texture in classification was studied, with an application in the classification of parquet defects. The development of a Matlab toolbox for texture analysis using our operators was in progress.

Techniques for color-based face tracking in varying illumination conditions have been investigated. Our approach makes use of a chromaticity constraint called skin locus to select pixel candidates for adapting the face color distribution to the color shift caused by illumination change. The chromaticity constraint is a function in the chromaticity space which defines possible object colors in a certain color temperature range. The technique has been tested on several videos and has been shown to be successful in situations in which non-adaptive methods and adaptive methods based on spatial constraint can totally fail. New research activity concerning face recognition in varying environments was begun.

In our research on greenhouse plant vitality it was found that key indicators can be determined from the reflectance spectra. These include the water content, photosynthesis activity, and cues on stress. The measurement instrument used is an imaging spectrograph with 14-bit dynamic range that enables measuring phenomena that previously were detected only via fluorescence spectrometry. The current goal is to find spectral indicators to discriminate between directly illuminated leaves and those in a shadow, and between central and peripheral regions of leaves. This information is needed to increase the reliability of vitality measurements.

In wood surface inspection, a non-supervised training principle has been demonstrated to result in an essential improvement of defect detection and classification accuracy. At the lower level, the approach relies on our previous color-based inspection techniques, and the use of new texture methods is being investigated. The development of an easy to use software package for demonstrating the results is in progress. It is planned to be distributed to system integrators, end-users, and researchers in the field.

In document image analysis, an experimental system for automatic ground-truth generation was developed in order to evaluate the skew-tolerance of page segmentation methods. It is currently publicly available via the Internet. Research on extracting text information from document images with complex colored and textured backgrounds was begun. Simple texture-based methods using edge information were under development, providing a very promising performance in the preliminary experiments.

Image sequence analysis

A novel method for detecting and tracking humans based on particle filters has been developed. The method utilizes motion as a cue for segmenting objects in a scene into foreground or background. The camera can be either stationary or it can be moving. The basic idea is to estimate multiple motions from image sequences including also the camera motion. Unlike most of the tracking methods, the new approach does not require any explicit image features such edges or contours. Due to this property, the method could be suitable for many practical applications such as visual surveillance and robot navigation.

Examples of recent work in human tracking utilizing different modalities (left: color, middle: motion, right: shape).

In video coding, transform domain motion estimation techniques have been investigated. The main goal is to develop solutions that are computationally efficient and require low power consumption in hardware implementations. Several transform schemes have been tested using software implementations. The results indicate that there seems to be much potential in the new approach. Furthermore, a content based rate-control technique has been developed for video coding purposes. This technique is especially well suited for low bit-rate mobile applications with wireless communication channels.

The group has developed SOM (self-organizing map) based tools for video classification and indexing. A graphical user interface for training and visualizing the layered SOM structure is implemented with Java. The underlying video processing has been designed to extract color, motion and audio features from the video bitstream without completely decoding it. In this manner, the solution becomes computationally more efficient, which is important when dealing with large video databases.

The Matlab toolbox we have developed for geometric camera calibration was revised. A new and more powerful algorithm has been created which directly utilizes the geometry of the known control points. A variant of the toolbox has been implemented for geometric calibration of the line-scan cameras. Also a method for detecting and estimating the pose of the electrical components on a printed circuit board has been developed to be used in conjunction with the calibration toolbox.

Intelligent systems

The research on context-aware systems concentrates on software architecture, 3-D sensing technology, machine vision, context recognition and control methods. Distributed software architecture is being developed for mobile context-aware systems. The architecture offers well defined and reusable interfaces for different resources like sensors, actuators, computing devices and user interfaces. The location and implementation of a software component is transparent for the rest of the system.

The architecture is being applied both to systems controlling mobile service robots and to ubiquitous systems serving mobile users. In 2000, the work continued on basic robot resources like motion control, range sensors, dead-reckoning and vision. Work on a system serving a mobile user was begun; the basic architecture was improved and a software component offering an augmented reality user interface was developed.

A robust color tracking system was developed for a mobile robot, utilizing the expertise of the group in color vision. The system is capable of tracking objects of different colors in various lighting conditions. With this tracking system, a mobile system can focus its attention on the colored objects in the scene, and track a person based on skin color as well. For 3-D environment modeling, an experimental system is being developed which combines 3-D information provided by structured lighting with 2-D information provided by intensity (color) images. During this reporting period, a graphical user interface was developed. The next step is to mount the system in a mobile robot.

Our recent work on control methods has focused on evolutionary methods. Novel methods are being studied to construct mobile robot controllers using artificial neural networks. The methods use a dynamic topology of neurons for adapting to the environmental changes. Although a lot of work has to be done, the preliminary results are promising. One of the main research challenges is the scalability of the methods to create an intelligent robot. To support this study, the development of a novel robot simulator has begun. To be able to utilize the group's strong background in machine vision, a virtual color camera was implemented as one of the robot's sensors.

Previous work on situated control architecture was continued. Our control system for playing simulated soccer acted as a qualifying competitor for the RoboCup'99. A student group participated in the simulation league of the fourth world cup of robotic soccer, RoboCup'00, held in Melbourne, Australia. In this competition the games of the simulation league were shown to the audience using the 3D visualization software developed by the student group.

In context recognition, a data set was collected from the actions of a user, as well as from the environment the user was in. The data set was preprocessed and different methods were tested for detecting the context of the user. Analysis was done by statistical signal processing techniques and neural networks. Neural networks were applied in other areas of signal analysis as well; in developing spatially and temporally adaptive learning techniques. Such techniques are needed as a static and deterministic system cannot cope with all the temporally changing situations in, for example, industrial environments or notably dynamic environments, such as the human physiological system.

New data mining methods are being developed for steel plant quality control. Data mining contains three closely related workflows: data preprocessing for data mining study, modeling of temperatures and dimensions after the finishing mill, and furnace control data study. The data preprocessing aims at finding variables and data items that make a significant contribution to defective coils. This study is carried out using statistical analysis, correlation study and the self-organizing maps. The modeling study strives to predict end product quality parameters with neural networks. The furnace control study surveys the dependencies between furnace control variables and develops new adaptive furnace control algorithms. The analysis is done using graph theory and neural networks.

Research into ECG signals has been conducted. The main goal in the project is to study heart rate variability of patients and healthy people in order to develop methods for automatic diagnosis and prognosis for clinical use. The medical part of the research is performed in the Oulu University Hospital while our group is responsible for signal analysis algorithms. The work is highly international, including universities from Europe and the USA. During 2000, an invited review paper on heart rate variability methods was published in an international scientific journal, and many journal papers applying our algorithms to clinical data were published.

New research on EEG signals has been launched in 1999. The goal of this study is to develop methods for automatic assessment of the depth of anaesthesia and physical reaction to pain in clinical operations. This goal is approached by performing adaptive segmentation and classification of evoked responses and phase synchronization of cortical cell populations. During 2000, five conference papers and two journal articles were published. The medical part of this research is conducted at the Oulu University Hospital while our group is responsible for algorithm development. The work involves other Finnish universities and industrial companies.

Exploitation of Results

The results of our research were applied to real-word problems in many projects, often in collaboration with industrial and other partners. Some examples of exploitation are described below.

Inspection methods developed by the group have been used in the wood grading system being developed for VTT Building Technology, and the non-supervised learning and user-interface approach developed by the group is used in the 3-D steel surface inspection systems made by Thermo Radiometrie.

The expertise of the group has been used in the development of an MPEG-4 codec solution by Hantro Products. The codec is intended to be integrated into wireless communications devices.

The European Esprit project called "Intraoperative real-time visualization and instrument tracking for MRI" (IRVIT) carried out in 1998-99 demonstrated the feasibility of an interactive MRI intervention system, which features real-time visual feedback. A follow-up project entitled "Advanced Minimally Invasive Therapy Using MRI" (AMIT) targeting clinical trials was begun during the reporting period.

The group's expertise in robotics was applied in developing a mobile robot for domestic help. A teleoperated robot serves as the remote eyes of the elderly and those who take care of them. During the reporting period, the main task was to develop teleoperation capabilities for the robot. A system for controlling the robot remotely over the Internet was implemented. The first successful usability tests were performed. The work was carried out as a part of a project the goal of which is to create new product and service concepts for helping the elderly and disabled in their everyday lives. As a spin-off of this work, the development of a wheeled walking aid offering services based on modern information technology was started during 2000. The services envisaged for such a system include video calls, safety monitoring and controlling the environment.

Future Goals

In December 2000, the MVIS group was chosen by the Academy of Finland for the second round in the competition for national centre-of-excellence status and funding. For the final application submitted in February 2001, the group prepared its long-term research and operation plan covering the period 2002-2007. The group will continue to strengthen its long term research and researcher training. We will also continuously seek opportunities for exploitation of our research results by collaborating with partners from industry and other research institutions in national and international research programs and projects.

Personnel

professors & doctors 11
graduate students 25
others 28
total 64
person years 42

External Funding

 Source

FIM

Academy of Finland  1 466 000
Ministry of Education 1 381 000
Tekes 3 280 000
domestic private 1 018 000
EU + other international  567 000
total 7 712 000

 

Doctoral Theses

Kauppinen H (1999) Development of a color machine vision method for wood surface inspection. Acta Universitatis Ouluensis C141. (Thesis was defended in 1999, but formally accepted in 2000.)

Selected Publications

Haverinen J & J Röning J (2000) A 3-D scanner capturing range and color: design and calibration. Proc. Vision Interface (VI 2000), May 14-17, Montréal, Quebec, Canada, 378-381.

Heikkilä J (2000) Geometric camera calibration using circular control points. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(10): 1066-1077.

Heikkilä J, Sangi P & Silven O (2000) Camera motion estimation from non-stationary scenes using EM-based motion segmentation. Proc. 15th International Conference on Pattern Recognition, September 3-8, Barcelona, Spain, 1:370-374.

Marszalec E, Martinkauppi B, Soriano M & Pietikäinen M (2000) Physics-based face database for color research. Journal of Electronic Imaging 9(1): 32-38.

Mäkikallio TH, Tulppo MP, Seppänen T & Huikuri HV (2000) Analysis of nonlinear heart rate dynamics in cardiac arrhytmias,. Invited paper for Hertzschrittmachertherapie und Elektrophysiologie, 11:131-138.

Ojala T, Pietikäinen M & Mäenpää T (2000) Gray scale and rotation invariant texture classification with Local Binary Patterns. Proc. 6th European Conference on Computer Vision (ECCV 2000), June 26-July 1, Dublin, Ireland, 1:404-420.

Pietikäinen M, ed. (2000) Texture Analysis in Machine Vision. In: Series in Machine Perception and Artificial Intelligence - Vol. 40, World Scientific Publishing Co., 261 p.

Pietikäinen M, Ojala T & Xu Z (2000) Rotation-invariant texture classification using feature distributions. Pattern Recognition 33(1): 43-52.

Sauvola J & Pietikäinen M (2000) Adaptive document image binarization. Pattern Recognition 33(2): 225-236.

Soriano M, Martinkauppi B, Huovinen S & Laaksonen M (2000) Skin detection in video under changing illumination conditions. Proc. 15th International Conference on Pattern Recognition, September 3-8, Barcelona, Spain, 1:839-842.

Tamminen S, Pirttikangas S & Röning J (2000) The self-organizing maps in adaptive health monitoring. Proc. International Joint Conference of Neural Networks (IJCNN2000), July 24-27, Como, Italy, 4:259-266.

Tuulonen A, Alanko H, Hyytinen P, Veijola J, Seppänen T & Airaksinen PJ (2000) Digital imaging and microtexture analysis of the nerve fiber layer. Journal of Claucoma 9(1): 5-9.

Vähä P, Röning, J, Heikkilä T, Kerva J & Rehu J (2000) Mechatronic systems techniques for intelligent paper roll manipulator systems. In: Mechatronics Systems Techniques and Applications: Electromechanical Systems Vol. 4 (Ed. CT Leondes), Gordon and Breach Science Publishers, 231-297.