Infotech Oulu
Infotech Oulu LogoUniversity of Oulu
Main Page
Lecture Series
Courses
 

Machine Vision and Intelligent Systems

Professor Matti Pietikäinen and Professor Juha Röning,
Information Processing Laboratory and Computer Engineering Laboratory,
Department of Electrical and Information Engineering, University of Oulu
mkpee.oulu.fi, jjree.oulu.fi
http://www.ee.oulu.fi/mvis


Background and Mission

Machine vision is a continuously growing area of research dealing with processing and analyzing of image data. It plays a key role in the development of intelligent systems. The principal goal of research on intelligent systems is to develop machines that have the ability to perceive, reason, move, and learn from experiences.

The mission of the MVIS group is to carry out leading edge, long term research on machine vision and intelligent systems technology, with an aim of bringing the results of the research close to applications.

The research on machine vision focuses on methods and systems for analyzing images and image sequences. The aim is to develop new approaches, which provide the leading performance, are robust with respect to changes in environment and are computationally efficient. The research on intelligent systems concentrates on mobile and context-aware systems, data mining methods and secure programming. Our aim is to increase systematically our methodological understanding and to develop components for an intelligent environment that gives versatile services for its inhabitants.

The research in the above areas provides the basis for original applied research in various areas, including visual inspection, mobile communications, multimedia, intelligent interfaces and medical engineering.

The group co-operates with many international and domestic partners. In applied research, the group has played central roles in European Esprit projects and several joint projects funded by the National Technology Agency (Tekes) and industry. The group and its members are active in the scientific community. For example, in 2002 the group co-organized with the MediaTeam group an international Infotech conference, MUM 2002 (1st International Conference on Mobile and Ubiquitous Multimedia, 11-13 December); Prof. Matti Pietikäinen served on the editorial boards of two major journals; and several members of the group were on the committees of international conferences. Olga Kouropteva won the award for the best Finnish Master’s thesis in pattern recognition in 2001 (Pattern Recognition Society of Finland).

The activities of the MVIS group are led by professors Matti Pietikäinen (Director), Juha Röning (Associate Director), Olli Silvén, Janne Heikkilä, Jukka Riekki and Tapio Seppänen.

 

Scientific Progress

Machine vision

Image texture analysis is an important fundamental problem in machine vision. During the past few years, the group has developed theoretically and computationally simple but very efficient nonparametric approaches to texture analysis based on Local Binary Patterns (LBP) and signed gray level differences. In 2002, the joint use of color and texture in classification was studied, suggesting separate processing of color and pattern information. Outex, the new large texture image database created for empirical evaluation of texture analysis algorithms was further developed (http://www.outex.oulu.fi). The performances of the MPEG-7 texture descriptors and the LBP method were empirically evaluated in the retrieval of 319 Outex textures. LBP clearly outperformed the MPEG-7 descriptors both with respect to the retrieval accuracy and the computation time. Multiscale versions of the LBP operator were studied. A novel method, based on cellular automata, was proposed for this purpose. Research on view-based recognition of 3D textured surfaces was begun. Our method provided the leading performance for CUReT textures and very promising results for outdoor scene images. A method for real-time surface inspection according to texture was developed, as well as a methodology for optimizing color and texture features for real-time inspection applications.

Techniques for color-based skin detection and face tracking under drastic illumination changes were further investigated. Our approach is based using a chromaticity constraint called a skin locus to select pixel candidates. This method has been applied to both image sequences and still images. In the first case, the skin locus is used for adapting the face color distribution. The skin locus performed very well in a comparison with other state-of-the-art methods that was carried out for both sequential and still images. Research concerning the distributions of all 8-bit RGB values in different color spaces was performed. These color spaces were also tested for the use of the dichromatic model and mixtures of light sources and skin spectra.

Research on the recently proposed unsupervised learning and dimensionality reduction method, locally linear embedding (LLE), was continued. A nonparametric supervised version of LLE was developed. In cooperation with the Pattern Recognition Group of the University of Delft, the methods of supervised LLE independently developed by both groups were unified in a common framework and applied to a number of benchmark data sets. The sensitivity of the original unsupervised LLE to outliers was also studied and a robust extension to tackle this problem was proposed.

The visualization-based approach for surface inspection was further investigated. The method combines an intuitive user interface with an unsupervised classifier. The idea is to map the unlabeled training data into two dimensions with some dimensionality reduction method, like the self organizing map (SOM). Then the training data is visualized on the 2D map where the user determines the boundaries between classes. The approach does not require labeling of separate samples, which is often an inconsistent and error prone task reducing the accuracy achieved, but the decision of the class boundaries is based on observing the data set as a whole. With this approach, the retraining for changing material or imaging conditions involves simply moving the decision boundaries on the 2D map. The approach is also computationally attractive for real-time applications.

Research on visualization-based paper characterization according to texture was in progress. A new methodology based on combining LBP-based texture feature extraction with unsupervised training and classification using SOM was developed. The preliminary results were excellent, reducing the classification error to under a fortieth part of that obtained with previous methods. In addition to the excellent classification accuracy, the method also offers a self-intuitive user interface and a synthetic view to the inspected data.

 

Paper characterization according to texture.

 

Our research on grain size distribution determination is based on using a non-segmenting approach and unsupervised classification methods. Image grain size information is extracted with efficient texture features like LBP and spectral information. Two and three fraction mixtures have been tested to evaluate the performance of the features. The purpose is to develop techniques for on-line grain size measurement in industrial processes.

In the research on plant vitality measurement, the SOM classification method, with several spectral features as input, was used to discriminate between plants with different vitality characteristics. Other dimensionality reduction methods, like LLE, are planned to be considered in future research.

The video processing research was focused on motion estimation and video stabilization. The main objective in motion estimation was to develop a hardware module for performing computationally the most intensive part of video coding more efficiently without loss of accuracy. As a result, a low-power solution for real-time block motion estimation for CIF-sized frames was designed that utilizes a novel transform based approach. Another result in motion estimation research was an improved method for eliminating poor candidate vectors in spatial domain full search block motion estimation with the sum of squared differences (SSD) error criterion.

In video stabilization, a new framework was developed. This framework is mainly targeted for mobile devices with video encoding capabilities. The feasibility of this approach has been successfully demonstrated with a software implementation and a large set of test material. The results have shown that the video quality has improved significantly while the video bit-rate has dropped substantially due to the attenuation of camera vibrations. The main advantages of the approach developed are the optimal separation of the intended and unintended motions, and the small overhead needed when stabilization is used together with a video encoder.

In transform domain pattern analysis, the properties of the Multi-scale autoconvolution (MSA) transform developed in our group were studied. In practical applications, the transform requires some preprocessing of the images and selection of the appropriate set of transform coefficients. This research is still in progress. Other possible image transforms have also been investigated to achieve necessary invariances against geometric transformations, and to solve the alignment problem between pairs of images under various geometric distortions.

 

Pattern recognition using multi-scale autoconvolution.

 

New research on machine vision for sensing and understanding human actions was begun. Our project proposal on this topic was accepted for the Proactive Computing program of the Academy of Finland. The general objective is to investigate the capabilities of machine vision in proactive computing and to develop solutions needed for building emerging applications. An embedded vision module is planned to be developed using vision software architecture based on event analysis. It will be provided with machine vision skills for sensing and identifying humans and their actions in varying conditions and with capabilities to wirelessly communicate with other units. The applicability of the approach will be shown in an intelligent room equipped with multiple cooperating vision units, demonstrating complex capabilities built on simple modules. A wireless local area network connected to the Internet is used for communication and database access.

 

An enbedded vision module.

 

Intelligent systems

The research on intelligent systems concentrates on mobile and context-aware systems, data mining methods and secure programming. These are applied in context-aware mobile systems, intelligent service robots and analysis of biomedical ECG and EEG signals.

The research on context-aware systems focuses on software architecture, 3D sensing technology, machine vision, context recognition and control methods. Distributed software architecture is being developed for mobile context-aware systems. The architecture offers well defined and reusable interfaces for different resources like sensors, actuators, computing devices and user interfaces. The location and implementation of a software component is transparent for the rest of the system. The architecture was further developed. A common interface was developed for all robotics resources. Furthermore, methods to dynamically construct state machines for robot controllers were studied.

The architecture is being applied both to systems controlling mobile service robots and to ubiquitous systems serving mobile users. In 2002, development of a general agent-based architecture, Genie of the Net, for managing services on behalf of the user has progressed. This work has been done in co-operation with VTT Electronics. The work on mobile robots continued with basic robot resources like motion control and vision. This has been realized in the Finnish Academy funded project "Robots Serving Humans". The goal of this project is to develop methods and components for the next generation intelligent service robots. In this project, we emphasize the role of robust senses (especially color vision) and learning (self-organization). The main application, and thus the test-bed for the developed methods, is intelligent telepresence, where the robot is a semi autonomous agent providing a convenient way to monitor and access remote environments for a human user.

As part of the future environment, we envisage home robots that can be used for different purposes, such as telepresence. In our research laboratory, we are developing a robot whose main purpose is to serve as a telepresence body, but which can also perform home aid functions. The goal is to develop new methods and devices that will improve the current telepresence equipment by providing a self-guiding, intuitive, and very easy-to-use user interface and by creating a realistic feeling of being present in the remote location. During the reporting period, work on speech recognition over a videoconference connection and robot sensor visualization via the video stream of the videoconference has been done. Our techniques allow the robot to be remotely controlled with generally available videoconference software applications such as Microsoft Netmeeting.

Evolutionary methods to evolve neural controllers for a mobile robot were studied. An approach called the Stochastic Evolutionary Neuron Migration Process (SENMP) was used to evolve neural control structures for mobile robots and to gain new insights into adaptation in neural networks. The neural structures were able to solve real problems in perception and control. The feasibility of the approach was demonstrated by evolving robust navigation behavior for a simulated and a real mobile robot.

A graph theoretical approach was studied for creating dynamic robot formations. The approach uses a leader-follower scheme, allowing multiple robots to move in formations and change the formations when necessary due to environmental conditions.

A new control architecture called Bender has been developed for mobile robots. It is developed as a general architecture for robots in our laboratory. Bender is a real-time, dynamic, and modular control architecture. It is the most recent member in the family of control architectures developed in our laboratory, using features from SAMBA and COCOA and building on top of the distributed architecture discussed above. Bender was primarily designed to be a control architecture allowing easy implementation of new methods and algorithms. It also aims to be a general framework for different mobile robots, as it should work in robots of different kinds and sizes. Dynamic modules provide a flexible way for creating different control systems from fully reactive to deliberative ones. The modules are usually created when the system starts, but they can also be created and connected to other modules dynamically during execution for a special purpose. The modules receive and send information to other modules in a unified format, called markers. Markers are XML formatted structures that refer to physical objects or abstract things, and they are also used for creating an environment model core.

Bender is a part of the distributed software architecture providing a way to distribute modules into a network, as some of the modules can be started in other machines and accessed via CORBA interfaces implemented for all robotics resources, ranging from color cameras to motion control systems. Furthermore, a dynamic state machine architecture was developed that allows the user to create and modify online a complete state machine, utilizing all distributed robotics resources by simply writing and executing an XML formatted description of the state machine.

In the context-aware research, the group is also participating the in CAPNET research program. The purpose of CAPNET (Context-aware Pervasive Networking) research program is to create a foundation for new information and communications technologies for business in the field. Three research groups from the University of Oulu participate in the research program: MediaTeam from the Information Processing Laboratory, Intelligent Systems from the Computer Engineering Laboratory, and Human Computer Interaction from the Department of Information Processing Science.

The focus of the CAPNET program is on context-aware mobile technologies for ubiquitous computing. These are technologies, which allow communication anytime and anywhere, with any kind of terminal device, automatically taking into account the characteristics of the network and the terminal. The Intelligent Systems Group participates in the CAPNET program by studying context-awareness and developing software architectures. Context-awareness research concentrates on identifying the user’s routines utilizing location information acquired through WLAN, and a pressure sensitive floor on the premises of the Tietotalo Robotics Laboratory.

 

A pressure sensitive floor in the Robotics Laboratory.

 

New data mining methods were developed for steel plant quality control and spot welding processes. The steel plant quality control study has continued. The two-year VALTA project in co-operation with Rautaruukki Steel in Raahe contains three application areas: furnace control study, predicting the scaling of the steel strips and breaking strength modeling for steel plates. The research project is partly a continuation of the earlier HiTech Steel project.

The furnace control study has progressed to intelligent furnace control system development. During this period, the usability of prior information was studied and the development of the steel slab temperature predicting system was continued. The impact of temperatures of the ceiling and heating time to the final slab temperature was modeled. The earlier neural network model predicted the temperature of the steel slab after the roughing mill, but during this research period the model has been improved to also predict the earlier temperatures.

 

A slab emerging from a furnace.

 

The predicting of the steel strip scale defects started in June. In the preprocessing state, the relationships of the variables were visualized with scatter plots and self-organizing maps. The association rules were also used. The variables causing the scaling seem to interact widely. The research continues. A predicting model for the breaking strength of steel plates was developed. Linear, log-linear and neural network models were used. The planning model for the breaking strength of steel plates has been improved and the first Excel based table models are in the testing phase at Rautaruukki Steel in Raahe. The research will continue analyzing the deviation of the breaking strength, which will be estimated with heteroscedastic regression models.

 

The deviation of the strength varies with the process variables.

 

In the research on applying data mining methods to steel industry measurements, we have also co-operated with Arizona State University where one of our researchers spent a year as a visiting researcher.

SIOUX (Intelligent system for dynamic online quality control of spot welding processes for cross-sectoral applications) is a two-year EU-project developing a real time quality control system of resistance spot welding joints. The aim of the system is to replace time and cost intensive destructive tests by using soft computing methods for quality estimation. The benefit is that the approach does not require any additional sensors, like for example ultrasound based methods, as it uses on-line process data combined with knowledge gathered from previous experiments.

 

An example of a welding experiment where the quality control developed in the SIOUX project would be needed.

 

We are building a dynamic database containing measurements from welding tests and developing data mining methods for analyzing the measurement data to achieve better welding quality. During the last period the database has been partly set up and methods for pre-processing and analyzing the measurements have been developed.

The system is an application example of one of our research topics - a smart archive. A smart archive is a system combining expertise in databases and data mining methods. The goal is to implement and bring this know-how to processes that produce large amounts of data and demand adaptive, continuous and largely autonomous supervision.

Research into ECG signals has been conducted. The main goal in the project is to study heart rate variability of patients and healthy people in order to develop methods for automatic diagnosis and prognosis for clinical use. The medical part of the research is performed in the Oulu University Hospital, while our group is responsible for signal analysis algorithms. The work is highly international, including universities from Europe and the USA. One graduate thesis (M.Sc.) was finished in 2002 on T-wave morphology research. Computer software for analyzing T-wave dynamics was written and several patient data analyses were conducted. The software is presently in extensive use at the University Hospital of Oulu. One conference paper was submitted on this result. Heart rate variability was investigated and as a result, one journal paper on heart rate variability methods was published, and two journal papers were revision-submitted for evaluation.

Research on the methods for automatic assessment of the depth of anesthesia and physical reaction to pain was continued. A new multi-sensor approach to this is being developed. New patient material was measured and analysis software was written. The work is performed in cooperation with the Oulu University Hospital and the Lappeenranta University Hospital while our team is responsible for algorithm and software development. During 2002, two journal articles were published.

A new research topic of muscle symphatetic nervous activity (MSNA) signal processing was established. The aim is to study the basic relations between heart rate, blood pressure, breathing and MSNA signals. One M.Sc. thesis was completed on the topic in which computer software was developed for this purpose. Ph.D. work has already been started. Two conference papers were submitted for evaluation in 2002.

Inside the Intelligent Systems Group, the Oulu University Secure Programming Group (OUSPG) has kept its focus on implementation level security issues and software security testing. Software implementation may introduce potential for unanticipated and undesired program behaviour, e.g. an intruder can exploit the vulnerability to compromise the computer system. The group has researched different approaches to testing implementations of protocols using black-box (i.e. functional) testing methods (PROTOS). The recent discoveries of the group have raised awareness of security issues globally, e.g. test results in SNMP (Simple Network Management Protocol, http://www.cert.org/advisories/CA-2002-03.html). As a result of PROTOS research, a spin-off company, Codenomicon Ltd., was founded in 2001 to develop commercial software testing products (http://www.codenomicon.com).

Interdisciplinary research was carried out on ethical and communication issues related to software vulnerabilities. A vulnerability communication survey attracted over 150 international participants, with over 50 entries from software vendors and prior to the late public outbreak of debate on ethics of software security a journal article on the very topic was prepared.

 

Exploitation of Results

The results of our research were applied to real-world problems in many projects, often in collaboration with industrial and other partners. Some examples of exploitation are described below.

In the wood inspection area, several board edgers using the visual training approach have been installed at sawmills by an industrial partner. These have become fully operational within hours of installation unlike earlier systems which required at least a several weeks training period.

The European Esprit project called "Intraoperative real-time visualization and instrument tracking for MRI" (IRVIT) carried out in 1998-99 demonstrated the feasibility of an interactive MRI intervention system which features real-time visual feedback. A follow-up project "Advanced Minimally Invasive Therapy Using MRI" (AMIT) targeting clinical trials was in progress during the reporting period. A start-up company has been founded by researchers, building on the foundation laid by the project.

The research in video sequence processing and analysis has contributed to recent video codec products that have been integrated into mobile communications devices. Although the key role of the researchers was mostly in enabling rapid product development, the result is a convincing demonstration of the benefits of long term research.

The group’s expertise in robotics was applied in developing a mobile robot for domestic help. A teleoperated robot serves as the remote eyes of the elderly and those who take care of them. During the reporting period, the main task was to develop teleoperation capabilities for the robot. A voice controlled service robot was successfully demonstrated. The purpose of the robot is to assist elderly people in their homes and provide a communication link to the health care personnel. A design project was launched with the University of Lapland to further develop the appearance of the robot, and make it suitable for various applications and research studies regarding human-robot interaction.

 

A mobile robot for domestic help.

 

The development of a wheeled walking aid offering services based on modern information technology has continued during 2002. The project is in co-operation with the University of Lapland. The services envisaged for such a system include video calls, safety monitoring and controlling the environment.

During 2002 significant practical penetration for developed theories and procedures were achieved in the software industry and security community. First, home grown testing methodology was applied in a SNMP test-suite, where the OUSPG, CERT/CC and tens of vendors worked together on a model developed in OUSPG to address serious and widespread problems. Second, in co-operation with the MediaTeam, a test-suite was developed for the Session Initiation Protocol (SIP). This was introduced in a novel way and also well received in a global SIP Interoperability Meeting of protocol pioneers and vendors (October 2002).

 

Future Goals

We will continue to strengthen our long term research and researcher training. We will also continuously seek opportunities for exploitation of our research results by collaborating with partners from industry and other research institutions in national and international research programs and projects. In order to further sharpen its focus, the MVIS group was recently divided into two separate but closely cooperating groups: Machine Vision (http://www.ee.oulu.fi/mvg) and Intelligent Systems (http://www.ee.oulu.fi/research/isg). Both of these were accepted to Infotech Oulu for the period 2003-2005.

 

Personnel

professors & doctors

12

graduate students

29

others

40

total

81

person years

56

 

External Funding

Source

EUR
EUR

Academy of Finland 

355 000

Ministry of Education

340 000

Tekes

408 000

domestic private

256 000

EU + other international 

203 000

total

1 562 000

  

Doctoral Theses

Martinkauppi B (2002) Face colour under varying illumination - analysis and applications. Acta Universitatis Ouluensis C 171, 104 p + App.

Repo T (2002) Modeling of structured 3-D environments from monocular image sequences. Acta Universitatis Ouluensis C 174, 133 p.

 

Selected Publications

Machine vision

Hadid A, Pietikäinen M & Martinkauppi B (2002) Color-based face detection using skin locus model and hierarchical filtering. Proc. 16th International Conference on Pattern Recognition, August 11-15, Quebec City, Canada, 4:196-200.

Heikkilä J (2002) Multi-scale autoconvolution for affine invariant pattern recognition. Proc. 16th International Conference on Pattern Recognition, August 11-15, Quebec City, Canada, 1:119-122.

Kouropteva O, Okun O & Pietikäinen M (2002) Selection of the optimal parameter value for the locally linear embedding algorithm. Proc. of the 1st International Conference on Fuzzy Systems and Knowledge Discovery (FSKD’02), November 18-22, Singapore, 359-363.

Niskanen M, Kauppinen H & Silvén O (2002) Real-time aspects of SOM-based visual surface inspection. Proc. SPIE Vol. 4664 Machine Vision Applications in Industrial Inspection X, January 21-22, San Jose, CA, USA, 123-134.

Ojala R, Blanco Sequeiros R, Klemola R, Vahala E, Jyrkinen L & Tervonen O (2002) MR-guided bone biopsy: preliminary report of a new guiding method. Journal of Magnetic Resonance Imaging 15:82-86.

Ojala T, Mäenpää T, Pietikäinen M, Viertola J, Kyllönen J & Huovinen S (2002) Outex - New framework for empirical evaluation of texture analysis algorithms. Proc. 16th International Conference on Pattern Recognition, Quebec, Canada, 1:701-706.

Ojala T, Pietikäinen M & Mäenpää T (2002) Multiresolution gray-scale and rotation invariant texture classification with Local Binary Patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(7):971-987.

Toivonen T, Heikkilä J & Silven O (2002) A new algorithm for fast full search block motion estimation based on number theoretic transforms. Proc. 9th International Workshop on Systems, Signals and Image Processing, November 7-8, Manchester, United Kingdom, 90-94.

Intelligent systems

Elsilä U & Röning J (2002) Knowledge discovery in steel industry measurements. Proc. STarting Artificial Intelligence Researchers Symposium (STAIRS 2002), July, Lyon, France, 197-206.

Haverinen J & Röning J (2002) Adaptation through a stochastic evolutionary neuron migration process (SENMP). Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems 2002, Sep 30 - Oct 4, 2002, Lausanne, Switzerland, 1008-1013.

Jäntti V, Mustola S, Huotari A-M & Koskinen M (2002) The importance of looking at the EEG when presenting univariate variables to describe it. British Journal of Anaesthesia 88(5): 739, discussion 739-40.

CERT® Advisory CA-2002-03 Multiple Vulnerabilities in Many Implementations of the Simple Network Management Protocol (SNMP) http://www.cert.org/advisories/CA-2002-03.html

Mäkikallio TH, Huikuri HV, Myerburg RJ, Seppänen T, Kloosterman M, Interian A Jr., Castellanos A & Mitrani RD (2002) Differences in the activation patterns between sustained and self-terminating episodes of human ventricular fibrillation. Annals of Medicine 34: 130-135.

Riekki J, Huhtinen J, Ala-Siuru P, Alahuhta P, Kaartinen J & Röning J (2002) Genie of the net, an agent platform for managing services on behalf of the user. Computer Communications Journal, Special Issue on Ubiquitous Computing, Elsevier Science.

Särkelä M, Mustola S, Seppänen T, Koskinen M, Lepola P, Suominen K, Juvonen T, Tolvanen-Laakso H & Jäntti V (2002) Automatic analysis and monitoring of burst suppression in anaesthesia and ICU. Journal of Clinical Monitoring and Computing 17: 125-134.

Takanen A, Raasakka, Laakso M & Röning J (2002) Ethical and legal aspects in software security: The illusion of professionalism in development and testing. Ethics and Information Technology.