University of Oulu

Intelligent Systems Group (ISG)

Professor Juha Röning and Professor Jukka Riekki,
Computer Science and Engineering Laboratory, Department of Electrical and Information Engineering, University of Oulu

juha.roning(at), jukka.riekki(at)

Background and Mission

The main scientific objective of the basic research conducted in the Intelligent Systems Group is to generate new applicable knowledge on intelligent systems, and to generate positive societal impacts by applying this knowledge using scientifically plausible methods and state-of-the-art technology. We conduct research on spatial computing, collective intelligence, sensor networks, optimization of industrial processes, mobile robotics and cybernetics, human-computer interaction (HCI), human-robot interaction (HRI), computer networks, and security in complex information processing systems.

Our strategic research themes, whereby we aim to conduct world class basic research, are: 1) Safety and security on all levels of intelligent systems, 2) Data mining with a special focus on optimization of industrial processes and well-being, 3) Human-environment interaction with a special focus on novel physical user interfaces, robot-environment and human-robot interaction, 4) Mobile robotics and cybernetics with a special focus on spatial computing, adaptation, and field robotics, and 5) Sensor networks with a special focus on understanding behaviours and patterns in our everyday environment.

We have conducted basic research related to these research themes for over ten years. Our team consists of three professors, five postdoctoral researchers and 21 doctoral students. The annual external funding of the group is more than two million EUR in addition to the basic university funding. There have been 14 completed doctoral degrees from the group. From the research of the group, seven spin-off companies have been established so far: Codenomicon, Clarified Networks, Hearth Signal, Nose Laboratory, Nelilab, Atomia and Probot.

We co-operate with many international and domestic partners. In applied research, we are active in European projects. In addition, several joint projects are funded by the Finnish Funding Agency for Technology and Innovation (Tekes) and industry. We are a research partner in the Cooperative Traffic ICT SHOK, the Devices and Interoperability Ecosystem (DIEM) ICT SHOK and the Cloud Software Program ICT SHOK where we are responsible for the cloud security theme. In the II City project, we collaborate with the University of Lapland and Sonic Studio from Piteå, Sweden, and in the Pervasive Service Computing project with Shanghai Jiao Tong University from China. These projects are funded by the European Union (Interreg IVA North) and the Academy of Finland (the MOTIVE program). Also we have started new collaboration with Tokyo University of Agriculture and Technology in the project Interactive Context-aware System for Energy Efficient Living funded by the Academy of Finland. Prof. Jukka Riekki visited the Huazhong University of Science and Technology (HUST) in Wuhan, China and Shanghai University for one month in winter 2010 and gave invited talks during the visit.

We are active in the scientific community. For example, Prof. Juha Röning co-chaired numerous international workshops in the software security area, including the 6th Crisis Management Workshop (CRIM 2010) in Oulu, Finland. He acted as a member of the SAFECode International Board of Advisors, 2010 and as a chief judge in the European Land Robot Trial (Elrob 2010) in Hammelburg in May. Elrob is the biggest outdoor robot event in Europe, and participants are research facilities and companies that represent the state-of-the-art in Europe in this research area. Prof. Jukka Riekki co-chaired the Third Asia-Europe Workshop on Ubiquitous Computing 2010 (AEWUC’10) and the Pervasive 2010 Poster Session, both organized in May 2010 in Helsinki, Finland, and the UBI Challenge Workshop at Ubicomp 2010 organized in September 2010 in Copenhagen, Denmark. Several members of the group were also on the committees of international conferences.

Scientific Progress

Research on Prototyping: from a Smart Environment towards Remote Distributed Intelligence

Verification of the developed methods and models in prototypes is an important part of our research. To support this activity, we develop software and hardware architectures for smart environments. In addition to verification, prototypes speed up the commercialization of the research results. In prototyping, we have set and tackled the following objectives:

Developing Hardware And Software for Prototypes And Commercial Devices: The goal is to make designing of embedded systems faster and easier, while preserving the commercial applicability of the resulting device. This has been approached via the Embedded Object Concept (EOC). EOC is a concept that utilizes common object-oriented methods used in software by applying them to combined Lego-like software-hardware entities. This approach enables people without a comprehensive knowledge in electronics design to create new embedded systems. For experts, it shortens the design time of new embedded systems. The conceptual idea of embedded objects has been successfully implemented with the Atomi II framework, which provides the so called Atomi modules.

The EOC research has proceeded by making the concept easier to grasp for new users. This has happened by developing the documentation, tutorials and example cases that are to be presented to local companies and research facilities in workshops and lectures.

Research on Nanoresolution Tools for Interdisciplinary Applications: The project “Nanoresolution tools for interdisciplinary applications” (NRT) develops novel research, manipulation, and manufacturing methods for micro- and nanotechnology components and instruments. This means ever smaller objects, which are placed on surfaces not measurable with existing tools. A network of contacts and collaborators is used to select the objectives where commercial instruments are not yet available. One basis of the selection is a study of the commercial potential and value. Commercial potential affects the cost and selection of the components to be used thereby enabling the commercialization process. In order to improve the regional effect of the project, market studies are carried out.

The tools developed in NRT are based on a Scalable Modular Control Architecture (SMCA) that is being developed in the project, and the previously mentioned embedded objects. SMCA is a generalized modular architecture for both the device hardware and the control software on a PC. The architecture is extensible, scalable and portable, and it enables reuse of modules. It is hierarchically layered hybrid architecture to implement research equipment. SMCA enables swift changing of actuators, sensors and tools with minimal effort, thus being an ideal frame for various applications.

Implementation of SMCA is included in the framework within the project. The framework aims to improve the quality of the developed tools and enable fast prototyping. To achieve these goals, the framework includes reusable components that perform the tasks needed in all developed measurement tools such as data management, data visualization and configuration of devices. It also supports run-time specialization of measurement tasks to enable the use of the same hardware to perform different measurement tasks easily. Future work with SMCA will concentrate on applying the architecture in new areas, and the dissemination of the technologies for the local companies and research facilities.

Research on Mobile Robotics

At the end of 2010, the Academy of Finland granted funding for the Evolutionary Active Materials (EAM) project. EAM is a joint effort of the Computer Science and Engineering laboratory (CSE) and the Microelectronics and Materials Physics laboratories. The aim of the EAM project is to develop novel evolutionary computation (EC) based design methods for active and versatile materials and structures, and realise the first components through a novel holistic design process utilising constantly increasing computation power, the development of multi-physics simulators, and EC techniques such as genetic algorithms. One objective of the project is to accelerate the paradigm shift from the conventional design process for active materials towards a new goal driven holistic design process using the full potential of new materials, material combinations, and nonlinear dynamics of materials combined with complex geometries. The results of the project will be demonstrated first in the design of energy efficient piezoelectric actuators.

Global pose estimation is of fundamental importance in various mobile robot applications where the accurate pose of the robot in a given environment is needed in order to successfully perform application specific tasks. Today, numerous techniques exist for indoor pose estimation. The most common techniques are based on range sensors and computer vision and these techniques have been successfully applied in various environments and mobile robot applications. Despite the fact that good solutions already exist for indoor pose estimation, there is room for new techniques which can make the pose estimation more accurate when combined with other techniques, more robust against possible changes in environmental conditions, and perhaps provide more cost efficient solutions for pose estimation. We proposed a global indoor pose estimation technique utilizing the ambient magnetic field. The technique is based on the well known observation that magnetic field fluctuations commonly exist inside buildings. These fluctuations arise from both natural and man-made sources, such as steel and reinforced concrete structures, electric power systems, electric and electronic appliances, and industrial devices. Assuming the anomalies of the magnetic field inside a building are nearly static and they have sufficient local variability, the anomalies provide a unique magnetic landscape that can be utilized in global pose estimation. Our experiments suggest that the ambient magnetic field may remain sufficiently stable for longer periods of time giving support for pose estimation techniques utilizing the local fluctuations of the magnetic field.

Our work on indoor positioning techniques making use of the ambient magnetic field continued in 2010. A new magnetic field mapping instrument has been developed. The new mapping instrument can be used to collect large amounts of data about the three-dimensional structure of the magnetic field.

A key issue in magnetic field localization is the building of a map of the surrounding magnetic fields. In contrast to the Tuli project, where the mapping is performed manually, in the robotics domain this is referred to as a simultaneous localization and mapping (SLAM) problem. During the year 2010, we published two papers focusing on experimental studies in the magnetic field SLAM and on theoretical issues relating to near-optimal SLAM exploration. Results from our experimental studies (see Figure 1) showed that indoor magnetic fields contain enough spatial variation for accurate mapping. In near-optimal SLAM exploration studies, we developed new methods to autonomously model continuous spatial processes while having uncertain localization information. These methods can be utilized in time-efficient magnetic field SLAM.


Figure 1. Magnetic field SLAM in the CSE lobby. The upper left image presents robot’s trajectory using only raw odometry information from the robot’s wheel encoders. The upper right image presents a corrected trajectory where we have used rao-blackwellized particle filters in magnetic field SLAM.


As part of our SLAM work, we have proposed a sub-modular sensing quality function which extends studies from discrete sensor placement into an autonomous sampling scheme where sensing sites must be visited frequently. This is beneficial in a SLAM context where the sensing sites themselves bear uncertainties. Also in time critical applications, the modelling accuracy has to be balanced with the sensing time. Our SLAM studies are inspired by our research on indoor mobile robot localization, utilizing ambient magnetic fields which can be modelled by three orthogonal GPs providing a flexible framework for localization and SLAM. We have proved that for applications where sensing sites must be visited frequently, mutual information provides near-optimal solutions. We have extended this into metric probability spaces where sensing sites are treated as random variables. We have shown that with particle filter discretization, this framework can attain near-optimality.

Research on Human-Environment Interaction

Our current work on human-environment interaction focuses on physical user interfaces and human-robot interaction. In physical user interfaces, mobile terminals are used as physical objects rather than as traditional I/O devices. Our current studies concentrate on touch-based and gesture-based interaction: users interact with the local environment by touching objects with their mobile terminals and moving their terminals in the air. The touch-based user interfaces use NFC technology (i.e. RFID technology for mobile phones): an act of touching brings an RFID reader near an RFID tag and hence the data in the tag is read and delivered to the system. The objects that can be touched are advertised in the environment by RFID icons, graphical icons resembling the icons of the graphical user interfaces of computers and other user terminals. Gestures are recognized from the sensor data produced by mobile terminals’ acceleration sensors.

In 2010, we continued research on physical user interfaces in the DIEM ICT SHOK together with researchers from VTT, the University of Tampere, and the Tampere University of Technology. The DIEM ICT SHOK UI work package studies multimodal interaction for interactive spaces. Our main research area in this workspace is interaction using NFC technology. During 2010, our work package developed and tested the Ambient Conference Assistant, an application offering information to conference participants, supporting interaction between the participants, and between the participants and organizers as well. Our group developed for this application the Interactive Poster service (Figure 2). This service provides a physical, touch-based user interface for interacting with a poster. With their mobile phones, visitors and poster presenters touch graphical icons on the poster and on a control panel placed next to the poster. For example, by touching one of the icons, a visitor can play and control videos related to the poster in a display nearby. The visitor can also “pick” multimedia files from the poster, listen to the poster abstract, comment on the poster, and get the author’s business card. This application was deployed and tested in two conferences held in Finland during 2010: Pervasive 2010 and Mindtrek 2010.


Figure 2. The control panel for the Interactive Poster at Pervasive 2010.


We also developed an application supporting three to five-year-old children in their efforts in learning to read. This application was developed and tested in collaboration with the Department of Education at the University of Oulu. We equipped the nametags of children in a kindergarten with RFID tags and star icons advertising the positions of the tags. When a child touched a star on a nametag with an NFC phone, the phone said aloud the name in that nametag. The aim was that the children learn to associate the written name with the pronunciation of the name. In the second mode, the phone presented a name and the task was to touch the corresponding nametag. The pilot study showed improvement in the ability to recognize letters; suggesting that this type of application might be useful in the early stages of learning to read.

In the II City project we developed two novel applications in collaboration with Sonic Studio from Piteå, Sweden, and the University of Lapland. The Audio Guide directs people to selected targets using audio instructions (without speech); freeing the people to be present and interact with their local environment. The City Sound Generator is an artistic application representing the city’s state as audio. It embodies playful interaction, breaking some known usability principles to enhance its artistic impact. With this application we are studying new, playful ways of communicating information to the local community. The application produces music from the local temperature, amount of people near the display and the season. Users can change the values by melting the cubes with their hands and by turning the record and this way experience what different temperatures, amounts of people, and seasons sound like (Figure 3).


Figure 3. The City Sound Generator user interface for a large touch display.


Research on Sensor Networks

The research on sensor networks is targeted at understanding behaviours and patterns in our everyday environment. We are studying sensor data processing platforms, sensor data processing algorithms, lightweight knowledge representations suitable for sensor networks, and reasoning. In 2010, we continued developing the sensor data processing platform in the Cooperative Traffic ICT SHOK. This research was conducted in the Sensor Data Fusion and Applications (SDFA) project together with the University of Jyväskylä, the Tampere University of Technology, the Information Processing Department of the University of Oulu, and VTT. We completed a second prototype that utilizes data from built-in acceleration sensors and GPS modules of mobile phones to analyze the travel mode of a person, and the road condition when the phone is carried in a car (Figure 4). The data uploaded from the mobiles is stored in a database and processed in real time on a server, which also exposes an interface to deliver the results to clients. The server integrates several modules developed by the consortium, including map matching of GPS coordinates to digital map data, travel mode detection and road condition analysis. Traffic data is also streamed to the server from the national DigiTraffic web service. A client prototype was developed to visualize the processing results in real time on a map in a web browser.


Figure 4. An SDFA prototype collecting data with mobile phones attached to vehicles and analyzing road conditions from the collected data.


In the UBI program, we continued the development of a platform on collecting, processing and storing sensor data from a heterogeneous wireless sensor network. The first prototype for measuring the energy consumption in households, using wireless sensor nodes over IEEE 802.15.4 networks with 6LoWPAN protocol stack, was completed in early 2010 together with MediaTeam Oulu. In total, 11 households in Oulu participated in the first pilot testing and the initial results confirm the usefulness of the system.

In the Pervasive Service Computing project, we continue the development of service-oriented infrastructure unobtrusively supporting people in their daily activities. Sensor networks are an important part of this infrastructure; providing measurements about the environment and the people. This work involves collaboration with MediaTeam Oulu and Shanghai Jiaotong University. Several prototypes were developed to demonstrate the capabilities of dynamic service composition and coordination. Moreover, we are moving towards deploying service compositions in the Cloud. In 2010, we concentrated on developing infrastructure for context-based service creation and composition. This infrastructure facilitates the building of services by utilizing rules and context information that is obtained through processing data produced by sensor networks.

Research on Human-Robot Interaction

Two ongoing projects, the Academy funded AFHRI (Affective-Human-Robot interaction) and the European Regional Development Fund (ERDF) funded Minotaurus combines the theoretical and practical aspects of creating service robotics of the future. Both projects co-operated with the Machine Vision Group (MVG). Human-Machine communication and robot’s operation in real world conditions are major topics. Key aspects are putting research results into practice, and integrating them as a part of the operation of the robot. The main challenges are representation of information and adapting a set of algorithms to the current state in the surrounding environment. Human friendliness of the outward appearance of the robot is driving the design of the robot.

Interaction with a machine has been studied from various perspectives. The combination of machine vision, speech recognition and synthesizing, touch and touchless interactions, along with the robot’s operation in the environment requires a software platform that processes, distributes and stores information efficiently. Real-time Linux based operating system services along with general purpose representation for information (called Markers) have been developed to support integration of the algorithms. Marker- representation is used for representing the output of sensor processing algorithms, for representing a model of environment and obstacles around the robot, and information related to the robot’s tasks.

In the Minotaurus project, speech and natural language processing have been studied to be able build a natural interface between a robot and human. At the first stage, the research has been concentrating on speech recognition where a small vocabulary model, based on a CMU Sphinx engine, is used to recognize greetings, commands, and questions. Furthermore, a simple language understanding model, based on predefined rules and regular expressions, is implemented to be able detect current speech action to form responses in human-robot dialogue. Responses are based on the robot’s internal knowledge as well as external knowledge from the web. Besides speech recognition and dialogue, speaker orientation is estimated using multiple microphones and phase correlation analysis to be able to detect a current active speaker and her orientation. In the future, work will be concentrating on large-scale speech recognition and language processing based on statistical machine learning to be able to implement more natural communication techniques between a human and robot without relying on a set of fixed rules.

The project includes also instrumentation of a mobile robot that can operate among people. The basic mobile platform for the robot is Segway RMP200, a two parallel wheeled balancing robot (Figure 5). It is equipped with onboard computers, a display, and several sensors like cameras, lasers and inclinometers. One of the main sensors is the Microsoft Kinect, a cheap depth and colour camera with orientation sensors. It is used for creating 3D model of the surrounding environment, and detecting human pose.


Figure 5. A test setup for the Human-Robot Interaction.


Research on Data Mining

Data mining research challenges are divided into three mutually supportive categories; the research on algorithms producing knowledge, the software running the algorithms and the knowledge bases storing the acquired knowledge. When put together, these three categories form a strong combination which can be applied to virtually any phenomenon where data can be processed into knowledge.

The research on algorithms has been focused on advancing methods for time series analysis, variance modelling and novelty detection. Special interest was directed towards establishing data driven methods for these areas, where the exact shape and nature of the observed data can be used to characterize the phenomena under study. In software research, the implementation of the first version of new software architecture for running the algorithms reached a successful conclusion. The architecture presents the algorithms as information generating logical devices to which the information measuring physical devices connect. A suitable combination of logical and physical devices can thereafter be used to form a data mining software application. In knowledge base research, the general ideas behind the role of knowledge bases were established, and the implementation of the first version of a concrete knowledge base was carried through to a successful end.

During 2010, the combination of the three data mining focus areas (algorithms, software and knowledge bases) proved its efficiency. Prototypes were implemented utilizing all the focus areas while the interfaces developed enabled modifications in the three areas individually. For example, inside the prototypes, the algorithms in use could be modified without extra work in software implementation or effect on the knowledge base structure.

The actual research has focused on four projects, XPRESS (2007–2010), MIDAS (2008–2010), PISKET (2009–2011) and prob2E (2010-2012).

Computer Vision / Augmented Reality: In the online car door assembly guidance system presented in Figure 6, computer vision and machine learning methods are applied to track the pose of the car door and to recognize if particular door parts are installed. Model-based tracking where a CAD model of the door, along with edge-based image data is applied to estimate the door position and orientation, as well as, to segment different door parts. Support Vector Machines (SVM) are trained to recognize the existence of door parts with a view to detecting the correct work phases. In SVM, a kernel for structured data is applied to image patches of local interest points. These methods are capable of detecting installed car door parts automatically for monitoring assembly and to recognize work phases completed correctly. In Figure 6, the car door frame and part segmentation images can be seen before and after the worker has placed the third part during the assembly scenario. The eight rectangles on the door frame pictures are the segmented part areas. The colour of the rectangle is red if the part has not been recognized and green if the part has been successfully identified. The blue dots on the part area images are the detected interest points used in the part recognition. The recognition prototype system was successfully demonstrated to project partners.


Figure 6. Car door frame and part segmentation images before and after the worker has placed the third part during the assembly scenario.



Sensor signal analysis with synchronized video: Human motion recognition using wearable sensors is a growing field. Commercial applications are starting to emerge in gaming, health, sports and consumer electronics. The process of developing a human motion recognition system, and designing the recognition intelligence, includes collecting and labelling the data. Labels presenting the ground truth about the classes of the data sequences can be used to train the recognition system and evaluate the recognition results it produces.

In 2010, ISG completed the MIDAS-project (Methods for Innovative Data Analysis Solutions, funded by the European Structural Fund Programmes, Polar Electro and Ruukki) in which we examined how the labelling process could be improved. Video based labelling methods are widely used, because the videos provide easily perceivable meta information. In the MIDAS-project, the video analysis approach is developed further by synchronizing the video with the inertial data that is labelled. The technical feasibility of the developed synchronized video analysis approach for human inertial data was proved by instantiating it as a labelling tool. In an experiment, it was found to be completely appropriate for labelling more complex human inertial data. The synchronized video analysis approach showed better results in labelling quality compared with other labelling approaches. Interviews and focus groups pointed out that data being labelled with high accuracy using a video based approach is considered to be valuable. On the other hand, the video based approaches are still criticized for the amount of effort. The labelling tool developed is now utilized in other research projects in the university and in industrial R&D by Polar Electro.

Gesture/Activity Recognition: In assembly guidance, the wearable inertial measurement units can be used in two different ways: using gestures as control commands, and monitoring the work tasks performed by the worker. In some situations, gesture recognition is a good option for handling human-computer interaction because it enables natural interaction and no input devices, such as a keyboard and a mouse, are needed. In this scenario, the gestures are recognized by matching the shapes of wrist movement to predefined time series templates (Figure 7). The second approach uses wearable sensors for activity monitoring to guide workers in real-time. The recognition is done using acceleration and angle speed information, and the results have been promising. Both approaches were successfully demonstrated to project partners in summer 2010, and the results were also presented at international conferences.


Figure 7. Gestures to control a user interface and the corresponding tri-axial acceleration.


The XPRESS project (2007-2011), financed from the 6th Framework Programme of the European Commission, has entered its final integration phase. The overall goal of the project is to develop a new kind of intelligent manufacturing system composed of hierarchically organized autonomous units called manufactrons. Two ISG members participated in a week-long integration meeting at the premises of the Fraunhofer Institute in Stuttgart, Germany. The objective of the meeting was to bring together the contributions of several project partners into a functional demonstration system that covers every level of the manufactron hierarchy. ISG contributed two major components to the system: a human-machine interface and a knowledge system.

The human-machine interface (HMI) gives guidance and feedback to a factory worker executing a manufacturing task such as welding or manual assembly. It integrates the results of several areas of research: gesture and activity recognition, machine vision, augmented reality and user interfaces. Besides a presentation of the task execution protocol and a post-execution quality summary, a worker using the HMI receives real-time instructions adapted to monitoring information representing the progress and current status of the work. In an assembly context, for instance, the interface can show the worker which part to pick up next and where to attach it. The capabilities of the HMI were demonstrated at the integration meeting, and later in the year at a meeting in Oulu with a representative of Centro Ricerche Fiat, an XPRESS partner representing the automotive industry.

All manufactrons in the XPRESS framework receive their assignments in the form of task description documents (TDD). Once received, a TDD must first be transformed into a method description document (MDD), which specifies to the manufactron how the task should be executed. This operation is called task-to-method transformation (T2MT) and performing it is the responsibility of the knowledge system (KS). To solve the T2MT problem, previous results on software architectures and knowledge bases were combined with new research on knowledge representation and retrieval. Metadata annotations and a specially devised three-tier query scheme are used to match methods with tasks. The resulting KS architecture was used to implement T2MT in two XPRESS cases, developed in collaboration with two German XPRESS partners: the Augsburg-based company KUKA and the Karlsruhe University of Applied Sciences. A report on this work was presented at an IEEE international conference.

Work on the new application platform of the ISG, the Device-Based Software Architecture (DBSA), also continued, although the focus shifted in 2010 from extending the architecture with new features to adapting it to the requirements of different application domains. Both the human-machine interface and the knowledge system developed in the XPRESS project were built on modified versions of the DBSA platform. The results of this work show the architecture to be a solid foundation for various data mining, signal processing and knowledge management applications. DBSA was introduced to the academic community in a paper presented at an ACM international conference.

The future of living and housing: During 2010 a new project for researching the future of living has been prepared. The project title is “Interactive Context-aware System for Energy Efficient Living” (INCA) and it is funded by the Academy of Finland. The goal for the project is to develop an interactive and context-aware feedback and control system for rationalizing energy efficient living. The system itself collects personalized data from the environment and offers information about consuming habits for the individuals. With the help of the information, it is possible to advise people on more energy efficient living and to give recommendations for avoiding unnecessary energy consumption.

The research in the INCA project is performed jointly with Associate Professor Kaori Fujinami from the Tokyo University of Agriculture and Technology. From the data mining perspective, the group will be concentrating on developing methods to discover instant and long-term habits and anomalies in water and electricity consumptions associated with the users’ context information such as identity, location, and activities, automatically learned and detected from the user’s behavioural patterns measured by visual and ambient sensors attached to an intelligent home environment. Figure 8 presents a context-aware system to be developed in the project for energy efficient living.


Figure 8. The context-aware system for energy efficient living.


Activation for physical activity and health: In the CallUp&Go (Communal, ICT based Activation for Physical Activity and Health funded by Tekes) the goal is ICT-based activation on the physical activity and health among conscription aged men in Finland. In other words, the aim is to persuade young men to be physically more active, and the persuasion is done through gamification. A gamified computer-platform will combine health information, game technology, tailoring and activity monitors. In the project, different combinations of sensor and game technologies, social media and their interfaces will be studied and an interactive social media prototype will be developed. The prototype consists of a video game, ICT systems, content services, and performance and activity monitors, both existing ones and those that are still to be constructed. The developed game is not going to be a typical video game, to proceed in the game subjects need to exercise. The novelty in this is that unlike in current exercise-based games, like Wii Sports, Wii Fit, Blobo Fun & Fit, EyeToy Play Sports, subjects do not exercise in front of TV or monitor, instead they need to go out and exercise, see Figure 9. This way the subjects gain points in the game based on their physical activity and they can proceed to the next level. The ICT platform will be set up using multidisciplinary collaboration by consulting local enterprises with related top expertise. The ISG’s role is to construct a new innovative computer-platform for the project and develop methods to monitor physical activity. The game developed in CallUp&Go will be tested by a limited number of real users. CallUp&Go is a part of a bigger project called MOPO.


Figure 9. The game: the player earns points based on his physical activity. After collecting enough points he proceeds to the next level.


Data mining methods for steel industry applications: The Intelligent Systems Group has long experience in advancing data mining methodologies in the steel industry, and is a member of the Centre for Advanced Steels Research – CASR, which is one of the interdisciplinary umbrella organizations of the University of Oulu. In 2010, advanced data analysis solutions for the needs of the steel industry were developed in three active projects.

In 2010, ISG finished a long lasting project in which a generic system for the semi-automatic maintenance of statistical prediction models was developed. The system developed makes use of production data to keep up-to-date applications that employ statistical predictive models to control, optimize and plan the production. The system analyzes the production data and monitors the performance of models. When a need for model update actions is detected, the system serves as a tool and graphical user interface which process engineers can easily use to generate a new model candidate, validate it and deliver it to the applications that optimize, plan and control the production process. The model candidates and earlier models are stored in a model archive and the system works as a diagnostic tool that helps to compare different models and to select the best model for production use. The system developed is currently applied to keep up-to-date prediction models utilized in the planning of steel plate products and product variations.

Currently, the system developed provides support for the modelling of distribution properties of a continuously measured response using neural networks or generalized linear models. The component-wise implementation allows for extending the algorithm library to support other model types and maintenance paradigms. The system can be a cost-efficient method of keeping prediction models up-to-date in a wide variety of industrial applications.

The quality of steel plates can be improved by optimizing the rolling pass schedule. An accurate prediction for rolling temperatures and loads of the rolling is necessary. The statistical prediction of rolling loads and temperatures is demanding and application of advanced functional data analysis methods is needed. In the PISKET project (Improving Pass Scheduling Calculation Taking into Account Flatness) prediction models are developed to predict the rolling temperatures and loads. In 2010, a model for predicting the rolling temperatures has been developed. The major challenge has been in the selection of a feature set that describes the deformation history of the earlier pass schedule, but is invariant to the pass number. The models developed will be utilized in the Ruukki steel plate mill.

Production efficiency with probability predictions: In 2010, ISG has launched a new project (prob2E) to develop further a probability prediction approach to develop and utilize statistical models. The aim of the project is to develop strategies for applying disqualification probability predictions in the optimization of production processes. Statistical methods that focus on accurate modelling of the dependence of deviation and tail distribution shape from input variables are employed as the data analysis tools. The expected result of the project is know-how on developing probability prediction models and on applying them in the optimization of the manufacturing industry processes. The main hypothesis is that decision making that is based on predicted disqualification risk decreases disqualifications, and decreases variation in the product quality characteristics. The hypothesis and related economic benefits are validated by applying the developed probability prediction procedure into three different manufacturing industry processes (cheese manufacturing, steel strip rolling and the tempering of steel bars). The project is funded by Tekes, Ruukki, Ovako, and Valio.

Research mobility: Dr. Perttu Laurinen was on a one year research visit (March 2009 - March 2010) at the Idiap Research Institute, affiliated with EPFL, in Switzerland. The purpose of the visit was work on developing new methods for labelling and identifying shapes of interest from time series data. The integration of multiple data sources (like video, audio and accelerometer signals) was of especial interest. The results can be used to recognize and label events from a numerical time series with the help of synchronized audiovisual recording. The visit was part of the MIDAS-project.

Eija Haapalainen was on a one year research visit (Asla-Fulbright scholarship) at Carnegie Mellon University and has been working on a study that aims to measure the cognitive load of a person, based on physiological measurements. In the study, a large number of different biosignals are examined, and the best measure or set of measures for assessing the cognitive load is searched for. The signals being measured include pupil size, heart rate, skin temperature and conductance as well as electrical brain activity. These signals are recorded while the test subject solves elementary cognitive tasks of different levels of difficulty. The prospective results of the study have broad applications in systems reasoning about human attention. The goal of the project is to develop methods for assessing the round-the-clock activity level of a person and to measure energy expenditure based on this information.

Research on Software Security

Within the Intelligent Systems Group, the Oulu University Secure Programming Group (OUSPG) has continued research in security and safety in intelligent systems. Security and safety challenges in intelligent systems are threefold: increasing complexity leads to unforeseeable failure modes, quality is not the priority and awareness is lacking. We have approached the challenges from these three directions in our research.

Complexity - Model Inference and Pattern Recognition: We work under the premises of unmanageable growth in software and system complexity and emergent behaviour (unanticipated, not designed) having a major role in any modern non-trivial system. We have worked on natural science approaches to understanding artificial information processing systems. We have developed and applied model inference and pattern recognition to both content and causality of signalling between different parts of systems.

Quality - Building Security In: Software quality problems, wide impact vulnerabilities, phishing, botnets, criminal enterprise have proven that software and system security is not just an add-on despite past focus of the security industry. Instead, security, trust, dependability and privacy have to be considered over the whole life-cycle of the system and software development from requirements all the way to operations and maintenance. This is furthermore emphasized by the fact that large intelligent systems are emergent and do not follow traditional development life-cycle. Building security in not only makes us safer and secure, but also improves overall system quality and development efficiency. Security and safety are transformed from inhibitors to enablers. We have developed and applied black-box testing methods to set quantitative robustness criteria. International recognition of the Secure Development Life Cycle has provided us with a way to map our research on different security aspects.

Awareness - Vulnerability Life Cycle: Intelligent systems are born with security flaws and vulnerabilities, new ones are introduced, old ones are eliminated (Eronen 2009, Askola 2008). Any deployment of system components comes in generations that have different sets of vulnerabilities. Technical, social, political and economic factors all affect this process. We have developed and applied processes to handling the vulnerability life-cycle. This work has been adopted in critical infrastructure protection. Awareness in vulnerabilities and processes to handle them all increase the survivability of emergent intelligent systems for developers, users and society.

These research goals are reached through a number of research activities.

Secure Software Development Lifecycle in Cloud Computing as part of the Cloud Software project we approach all three goals by researching practical ways of building security into complex Cloud Computing services, from the design phase to actual operational use.

Situational awareness of modern information networks and applications in the MOVERTI project where we approach information systems from all three viewpoints, and have developed methods for combining measurements from network traffic, vulnerability feeds and scans, and socio-economic networks to gain situational awareness of modern information systems.

Identification of protocol gene: This research, PROTOS-GENOME, approaches the problems of complexity and quality by developing tools and techniques for reverse-engineering and identification of protocols based on using protocol genes - the basic building blocks of protocols. The approach is to use techniques developed for bioinformatics and artificial intelligence. Samples of protocols and file formats are used to infer structure from the data. This structural information can then be used to effectively create large numbers of test cases for this protocol. In 2010, the project further developed the existing methodology resulting in improvements in efficacy and discovering a number of vulnerabilities in web browsers.

Hash function security research is a part of cryptography, where hash functions and their properties are studied. Hash functions play a major role in many modern communication protocols, and are at the moment a very hot topic since NIST (National Institute for Standards in Technology) proposed a competition for a new and more secure hash function standard, SHA-3. This competition vets the one best algorithm from 64 different propositions. At OUSPG, the study of hash functions has led to some practical results, but mostly concerning theoretical advances in the field of hash function security. The results gained form this research and the cryptographic expertise are then applied to the practical information security work in OUSPG.

Exploitation of Results

The results of our research were applied to real-world problems in many projects, often in collaboration with industrial and other partners. Some examples of exploitation are described below.

The Intelligent Systems Group utilizes a robotics laboratory and a pressure-sensitive floor (EMFi material) installed in our laboratory as part of a smart living room. Other equipment includes a home theatre, two degree-of-freedom active cameras, four mobile robots and one manipulator, a WLAN network, and various mobile devices. Our aim is to gradually build a versatile infrastructure that offers various generic services for pervasive applications. Naturally, this kind of environment enables realistic experiments that lead to a better understanding of such applications.

The generic system for the semi-automatic maintenance of statistical prediction models is now in use for product planning at the Ruukki steel plate mill. The system enables us to keep up-to-date these models which are important for ensuring that the products fulfil the customer needs. As new products and production methods are continuously developed and novel process settings are taken into use, data from previously unseen conditions occur, and thus there is a constant need to update the prediction models. Another major reason that causes a need for model updates is process drift. The system developed generates semi-automatically new model versions when the newest data makes it possible to improve the prediction accuracy. Each model version is a dynamic link library (DLL) that can be directly plugged in to the on-line applications that actually utilize the models. The system will give economic benefits because improved prediction accuracy in the product planning decreases production costs and the costs needed to maintain the good prediction accuracy will now be small.

The embedded objects implemented according to the Atomi II Framework specification are called Atomi objects. Several different Atomi objects have been created for real life tests. The Atomi objects have been used in many projects, and they have proved to be very usable.

The Atomi objects are being used both in pure research projects, and in projects that aim ultimately at commercial products. The Atomi objects have been applied to telepresence robots, nanoscale manipulation and measurement technology, a hand held medical device, and several robot applications (Figures 10 and 11).


Figure 10. The Atomi modules have been used in several research projects. They have made prototyping of new innovations easy and fast.


Figure 11. Micromanipulation platform includes a PC, a probe, an actuator device and a controller device.


The SMCA and the framework has been declared open source, and the source is freely available at This enables their use in both academic and commercial research and development. Many applications have been built with the developed techniques, and most of them have been disseminated in different journals and conferences. Currently new applications are being developed.

The latest work on both SMCA and Atomi objects has concentrated on improving the documentation, wiki pages and tutorials for making their use easier. This work aims to spread the technologies for the use of local companies and research facilities in order to enable them to gain benefit in research and development work and prototyping. At the same time this aims towards a large scale use test in order to improve the technologies further.

During the reporting year, the group continued utilizing outdoor robotic systems. Development and utilization of Mörri, a multipurpose, high performance robot platform continued. More focus was put on perception in natural conditions, representation of detections, knowledge, and environment model of operating environment. The software architecture further developed the earlier work on Property Service Architecture and the Marker concept as general purpose representation was further developed.

Future Goals

We will continue to strengthen our long term research and researcher training. We will also continuously seek opportunities for the exploitation of our research results by collaborating with partners from industry and other research institutions on national and international research programs and projects. The group is a founding member of the European Robotic Network of Excellence (EURON). The group is a contract member of EURON II which was approved for the EU’s FP6 as a Network of Excellence.

We will strengthen our international research co-operation. Within the last years, the group has created collaboration projects with Japanese researchers. In 2010, we continued the collaboration and developed it further through a memorandum of understanding with Keio University. Our aim is to continue and strengthen the collaboration in 2011. We will continue the collaboration with Sonic Studio from Sweden and Shanghai Jiao-Tong University from China as well. With the University of Tianjin, China we have a joint project in which methods and a system will be developed for vision-based navigation of Autonomous Ground Vehicles, which utilize an omni-directional camera system as the vision sensor. The aim is to provide a robust platform that can be utilized in both indoor and outdoor AGV (Autonomous Ground Vehicles) applications. This co-operation will continue.

In the USA, we will continue to co-operate with the Human-Computer Interaction Institute in Carnegie Mellon University. A doctoral student from ISG is currently on a one year research visit to the institute and is co-operating with Assistant Professor Anind K. Dey. The research is on human modelling in the area of human machine interaction. Shorter research visits to European partners in EU-funded projects are also planned.

In 2011, the aim is to utilize more widely the know-how from sensor technology and data mining. New application areas will be studied including rehabilitation, exercise motivation and energy efficiency in households, and the benefits of our expertise will be highlighted to the actors in the areas.


professors & doctors


graduate students






person years


External Funding



Academy of Finland

323 000

Ministry of Education and Culture

206 000


872 000

domestic private

393 000


377 000


2 171 000

Selected Publications

Davidyuk O, Georgantas N, Issarny V & Riekki J (2010) MEDUSA: Middleware for end-user composition of ubiquitous applications handbook of research on ambient intelligence and smart environments: trends and perspectives. IGI Global (Eds. Mastrogiovanni F & Chong NY).

Davidyuk O, Sanchez I & Riekki J (2010) CADEAU: Supporting autonomic and user-controlled application composition in ubiquitous environments pervasive computing and communications design and deployment: technologies, trends, and applications. IGI Global (Ed. Malatras A).

Gilman E, Sanchez I, Saloranta T & Riekki J (2010) Reasoning for smart space application: comparing three reasoning engines CLIPS, Jess and Win-Prolog. Proc. 10th IEEE International Conference on Computer and Information Technology (CIT 2010).

Haapalainen E, Kim S, Forlizzi J & Dey A (2010) Psycho-physiological measures for assessing cognitive load. 12th ACM International Conference on Ubiquitous Computing (Ubicomp 2010), September 26-29, 301-310.

Halunen K, Kortelainen J & Kortelainen T (2010) Combinatorial multicollision attacks on generalized iterated hash functions. Eighth Australasian Information Security Conference (AISC 2010), ACS, Brisbane, Australia, 105:86-93 (Eds. Boyd C & Susilo W).

Halunen K & Röning J (2010) Preimage attacks against variants of very smooth hash advances in information and computer security. Springer Berlin / Heidelberg, 6434: 251-266 (Eds. Echizen I, Kunihiro, N & Sasaki R).

Hosio S, Jurmu M, Kukka H, Riekki J & Ojala T (2010) Supporting distributed private and public user interfaces in urban environments. Proc. 11th Workshop on Mobile Computing Systems and Applications (HotMobile 2010), Annapolis, MD, USA, 25-30. (Best Presentation Award).

Hosio S, Kukka H & Riekki J (2010) Social surroundings: bridging the virtual and physical divide. IEEE MultiMedia 17(2): 26-33.

Suutala J, Fujinami K & Röning J (2010) Persons tracking with Gaussian process joint particle filtering. IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2010), Kittilä, Finland, August 29 - September 1, 160-165.

Aubert J, Schaberreiter T, Incoul C, Khadraoui D & Gateau B (2010) Risk-based methodology for real-time security monitoring of interdependent services in critical infrastructures. IEEE Computer Society, 262-267.

Juutilainen I & Röning J (2010) How to compare interpretatively different models for the conditional variance. Journal of Applied Statistics 37(5): 983-998.

Kantola J, Perttunen M, Leppänen T, Collin J & Riekki J (2010) Context awareness for GPS-enabled phones. Proceedings of ITM2010, Institute of Navigation, January 25-27.

Kemppainen A, Haverinen J, Vallivaara I & Röning J (2010) Near-optimal SLAM exploration in Gaussian processes. 2010 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, 7-13.

Kortelainen J, Halunen, K & Kortelainen T (2010) Multicollision attacks and generalized iterated hash functions. Journal of Mathematical Cryptology.

Kätevä J, Laurinen P, Rautio T, Suutala J, Tuovinen L, Röning J (2010) DBSA - A device-based software architecture for data mining. Proceedings of the 2010 ACM Symposium on Applied Computing (SAC 2010), 2273-2280.

Leppänen T, Perttunen M, Riekki J & Kaipio P (2010) Sensor network architecture for cooperative traffic applications. Proceedings of 6th International Conference on Wireless and Mobile Communications, September 20-25, 400-403.

Mikkola H, Rahikkala A, Kumpulainen K & Riekki J (2010) ActiveAquarium – virtual fish reflecting your physical activity. Proceedings of the 20th Annual Conference of the European Teacher Education Network (ETEN) (Eds. Kumpulainen K & Toom A), 7-18.

Thepvilojanapong N, Konomi S, Yura J, Iwamoto T, Pirttikangas S, Ishida Y, Iwai M, Tobe Y, Yokoyama H, Nakazawa J & Tokuda H (2010) Aquiba: An energy-efficient mobile sensing system for collaborative human probes. DASFAA 2010, Part II (Ed. Kitakawa H), Springer-Verlag, Berlin / Heidelberg, April, LNCS 5982: 420-423.

Pietikäinen P, Karjalainen K, Roning J & Eronen J (2010) Socio-technical security assessment of a VoIP system. IEEE Computer Society, Los Alamitos, CA, USA, 141-147.

Siirtola P, Laurinen P, Koskimäki H & Röning J (2010) Recognizing user interface control gestures from acceleration data using time series templates. The 7th International Conference on Informatics in Control, Automation and Robotics (ICINCO 2010), Funchal, Madeira-Portugal, June 15-18, 176-182.

Pirttikangas S, Tobe Y & Thepvilojanapong N (2010) Smart environments for occupancy sensing and services. Chapter in Handbook of Ambient Intelligence and Smart Environments (AISE) (Eds. Nakashima H, Aghajan H & Augusto JC), Springer, 825-849.

Riekki J, Sanchez I & Pyykkönen M (2010) Remote control for pervasive services. International Journal of Autonomous and Adaptive Communications Systems 3(1): 39-58.

Stenlund L, Riski K, Seppä J, Pudas M, Vähäsöyrinki M, Tuhkanen V & Röning J (2010) Traceable characterization of a bending millimetre scale cantilever for nanoforce sensing. Measurement Science and Technology 21(7).

Su X, Fucci D & Riekki J (2010) A framework to enable two-layer inference for ambient intelligence. International Symposium on Ambient Intelligence (ISAmI 2010) (Eds. Augusto JC et al), Advances in Soft Computing 72, Springer Berlin / Heidelberg, 29-36.

Su X & Riekki J (2010) Transferring ontologies between mobile devices and knowledge-based systems. Proc. 8th IEEE/IFIP International Conference on Embedded and Ubiquitous Computing (EUC 2010), December 11-13, Hongkong, China, 127-135.

Su X & Riekki J (2010) Bridging the gap between semantic web and networked sensors: A position paper. 3nd International Workshop on Semantic Sensor Networks (SSN 2010), with 2010 International Semantic Web Conference (ISWC), November 7-11, Shanghai, China.

Kortelainen T, Kortelainen J & Halunen K (2010) Variants of multicollision attacks on iterated hash functions, Inscrypt 2010, Springer-Verlag.

Tamminen S, Juutilainen I & Röning J (2010) Modelling of Charpy V test rejection probability. Ironmaking & Steelmaking 37(1): 35-40.

Tamminen S, Juutilainen I & Röning J (2010) Quantile regression model for impact toughness estimation. 10th Industrial Conference on Data Mining (ICDM 2010), Berlin, Germany, July 12-14, 263-276.

Tokola T, Halunen K & Röning J (2010) Benefits and arrangements of Bachelor’s thesis in information engineering. Koli Calling 2009.

Tuovinen L, Talus T, Koponen E, Laurinen P & Röning J (2010) A task-to-method transformation system for self-configuration in manufacturing. Proceedings of the 8th IEEE International Conference on Industrial Informatics (INDIN 2010), 245-252.

Turunen T, Pyssysalo T & Röning J (2010) Mobile AR requirements for location based social networks. The International Journal of Virtual Reality, 67-78.

Vallivaara I, Haverinen J, Kemppainen A & Röning J (2010) Simultaneous localization and mapping using ambient magnetic field. 2010 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, Salt Lake City, Utah, USA, 14-19.

Huikari V, Koskimäki H, Siirtola P & Röning J (2010) User-independent activity recognition for industrial assembly lines - feature vs. instance selection. Proc. 5th International Conference on Pervasive Computing and Applications (ICPCA 2010), Maribor, Slovenia, December 1-3.

Zhou J & Riekki J (2010) Context-aware pervasive service composition. IEEE Proceedings of 1st Int. Conference on Intelligent Systems, Modelling and Simulation, ISMS2010, January 27-29, Liverpool, UK, 437-442.

Zhou J, Gilman E, Palola J, Riekki J, Ylianttila M & Sun J (2010) Context-aware pervasive service composition and its implementation. Personal and Ubiquitous Computing, 1-13.

Zhou J, Gilman E, Riekki J, Rautiainen M & Ylianttila M (2010) Ontology-driven pervasive service composition for everyday life. ISoLA, Part I (Eds. Margaria T & Steffen B), LNCS 6415, Springer-Verlag, 375–389.

Zhou J, Gilman E, Ylianttila M & Riekki J (2010) Pervasive service computing: visions and challenges. Proc. 10th IEEE Int. Conference on Computer and Information Technology (CIT2010), June 29 - July 01, Bradford, UK, 1335-1339.

Collin J, Kaipio P, Kutila M, Leppänen T, Markkula J, Mazhelis O, Perttunen M, Pirttikangas S & Riekki J (2010) Älyä liikenteeseen. Prosessori (11): 22-25.