Bielefeld University

Bielefeld University (UNIBI) was founded in 1969 with an explicit research assignment and a mission to provide high-quality teaching. A strategic research focus of the university is interactive intelligent systems, it’s two central research institutes  Research Institute for Cognition and Robotics (CoR-Lab) and Center for Cognitive Interaction Technology (CITEC). Founded in 2007, both institutes are dedicated to research in the key areas of interaction technology, robotics and artificial intelligence with a focus on transfer to applications in socio-technical systems. Bielefeld University is located in Northern Germany, where almost 200 associated researchers work in both basic and applied research on intelligent systems, very often in collaboration with industry, the public sector, or civil society.

https://vedliot.eu/

The Smart Mirror serves as an interface between residents and a smart environment. The solution is an interactive mirror, as well as a central IoT control point for connecting to various smart applications that are integrated in a home, in a corporation, or in a city. With high computational power, UNIBI’s Smart Mirror uses deep learning algorithms and distributed AI to perform recognition of objects, faces, gestures, and speech. The solution thereby adds a range of smart features to its environment, and automates and performs action commands provided by the user.

The Smart Mirror is a consumer IoT product, which is applicable for use both in private and public smart environments. The solution is currently a research use case and is therefore not yet commercialised.

Illustration 1: The Smart Mirror relies on a modular hardware platform that can be used in a range of IoT applications, which is being developed as part of the VEDLIoT project.

IoT incentive

The incentive for the researchers to develop the Smart Mirror was an urge to tap into the accelerating smart environment trend and explore advanced technological functionalities that were not possible in the past. By building a hardware platform that caters to the continuous development in smart technology, the institute expands the prospects for efficient automation in homes, companies, and cities. This aims to save energy, increase productivity and security, support self-sustained living, facilitate entertainment, and simplify life for the general public.

“The hardware platform does not only apply to the Smart Mirror but to the whole range of IoT technology. This can really make a difference in saving energy and having technology become green.”

Jens Hagemeyer, Team Leader at Bielefeld University

Smart environments are based on the automation of subsystems that consist of numerous sensors, information processing schemes, and actuators. Modern smart applications collect as much information about their surroundings as possible to derive insights on present situational factors and to predict future events and behaviours. The concept of smart technology has transitioned from isolated applications that function as simple embedded systems, towards cyber-physical systems that gather and process large amounts of complex data from a high number of distributed smart devices. This involves linking numerous devices, including speakers, cameras, computers, smartphones, tablets, televisions, security systems, appliances, and much more.

The Smart Mirror essentially serves as a hardware platform that can connect to any IoT device and be used for any kind of IoT data processing. It is vital in aiding AIoT systems in particular that combine compute intensive AI algorithms and IoT technology. These AIoT systems are a prerequisite in reaching the societal aim on cultivating smart homes, transforming companies into smart industries, and building smart cities. However, AIoT systems are heavily energy-consuming, pose privacy concerns, and the computing capabilities of current hardware platforms are insufficient to handle the significant amounts of data generated by these systems. UNIBI’s effort to make these systems more functional, compliant, and energy efficient – exemplified by the Smart Mirror use case – is a potential contribution that can essentially result in facilitating more effective industries and cities and making life of the general public easier and more enjoyable.

Video: Video introduction to the VEDLIoT project and [from 01.45] the Smart Mirror solution developed by UNIBI.

In 2016 UNIBI started a number of research apartments to study smart environments, as well as to explore the possibilities and potentials of home automation and build solutions of the future. Hence, a number of smart subsystem applications were integrated in the building and its furnishings, and the first prototype of the Smart Mirror was developed to be the central control point for the subsystem applications.

“We try to imagine how life in the future will be and what kind of autonomy we can build into an apartment.”

Nils Kucza, PhD student at Bielefeld University

The central AI capabilities enable all sensors, processing units, and actuator nodes of the smart subsystems in the apartment to ‘talk together’, exchange data, and elicit the action commands given by the user. The premise for this systematic communication is the deep learning intelligence of the Smart Mirror, providing the ability to process interaction of different users and different smart applications, and combine different interaction schemes (e.g., switching between text, touch, audio, video input and output). Providing this functionality is highly computationally intensive and the collected data contains personal and highly sensitive information. Cloud-based processing is, therefore, undesirable, and to address this issue, UNIBI has developed the Smart Mirror on an edge computing platform.

Since releasing the first prototype of the Smart Mirror, the solution has been refined and further developed throughout the European LEGaTO and VEDLIoT projects. Resource-efficiency and technological capabilities have been significantly enhanced in the most recent prototype of the Smart Mirror. For example, the algorithms that perform object, face, gesture, and speech detection require a high volume of computing power to ensure a seamless user experience – this relies on a power consumption of 50 W at 10 FPS [1], which is 11 times lower than prior versions.

The IoT solution

The physical Smart Mirror consists of a range of hardware components, including a semi-transparent foil display with an underlying monitor, a microphone, and a RGBD [2] camera. These integrated components collect data on the surroundings of the mirror in terms of audio and visual stimuli. This data input is transmitted to the central hardware of the mirror, i.e., the hardware that makes the mirror smart- the edge processing unit. This unit process the data input and, in combination with several neural networks, runs algorithms that enable face recognition, object detection, gesture detection, and speech recognition. The processing happens locally and combines GPU[3], CPU[4], and FPGA[5] computation.

The Smart Mirror connects to the internet via Wi-Fi or Ethernet, which allows the solution to also connect to other smart applications that may be installed in the building. When connected to other smart applications, the Smart Mirror retrieves data collected by these separate applications and process the data input in combination with the neural network algorithms.

Illustration 2: Key enabling technologies for the Smart Mirror solution.

Based on the various data inputs and the processing, AI algorithms enable the Smart Mirror to transmit information in a visualised format to the mirror display and to elicit action commands for controlling the connected smart applications.

• Display information: The user is able to see his own mirror image like in a normal mirror; however, this mirror generates a virtual image using 3D point clouds, which means that a user is able to see one’s physical self in a digital 3D representation. Underneath the mirror image, a variety of features are shown via the monitor, making the mirror display just a graphical overlay. These features are displayed in individually realized information and control modules based on the identity of the user (which is detected through face and/or speech recognition). The display of the Smart Mirror thereby enables the respective user to access personally configured information such as public transport timetables, local weather forecasts, latest news, sports results, calendar reminders etc.
• Action command: The user is able to use the Smart Mirror as a central controller for all connected smart applications in the building. Using the personalised control modules, the user can elicit action commands via speech or gesture for controlling the entire intelligent environment with applications, such as regulating temperature and lighting conditions, opening/closing the entrance door, turning on/off the oven, shopping online etc.

As the user increases their frequency of interaction with the Smart Mirror, the information and services will become more personalized. Any interaction contributes to the training of the processing algorithms, and when sufficient data input is attained, the solution will be able to present recommendations and elicit actions autonomously. Hence, the Smart Mirror may suggest dinner recipes based on the contents of the refrigerator, propose clothes from the wardrobe that is suitable for current weather conditions, generate shopping lists based on the contents of the kitchen cabinets, remind residents to bring objects such as keys and phone when leaving the home etc. – all based on the preferences, habits, and inclinations of the user.

Illustration 3: Users of the Smart Mirror are not only able see their own reflection in 3D, but also access personalised information for example on the local weather and their private schedule.

Technological capabilities and data precautions of the Smart Mirror differ according to its use as either a private or a public consumer IoT product.  Hence, the available features of the solution are framed by the context in which it is being used, as it could be both in a private context such as a home or a company, or in a public context such as in a hospital, a hotel or an open public space as a train station. The context of use will affect the learning models due to differing interaction frequency and sources (few vs. many users), input data quality (background noise and movements), and requirements on data integrity (varying privacy concerns).

Data privacy and security of the system is a critical aspect of the Smart Mirror as very strict requirements are imposed to ensure the integrity of privacy (this applies in particular to public city installations). Although the amount of data that is stored by the Smart Mirror is very limited, some data storage is necessary in order to train the algorithms to enable face, object, gesture, and speech recognition, as well as to deliver personalised features. To ensure personal privacy, only metadata is being stored, and all privacy-critical data processing happens close to the data collection point. Hence, a significant attribute of the Smart Mirror in this respect is that sensors, processing units and actuator nodes are then distributed over a local area, where edge computation guarantees that all processing happens locally, and that no data gets into the cloud.

The notion of a Smart Mirror is not new, but existing solutions are still bound to cloud computing, and none of them use neural networks and AI algorithms like the solution presented by UNIBI.

“The overall idea of a smart mirror is not new. To build a smart mirror which does all the processing locally so you do not need the cloud – using efficient hardware for it – that is something which has not been done before.”

Jens Hagemeyer, Team Leader at  Bielefeld University

 

Outcome of the IoT solution

The development of UNIBI’s Smart Mirror solution is a significant contribution to the societal agenda of promoting smart environments and accelerating automation in homes, companies, and cities. By providing a hardware platform that empowers complex AIoT systems to function more efficiently and sustainably, the solution aids future advances in the smart technology domain, and thereby facilitates the valuable outcomes that this will indisputably effect.

“This is not significant for one mirror – I mean apart from affecting your electricity bill… I see it more on the resource efficiency side of things. Enabling energy efficient processing, enabling energy efficient computing, is something!”

Jens Hagemeyer, Team Leader at Bielefeld University

For users of the Smart Mirror solution, outcomes are manifold. Integrated into a smart home environment, the mirror facilitates home automation, which leads to:

• Convenient and integrated living for technology enthusiastic people: By acting as a central control unit to increase application efficiency for the user, the mirror can be a valuable gadget in private homes as well as in hotels and student accommodations of the future.
• Enablement of self-determined living for elderly and disabled people: By supporting everyday activities and behaviour of individual users, the mirror can be a valuable helping tool in private homes as well as in nursing homes and senior accommodations.

Integrated into a smart city environment, the mirror facilitates a range of intelligently enhanced applications, both as a commercial as well as a public service. For example in a reception hall of an institution, university, or corporation, the mirror can detect arriving guests and provide them with information on the room or office they are awaiting – or in a fashion clothing store, the mirror will enable customers to see themselves in various outfits without them having to try on the clothes physically – or in a hospital, the mirror can detect pulse and breathing rates of patients and alert the personnel in case of anomalies – or in a public train station, the mirror can provide instructions on how travelers reach their destination the fastest, easiest and cheapest way.

For end-users of the Smart Mirror, the beneficial outcomes are the derived effects summarised in the list below:

• Save energy and costs: Detection of resource and energy consumption in the building. Automation of optimisation actions and related cost drivers.
• Increase productivity: Connecting subsystems and centralising applications management. Automating action commands and receiving recommendations.
• Increase security: Linking and restriction system use, building access, and specific action commands to individuals (using face and speech recognition).
• Self-sustained living: Automated daily-life reminders, detection of behavioural anomalies, AI facilitated guidance, and support through conversation,
• Entertainment: Hub for smart applications. Unlimited provision of features, services, content, and information based on personalised preferences.
• Life simplicity: Smarter homes, companies and cities enabled by automation of repetitions tasks, and decision-making support and recommendations on complex tasks. Easy integration into daily life and intuitive operation using gestures or speech.

Illustration 4: Neural networks and AI algorithms of the hardware platform are trained to enable the Smart Mirror to recognize faces, objects, gestures and speech.

UNIBI distinguishes between a user segment and a customer segment for the Smart Mirror. The solution is open source and should be perceived as a hardware component in a smart environment – not as a stand-alone product. Hence, the end-users are foreseen as consumers rather than direct customers of the commercialised Smart Mirror, whereas customers are more likely to be B2B providers of smart home, company, and city solutions that integrate it as a central platform.

“This Smart Mirror is a central control interface for a potential smart hub. So it can be seen as a kind of personal assistant.”

Jens Hagemeyer, Team Leader at Bielefeld University

To customers, UNIBI’s Smart Mirror possess three unique selling points that are all bound by the competitive advantage of the edge computing competence. First is privacy data sensitivity, second is efficiency and third is reliability – all adding up to a superior and streamlined AIoT system, which is enabled by the platform’s local processing and independency on cloud and external servers.

Although the Smart Mirror is not yet commercially available to the market, research shows promising prospects, and demonstrations attract great interest from both industry and the general public. UNIBI hopes for partners to leverage the hardware platform and integrate it in their own solutions and/or build on top of it – effecting an increase in more resource efficient and environmentally friendly technology development as well as accelerating scalability of the smart environment concept.

“What we want to get out of this is for sure cool research results – but exploitable research results, which can then be implemented by companies.”

Jens Hagemeyer, Team Leader at Bielefeld University

Best practice learnings

The main solution offered by UNIBI is the hardware platform that is exemplified by the Smart Mirror. With a value proposition of high-efficient computing capability, the Smart Mirror represents a solution of the future, and as digitalisation and automation accelerate globally, so does a basic understanding, openness towards, and request for novel technology solutions. An essential learning for UNIBI  in this respect has been the power of using this demand for effective AIoT systems to tap into an emerging market gap. Operating among first movers in the respective field, rapid experimentation, prototyping, and testing has, according to UNIBI , played a significant role in reaching the current state of achievement.

UNIBI  considers a number of points to be vital for the immediate success of their hardware platform and the Smart Mirror. Taking departure in current state of development and the results collected from research and demonstrations, these points are compiled into a list of recommendations below:

1. Solve problems with consumer use cases

UNIBI  fundamentally solves a backend efficiency problem by refining and advancing the computational capabilities of IoT technologies and subsystems.

“That is the typical thing that we do for IoT use cases, applying AI or deep learning or whatever you want to call this, to a certain problem.”

Jens Hagemeyer, Team Leader at Bielefeld University

With the Smart Mirror use case, the technological problem solving is wrapped into a consumer-oriented value proposition being the enablement of smart environments. For UNIBI , this builds a solid foundation for broad communication of the solution’s value potential, generating interest and derived demand for the hardware platform.

2. Think compatibility and heterogeneity into solutions

“Think about distributed solutions, think about the heterogeneity in hardware, to make your processing and use case efficient in terms of computational efficiency.”

Jens Hagemeyer, Team Leader at Bielefeld University

In the mind of some solution developers, compatibility in technological subsystems is not a concern, and technological heterogeneity is considered potential blasphemy to business. From experience, UNIBI  can call this perception an immense misunderstanding since a vital aim of the smart domain is to maintain a cooperative processing environment, and in particular when looking towards a future with an immense number of connected devises and applications.

3. It is all about the hardware:

“It comes down to better hardware I guess.”

Nils Kucza, PhD student at Bielefeld University

The hardware platform is the principal cause of success for the Smart Mirror according to UNIBI. Hardware developers in the domain have traditionally been looking at the functional side of computation to achieve good results. UNIBI, however, advises developers to also consider efficiency in terms of resources, energy consumption, and environmental impact – do not just develop a functional solution, but develop a sustainable solution.

“Keep in mind that computation and computational power do not come for free. That is where the efficient hardware and heterogeneous solutions comes in.”

Jens Hagemeyer, Team Leader at Bielefeld University

End notes

[1] FPS (Frames per Second): the frequency (rate) at which consecutive images (frames) are captured or displayed. The human visual system can process 10 to 12 images per second and perceive them individually, while higher rates are perceived as motion.

[2] RGB-D:  Specific type of depth-sensing device that work in association with a RGB (red, green and blue colour) sensor camera. They are able to augment the conventional image with (D) depth information (related with the distance to the sensor) in a per-pixel basis.

[3] GPUs (Graphic processing units): Originally designed for graphics; now used in a wide range of computationally intensive applications.

[4] CPUs (Central processing units): Traditional sequential processor for general-purpose applications.

[5] FPGAs (Field programmable gate arrays): Flexible collection of logic elements and IP blocks that can be configured and changed in the field.

Copyright notice:  © 2020 – 2023 EU-IoT Consortium.

This material was produced as part of the EU-IoT project, grant ID 956671, and is funded by the Horizon 2020 Framework Programme under topic ID ICT-56-2020.

EU-IoT is the European IoT Hub. The EU-IoT project works towards growing a sustainable and comprehensive ecosystem for Next Generation Internet of Things.

Source of origin: Information to document this use case originates from the H2020 call: ICT-56-2020 Next Generation Internet of Things (RIA); Project VEDLIoT 957197; Period: 01-11-2020 to 31-10-2023; EC contribution 8 mil. EUR.