After the AI camera enters the classroom, it becomes more and more difficult to pretend to listen carefully. Recently, researchers from HKUST and Kazakhstan have jointly developed a system that uses AI cameras to record and analyze students’ emotional changes. Mind Course “. This article comes from public micro-channel number: Almost Human (ID: almosthuman2014) , author: Synced, from FIG title: FIG insect < / span>
Many people have become accustomed to the monitoring work that AI can do in the classroom.
“A professor looks at his computer after the lecture and with a piece of software, he can see the emotional changes of the students in this whole lesson. After 30 minutes, most of the students have lost interest and started to lose their mind. This is probably the time when he talked about the problem. So the professor made a record to remind himself not to run the problem in the future.
Most classrooms in reality are not like this, but with the development of technology, such scenarios will become more and more common.
Recently, a paper on classroom monitoring technology was published in “IEEE Transactions on Visualization and Computer Graphics”. In the paper, researchers from the Hong Kong University of Science and Technology, Harbin Engineering University, and other institutions proposed a method called EmotionCues.system. The system mainly focuses on how to record students ‘facial expressions, and analyzes students’ emotional changes and concentration in the classroom based on this.
Qu Huamin, one of the authors and a computer professor at the Hong Kong University of Science and Technology, said that the system “provides a faster and more convenient way for teachers to measure student participation in the classroom.”
The original intention of this research is “goodwill”: Relying on the system to monitor students’ emotional feedback in the classroom, to judge when students start to feel bored, and when to concentrate more, to remind teachers how to Improve classroom content and quality.
The research team tested the proposed system in two classrooms. One classroom was a student from the Hong Kong University of Science and Technology, representing a group of college students; the other classroom was a kindergarten in Japan, representing a group of young students.
The test found that this visual analysis system is better at detecting those “obvious emotions”, such as the joy when learning interest is strong. However, the system’s ability to interpret “anger” or “sadness” is still lacking. Students may simply focus on the content of the class and frown just because of in-depth thinking, but they are easily interpreted as “anger” by the system.
System workflow
The following figure shows the workflow of the entire system, including the two stages of data processing and visual exploration.
Data processing process
The first stage is to process a series of raw data and use computer vision algorithms to extract emotion information, including face detection, face recognition, emotion recognition, feature extraction and other steps.
In the face detection step, researchers use MTCNN (Multi-task cascade convolutional network, a deep convolution for predicting the position of the face and Landmark Network) to detect faces in each sample frame.
In the face recognition phase, the usual method of face contrast is to vectorize the image. The researchers used facenet (a deep learning model that is more complete in facial recognition) , which can directly learn from facial images to compact European Mapping of space.
At the stage of emotion recognition, researchers chose to use a classification model for reasons of intuitiveness and comprehension. They fine-tuned a CNN model (ResNet-50) using the FER 2013 dataset. This data set has been widely used for facial expression recognition.
Considering that emotion recognition may not be so accurate, the researchers singled out some influencing factors (such as face size, occlusion, image resolution, lighting conditions Etc.) and visually encode them in the system to determine the emotional status of the students.
These factors may play a key role in systematic sentiment analysis. For example, a person who is far away from the camera has a smaller area in the video, and is more likely to be misidentified. In addition, if a person’s face is often blocked by others, there will be a higher risk of misjudgment. Researchers have integrated these factors into the system analysis process and provided richer interactive functions to improve system performance.
Interactive Visual System
The second stage is to design an interactive visual system based on the five major requirements. This system can support two-level granular video visual analysis, including the overall emotional evolution of a student and the individual emotional evolution of a student.
The researchers implemented a web-based system based on the Vue.js front-end framework and the Flask back-end framework, as shown in the figure below. The system includes three views: summary view (summary view, Figure 3a-b) ; people view (character view, Figure 3c) and video views (video view, Fig. 3d) .
It is important to provide the teacher with an overview of the student ’s emotional changes, so the researchers designed a summary view to let the teacher see the static and dynamic evolution of student emotions. Figure (a) shows the student’s sentiment file, which is used to show the student’s sentiment distribution. (static summary) ; Figure (b) shows Student’s emotional change curve (dynamic summary) .
People view visually displays the emotional state of the selected target person through portrait-type glyphs. The differences between different emotional portraits enable users to identify and compare the characteristics of different people. As shown in the figure below, the researchers used a customized pie chart in the design:
With this customized pie chart design, users can easily observe detailed emotional information and the factors that affect them. At the same time, the screen snapshot function makes it easier to compare emotional information between different people. If the user wants to see the details, they can click on the snapshot of interest to view it. An example of the snapshot is to the left of the character view (Figure c) .
In the system, the researchers provided the original video for users to browse in the video view “Improve” teaching or “monitor” teaching?
The original intention of this research is to help the lecturer collect student feedback and improve teaching quality. But can the facts really do what they want?
Compared to analyzing emotions based on video records, in domestic classrooms, there is a more exaggerated “smart headband”.
In a classroom in a primary school in Jinhua, Zhejiang, each seated student wears a black headband that looks like a “golden hoop”, lights up red when he is focused, and lights blue when he is distracted. Send it to the teacher’s computer every 10 minutes and sync it to the parent WeChat group, so that parents outside the school can keep track of the child’s class status at any time.
But this headband, or this kind of classroom monitoring technology, faces a lot of skepticism. For example, ethical issues: it exposes students’ personal emotions in the classroom, allowing teachers to know who is focusing or not focusing in the classroom. This involves the privacy of students.
In addition, in a 40-minute course, it is impossible for a student’s attention to maintain full concentration. It is not meaningful to continuously monitor the student’s attention and correct any behaviors that are not focused.
On the other hand, this kind of monitoring system may distract teachers and students, because people in them will feel that they have a pair of eyes staring at themselves all the time. If you wear a gold hoop, this emotion will become more obvious. This feeling of being monitored in real time can affect classroom participants’ freedom of expression to some extent.
Reference link
https://spectrum.ieee.org/the-human-os/biomedical/devices/ai-tracks-emotions-in-the-classroom
https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8948010
This article comes from WeChat public account: Heart of the Machine (ID: almosthuman2014) , author: Synced