domeet webmaster

After the AI ​​camera entered the classroom, it became more and more difficult to pretend to listen carefully.

Editor’s note: This article is from WeChat public account “ Heart of the Machine ” (ID: almosthuman2014), author Synced, participating: Zhang Qian, egg sauce, Jamin.

After the AI ​​camera entered the classroom, it became more and more difficult to pretend to listen carefully. Recently, researchers from the Hong Kong University of Science and Technology and Harbin Engineering have developed a system that uses AI cameras to record and analyze students’ emotional changes. course”.
About AI in the classroom Many people have become accustomed to the completed monitoring work.

“A professor looks at his computer after the lesson. With the help of a software, he can see the emotional changes of the students in the whole lesson. After 30 minutes, most of the students have lost interest and started to lose their mind. This is probably the time when he talked about the off-topic. So the professor made a record to remind himself not to run off-topic in the future. “ Most of the classrooms in reality are not like this, but with the development of technology, such a scenario Will become more common.

Recently, a paper on classroom monitoring technology was published in “IEEE Transactions on Visualization and Computer Graphics”. In the paper, researchers from the Hong Kong University of Science and Technology, Harbin Engineering University and other institutions proposed a system called EmotionCues. The system mainly focuses on how to record students ‘facial expressions, and analyzes students’ emotional changes and concentration in the classroom based on this. One of the authors, Qu Huamin, a computer professor at the Hong Kong University of Science and Technology, said that the system “provides a faster and more convenient way for teachers to measure student participation in the classroom.”

The original intention of this research was “goodwill” : Rely on the system to monitor students’ emotional feedback in the classroom, to determine when the students start to feel bored, and when to concentrate more, to remind teachers how to improve the content of the classroom and improve the quality of teaching.

The research team tested the proposed system in two classrooms. One classroom was a student from the Hong Kong University of Science and Technology, representing a group of college students; the other classroom was a kindergarten in Japan, representing a group of young students.

Tests have found that this visual analysis system performs better in detecting those “obvious emotions”, such as the sense of joy when learning interests are more intense. But the system still lacks the ability to interpret “anger” or “sadness”. Students may simply focus on the content of the class and frown just because of in-depth thinking, but they are easily interpreted as “anger” by the system.

System Workflow The following figure 2 shows the workflow of the entire system, including the two stages of data processing and visual exploration.

Figure 2.

Data processing process

The first stage is to process a series of raw data and use computer vision algorithms to extract emotion information, including face detection, face recognition, emotion recognition, feature extraction and other steps. In the face detection step, the researchers used MTCNN (Multi-Task Convolutional Convolutional Network, a deep convolutional network for predicting faces and Landmark locations) to detect faces in each sample frame. In the face recognition phase, the common method of face contrast is to vectorize the image. The researchers used facenet, a deep learning model that is more sophisticated in facial recognition, which can directly learn the mapping from facial images to compact European-style spaces.

At the stage of emotion recognition, researchersFor understanding, a classification model was chosen. They fine-tuned a CNN model (ResNet-50) using the FER 2013 dataset. This data set has been widely used for facial expression recognition. Considering that emotion recognition may not be so accurate, the researchers singled out some influencing factors (such as face size, occlusion, image resolution, lighting, etc.) and performed them in the system Visual coding to judge students’ emotional situation.

These factors may play a key role in systematic sentiment analysis. For example, a person who is far away from the camera has a smaller area in the video, and is more likely to be misidentified. In addition, if a person’s face is often blocked by others, there will be a higher risk of misjudgment. Researchers have integrated these factors into the system analysis process and provided richer interactive functions to improve system performance.

Interactive vision system The second stage is to design an interactive vision system based on the five requirements (see the paper for details). The system can support classroom video vision of two granularities Analysis, including the overall emotional evolution of a student and the individual emotional evolution of a student. Researchers have implemented a web-based system based on the Vue.js front-end framework and the Flask back-end framework, as shown in Figure 3 below. The system includes three views: summary view (Figure 3a-b); character view (Figure 3c) and video view (Fig. 3d).

Figure 3. It is very important to provide the teacher with an overview of the student’s emotional changes, so the researchers designed a summary view to let the teacher see the static and dynamic evolution data of the student’s emotions. Figure 3 (a) shows the student’s emotion profile, which is used to show the student’s emotion distribution (static summary); Figure 3 (b) shows the student’s emotion change curve (dynamic summary). Personal view visually displays the emotional state of the selected target person through portrait-type glyphs. The differences between different emotional portraits enable users to identify and compare the characteristics of different people. As shown in Figure 5 below, the researchers used a customized pie chart in the design:

Figure 5: Visual illustration of emotional changes. With this customized pie chart design, users can easily observe detailed emotional information and interested in it. Influencing factors. At the same time, the screen snapshot function makes it easier to compare the emotional information between different people. If the user wants to view the detailed information, he can click the snapshot of interest to view it. The example of the snapshot is located in the people view (Figure 3c) Left. In the system, the researcher provides the original video for users to browse in the video view (Figure 3d). At the same time, the user can play the video at different speeds. When the user pauses the video, each frame The corresponding face will be highlighted. The user can also select the interesting part for further exploration and mining based on his observation of the emotional flow.

“Improving” teaching, or “monitoring” ” Learn? The original intention of this research is to help the lecturer collect student feedback and improve teaching quality. But can the facts really do what they want?

In contrast to analyzing emotions based on video recordings, there are more exaggerated “smart headbands” in domestic classrooms. In the classroom of a primary school in Jinhua, Zhejiang, each seated student wears a black headband that looks like a “golden hoop”, lights up red when he is focused, and lights blue when he is distracted. This attention score Send it to the teacher’s computer every 10 minutes and sync it to the parent WeChat group, so that parents outside the school can keep track of the child’s class status at any time.

But this headband, or this kind of classroom monitoring technology, faces a lot of skepticism. For example, ethical issues: it exposes students’ personal emotions in the classroom, allowing teachers to know who is focusing or not focusing in the classroom. This involves the privacy of students. In addition, in a 40-minute course, it is impossible for a student’s attention to remain fully focused, and it is not meaningful to continuously monitor the student’s attention and correct any attention-deficit behavior. On the other hand, this kind of monitoring system may distract teachers and students, because the people in them will feel that they have a pair of eyes staring at themselves all the time. If you wear a gold hoop, this emotion will become more obvious. This feeling of being monitored in real time can affect classroom participants’ freedom of expression to some extent.

Reference link:
https://spectrum.ieee.org/the-human-os / biomedical / devices / ai-tracks-emotions-in-the-classroom https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8948010