A Study of Human Emotion Recognition Methods Based on Digital Image Analysis

Introduction. Recognizing basic emotions from facial images is a highly sought-after computer vision task, with both research and regulatory implications. From July 2024, new vehicles in categories M (passenger cars) and N (trucks) in the EU have been required to be equipped with driver monitoring systems, in accordance with Regulation (EU) 2019/2144¹. For this very reason, computer vision techniques that analyze facial expressions, gaze direction, and head position are widely used in distraction detection systems, including the Advanced Driver Distraction Warning (ADDW) and Euro NCAP DMS systems. These same facial expression analysis methods are also in demand in medical diagnostics, operator monitoring in transport, and educational analytics. In these areas, automated analysis provides higher reproducibility compared to surveys and subjective observation.

The practical appeal of such systems is supported by a well-established theoretical foundation. In particular, human facial expressions are varied and spontaneous, complicating their interpretation. However, despite cultural and individual differences, standard facial expressions are recognized in the literature as stable indicators of emotional state [1]. It is the universality of such expressions that forms the basis of facial activity coding systems and provides the theoretical basis for automatic recognition algorithms. In parallel, multimodal approaches are being developed that use physiological signals to objectively assess emotional reactions. They complement visual methods and expand the capabilities of integrated human state assessment.

The current state of face recognition research at the methodological level is covered in several review publications [1]. Among the classical feature representations of the face, a special place is occupied by local binary patterns (LBP), which describe the textural heterogeneities of the face and are invariant to monotonic changes in illumination [2], as well as histograms of oriented gradients (HOG), which characterize the contour geometry through local distributions of gradients [3]. Historical continuity with them is formed by cascade Viola-Jones detectors on Haar-like features with the AdaBoost algorithm, which provide the implementation of face detection in consumer cameras, as well as the SIFT descriptor, which detects scale-invariant feature points in the vicinity of facial landmarks.

The applied maturity of this field is confirmed by reviews of modern methods of automatic facial expression analysis [4], clinical studies on emotion recognition [5], and mature commercial emotion analysis systems – objective assessment of facial expressions is already in demand in practice. However, commercial systems themselves do not answer the methodological question – their algorithms are closed and do not allow assessing the contribution to accuracy of an individual classical descriptor. And published comparisons of LBP and HOG are obtained under heterogeneous conditions and are poorly comparable with each other. This, in turn, raises the question of how classical descriptors behave under strictly uniform assessment conditions.

In parallel, image analysis methods are being developed in related areas of computer vision — video surveillance software systems [6], driver monitoring [7], 3D motion capture [8], optical flow parameter identification [9], and license plate recognition [10]. Experience from these studies shows that accurate feature localization and robust classification remain critical requirements regardless of the target task.

Against the backdrop of these general requirements, emotion recognition remains an open challenge even for convolutional neural networks (CNNs), which are widely implemented in computer vision. A comprehensive analysis [11] notes that deep models primarily face overfitting due to limited availability of training data and non-expression-related variations such as illumination, head rotation, image resolution, and identity bias. Furthermore, creating datasets with emotion labeling is labor-intensive and expensive, and in transport and medical security, explainability of decisions is additionally required, limiting the use of black box models. A separate issue remains the model memory of specific subjects, which is particularly noticeable in small datasets [12].

To summarize, a consistent pattern can be identified: for facial expression recognition, a single classical descriptor is inferior to a trained representation. LBP provides more robust performance under variable lighting conditions due to its invariance to monotonic luminance transformations [2]. HOG, on the other hand, better describes the contour geometry of the face and maintains its leadership on laboratory datasets [3]. However, both methods, when used alone, are inferior to convolutional neural networks on large datasets [4]. Relatively high accuracy is achieved through combining LBP features with trainable representations based on CNN or LSTM (long short-term memory) [7]. Similar conclusions are integrated in a summary analysis of modern facial expression recognition methods [11]. At the same time, cascade Viola-Jones detectors and their development still provide acceptable quality of face localization at low computational costs, which is confirmed by modern facial authentication systems based on classical features [13]. Comparison of application conditions, sizes of training samples and verification protocols in different sources is complicated by their methodological heterogeneity.

From the above, it is clear that several aspects of classical LBP and HOG descriptors in emotion recognition remain insufficiently described: robustness of metrics during stratified cross-validation on small datasets, nature of systematic errors between visually similar emotions, and limits of applicability under conditions with variable illumination and uncontrolled viewing angles. Without this information, it is impossible to meaningfully integrate classical features into hybrid recognition schemes, therefore, there is a need for their direct comparison under uniform and reproducible conditions.

Therefore, the objective of this study is to determine the applicability limits of classical LBP and HOG descriptors for recognizing seven basic emotions on the JAFFE dataset, and to characterize the observed classification errors.

To achieve this objective, the study addresses five tasks:

1) comparison of the theoretical properties of LBP and HOG as applied to facial expression recognition – robustness to illumination, sensitivity to subject identity, and feature representation dimensionality;

2) implementation of a unified image preprocessing and feature extraction pipeline for LBP and HOG for the JAFFE dataset, providing reproducible results;

3) classification of seven basic emotions using a linear-kernel support vector machine with five-fold stratified cross-validation;

4) interpretation of confusion matrices and the Accuracy and Macro-F1 metrics with reference to a psychophysiological explanation for systematic mixing of emotion classes;

5) formulation of recommendations for the application of LBP and HOG for typical scenarios, shooting conditions, and ways to incorporate classical features into hybrid architectures.

The methodology of this study is based on the canon of classical digital image processing [14] — brightness normalization, histogram equalization, extraction of local texture features and gradients.

Materials and Methods. Dataset. The open research dataset JAFFE (Japanese Female Facial Expression, Lyons et al., 1998; Kyushu University, Japan), distributed for scientific purposes, was used as the experimental base. The dataset contained 213 grayscale images of ten Japanese female subjects with seven basic emotions: anger, disgust, fear, happiness, neutral, sadness, and surprise. The classes were distributed almost evenly, from 29 to 32 images per emotion. The images were acquired in laboratory setting under identical lighting and frontal perspectives using staged expressions. This provided uniformity of the footage but limited the direct transfer of the results to field scenarios. The input JAFFE images were converted to a grayscale palette with a resolution of 256 × 256 pixels. Histogram equalization was used to normalize brightness. Faces were aligned according to eye coordinates obtained using a landmark detector based on the Dlib library (model shape_predictor_68_face_landmarks).

Feature Extraction. Two complementary classical descriptors were calculated on the images prepared in the following way. Local binary patterns (LBPs) were calculated using a uniform encoding with radius R = 1 and neighbor count P = 8; histograms were constructed using disjoint 16 × 16-pixel blocks and combined into a common feature vector of dimension 15,104. Histograms of oriented gradients (HOGs) were formed with 8 × 8-pixel bins, 2 × 2 bins, nine orientation bins, and L2-Hys block normalization (default value in scikit-image).

Classification and Quality Assessment. The resulting feature vectors were fed to a support vector machine (SVM) classifier with a linear kernel. The regularization parameter C was tuned over the grid {0.01; 0.1; 1; 10}, with the optimal value selected via inner three‑fold cross‑validation nested within the outer stratified five‑fold cross‑validation loop. Quality was assessed using the mean and standard deviation of Accuracy and Macro-F1 across five outer folds. Confusion matrices and the distribution of precision, recall, and F1 scores across seven classes were additionally constructed.

Software Implementation. The pipeline was implemented in Python 3.10 using scikit-learn 1.3 (SVM, cross-validation) and scikit-image 0.21 (LBP, HOG) libraries. The initial value of the pseudorandom number generator was fixed (random_state = 42), which provided reproducibility of sample fold splitting and repeatability of hyperparameter selection.

Research Results. The HOG method combined with linear SVM achieved complete separability of seven emotional classes: Accuracy = 1.000 ± 0.000 and Macro-F1 = 1.000 ± 0.000 (mean ± standard deviation over five outer folds; Table 1, Fig. 1). For all seven classes, precision, recall, and F1-score metrics were equal to 1.00. No interclass intersections were found in the confusion matrix. The final HOG feature vector had 34,596 elements; in all five external folds, internal cross-validation selected the regularization parameter value C = 1. The average feature extraction time per image was about 0.018 s, classifier training on the training fold was 0.42 s, and prediction per image was 0.003 s.

The LBP method in the same protocol showed Accuracy = 0.300 ± 0.041 and Macro-F1 = 0.202 ± 0.038 (Table 1). For five outer folds, the Accuracy values were 0.256; 0.310; 0.310; 0.357 and 0.267. The dimension of the final LBP feature vector was 15,104 elements; internal cross-validation selected value C = 0.1 in three outer folds and C = 1 in the other two. The average feature extraction time per image was about 0.009 s; classifier training – 0.18 s; prediction – 0.002 s.

The distribution of precision, recall, and F1 values across seven JAFFE classes for the LBP method turned out to be uneven (Table 2). The maximum F1 values were observed for the “disgust” (0.33) and “happiness” (0.30) classes; the minimum values were for the “fear” (0.11), “sadness” (0.14), and “neutral” (0.14) classes. The minimum precision values were recorded for the “neutral” (0.12), “fear” (0.13), and “surprise” (0.14) classes. The minimum recall values were for the “fear” (0.10), “sadness” (0.11), and “neutral” (0.17) classes. For the “surprise” and “neutral” classes, recall exceeded precision by more than 0.05; for the “sadness” class, precision exceeded recall by 0.07.

In the LBP confusion matrix (Fig. 2), the “fear” class was assigned to “surprise” in 24.1% of cases, to “anger” – in 14.8%, and to “neutral” – in 11.1%. The “sadness” class was assigned to “neutral” in 27.6% of cases and to “anger” – in 10.3%. The “neutral” class was assigned to “sadness” in 16.7% of cases. The largest diagonal values of the confusion matrix were recorded for the “disgust” and “happiness” classes – about 33% and 30%, respectively.

Discussion. Comparison of our results with other facial expression recognition studies shows that direct quantitative benchmarking requires careful qualification regarding the experimental conditions. In [15] on JAFFE, carefully selected LBP parameters were used: radius R = 2, partitioning into 42 subregions of 18 × 21 pixels, uniform encoding. With this set and linear SVM, the Accuracy indicators are higher than those of pipelines with typical settings. The present work is based on a different configuration: radius R = 1, noncrossing blocks of 16 × 16 pixels in an image of 256 × 256, the final feature vector of dimension 15,104. This is a typical representation of LBP with an out-of-the-box descriptor implementation. The configuration was selected to verify the response of the method under standard conditions, and reproducing quantitative values from [15] was outside the scope of this study. The mismatch with the reference data consequently confirms the inherent sensitivity of LBP to the radius and partitioning scheme.

In addition to direct comparisons, hybrid architectures are a significant benchmark: the use of LBP at the input layers of convolutional or recurrent neural networks can provide significantly higher quality indicators compared to an isolated classifier [12]. In this context, the results obtained for the HOG method within the framework of this protocol should be interpreted not as evidence of the universal superiority of the said descriptor, but as a marker of the high sensitivity of the gradient profile to preserving the identity of subjects in the training and test samples; while the quality of coding of emotional states with such a division of data turned out to be a secondary factor.

To more accurately assess the generalization ability of the algorithms, it is advisable to use a leave-one-subject-out (LOSO) experimental protocol, in which the training and test sets do not contain images of the same faces. The five-fold stratified cross-validation used in this study allows for the presence of identities in both subsets, which results in an artificial inflated metric for HOG. Under such conditions, comparative diagnostics of the methods is characterized by systematic bias.

The error detection structure analysis shows that the class confusions recorded in the LBP confusion matrix have a clear psychophysiological basis. Thus, the “fear-surprise” and “sadness-neutral” pairs share overlapping sets of active facial muscles under the FACS system. In the first case, eyebrow raising and eye widening are observed, while in the second case — a subtle, low-intensity activation of specific action units (AU1 + AU4 + AU15) associated with sadness in static imagery. Textural features of LBP capture local heterogeneities and inevitably reflect this visual similarity. Thus, the recorded error is determined by the limitations of the feature space, while the classifier itself functions correctly.

The combined observations presented demonstrate that standard five-fold cross-validation has limited efficiency when evaluating facial expression recognition methods on compact datasets. Even with strict adherence to existing methodological requirements, the final results tend to overestimate the generalization quality to new individuals. Switching to a LOSO protocol, incorporating larger databases with variable registration conditions, and strict control over the absence of overlapping identities between samples are essential for obtaining an objective assessment.

The findings obtained are based on conditions that limit their sweeping generalization:

• the JAFFE dataset used is characterized by a relatively small volume (213 images of ten subjects) and ethnic homogeneity, which precludes extrapolation of the numerical results to populations with different anthropometric parameters;
• the images were taken in a laboratory environment under stable lighting and a frontal perspective with simulated expressions, while field scenarios involve dynamic lighting, random perspectives, and spontaneous reactions;
• the classification procedure is implemented by a single type of algorithm — linear SVM — without comparison with other types of kernels or deep learning architectures;
• the object of analysis was exclusively static images without taking into account the temporal dynamics of facial contractions available in the video stream.

Taking these factors into account, the results obtained confirm the feasibility of using the LBP and HOG methods under specific conditions determined by the physical nature of each determinist. The application of HOG is demonstrated within the context of laboratory-based tasks involving a fixed set of participants and controlled lighting conditions, where class separability in terms of identity-related features is feasible. As a standalone tool, LBP yields lower accuracy; however, it provides a more robust estimation under unstable illumination and uncontrolled viewpoints. This makes LBP an efficient component of feature representation within hybrid systems incorporating trainable models.

Conclusion. The study addressed the following objectives: comparing the properties of LBP and HOG for facial expression recognition, implementing a unified reproducible preprocessing and feature extraction pipeline, classifying seven basic emotions using a linear SVM, constructing confusion matrices, and formulating recommendations for using descriptors. In a unified protocol on JAFFE, the HOG method with a linear SVM achieved Accuracy = 1.000 ± 0.000, which is explained by subject identity leakage during stratified cross-validation. LBP, using the same protocol, yielded Accuracy = 0.300 ± 0.041 and robust systematic mixing of “fear – surprise” and “sadness – neutral” pairs, consistent with the FACS description of overlapping sets of facial muscles.

In terms of practical implementation, a model based on HOG and linear SVM is applicable to solving local laboratory problems with a constant set of test subjects, for example, on research benches or in pilot versions of medical interfaces. Using LBP in conjunction with linear SVM in isolation is less effective in terms of accuracy, but is justified as a component of hybrid architectures (including convolutional and recurrent networks) due to its high computational efficiency and invariance to monotonic illumination fluctuations. The obtained results can be valuable for the design of driver monitoring systems compliant with the requirements of Regulation (EU) 2019/2144, as well as for the methodological justification of quality assessment protocols on small datasets. Given the identified limitations, the prioritized direction for future research involves transitioning to Leave-One-Subject-Out (LOSO) protocols and expanding the experimental dataset with data acquired in uncontrolled environments.