Contents
Meet by clothes, and see off by mind. Do not judge a book by its cover. Appearances are deceptive. All these common expressions in one way or another refer to the ability of a person to make assumptions about another person based solely on external factors. Having met a stranger, we assume his character, status and even intellectual abilities depending on appearance, mostly depending on features and facial expressions. There is nothing reprehensible in this, since it is most often not a completely conscious process. But is there any logic in such “fortune-telling” by faces, and to what extent such judgments correspond to reality? Often assumptions about a person turn out to be wrong, but the very fact of their presence is an important social aspect. Scientists from the Stevens Institute of Technology (USA) have developed an algorithm that can simulate a person’s assessment of his face in order to understand what assumptions strangers will make about a particular person, based only on appearance. How accurately does the algorithm predict human judgments, what assumptions have been made, and how else can the algorithm be used? We will find answers to these questions in the report of scientists. Go.
Research basis
If you look at the world of wildlife, you can confidently declare its diversity. External interspecific and intraspecific differences play an important role in various aspects of the life of a creature (hunting, finding a partner, avoiding dangers, competition for territory, etc.). Some species prefer to be inconspicuous, while others show off their most prominent features (for example, male peacocks with their incredible fan tails). As for a person, the picture does not change much: someone dyes their hair in bright colors, someone prefers to dress in all gray and be invisible, someone does not think about all this at all. However, no matter what tactics a person chooses, one aspect will always attract the most attention — the face. Yes, when we see a stranger with bright green hair, we will first glance at the hair, but the second (much longer and more analyzing) look will fall on his face. And that begins an unconscious analysis of who this extravagant green-haired type can be.
Faces are one of the most important stimuli people face. The first thing that babies visually begin to distinguish is faces. And the processing of information related to faces involves special processes in the human brain. Looking at faces, we use certain attributes that we attribute to them, often unconsciously: thin, tired, bright, smart, etc.
These attributes can be divided into two conditional categories: objective and subjective. In the first case, we evaluate age, physique, gender. In the second, everything is much more curious, since we apply attributes that subsequently answer the question — can the owner of this person be trusted or not.
More often than we would like to admit, the subjective attributes we attribute to this or that person do not correspond to reality. Nevertheless, all people on the planet sin with such inaccurate judgments, regardless of religion, nationality, orientation and education. Gu.e. In other words, judging people by their faces is (often erroneously) inherent in each of us.
Scientists say that since any person can be judged by such attributes, these psychological parameters are universal in the sense that they are implicitly defined in the space of almost all possible faces, contexts and conditions of observation. These factors combine to form a diverse «landscape» of stimuli, making it difficult to capture the relevant psychological content in its entirety.
The importance of face attribute analysis has led to the proliferation of scientific face modeling techniques, which can be broadly classified into two approaches. The first is based on the use of photographs of faces, often linked together by annotations of landmarks. The second generates artificial faces using parametric XNUMXD models.
Photographs provide more realism, but are limited by the available facial stimulus datasets that serve as the basis for interpolation, and by the interpolation algorithms themselves, which often require high-quality annotations of landmarks that are unattainable without human intervention. Artificially created faces are not subject to these restrictions, but they lack variety and realism. Consequently, none of the approaches provides workable models that express the full richness and diversity of human faces.
If we minimize the human factor, namely, apply machine learning (neural networks, for example), then we can get a system that can model faces using a selection of photographs as input data. This is the third approach to face modeling. However, even with the most accurate representative face models, they are extremely difficult to relate to how those faces would be perceived and judged by real people. Simply put, how a person makes assumptions about another person, judging by his face, this is a process that is difficult to translate into machine thinking.
The authors of the work we are considering today believe that the key to unlocking the scientific potential of these models is large-scale data sets on human behavior, unattainable using traditional laboratory experiments. In particular, such large datasets provide enough evidence to determine a reliable mapping between expressive multidimensional representations from machine learning models and mental representations of human faces.
Scientists have quantified an upper bound on the reliability of face mapping in terms of the reliability of inferences about underlying attributes. They then determined how this reliability scales with the number of faces assessed, the number of assessments per face, and the dimension of the feature space. The resulting mapping was then used to predict and manipulate the perception of arbitrary faces. In other words, the scientists could adjust the photo of the face so that the machine made a different judgment about it.
Such a mapping can be computed for any psychologically meaningful inference about attributes. In this paper, scientists have focused on three classes of such conclusions.
First, there are inferences determined by subjective impressions of objective properties (such as age and build). These more objective properties, which also include hair styling, the presence of accessories (such as glasses), gaze, and facial expressions, are commonly studied in computer vision, where they are referred to as «attributes» or «soft biometrics.»
This is followed by inferences about subjective and socially constructed attributes such as dependability and masculine/feminine, and so on.
Finally, there are conclusions about wholly subjective attributes, such as «acquaintance», where the observer is the only source of truth about the observed person (and its owner).
The study used online crowdsourcing to generate attribute inference scores for just over 1000 synthetic (though highly naturalistic) facial stimuli across 34 attributes (features), with scores from at least 30 unique participants per attribute-stimulus pair, for a total of 1020000 human judgments.
Results of the study
Attribute Inference Structure
In order to study the structure of attribute inferences, it was necessary to calculate the correlation between the average values of the face scores for each pair of attributes (image No. 1).
Many attributes were highly correlated, including happy-outgoing (r=0.93) and dominant-reliable (r=-0.81). While others were largely unrelated: smart-attractive (r = 0.01), smart-reliable (r = 0.02), liberal/conservative-believing (r = 0.08), trusting-attractive (r = 0.05).
While some of these correlations are consistent with previous studies, others are not. First, although previous work has shown that judgments of reliability and dominance are often negatively or very weakly correlated (of the order of -0,2), in this study the correlation (-0.81) was found to be much stronger. Second, judgments of intelligence or competence have previously been found to be highly positively correlated with judgments of attractiveness and reliability (with values up to ≈ 0.8), while only minor correlations between these attribute judgments were found in this work.
One of the explanations for such discrepancies may be that the facial stimuli used in this work are more diverse than in previous works (especially in terms of age, since children’s faces were not previously used). This explanation is quite plausible, given that the correlation structure of judgments about the faces of children differs from the structure of judgments about the faces of adults.
To test this hypothesis, the researchers recalculated inter-attribute correlations on data subsets with a limited age range. The inclusion of child faces was found to partially explain some inconsistencies (eg, smart-attractive) and not explain others (reliable-dominant).
It is also worth noting that memorable faces were more attractive, as evidenced by the positive correlation between the corresponding ratings. This finding is inconsistent with research showing that actual face recall is negatively correlated with attractiveness to the extent that recall predictions are correct. Finally, familiar faces were considered more attractive, consistent with earlier findings that a person is much more comfortable with a normal (non-model) face.
The attribute «outside» (regardless of whether the photo was taken indoors or outdoors) was included in the analysis to assess possible confusion when using naturalistic face photographs. This attribute was found to be the least correlated with other attributes, showing the lowest maximum absolute correlation for each attribute (eg, outside-confidence r = 0.20).
In comparison, the attribute with the next lowest high was thin/fat (thin/fat-attractive, r = 0.43), which, despite doubling in magnitude, was one of the easier attributes to predict (image #2). In addition, the fact that «outside» had the lowest correlation with all other attributes (r = 0.08) indicates a minimal contribution of contextual effects due to natural background and lighting.
Attribute Inference Prediction
To model the attribute, one had to start with multivariate representation vectors zi = {z1,…zd} assigned to each synthetic face (i) in the stimulus set using a pretrained modern GAN* (from the generative adversarial network).
GAN* (generative adversarial network) is an unsupervised machine learning algorithm built on a combination of two neural networks, one of which (network G) generates samples, and the other (network D) tries to distinguish correct («genuine») samples from incorrect ones. .
GAN learned the mapping of each such vector to an image through extensive training on a large database of photographs of real, non-synthetic faces. Then, each psychological attribute, measured by mean scores (yi), was modeled as a linear combination of characteristics: yi = w0 + w1z1 +… + wdzd. The weight vector wk = {w1,… wd} represents the attribute as a linear dimension traversing the representational space and is tuned using cross-validation.
It is worth noting that survey participants partially disagree in their assessments of judgments, which makes it difficult to formulate an accurate forecast. To better understand the prediction ceiling imposed by limited inter-rater reliability, it was necessary to calculate the reliability for each attribute using a half-split method, averaging the square of the correlation between the means of 100 random score splits for each image.
Curiously, the “familiar” and “similar to you” models showed the smallest gaps between efficiency and reliability. This indicates that their unpredictability is not due to the poor quality of the model or the absence of useful input features. Rather, it seems likely that familiarity, more than other attributes, is based both on a general concept or experience, and on a much broader personal concept or experience; only the former can be predicted for the participants in the population.
Attributes corresponding to some racial or ethnic social categories showed a larger gap between the reliability and performance of the model than other attributes. One possible reason for this gap is sampling bias in the stimulus generator.
Factors affecting the effectiveness of forecasting
In order to characterize the factors influencing the efficiency of forecasting, a study was made of the influence of the number of assessed persons on the effectiveness of forecasting (on top of image No. 3).
Efficiency curves were generated by fitting models for each of 30 random samples of images (from 100 to 1000 pieces). Most of the attributes benefited from an increase in the number of persons evaluated.
The scientists then examined the relationship between the number of ratings received from unique participants for each face and predictive performance (middle in image #3). Efficiency curves were generated by fitting models to downsampled datasets (from 5 to 30). The increase in efficiency due to the number of ratings decreased with the increase in the number of unique ratings, but slower than the increase due to the number of faces.
Finally, the relationship between the number of extrinsic features (512 in total) and predictive performance (below in Image #3) was explored. Efficiency curves were generated by fitting models using reduced feature sets obtained by principal component analysis (from 10 to 512).
In most cases, there was a rapid saturation of efficiency, but in some cases there was a slight improvement with an increase in the number of features. Evaluation of various saturation profiles showed that 10 features are quite enough for a satisfactory level of forecasting efficiency. At the same time, an increase in the number of features only contributes to an increase in this level.
Manipulating Attribute Inferences
Since the learned attribute vectors correspond to linear dimensions, it is possible to manipulate an arbitrary face represented by features zi with respect to attribute k using vector arithmetic: zi + βwk, where β is a scalar controlling the positive or negative modulation of the attributes.
A symmetric range of β around 0 was applied to each attribute vector to manipulate a number of face representations in both negative and positive directions, and decode the results for rendering using the same neural network decoder/generator component that was used to obtain the representation.
The results of the above transformations are shown in the image above. All manipulations were amazingly smooth and efficient with respect to each attribute parameter. For example, in the aspect of the “confidential” attribute, manipulation changed the look, smile, shape and femininity of the face. If there was a task to increase the “intelligence” attribute, then the algorithm tried to add points to the face and change facial expressions in general.
It is worth noting that manipulations with the output of attributes can affect both internal facial features and external features. When only the internal elements of a face are changed, it is not because the GAN only manipulates the internal elements, but because the external elements are orthogonal or irrelevant to this attribute inference in the region of the face being processed.
At the conclusion of their study, the researchers posed an interesting question: Do the above-generated attribute models reliably change participants’ perceptions of transformed faces? To answer it, scientists conducted a series of experiments involving 1000 people.
In each of the experiments, one of two face types (artificial or real) was combined with one of 10 different attributive measures chosen to represent a wide range of different models and levels of objectivity/subjectivity (age, femininity/masculinity, thin/fat, trusting, attractive, dominant, intelligent, sociable, memorable and familiar). As with the attribute modeling experiments, 50 unique synthetic faces were randomly generated for the artificial face experiments. In each trial, participants were shown one face and asked to rate it.
Each of the faces shown to the participants went through several stages of manipulation of attributes in order to give out as a result three levels of expression of one or another attribute. If the transformation of the attribute model does change the observer’s judgment of the face, then changes in the assessments on the part of the participants should be noticeable during the experiments.
An analysis of the results of the experiments showed that the manipulation of attributes really changes the participants’ perception of a particular person. At the same time, a linear trend was observed due to an increase in the level of manipulation of one or another attribute. Therefore, a person supposedly characterized as «trustworthy/trustworthy» could be transformed into «untrustworthy» and vice versa by manipulating attributes.
For a more detailed acquaintance with the nuances of the study, I recommend looking into the report of scientists.
Epilogue
In the work we reviewed today, scientists demonstrated an algorithm they created that was trained to model human value judgments in response to a demonstration of faces. This algorithm was supposed to predict how a person characterizes another person according to the features (attributes) of his face.
The preparation phase used data from more than 1000 people who were asked to look at photos of faces and rate them. As a result, individuals (or rather their owners) were assigned various characteristics: age, physique, intelligence, degree of confidence, degree of attractiveness, etc.
The data obtained was used to train the GAN neural network, which could later imitate the value judgments of real people. As a result, the resulting algorithm could independently evaluate and attribute certain characteristics to persons.
It turned out to be much more interesting that changing certain parameters (attributes) of a face using this algorithm can radically change its assessment. For example, a face that does not inspire confidence, with the help of an algorithm, miraculously turns into the face of a person to whom you are ready to entrust the keys to an apartment.
Developers are aware of the danger of such functionality. They themselves state that the manipulation of facial attributes can be used, for example, in election races, making the face of one candidate more trustworthy, and the face of a competitor more repulsive. Therefore, scientists immediately filed a patent for their development and began the process of creating a company to license the algorithm for pre-approved ethical purposes.
Despite concerns about the harmfulness of such technology, scientists intend to continue to work on it. In the future, they hope to improve the algorithm so that it can accurately predict a particular person’s value judgments in response to specific faces.