In the Eye of the Beholder
Professor Shinsuke Shimojo
Professor of Biology, California Institute of Technology
Of all the organs in the human body, the brain is one the most mysterious. While
a complete description of consciousness may not be within our grasps, scientists have made extraordinary progress is understanding the fundamental processes such as signal processing in our nervous systems. Well joining us today is Professor Shinsuku Shimojo, Professor of Biology at the California Institute of Technology and he studies Computational and Neural Systems. Frank Ling (FL) talks with Shinsuke Shimojo (SS) about his work in visual perception and his recent paper on the concept of beauty.
FL: Professor Shimojo, thanks for joining us today.
FL: So first of all, maybe you could give us a brief background. What exactly is psychophysics?
SS: My original background is psychology, more specifically, experimental psychology and traditionally speaking, psychophysics is a methdology in which you study a human experience of perception or memory or emotional judgement or anything mental from objective, behavioral, and quantitative ways. So, that is my definition of psychophysics. The reason why it is called psychophysics, a peculiar name, is we are studying the relationship between the psycho-world (the mental world) in relation to quantifiable physical environment (physical energy). So that is why it is called psychophysics.
FL: And you make a distinction between cognitive and neuroscience. What exactly is the difference?
SS: My field, psychophysics, in the current cognitive map of brain science is sort of falling between cognitive science and brain science, and trying to bridge the gap in between. The reason is because cognitive science mainly studies functions, including human and animals, but in relation to algorithms and computation, which can be basically simulated by computers, so that is essence of cognitive science. Brain science or neuroscience, as you may know, studies the materialistic, physiological, or genetic basis of the mental world. Those two are different. One is structure, the other is function. They no doubt, in a very very close and intrinsic relationship, not many people really understand how the relationship goes. So that is the kind of study we are trying to do.
FL: Recently, you had a paper out in December 2003 issue of Nature Neuroscience. In this paper, when a subject is asked to choose between two faces, they will spend more time gazing at the one that they find more attractive, and choose it eventually. Could you tell us a little bit about these findings?
SS: This is actually our first attempt to study anything related to emotion. Emotion, I think, as well as consciousness, are the two last challenges to scientists in cognitive brain science. The reason is because both are very subjective and science by definition and it’s methodology is objective. So how do you study subjective using objectiveness? It is a challenge. Recently, we came up with this idea that even though preference judgement or some aesthetic judgement is entirely subjective, there may be some somatic precursor, meaning body or physiological precursor, which proceeds to conscious awareness or conscious decision of the person. If this is the case, you can of course study the process objectively. There is a reason that this may be the case. For example, when one realizes that he or she likes somebody, it’s typically the case that he or she doesn’t really know why. Suddenly, he or she realizes that is the case. You know, it’s like a sudden incidence or a sudden occurrence. The reason is not obvious to the first person mainly due to implicit subconscious processes including lots of physiological, bodily action and neural processing, which is implicit again. This is the target in our case. To be more specific, I wonder if some of you may remember Williams James, the classical psychologist, asked as to whether one cries because one is sad, or one is sad because one is crying. Which is true? Which is the causal relationship? Of course, ordinary people would think typically that one cries only because one feels sad. A lot of physiologist and psychologist historically argue for the opposite causal link as well. Once you cry, for whatever reason, you feel even sadder and we wonder if the same applies to the relationship between gaze and liking. Just to rephrase the question, one could ask whether one looks at the particular thing longer because one likes it better or since one likes it better because one looks at it longer. The conclusion first, is that in fact both ways. The causal links, pathways in the brain, are both ways and this is like a positive feedback pathway.
FL: One can influence the other.
SS: One influences the other, the other influences the first one and then it’s positive feedback and you are conscious awareness of liking or judgment of liking is just the final outcome of this cascade like process.
FL: So, smiling can also make you happy?
SS: Exactly, it’s called the role playing paradigm in cognitive social psychology. When you feel sad, you smile and it helps you to be happier. There is evidence for this, but it’s related to this kind of study. Our case is unique in that it attempts to find a link between bodily orienting mechanism, which is well studied, well established on one hand, and subjective cognitive experience of liking and I can perhaps describe what exactly we did. So imagine that you are the subject. You are sitting in our lab and you are looking at two faces on a computer screen. These two faces in one case is pretty well matched in baseline attractiveness and race and gender and everything else. In some other experiments, they didn’t match. One is beautiful from anyone’s viewpoint, the other is ugly from anyone’s viewpoint. You have two faces, you are asked as a subject to freely observe these two until you know exactly which face you like better, which face you are attracted to more. And then, when you are aware of this, you have to press the button (left button or right button) accordingly to indicate your judgment. So the task is very simple. Meanwhile we are measuring your eye movements before your judgment. The question is the following: Is there any aspect of the eye movement patterns before the subject’s conscious decision which predicts the decision as to whether the left or right face is more attractive. We were trying to know the observer’s decision before the observer is consciously aware. It’s quite a bold attempt but as a result, we found that we could do it, but not with a huge amount of time ahead. It’s about one second or less, but still we can do it. The way we analyze the data is the following. So we measure eye movement pattern in each trial and then we average the likelihood of the eye staring at the particular face. Let’s say, the finally chosen face as a function of time to the final decision. So, in another words, we analyze the eye movement data time-locked to the final decision, not to the onset of the stimulus. And this turned out to the key to success.
FL: So you find a correlation?
SS: That’s right, so because we are interested in the coordination of eye movement, not so much to the onset of stimuli, but very much to the final decision. For this reason, we time-lock our analysis to the final decision. As a result, if you plot the likelihood to gaze staying at the finally chosen face as a function to time to the final decision. It turned out of, course, initially you are looking at both faces back and forth so it’s 50% and then some random noise. But then, about one second or less ahead of the final decision, then the likelihood start increasing toward 80% or more than 90% in particular cases to the eventually chosen face. Then when you gaze by us, it’s going beyond 85% or 90% depending on the condition, then you are finally ready to press the button. So this is the finding and the way we interpreted this result, the main finding, is again both ways causal relationship. Some kind of positive feedback pathway and then conscious awareness is just going beyond the threshold towards the end of this loop. So that is the main finding.
FL: You have a second part to your study, the gaze manipulation. Perhaps, you can talk about that a little bit.
SS: Before I step into it, I have to say two things. One is because of our theory — it’s somewhat a post hoc theory — we need to empirically prove the other pathway, the counterintuitive pathway. That is one is seeing therefore one like it better pathway, which is opposite to the intuition. We need to have a little more direct evidence and the direct evidence is to manipulate the observer’s preference by simply manipulating the gaze pattern. But the second is sort of a warning. We are aware this may sound terrible from ethical viewpoint. One is worried about free will and freedom and stuff like that. Are we trying to manipulate someone’s mind?
First of all, we were only successful to some limit. And as I will tell you it’s only 60% versus 50%, so we really didn’t tap on free will and I doubt this will really work outside of the laboratory, but nonetheless if our theory is correct, then we should see some increase or some effect of gaze manipulation on one’s preference.
Okay, so here is the way we did it. This time, there is only one face either on the left side or the right side of the screen alternating quickly. One is presented shorter, the other is presented longer and the subject was asked to naturally follow those faces with their gaze. This is repeated either two times, six times, or twelve times. Of course the more repetition, we expect more effects and towards the end of each trial, the observer saw two faces presented briefly and then asked to make a quick judgment over which side is more attractive. Now, it turns out that twice repetition of the shorter/longer face presentation did not have any effect. That is the subject’s chance or likelihood of choosing the longer presented face was roughly 50% and not statistically deviated from it, but if we repeated this six cycles or twelve cycles,, then suddenly, the choice is about 60% towards the longer presented and therefore longer gazed face. By this margin and this was true in twelve out of fifteen individually statistically significant subjects, so it is groupwise significant and individual-wise in most cases, significant result. So, with this margin of 50 to 60 percent. With this three exception out of fifteen subjects, we could manipulate successfully one’s preference accordance with our theory. So, for us this is the evidence that there is bidirectional causal pathways in the brain when one attempts to make a decision of preference. Now, I have to add that this can not be explained by any other artifact, so for the first main finding of gaze cascade itself, it happens only when one is asked to choose the more attractive face, never happens when one is asked to choose less attractive faces or round faces. It’s only when you are trying to find the more attractive one that this happens. For the manipulation experiment, the second part, we were worried it could be the effect of near exposure because there is another finding, no one in the literature, that whenever one is exposed to a particular stimulus longer or more frequently, and automatically the liking goes up, so in our case, you know, one face is presented shorter, the other face is presented longer. Maybe the longer face is just becoming more attractive only due to this mere exposure. We have some other control experiment in which the presentation time was again short but the subject is not allowed to move a gaze. In that case, we don’t get any effect of manipulation. It’s not so much of which face is presented longer, but it’s rather which face the observer gaze longer. So that’s why we think it’s not mere exposure but the gaze itself. In order to like somebody, meet the person more frequently. That certainly helps but that is not enough. You active audience towards this person. You know look at this person. Talk to you this person. Your entire body, mind and behavior oriented towards this person. It will make your impression better. So that’s the sort of out of the laboratory lesson which may or may not work again. If you take the experimental results seriously, that’s the kind of things that I can say.
FL: I’m just curious, have you found any differences in the perceptions between men and women? Children and adults?
SS: Yes, we analyzed all types of cross relationship between observer’s gender and stimulus material. As for the main finding, there are no differences. We haven’t tried age or children yet. I know that infants, even newborn infants, show this thing called preferential looking. Preferential looking is the phenomena that you can observe even in newborns. Whenever they are exposed to multiple stimuli, they compare and look at what is supposedly most attractive material or object, doesn’t have to be a face. So it seems like this is our intrinsic tendency. Almost innate tendency. But our new finding is that this contributes to our conscious symbolic, cognitive level of judgment. We, in a way, are finding connections to what we know about babies and what we know about adults. In reality, all adult used to be babies. This is not surprise.
FL: Are there any salient features which people would describe as attractive, for example, symmetry or…
SS: Yes, so it is an ongoing project that we are interested. There are tons of papers indicating that symmetry is very important, not just among humans but also among animals. Some argue that symmetry is not the entire thing. Some argue there is biological significance for symmetry. To us, what makes more sense is the past experience because there is another theory claiming that everybody has one’s own attractiveness template in the brain, which is based upon his or her past experiences of faces. In fact, the attractiveness template can be approximated by weighted average of the entire face experience in the life. So, that is kind of interesting because then you are more exposed and you sort of interact or orient to that face more, then you have stronger trace in the brain and that defines attractiveness. So our theory is not particularly proving or disproving this theory of attractiveness template, but it adds a lot to this attractiveness template theory because it points out to the other side of the causal relationship.
FL: I guess we are running out of time today. Are there last words you’d like to add about yourself or your work?
SS: I would like to add a couple of interesting new findings beyond this published paper. Everybody asks, “Is this limited to faces?” Even in the paper, we reported that you can get even stronger gaze cascade effect that I described with entirely abstract geometric figures, so we used Fourier descriptors that is mathematically defined shape generator and we created many symmetrical shapes and we get very strong gaze cascade in close link to preference judgment of these figures. We recently started trying commercial products such as jewelry, cars, furniture, stuff like that and our basic finding confirms that. It seems like whenever we are asked to choose whichever is more attractive regardless of the material, whether it is facial map, we generate this cascade. In some other studies, we showed that there is no way we avoid gaze cascade. For example, even if one face is most attractive, the other face is ugliest, we still need to generate gaze cascade in order to say consciously that this face is more attractive than the other. So, it is a necessary component of our brain to make any attractiveness judgment. So this might be worth mentioning.
FL: This is excellent, Professor Shimojo thanks very much for this interesting discussion. Thanks for joining us today.
SS: You are welcome.