The Ethics and Aesthetics of the Image Interface
Copyright © Paul Brown 1990
This paper was written in 1990 and a revised version was published in
ACM SIGGRAPH Computer Graphics, Vol. 29, No. 1, February 1994
All Rights Reserved
The Computer is a Communication Device
The Application of Semiosis
Commentary on the Semiotic Relationships
Now that we have developed and are introducing media-rich computer human
interfaces it is important that we develop a taxonomy that can help us
appreciate the power and value of computer systems in order to better quantify
them and assess their creative and ethical impact. To date the application of
semiosis has been to natural languages and human-human communications. The
author suggests that this can now be applied to artificial languages and in
particular their use in human computer interactions, imaging and visualisation.
The Semiology of Charles Sanders Peirce can be applied to this field and may be
used to assess the aesthetic (creative) and ethical implications of computer
human interface (CHI), imaging and visualisation technology and the way it is
being developed and used.
The two potentials that seem to essentially identify the uniqueness of
the digital computer system are:
Where the term "artificial intelligence" (AI) is used
extremely loosely to stand for any computational construct (database or
process) of interest or value.
- communication between two, or more, humans that is mediated and/or enhanced
by an "artificial intelligence" and;
- communication between a human, or group of humans and an "artificial
An example might include the simulation of air turbulence about a airfoil. The
computer software and hardware can be perceived as a device that is enabling a
researcher to communicate with a numerical process. The researcher provides
input (geometry, parameters, etc..) and receives output (visualisations,
graphs, statistical analyses, etc.). Furthermore the simulation purports to
map on to (or model) physical phenomenon in meaningful ways so the
computational process can be interpreted as enabling the researcher to perceive
aspects of the phenomenological world that would otherwise not be available for
scrutiny and similar therefore, in some respects, to other observational
instruments like the telescope or microscope.
Another example might include a group of people at remote sites who are playing
a networked game where characters they meet (or particular artefacts or moves)
may be simulations or other "real" human players (or their actions). All
players experience a consistent though artificial reality.
Both these examples illustrate processes that are provided uniquely by computer
processes and digital networks. Both are available now. In the very near
future both these kinds of experience will become more available as computer
performance increases, costs fall and network access grows. Given the
potential ubiquity of this technology it seems timely to consider the
importance of human values and to try and create a framework for discussion, or
taxonomy, that enables better understanding, development and application of
The success of the personal computer revolution was not only due to the
miniaturisation of components and of prices. Far more important was the
development of "user friendly" object-oriented and graphical user interfaces.
One of the most successful developments has been the use of metaphors. The
most famous is based on Alan Kay's Xerox PARC project, Smalltalk, and
popularised by the Apple Macintosh Desktop metaphor. Although this kind of
interface goes a long way towards reducing user intimidation it nevertheless
reinforces the misapprehension of the computer as a productivity/automation
device. Also, because of its strict definition and use of external referents
(like "desktops"), it limits human flexibility, creativity and exploration.
Kay himself has been quoted recently as saying: "you have to live with the
consequence of your metaphors".
This paper will attempt to identify the causes of this undesirable dichotomy
where sophisticated CHI's offer access only at the expense of deeper
appreciation and imaging and visualisation techniques more often confuse than
Pretty soon computers will have become tightly integrated with most
aspects of human life and endeavour. Already they are playing important roles
in education, employment and entertainment. Given the likelihood of a close
symbiosis developing between humans and computational constructs it is
important that we find a way of developing the computer human interface and of
presenting information in ways that:
It is important
that we should aspire to achieve these aims because:
- reduce user intimidation and encourage awareness and usage;
- do not disguise the inherent nature and potential of computer systems (ie.
that conform to an ethical paradigm) and;
- enhance (encourage) the users ability to develop novel and unique
solutions (that are not preprogrammed into the software or interface). (That
is to say they endow an aesthetic appreciation and usage.)
In order to better understand the effects of technology on human users
and to determine improved design parameters for user interface development it
is important to establish a taxonomy which provides classification and
establishes and helps analyse causality.
- everybody should have the opportunity to participate in the new
information routes that computer system enable. If we do not achieve this aim
we will be developing a society of "haves" and "have nots". This is both
undemocratic and undesirable;
- people should be aware that there are two sides to this flexibility of
access to information. The same freedom that allows us to plug plastic cards
into the wall and receive money almost anywhere in the world also allows
surveillance agencies to closely monitor our behaviour. Greater access to
information means loss of privacy. This self-evident truth is masked by the
"simple" metaphorical user interfaces that disguise computer systems as
friendly productivity enhancers.
- people should have the ability to contribute to the maximum of their
ability and not be constrained by predetermined limitations. It is important
to encourage creativity: people should be able to effect changes and make
significant contributions to the evolution of their world.
A number of possible ways to classify the development of computational
processing have already evolved. The First through Third Generations refer to
hardware - the use of valves, transistors and then integrated circuits. The
Fourth Generation was a fuzzy mixture of hardware (microprocessor-based
personal computers linked by networks) and software (application tools and
graphic user interfaces [GUI's] that created market confidence for high-volume,
low-cost computer systems). The Fifth Generation mixed in new hardware
architectures, improved user interfaces and artificial intelligence. The
recently announced Sixth Generation introduces issues concerned with artificial
perception and cognition.
Recent commentators have suggested that software generations (KAY84) or, in
particular, that the evolution of CHI provides an alternative analysis. John
Walker (WAL90) has suggested five significant stages in the historical
development of CHI: jackfields/plugboards; punch cards and batch processing;
timesharing and interaction; menus and; the GUI/WIMP (Windows, Icons, Mice and
Considering this analysis it would seem to me that the following taxonomy may
be of interest. It is based on the Semiology of the American philosopher
Charles Saunders Peirce and uses his three classifications of signs: Symbolic;
Iconic and; Indexical.
If the semiological categories are acceptable then we can qualify each
generation of user interface using these descriptions:
- The first generation interfaces that used keyboards and command/control
languages were concerned with Symbolic (linguistic, alphanumeric)
communications. The meaning or relevance (relationship between Signifier and
Signified) of the Symbol is defined by convention (user agreement) - they have
no intrinsic referentiality. The word H*O*U*S*E (or an "abstract" symbol) is
agreed to stand for a/the real house by those who share the language.
- The second generation (Menu-based, WIMP) interfaces involve Iconic
communications. Icons are simplified representations of "real" things. Iconic
interfaces allow gestalt methods of recognition and simple "point `n pick"
input. The iconic bonding to the reference "reality" is intuitive (at least
within cultural groups that share similar conditioning). A simplified
diagrammatic house is appreciated to stand for a/the real house. International
signage like railway and subway directions or Olympic logotypes use Iconic
- The newer interfaces involve media rich 2-D and 3--D interfaces including
multi-sensory and "immersion" technologies. These include the areas variously
described as hypermedia, multimedia, virtual reality, telepresence, Cyberspace
and artificial reality. They can include sophisticated methods for recognition
of human behaviours (handwriting, speech, gesture, expression, etc...) and
communication to the human includes both synthetic and reproductive graphics
presented with high fidelity and sometimes with wide field of view opaque and
transparent visual displays; binaural and spatial sound and voice synthesis
and both force and tactile feedback. These interfaces can be described using
Peirce's Indexical classification. This is a rich experiential and
existential set that has a close bonding with the underlying "reality" to which
it refers. The Indexical House Experience closely matches the experience of a
real house - this is a "natural" relationship.
- The Symbolic interface is hard to learn since it requires a knowledge of
the conventions intrinsic to the language being used. This can be an
intimidating experience. Nevertheless acquiring this level of skill enables
the user to exercise a very wide degree of freedom of achievement using these
tools. Because the user is intimate with the inner workings of computer
systems and their peripherals they are likely to gain a good appreciation of
their nature and of their social, political and economic consequence and
- The Iconic interface is intuitive but shallow. Although the user can
quickly learn to be productive using specific application tools they are
restricted to the limited set of opportunities that those tools signify/enable.
They are unlikely to acquire knowledge of the wider application, functionality
or sociopolitical implications of computer technology. Nor will they develop
general purpose creative-flexible skills (other than in the use of iconic
Tom Defanti, commenting on this interface at SIGGRAPH 88 said: "there are
too many people on this planet who think that computers are nothing more than
hyped up 3 by 5 file cards". Given the ubiquity of the technology and the
degree of power it endows on those who control it this widescale
misapprehension of computers as friendly beige productivity enhancers is
- The Indexical interface is very high bandwidth, rich, natural and
seductive. Since it is so new we have little evidence on which to base
qualification of its effects in practice. Nevertheless we can expect that the
interface has the potential of combining ease of use with wide variety of
achievement and both skill and knowledge development.
Its seductive nature will call upon developers to maintain the highest
possible ethical considerations to ensure that user achievement does not fall
prey to user addiction. There are many intrinsic limitations compared with
slower, but more detailed, symbolic communication.
It should also be
noted that higher-order interfaces can contain lower-order but that the inverse
is not true. That is to say an Indexical interface can contain Iconic or
Symbolic components; Iconic may contain Symbolic. The use of words in
Pull-down-menus or a programming window on a Desktop are both examples of
Symbolic in Iconic. The use of "roller blind" pull-down-menus in NASA's VR
demo is an example of Iconic in Indexical.
- Symbolic CHI
- Pro - intellectual, linguistic/alphanumeric, detailed, specific, precise,
intrinsic, unlimited variety, flexible, efficient, creative
- Con - slow, pedantic, difficult to learn, culture specific, intimidatory
- Use - specialist, developmental, creativity with the Metamedium
- Iconic CHI
- Pro - intuitive, visual/gestalt, attractive, productive, easy to learn and
- Con - application specific, disguises potential, addictive, restricted
variety, inflexible, high redundancy
- Use - professional applications, prototyping, creativity with the
- Indexical CHI
- Pro - natural, behavioural, seductive, fast, immersive, multi-sensory, high
variety, multi cultural, flexible, learn by experience, spontaneous, holistic,
- Con - seductive, highly addictive, very high redundancy
- Use - consumer applications, experiential creativity
As an aside I should mention that Kay, when developing Smalltalk was aware of a
similar classification system. He quotes the work of Jerome Bruner (BRU66) who
used the categories Enactive, Iconic and Symbolic to describe the development
of learning abilities in the child. Perhaps because he was unaware of the
semiological analysis he seems unable, or unwilling, to acknowledge the
inherent limitation of iconic communication: icons allow intuitive
comprehension but their gross simplification and tight bonding with the
signified object prevents then from offering the power of a more complex and
flexible, symbolic language (KAY90).
There is a parallel here in mathematical notation. The Chinese and Romans
developed iconic representation for number because they possessed calculating
machines (abacus) and only needed to record answers. In contrast the Vedic
culture developed a symbolic column sensitive notation which allowed written
forms of numerical manipulation which record and enable the process
(which the iconic forms prohibit). Another parallel is the Chinese iconic
(hieroglyphic) "alphabet" which contains many thousands of characters compared
to the English (Vedic/Islamic) symbolic alphabet which contains only 26
As I have hinted above there are significant problems with Iconic CHI.
Its very nature limits its application and prevents freedom of action outside
the bounds of a strictly defined application or process. Also in programming
Iconic interfaces their designer put emphasis on the computer system meeting
the expectations of the user rather than the user learning new skills. This,
in itself is a counter evolutionary development. Other significant problems
can by summarised:
Interface, at least in its current metaphorical application, doesn't encourage
development of the status quo. The measure of value is verisimilitude to
existing technology (paradigms) rather than creative exploration of new and
unique potentials that are intrinsic to the technology itself.
- the onus on adaptation (learning) has been put on tool for the first time
in human history. This restricts the innovation process and encourages a false
sense of security in the tool (BRO91);
- the digital programmable machine system is disguised as something else -
it hides its true nature. We have an immediate philosophical (ethical) problem
with the concept of essence, of truth to the medium. This encourages the
development of a belief system (it works!) in favour of a knowledge system (I
know how it works) indicating potential theological problems (BRO90) and also
reinforces the sense of false security;
- the Iconic user illusion encourages basic misapprehension of the real
issues: the unique potential of IT is overshadowed. Rather than discovering
new ways to solve problems the emphasis is placed on copying existing styles
and methods (BEE74). Innovation is limited;
- the implementation of the user illusion requires much more technology that
we really need to solve the problem at hand. It encourages waste, redundancy
and inefficiency. The potential power of the system is limited by CHI
- also of concern are human and ecological issues: third world women and
CFC's are both abused in the ULSI manufacturing process.
A number of similar problems arise from the obsession with verisimilitude. The
super-realism of much modern computer graphics imaging is often produced by
people who have little appreciation of the history of reproductive graphics or
of the evolution of perception. The history of painting can be interpreted
(somewhat crudely!) as the development of effective communication via
simplification. We also know from the contemporary record that early film
audiences fled in terror from a grainy, black and white, 16 frame per second
movie of a locomotive approaching the camera position. Now even the original
Star Trek shows from the sixties seem cliched and mannered in their use of sets
and special effects and audiences sit happily just a few feet away from
Cinerama screens showing the pyrotechnic gee-whizzery of films like Terminator
Is realism useful or necessary? Brenda Laurel (LAU92) comments that
researchers into rich media interfaces like virtual reality have discovered
that a high degree of simulated imitation in any one sensory channel is not
sufficient. Much simpler input, when it's multi-sensory and tightly integrated
(sound linked to vision for example) achieves a greater illusion of
Laurel also points out that kids can make do with simple wireframe graphics and
play for hours immersed in a fantasy landscape whereas adults require greater
degrees of stimulation to achieve similar depths of involvement. Maybe it's
not the performance of the system that's in question but rather the imagination
of the user! Lets remind ourselves that the panic caused by Orson Welles "War
of the Worlds" was created by a radio show. In bombarding ourselves with more
and more complex simulations and special effects maybe we are, albeit
unintentionally, saturating and cauterising our (and our children's)
Ever since Nancy Burson began exhibiting her digitally aged portraits of famous
personalities in the early 80's (BUR86) there has been a debate about the
ethics of simulation. This is particularly crucial in cases that involve legal
or political decision making. Although courts welcome the reconstructive
simulation of flight "black box" data intended to facilitate interpretation of
complex non-intuitive information there is considerable, and justifiable,
concern regarding the potential of photographic simulation in creating
misleading, or prejudiced, evidence.
The whole question of representation and truth has been brought home to the
computer graphics community recently in their relationship with scientists
using visualisation. A typical example is Antialiasing. Aliasing is a term
from signal theory that refers to a situation where a sampling frequency is
less that the frequency of that which is being sampled. The output is said to
be travelling under the "alias" of the sampling frequency. In computer
graphics a typical example is the fixed frequency of pixel boundaries and the
problem is manifest as jagged representations of edges and highlights.
Similarly the sampling of geometric surfaces leads to faceted networks of
polygons and limited colour space leads to quantisation banding.
Computer graphics researchers have invested considerable time and effort into
methods for antialiasing, developed splined surfaces and evolved colour map and
dithering techniques in order to overcome these limitations. Now however many
scientists, who need to keep track of their sample positions reject these
solutions as both unnecessary and confusing.
The problem of verisimilitude comes back once again to semiosis, the science of
signs, of the relationship between the Signifier and the Signified.
Communication is achieved when an adequate (and requisite) sign is adopted.
The discipline of computer imaging has not yet learned to wean itself from an
obsession with techno-glamor, of glitz for its own sake. Nowhere is this more
pronounced than at the large international graphics conferences where new PhD
graduates demonstrate their technical machismo. One reason for this may be
that the discipline has evolved as a part of the mathematical and scientific
community, who are primarily concerned with quantitative observation and
evaluation, whereas it correctly belongs within (or, at least needs a strong
input from) a community devoted to qualitative valuation like the arts and
The computer human interface, as well as imaging and visualisation
technology, can if they are not sensitively developed, be counter productive
and create confusion and information pollution. In encouraging short-term
expediency (in order to promote rapid adoption and usage) important long term
implications that would include evolutionary, social, theological, aesthetic
and ethical considerations, are often considered subsidiary if not altogether
overlooked. As interfaces and technology become more seductive, and as more
people become involved with computer usage, it becomes more urgent to make
consideration of these issues a primary research topic.
In particular we should reconsider an education system that imposes boundaries
between both academic and professional disciplines and between different modes
of learning and research. In order to tackle and solve the kind of problems
outlined above (as well as many other major problems of our age) we need a rich
crossover of skills and knowledge that no one discipline or method of study can
BEE74 Stafford Beer, Designing Freedom, London, Wiley 1974. Beer
describes the first attempts to automate accounting procedures which copied
19th century "triple entry" procedures and created overwhelming redundancy.
BRO90 Paul Brown, Communion and Cargo Cults, Proc. Second International
Symposium on Electronic Arts - SISEA, Groningen, the Netherlands 1990. In this
paper I suggest that Iconic interfaces, together with other aspects of recent
scientific discovery including Chaos Theory are precipitating a new
technological religion (belief system) and that virtual reality may provide its
equivalent of the communion experience.
BRO91 Paul Brown, Don't Look Now - You're Being Followed, MacNews
(Australia) November 1991. In my regular column "The Flip Side" I comment on
the unforeseen consequences of information technology usage. In this episode I
discussed the use of pro-active credit cards and networked consumer products
for public surveillance.
BRU66 Jerome Bruner, Toward a Theory of Instruction, 1966. Quoted in
BUR86 Nancy Burson, Richard Carling and David Kramlich, Composites,
Beech Tree Books (William Morrow) 1986.
KAY84 Alan Kay, Computer Software, Scientific American September 1984 V
251 #3 pp 41-47.
KAY90 Alan Kay, User Interface: A personal View, in Brenda Laurel (Ed.)
The Art of Computer Human Interface Design, Addison Wesley 1991.
LAU92 Brenda Laurel, On Finger Flying & Other Faulty Notions, in
Linda Jacobson (Ed.) CyberArts, Miller Freeman 1992.
TUF83 Edward Tufte, The Visual Display of Quantative Information,
Graphics Press 1983. Tufte coins the term chartjunk to refer to
unnecessary and confusing graphic elaboration of quantitative data.
WAL90 John Walker, Through the Looking Glass, in Brenda Laurel (Ed.)
The Art of Computer Human Interface Design, Addison Wesley 1991.
A revised version of this paper was published in
ACM SIGGRAPH Computer Graphics, Vol. 29, No. 1, February 1994.
SIGGRAPH is the Special Interest Group in Graphics and Interactive
Techniques of the Association for Computing Machinery (ACM). For
more information check:
Return to the top of page