The Ethics and Aesthetics of the Image Interface

Copyright © Paul Brown 1990
All Rights Reserved

Abstract
The Computer is a Communication Device
User Comprehension
Mission
The Application of Semiosis
Commentary on the Semiotic Relationships
Emerging Problems
Conclusion
References
About SIGGRAPH

Abstract

The Computer is a Communication Device

• communication between two, or more, humans that is mediated and/or enhanced by an "artificial intelligence" and;

• communication between a human, or group of humans and an "artificial intelligence".

An example might include the simulation of air turbulence about a airfoil. The computer software and hardware can be perceived as a device that is enabling a researcher to communicate with a numerical process. The researcher provides input (geometry, parameters, etc..) and receives output (visualisations, graphs, statistical analyses, etc.). Furthermore the simulation purports to map on to (or model) physical phenomenon in meaningful ways so the computational process can be interpreted as enabling the researcher to perceive aspects of the phenomenological world that would otherwise not be available for scrutiny and similar therefore, in some respects, to other observational instruments like the telescope or microscope.

Another example might include a group of people at remote sites who are playing a networked game where characters they meet (or particular artefacts or moves) may be simulations or other "real" human players (or their actions). All players experience a consistent though artificial reality.

Both these examples illustrate processes that are provided uniquely by computer processes and digital networks. Both are available now. In the very near future both these kinds of experience will become more available as computer performance increases, costs fall and network access grows. Given the potential ubiquity of this technology it seems timely to consider the importance of human values and to try and create a framework for discussion, or taxonomy, that enables better understanding, development and application of computer technology.

User Comprehension

This paper will attempt to identify the causes of this undesirable dichotomy where sophisticated CHI's offer access only at the expense of deeper appreciation and imaging and visualisation techniques more often confuse than communicate (TUF83).

Mission

• reduce user intimidation and encourage awareness and usage;

• do not disguise the inherent nature and potential of computer systems (ie. that conform to an ethical paradigm) and;

• enhance (encourage) the users ability to develop novel and unique solutions (that are not preprogrammed into the software or interface). (That is to say they endow an aesthetic appreciation and usage.)

• everybody should have the opportunity to participate in the new information routes that computer system enable. If we do not achieve this aim we will be developing a society of "haves" and "have nots". This is both undemocratic and undesirable;

• people should be aware that there are two sides to this flexibility of access to information. The same freedom that allows us to plug plastic cards into the wall and receive money almost anywhere in the world also allows surveillance agencies to closely monitor our behaviour. Greater access to information means loss of privacy. This self-evident truth is masked by the "simple" metaphorical user interfaces that disguise computer systems as friendly productivity enhancers.

• people should have the ability to contribute to the maximum of their ability and not be constrained by predetermined limitations. It is important to encourage creativity: people should be able to effect changes and make significant contributions to the evolution of their world.

The Application of Semiosis

A number of possible ways to classify the development of computational processing have already evolved. The First through Third Generations refer to hardware - the use of valves, transistors and then integrated circuits. The Fourth Generation was a fuzzy mixture of hardware (microprocessor-based personal computers linked by networks) and software (application tools and graphic user interfaces [GUI's] that created market confidence for high-volume, low-cost computer systems). The Fifth Generation mixed in new hardware architectures, improved user interfaces and artificial intelligence. The recently announced Sixth Generation introduces issues concerned with artificial perception and cognition.

Recent commentators have suggested that software generations (KAY84) or, in particular, that the evolution of CHI provides an alternative analysis. John Walker (WAL90) has suggested five significant stages in the historical development of CHI: jackfields/plugboards; punch cards and batch processing; timesharing and interaction; menus and; the GUI/WIMP (Windows, Icons, Mice and Pull-down-menu) interfaces.

Considering this analysis it would seem to me that the following taxonomy may be of interest. It is based on the Semiology of the American philosopher Charles Saunders Peirce and uses his three classifications of signs: Symbolic; Iconic and; Indexical.

• The first generation interfaces that used keyboards and command/control languages were concerned with Symbolic (linguistic, alphanumeric) communications. The meaning or relevance (relationship between Signifier and Signified) of the Symbol is defined by convention (user agreement) - they have no intrinsic referentiality. The word H*O*U*S*E (or an "abstract" symbol) is agreed to stand for a/the real house by those who share the language.

• The second generation (Menu-based, WIMP) interfaces involve Iconic communications. Icons are simplified representations of "real" things. Iconic interfaces allow gestalt methods of recognition and simple "point `n pick" input. The iconic bonding to the reference "reality" is intuitive (at least within cultural groups that share similar conditioning). A simplified diagrammatic house is appreciated to stand for a/the real house. International signage like railway and subway directions or Olympic logotypes use Iconic methods.
  

• The newer interfaces involve media rich 2-D and 3--D interfaces including multi-sensory and "immersion" technologies. These include the areas variously described as hypermedia, multimedia, virtual reality, telepresence, Cyberspace and artificial reality. They can include sophisticated methods for recognition of human behaviours (handwriting, speech, gesture, expression, etc...) and communication to the human includes both synthetic and reproductive graphics presented with high fidelity and sometimes with wide field of view opaque and transparent visual displays; binaural and spatial sound and voice synthesis and both force and tactile feedback. These interfaces can be described using Peirce's Indexical classification. This is a rich experiential and existential set that has a close bonding with the underlying "reality" to which it refers. The Indexical House Experience closely matches the experience of a real house - this is a "natural" relationship.

Commentary on the Semiotic Relationships

• The Symbolic interface is hard to learn since it requires a knowledge of the conventions intrinsic to the language being used. This can be an intimidating experience. Nevertheless acquiring this level of skill enables the user to exercise a very wide degree of freedom of achievement using these tools. Because the user is intimate with the inner workings of computer systems and their peripherals they are likely to gain a good appreciation of their nature and of their social, political and economic consequence and potential.

• The Iconic interface is intuitive but shallow. Although the user can quickly learn to be productive using specific application tools they are restricted to the limited set of opportunities that those tools signify/enable. They are unlikely to acquire knowledge of the wider application, functionality or sociopolitical implications of computer technology. Nor will they develop general purpose creative-flexible skills (other than in the use of iconic interfaces!).

  Tom Defanti, commenting on this interface at SIGGRAPH 88 said: "there are too many people on this planet who think that computers are nothing more than hyped up 3 by 5 file cards". Given the ubiquity of the technology and the degree of power it endows on those who control it this widescale misapprehension of computers as friendly beige productivity enhancers is undesirable.

• The Indexical interface is very high bandwidth, rich, natural and seductive. Since it is so new we have little evidence on which to base qualification of its effects in practice. Nevertheless we can expect that the interface has the potential of combining ease of use with wide variety of achievement and both skill and knowledge development.

  Its seductive nature will call upon developers to maintain the highest possible ethical considerations to ensure that user achievement does not fall prey to user addiction. There are many intrinsic limitations compared with slower, but more detailed, symbolic communication.

• Symbolic CHI

  Pro - intellectual, linguistic/alphanumeric, detailed, specific, precise, intrinsic, unlimited variety, flexible, efficient, creative

  Con - slow, pedantic, difficult to learn, culture specific, intimidatory

  Use - specialist, developmental, creativity with the Metamedium

• Iconic CHI

  Pro - intuitive, visual/gestalt, attractive, productive, easy to learn and use

  Con - application specific, disguises potential, addictive, restricted variety, inflexible, high redundancy

  Use - professional applications, prototyping, creativity with the application medium

• Indexical CHI

  Pro - natural, behavioural, seductive, fast, immersive, multi-sensory, high variety, multi cultural, flexible, learn by experience, spontaneous, holistic, experiential

  Con - seductive, highly addictive, very high redundancy

  Use - consumer applications, experiential creativity

As an aside I should mention that Kay, when developing Smalltalk was aware of a similar classification system. He quotes the work of Jerome Bruner (BRU66) who used the categories Enactive, Iconic and Symbolic to describe the development of learning abilities in the child. Perhaps because he was unaware of the semiological analysis he seems unable, or unwilling, to acknowledge the inherent limitation of iconic communication: icons allow intuitive comprehension but their gross simplification and tight bonding with the signified object prevents then from offering the power of a more complex and flexible, symbolic language (KAY90).

There is a parallel here in mathematical notation. The Chinese and Romans developed iconic representation for number because they possessed calculating machines (abacus) and only needed to record answers. In contrast the Vedic culture developed a symbolic column sensitive notation which allowed written forms of numerical manipulation which record and enable the process (which the iconic forms prohibit). Another parallel is the Chinese iconic (hieroglyphic) "alphabet" which contains many thousands of characters compared to the English (Vedic/Islamic) symbolic alphabet which contains only 26 characters.

Emerging Problems

• the onus on adaptation (learning) has been put on tool for the first time in human history. This restricts the innovation process and encourages a false sense of security in the tool (BRO91);

• the digital programmable machine system is disguised as something else - it hides its true nature. We have an immediate philosophical (ethical) problem with the concept of essence, of truth to the medium. This encourages the development of a belief system (it works!) in favour of a knowledge system (I know how it works) indicating potential theological problems (BRO90) and also reinforces the sense of false security;

• the Iconic user illusion encourages basic misapprehension of the real issues: the unique potential of IT is overshadowed. Rather than discovering new ways to solve problems the emphasis is placed on copying existing styles and methods (BEE74). Innovation is limited;

• the implementation of the user illusion requires much more technology that we really need to solve the problem at hand. It encourages waste, redundancy and inefficiency. The potential power of the system is limited by CHI overheads; and

• also of concern are human and ecological issues: third world women and CFC's are both abused in the ULSI manufacturing process.

A number of similar problems arise from the obsession with verisimilitude. The super-realism of much modern computer graphics imaging is often produced by people who have little appreciation of the history of reproductive graphics or of the evolution of perception. The history of painting can be interpreted (somewhat crudely!) as the development of effective communication via simplification. We also know from the contemporary record that early film audiences fled in terror from a grainy, black and white, 16 frame per second movie of a locomotive approaching the camera position. Now even the original Star Trek shows from the sixties seem cliched and mannered in their use of sets and special effects and audiences sit happily just a few feet away from Cinerama screens showing the pyrotechnic gee-whizzery of films like Terminator II.

Is realism useful or necessary? Brenda Laurel (LAU92) comments that researchers into rich media interfaces like virtual reality have discovered that a high degree of simulated imitation in any one sensory channel is not sufficient. Much simpler input, when it's multi-sensory and tightly integrated (sound linked to vision for example) achieves a greater illusion of immersion.

Laurel also points out that kids can make do with simple wireframe graphics and play for hours immersed in a fantasy landscape whereas adults require greater degrees of stimulation to achieve similar depths of involvement. Maybe it's not the performance of the system that's in question but rather the imagination of the user! Lets remind ourselves that the panic caused by Orson Welles "War of the Worlds" was created by a radio show. In bombarding ourselves with more and more complex simulations and special effects maybe we are, albeit unintentionally, saturating and cauterising our (and our children's) imagination's.

Ever since Nancy Burson began exhibiting her digitally aged portraits of famous personalities in the early 80's (BUR86) there has been a debate about the ethics of simulation. This is particularly crucial in cases that involve legal or political decision making. Although courts welcome the reconstructive simulation of flight "black box" data intended to facilitate interpretation of complex non-intuitive information there is considerable, and justifiable, concern regarding the potential of photographic simulation in creating misleading, or prejudiced, evidence.

The whole question of representation and truth has been brought home to the computer graphics community recently in their relationship with scientists using visualisation. A typical example is Antialiasing. Aliasing is a term from signal theory that refers to a situation where a sampling frequency is less that the frequency of that which is being sampled. The output is said to be travelling under the "alias" of the sampling frequency. In computer graphics a typical example is the fixed frequency of pixel boundaries and the problem is manifest as jagged representations of edges and highlights. Similarly the sampling of geometric surfaces leads to faceted networks of polygons and limited colour space leads to quantisation banding.

Computer graphics researchers have invested considerable time and effort into methods for antialiasing, developed splined surfaces and evolved colour map and dithering techniques in order to overcome these limitations. Now however many scientists, who need to keep track of their sample positions reject these solutions as both unnecessary and confusing.

The problem of verisimilitude comes back once again to semiosis, the science of signs, of the relationship between the Signifier and the Signified. Communication is achieved when an adequate (and requisite) sign is adopted. The discipline of computer imaging has not yet learned to wean itself from an obsession with techno-glamor, of glitz for its own sake. Nowhere is this more pronounced than at the large international graphics conferences where new PhD graduates demonstrate their technical machismo. One reason for this may be that the discipline has evolved as a part of the mathematical and scientific community, who are primarily concerned with quantitative observation and evaluation, whereas it correctly belongs within (or, at least needs a strong input from) a community devoted to qualitative valuation like the arts and humanities.

Conclusion

In particular we should reconsider an education system that imposes boundaries between both academic and professional disciplines and between different modes of learning and research. In order to tackle and solve the kind of problems outlined above (as well as many other major problems of our age) we need a rich crossover of skills and knowledge that no one discipline or method of study can provide.

References

BRO90 Paul Brown, Communion and Cargo Cults, Proc. Second International Symposium on Electronic Arts - SISEA, Groningen, the Netherlands 1990. In this paper I suggest that Iconic interfaces, together with other aspects of recent scientific discovery including Chaos Theory are precipitating a new technological religion (belief system) and that virtual reality may provide its equivalent of the communion experience.

BRO91 Paul Brown, Don't Look Now - You're Being Followed, MacNews (Australia) November 1991. In my regular column "The Flip Side" I comment on the unforeseen consequences of information technology usage. In this episode I discussed the use of pro-active credit cards and networked consumer products for public surveillance.

BRU66 Jerome Bruner, Toward a Theory of Instruction, 1966. Quoted in (KAY90).

BUR86 Nancy Burson, Richard Carling and David Kramlich, Composites, Beech Tree Books (William Morrow) 1986.

KAY84 Alan Kay, Computer Software, Scientific American September 1984 V 251 #3 pp 41-47.

KAY90 Alan Kay, User Interface: A personal View, in Brenda Laurel (Ed.) The Art of Computer Human Interface Design, Addison Wesley 1991.

LAU92 Brenda Laurel, On Finger Flying & Other Faulty Notions, in Linda Jacobson (Ed.) CyberArts, Miller Freeman 1992.

TUF83 Edward Tufte, The Visual Display of Quantative Information, Graphics Press 1983. Tufte coins the term chartjunk to refer to unnecessary and confusing graphic elaboration of quantitative data.

WAL90 John Walker, Through the Looking Glass, in Brenda Laurel (Ed.) The Art of Computer Human Interface Design, Addison Wesley 1991.

SIGGRAPH is the Special Interest Group in Graphics and Interactive Techniques of the Association for Computing Machinery (ACM). For more information check:

http://www.siggraph.org/

Return to the top of page