Stephen Grand

Virtual worlds are sterile places with limited usefulness if they do not contain living things. In general, research has focused on ways of bringing real human beings into virtual contact with each other, through shared virtual worlds. However, it is increasingly being recognised that, for many purposes, artificial life forms (plant, animal or human) are a highly desirable addition. The problem, of course, is how to construct and "program" these virtual creatures. Artificial Life takes a biological (or at least biologically plausible) approach to the problems of AI. Compared to traditional AI, this is a difficult and uncertain road, yet it is likely to be a more productive one in the long term. Nevertheless, fifty years of procedurally-oriented computer science, and traditional, symbolic approaches to AI have left a legacy from which it is hard to break free. Much work in virtual environments also owes its origin to existing computer graphics thinking, which in turn draws from traditional animation ideas. All of these things have a tendency to colour the viewpoint of system designers, and consequently it is possible that progress is being hampered by inappropriate paradigms.

Computer-generated graphics work is obviously primarily concerned with the appearance of things. First and foremost, workers focus on the visual rendition of form, through 3D graphics techniques. Given the immense challenges in generating fast, faithful, 3D displays, animation tends to take second place, and true behaviour a poor third. Because of this perspective, workers in virtual reality often approach the problem from the outside (the graphics on the screen) towards the inside (the underlying model). The graphical design comes first, the animation system is added to the graphical system, and any autonomous behaviour is generally appended to that (VRML2 is an example of this logic in action). This outside-in viewpoint can sometimes lead to a faulty approach, since it is the reverse of reality. Perhaps above all, it can lead to an insidiously "image-conscious" attitude to system design, and the results might more fairly be described as an emulation of the symptoms of reality than a model of reality itself. A better approach is to think from the inside-out; from a model of the underlying system outwards, towards the means of its visualisation. The behaviour should emerge from the representation, not be "painted on" from the outside.

Inside-out thinking implies that the internal representation is paramount, and the rendition must follow from it, not dictate it. But what kinds of internal representation are useful and how deep must the representation go? One way to examine this is to consider "orders" of simulation. The following hypothesis is simple to assert but contentious and difficult to defend. At the very least, however, it suggests a shift of viewpoint that can have useful practical consequences. It can be stated via an example:

The atom in the above example is a "first-order" simulation, in that it uses procedural concepts to "emulate" the behaviour of an atom. Just because such a procedure contains a rule that describes its reaction to an accelerating force, it does not mean that the code itself has inertia, just that it emulates the symptoms of inertia. The molecule, on the other hand, is a second-order simulation. Its properties are entirely an emergent consequence of the atoms of which it is made. When such a molecule exhibits some phenomenon, that phenomenon is in a real sense equivalent to any similar emergent phenomena caused by aggregates of real atoms, albeit operating in a different universe. The atom is a simulacrum; the molecule is an example of true virtual reality.

Central to this idea is a view of the world in which the entire structure of the universe, from the passage of a photon in space to the mind of a conscious creature can be described in terms of a complex and fuzzy hierarchy of meta-phenomena superimposed on simpler phenomena. The assertion is that one can simulate phenomena at any point on that hierarchy, but the simulation becomes "real" only at points higher up the scale. This applies continuously, so, for example, a simulated neurone is not really a neurone, but if many such neurones are aggregated to form a brain, it is not the brain's fault that its components are a sham—it will still be a brain. Should that brain then exhibit the properties of a conscious mind, we are being quite rational to describe it as conscious. Emulating the symptoms of consciousness by coding explicit "rules for conscious behaviour", on the other hand, is a first-order construct, and the result cannot be described as conscious.

In general, the more levels of abstraction (orders of simulation) between the procedural emulation of some structure and the aggregate behaviour that is actually the object of the exercise, the more realistic that behaviour will be. This includes its "pathological" behaviour—simulated people suffer from "bugs"; virtual people that are built from simulated components become "ill". See [1] for more on the philosophical argument and [2] for evidence that this approach can be applied in practical circumstances.

Following from this nth-order concept comes a "strong" view of cyberspace. "Procedural space" is a term that can be used to locate first-order simulations. The atom above is simulated by procedural rules, and therefore exists in procedural space. Outside procedural space, where second- and higher-order phenomena reside, lies a space that more closely parallels real Cartesian space. It seems reasonable to describe this as "cyberspace" (with some qualification regarding the term's more conventional usage). Cyberspace is the space in which nth-order simulations exist, and nth-order simulations are true virtual reality.

Regardless of how valid this idea is in literal terms, it has implications for simulation, as it seems to imply a conceptual distinction between first-order, procedural methods and higher-order, "structural" methods. Such a distinction in turn implies a boundary, and there is some logic to marking that boundary with a metaphorical "membrane". Membranes are one of the most crucial concepts in biology, and a number of things fall out of the notion that procedural space should be isolated from cyberspace by a membrane, just as cytoplasm is isolated by the membrane surrounding a biological cell. The principal assertion of this paper, and the rationale behind the modelling system to be described below, is that such a "cellular" metaphor has significant conceptual and practical advantages.

The notion of a cell membrane surrounding procedural space and separating it from cyberspace is not just pseudo-technical babble. Object-oriented programming systems (OOPS) go part way towards this by encouraging the programmer to encapsulate the properties and attributes of an object within a class boundary. OOPS systems do this for good reasons, primarily in order to reduce entanglement in program design, hence reducing software complexity.

Complexity is a major issue in most programs, and cyberspatial complexity will be a major issue in future virtual worlds, particularly those containing complex, autonomous artificial life forms. In particular, intelligent virtual worlds have a need for many classes of object to be able to interact with many other classes of object without explicit pre-programming of every interaction. For example, imagine a simulated townscape, populated by a mere 100 intelligent agents (people, dogs, birds, etc.) and 1000 assorted objects (doors, vehicles, traffic lights, books, etc.). Suppose the intelligent agents can interact with each dumb object in several different ways (a person can open or close a door). Suppose also that some objects can be used on other objects (a hammer may smash an egg), and that the sequence of interactions may be important (pulling the trigger before loading the gun has a different effect than pulling it after). It can readily be seen that the total number of possible interactions is astronomical, and because the agents are autonomous and have free reign over their actions, all possible interactions must be catered for. Not only that, but the agents have to be able to make sense of this rat’s nest of interactions. If every interaction is unique, there is no potential for generalisation, the key to intelligent behaviour.

A first step in managing the problem is to reduce entanglement by encapsulating the code into objects. It would be a brave designer who was willing to program in the behaviour of the entire world as if it were one vast object. It would be much more sensible to divide the program up in the same way as the conceptual world—one class for all cats, one for all policemen, and so forth. This disentangles each object from the others and allows the programmer to deal with each one individually.

But how does the programmer define the interactions between objects? In a conventional OOPS language, objects impinge on other objects by calling their member functions (called "methods" in some languages), and each class of object has its own unique set of functions (even if inheritance allows some methods to be shared). This can be a nightmare of complexity, since each object that might interact with another object needs to know what member functions it can invoke. It is also a serious inhibitor for an intelligent agent, since all possible interactions must be known in advance and hence the behaviour of the agent is largely preordained and can’t truthfully be called intelligent. Moreover, suppose the virtual world has to be designed in order to accommodate new, as yet unwritten objects. How can an agent know what interactions are possible when its methods are not known at the time the agent is constructed?

In terms of computer science, this problem of large numbers of arbitrarily interacting objects demands a new paradigm. In the operating systems world, this paradigm is called component-oriented design. A component differs from an OOPS object in that it is "mobile" (can come and go) and has incomplete prior knowledge of other objects with which it may need to interact. In operating systems, these components may represent radically different kinds of thing, yet a significant amount of reasonable expectation exists. So, for example, a word processor program is likely to contain "word" components, and these are likely, though not guaranteed, to support functions such as "italicise". Under these circumstances, the emerging method of interfacing components is to use interrogation. Every component must guarantee to support a member function (called IUnknown in Microsoft’s component model [3]), which can be interrogated to discover what other functions the component supports. However, this only works if the calling component has some idea how to make use of these functions. For our purposes we need something different.

Perhaps the only other realistic option is for all components to have essentially the same interface. If the interface is consistent across all objects in the virtual world, every object automatically understands how to interact with every other. This couldn’t work in the disparate world of computer operating systems, but might make sense inside a virtual world. Yet what should this hypothetical interface look like?

One of the most hideously complex yet reliable machines on the planet is the human body—over one trillion parts acting together to form a coherent whole. Clearly no one cell in the body knows about, cares about or controls all of the others. Such a top-down approach would result in a combinatorial explosion. Human beings are, like OOPS or component systems, encapsulated into objects (cells being the most obvious). Part of the "secret" that allows biological cells to interact without needing prior knowledge of each other's functionality is that all cells are essentially similar. Blood cells and kidney cells have strikingly different functionality, yet they are still cells, and consequently share a common "interface" that enables them to talk. Perhaps there are clues in the structure of multi-cellular creatures that help us to find a unified interface that makes sense in a virtual world.

In fact I have reason to believe this is true, plus some experience of what a few of those common interface features are. Some time ago I wrote a program called Creatures [2] (Figure 1), in which quite complex intelligent agents were constructed from aggregates of relatively simple building blocks (primarily neurones, chemoreceptors, chemoemitters, chemical reactions and genes). The creatures were capable of learning, thinking, directing attention, digesting simulated food, respiration, sexual behaviour, reproduction, inheritance, evolution, illness, and assorted other behavioural traits. All of these phenomena arose from "circuits" of these basic components, relying on a few basic interfaces to enable, say, chemoreceptors to attach to neurones and modulate their activity, etc. At the same time, these creatures lived in a virtual world comprising a reasonably wide assortment of objects, including toys, vehicles, food and of course other creatures. Again, these objects were constructed from a few basic types, some solitary (small objects like food) and some aggregate (vehicles and creatures’ bodies are made from multiple parts). Finally, the creatures and objects were contained within a single virtual world object.

Amongst the features of this toolkit are the ability to add new objects without the other objects (or creatures) needing a priori knowledge of them, a general robustness that allows the system to fail gracefully (if messages fail to be delivered, or objects stop working), plus of course the ability to construct second-order moderately intelligent, autonomous life forms. What the system lacked, however, was the final step that would have solved more general problems. I essentially had three main classes of object:

The virtual world. This was dumb and very limited in power and flexibility.

Objects. These were highly flexible, scripted objects. They could be created on the fly in the virtual world, by injecting scripts from outside the application. They couldn’t, however, be freely aggregated or formed into "circuits".

Neurones and their associated biochemistry. These were very fast to process, could be wired up into arbitrarily complex circuits and could be specified by genes (and thus recombined in new ways through inheritance). On the other hand, they could only execute simple functions individually. They were also not visible to the user, could not move under their own steam and could not interact via broadcast messages, as objects could.

Moving on from Creatures to deal with more general instances of intelligent virtual worlds (particularly 3D ones) it occurred to me to wonder whether these three object classes could be combined into one—a single class of object that would serve all these purposes in a general way. By having one unified interface between all objects at all levels of the hierarchy, everything would automatically be able to talk to everything else. By making use of biological metaphors that had already been more useful than expected in Creatures, such as chemical reactions, receptors and hormones, but widening them to encompass further kinds of interaction, I wondered how far it would be possible to go. Could this cell biology metaphor extend from the simulated biology of intelligent artificial life, right out to inanimate objects and even the virtual world itself?

In the Creatures world, an intelligent agent would be made from a compound object (a limited type of aggregate), its brain and physiology would be constructed from neurones and biochemical components, and it would live inside a virtual world object. However, the interactions between each level of that hierarchy were specific—neurones and creature objects talked to each other differently from the way objects talked to the virtual world. By building everything from the same building blocks, everything could talk to everything else, regardless of position in the hierarchy. Indeed, the hierarchy could be a great deal more varied and fuzzy.

I then started a project at my company (Cyberlife Technology Ltd.) to build a unified virtual world modelling toolkit called Origin, based on these ideas. Origin has been in development for roughly a year, and has proved the principles. We have used it in a number of serious applications, from simulating the behaviour of customers in a virtual bank branch to simulating neural network-controlled pilots in fighter aircraft (figure 2). At the time of writing, we feel we have learned enough to re-write the system from scratch, primarily to improve efficiency and increase scaleability. This has enabled us to incorporate a number of architectural improvements.

The design is too complex to discuss in detail in a short paper, however Figure 3 shows a rough schematic view of the two basic structures in Origin—cells and organelles. Cells are the basic building block from which all elements in cyberspace are constructed, either as single-celled objects or multi-celled (often jointed) aggregates. Organelles are a generalised mechanism for providing the inputs, outputs and internal functionality for cells. Organelles can be configured to represent several types of biologically inspired structure, for example chemoreceptors, sensors (eyes, ears, touch), neural synapses, chemical reactions and cell properties (rules for behaviour). Cells communicate with each other by means of direct connections (an organelle on one cell "synapses" onto a locus on another), scanned connections (a sensor organelle tracks cells in the vicinity and monitors their loci) or hormones (a kind of messaging system). Finally cells can communicate with loci in their parent cell (the cell that contains them) and these loci can interact to model spatially distributed systems such as chemical gradients, ground water seepage, etc.

To understand how Origin might be used, imagine the following virtual scenario. The world consists of a townscape, containing buildings. On the streets are cars and other vehicles. In the back seat of one of the cars is a virtual human, eating a sandwich.

Let us begin at the finest grained level. The virtual human is controlled by a neural network. Each neurone is a cell, and connects to other cells via synapses (organelles). The various sub-functions of the brain are supported by different regions of neurones, called lobes. Each lobe is also a cell, and handles some lobe-wide (i.e. diffuse) behaviour. For example the map of loci on the inside surface of a lobe can represent a field upon which chemical gradients can form, emitted during activity in certain neurones in the lobe. Such gradients might be involved in helping a part of the virtual human’s brain learn (in this case, probably about sandwiches). All of the brain lobes are contained inside a single "brain cell". None of these parts need to be visible to users of the virtual world, and so none are given meshes.

The brain exists inside a head, which of course is a cell. Organelles in this head act as the agent’s senses (eyes, ears, touch, taste). Unlike the brain, the head is visible, and so is associated with a mesh, textures, etc. The head is connected to other body parts, such as the trunk and the various limb segments. All these cells are siblings of each other, and are jointed together so that they can articulate. When the brain wants to move the sandwich to the agent’s mouth, it sends a message to the arm muscles, which cause the various arm joints to swivel and the hand to move towards the mouth. Similarly, movements of the legs would cause the agent to walk across the landscape (but for now the agent is sitting in a car). Inside the trunk cell is a set of chemical reactions (organelles) modelling the creature’s digestive and respiratory systems, etc. When the sandwich (which is a single-celled object) is placed in the agent’s mouth, it injects various nutrient chemicals. These get digested by the chemistry and provide the energy that the muscles are programmed to require in order to work. Surrounding the whole virtual person is a single, invisible bounding cell, which makes this aggregation of cells into a single thing, as far as other people or objects at the same level of hierarchy are concerned.

The car the person is sitting in is also made from various cells surrounded by a bounding cell. In this instance, the cells are not jointed to each other but can still animate individually (the wheels can rotate, etc.). The engine of the car is modelled using "biochemistry"—chemical reactions control the consumption of fuel, etc. Refuelling the car simply involves an injection of a chemical (as in the sandwich)—the same mechanism works in many places.

The car sits on the street, surrounded by building cells (some of which contain room cells). Finally, the whole thing is contained within the "universe" cell—the top of the cell hierarchy. This universe cell, because it contains the same functionality as all the others, can have its own complex behaviour. Maps of loci (accessible to sensor organelles on child cells) model such things as the formation of puddles when it rains.

Biology may seem to provide an odd set of building blocks for simulating non-biological objects, but the metaphors transfer well, and even notions such as biochemistry and genetics can play a part in providing useful modelling facilities. However, biological building blocks are self-evidently well suited to the construction of virtual organisms, and complex biologically inspired creatures and even humans can be constructed, that can breed, evolve and learn (given suitable second-order constructions). One of the most important aspects of this structure is that these creatures both interact with cells and are made of cells; the agent is totally grounded in its environment, since it and the environment are made of the same stuff. Because of this consistency, the agent can perceive its environment in a coherent way, and make rational sense of the movements, colours, sounds and other behaviour of all things around it, regardless of what kinds of things they are. Similarly, it can interact with all kinds of objects in a consistent fashion, and they will behave predictably and rationally. Such deep grounding is a prerequisite for true intelligence.

This paper has thus argued for an inside-out approach to the modelling of virtual environments, using nth-order constructs, in which procedural space is strongly isolated from cyberspace. By using the methods through which multi-cellular organisms are constructed, a complexity-damped, robust mechanism can be created, in which virtual environments, objects and organisms can be made from the same stuff, in one coherent system. Such building blocks lend themselves particularly to the simulation of complex organisms with truly emergent behaviour, supporting intelligence that is strongly grounded in a rational and self-consistent sensory environment. Such a system has been built and is now in use for creating industrial and entertainment virtual environments.

[3]. The Component Object Model: Technical Overview. Microsoft Corp. 1997. Web document at:
http://www.microsoft.com/oledev/olecom/Com_modl.htm