Written: 03/24/2008

Introduction

Every year, millions of video games fly off the shelves of retail stores around the world, and even more are purchased online. In 2007 alone the computer games industry posted a gross profit of $17.94 billion according to the NPD group, a number that far surpasses box-office cinema’s $9.7 billion gross profit (YVG Staff). After purchase, many hours are spent in the games, creating characters, building civilizations, defeating enemies, solving mysteries, and plumbing the depths of archaic dungeons in some distantly imagined world. As these games have progressed and hardware has improved, the experiences offered have become far more visceral and engaging as the artists behind these titles are given the freedom to imagine unique worlds, and they use today’s bleeding edge software and methodology to create them. According to nVIDIA Corporation, “As 3D graphics hardware has evolved over the years, new features have been continuously added resulting in steady improvements to image quality” (Technical brief pixel). Despite the universal appeal of these games, however, as they become more complex, and, more importantly, the processes used to create them become more intricate, the lay audience has less and less knowledge of how they are constructed and how they operate.

Compare this once again to the movie industry. Despite the intricacies of filmmaking, most people who watch movies have at least a fundamental knowledge of how they are made: a camera is used to capture images, usually actors conversing or interacting in some way. It’s common knowledge that a script is followed, and there are stunt men, and intricate lighting, and large set pieces etc. None of this should sound particularly new, and this understanding of the industry leads movie-goers to appreciate particularly good films. On the other hand, most people don’t know how the character they play in a video game is created, nor do they understand how the water in the game flows, or how the light reflects off windows, or even how the doors of the houses swing open. Does anyone outside of the industry understand what UV texturing is? Or could perhaps define a shader? This lack of awareness of the industry is understandable; the concepts behind game development are far less tangible than that of film making, but this ignorance is worrisome when compared to other artistic entertainment mediums. This lack of transparency is a problem because if no one understands the process that games endure from the conceptual phase to become the sometimes breathtaking products that grace retail shelves, then it is easy to dismiss them as some cheap, easy to create market fad. The lack of appreciation is damaging to the reputation and understanding of the gaming community as a whole and severely inhibits its progression as an art form. The artists within the industry need to take the initiative to promote the understanding of the art form just as the audience needs to be responsible for the absorption of this knowledge. The following is intended as a step in that direction by promoting education of the subject.

 

The Art Process

As a game artist myself, I am often asked a question that makes me want to slap my head in frustration: “So did you make that or did the computer make that?” The reason this is frustrating to me is that people, when they see those images on the computer screen or television, assume the art they view is created largely by the computer, when in fact the machine does nothing artistically. The other frustrating circumstance is when someone who has played a lot of video games feels like they are competent enough to create their own grade A title. While this is a step in the right direction, the assumption that one can build a car just because they can drive it is uneducated and reflects an ignorance of the intricacies of mechanics. The same is true for video game art. The thing that sets video game artists apart from many other artists is simply the wide range of skills required to produce finished assets for a game, and the sheer number of people required to ship a product. Thirty nine artists were used to create the assets for 2K’s “Bioshock” (Finley). Different game artists have to be able to fill many roles, from envisioning the final product they will make, sculpting it in 3D much like traditional sculpture, being able to slice it open and unwrap that 3D object, and painting like traditional painters. On top of that, they will also often have to give the object a bone structure, and then give it motion using the bones to create animation. The computer’s impact on these creative processes is minimal to none (with the exception of unwrapping the object).

 

The Vision

The entire process begins in the conceptual phase. While not actually seen by the end-user, this phase is one of the more important because it really determines the way the asset it depicts will look at the end of production.

During this phase, an artist (usually conceptual artists are fairly specialized) will draw sketches or paintings of environments or characters (Fig. 1, Fig 2).

Fig.1 http://www.philipstraub.com/death_dealer.htm	Fig.2 http://www.markbehm.com/

This is usually done as a mixture of both traditional and digital processes. Many conceptual artists sketch with paper and pencil to get the form of what the product will look like. They will then often color them digitally by painting in programs like Photoshop. This is nearly identical to traditional painting (in terms of the methods used, rather than the medium) since now, with the advent of tablets[1], artists have much more flexibility in their work. Tablets are basically a surface that tracks the motion of a special pen that then transposes everything onto the screen (Graphics Tablet). This is important because it gives artists the benefit of having a more tactile interface with their artwork that artists in more traditional, non-digital fields have always enjoyed. Aaron St. Goddard is an artist who employs this dual method. He says: “I work with natural media (paper, pencils, pens) for all my drawing, and Adobe Photoshop 7 for my coloring” (Concept Art).

The purpose of the conceptual art is many-fold. First it establishes the basic details: does character A have red hair? Is it tied up in a pony tail, or cropped short? Does his belt carry pouches attached to it, or is it more ornate with a huge buckle? Conceptual art gives the answers to all these questions by allowing artists to see a representation of the final product before creating it. Conceptual art essentially sets up a roadmap for the rest of the creative process.

On top of this vision, conceptual art helps in establishing the materials related to the asset, and how they interact with light. If character A is wearing a leather jacket, is it a soft suede leather that gently scatters the light it receives, or is it a harshly treated set of leather armor, complete with scratches and scuffs? While figuring out the way that a material interacts with light may seem fairly trivial, it’s vital to achieving the believability of the material. How could you tell glass from a concrete wall if it wasn’t smooth and didn’t reflect and refract light while letting some pass through so that it’s transparent? The way we are able to identify glass or any other material is through the way that light interacts with it. While this may have been far less important five years ago, it’s vital now as the expectations of the audience have risen and technology has met that increased demand on visual fidelity.

Concept art also is used to create believability in character or creature art. When designing a creature, a concept artist will want to make sure it moves correctly, and include details that would allow the viewer to interpret its functions. When designing creatures, Aaron St. Goddard tries “to create a level of realism in my designs which will make people think that it really can exist and function in the real world” (Concept Art).

And finally, and probably most importantly, concept art is used to establish the overall look of the asset. Knowing what the end product will look like is incredibly important for all games, but especially when creating a game with some sort of stylistic choice like a cartoon style game, or perhaps an anime game (Fig. 3).

Fig.3 Valve Software

It’s vital that the overall visual style of the game is established so that all of the art assets maintain that style. According to art director Michael Khaimzon with the German-based developer Crytek, to establish the look of their aliens in the game “Crysis”, they went through “hundreds” of concept sketches before they found what they wanted. Furthermore, says project director and executive producer Cevat Yerli, “we spent more time and money and resources ever in one creature than probably the entire budget of FarCry [their 2004 title]” (Making of Crysis).

 

Changing Dimensions

After a solid concept has been nailed down, work begins on the asset that will be seen in-game. The first step to this is creating the three dimensional object, called a mesh. In its purest form, this is very similar to traditional clay sculpting, but with some technical twists. Firstly, three dimensional space within the computer is represented along 3 axes: X, Y, and Z on a 3-space Cartesian Coordinate system. For example, the coordinates for a point in 3D space could be written as (3,-4,7), indicating that the point is located positive 3 units along the X axis, negative 4 units along the Y axis, and positive 7 units along the Z axis (Fig. 4).

Fig.4

While it’s important to understand this, the computer really does the legwork for the artist as far as the coordinate numbers are concerned, and the artist can manipulate the points visually (The Art of Maya).

To create a 3D mesh, the artist, called a modeler (or mesh modeler), will often begin with what’s called a primitive. A primitive is reasonably self explanatory: it’s a very simple geometric object. Some primitives include cubes, spheres, cones, planes, cylinders etc. It’s important to remember that in real-time, a computer is able to render only geometric objects, triangles[2] specifically, as they are the simplest shape in Euclidian geometry, and are planar. A triangle has three points, or vertices (which simply represent a defined point in 3-space). A line connecting two vertices becomes an edge and connecting two edges creates a face[3]. Faces indicate solidity within the mesh and can be manifested either as a triangle or quadrangle depending on whether it contains three or four vertices (Fig. 5).

Fig.5

This means that if you see a curve in a game, perhaps an arched doorway, that doorway is actually made of several perfectly straight connected lines bent at incremental angles in order to fool the eye into perceiving a curve. The more geometry used, the more convincing the curve. This is at a complete juxtaposition with traditional art, where, in the real world, perfectly straight lines simply do not exist; they must be approximated (Polygonal Modeling).

Another important thing to note in computer games is that volume is a complete illusion. There is no such thing. As mentioned, everything in a game world is made up of the completely flat, planar triangles or faces. The way that the illusion of volume is created is when an artist takes two triangles, combines them to form a quadrangle, duplicates that quadrangle six times, sets all those quadrangles at 90 degree angles to each other, and there’s a cube! A mesh modeler rarely wants the edges of the geometry to pass through the surface of other geometry because it can dispel the volumetric illusion. The easiest way to imagine this process is probably to picture that everything in the computer game is made up of pieces of paper; people, cars, houses, rain, glass, everything: it’s all made up of completely flat pieces. While this may sound daunting, technology has improved well enough that it is fairly easy for the artist to control these many flat geometric pieces (Polygonal Modeling).

The way that a modeler manipulates these polygons is by controlling the position, rotation, scale and amount of groups of vertices. Vertices are the points of the polygon and represent an arbitrary point in 3D space. For example, a triangle has three vertices (or ‘verts’) that can be controlled, one at each tip. Vertices essentially are define the length of a side of a polygon (an edge) by acting as end-points. Of course, simply moving the vertices gives only limited control (you still only have a four sided figure at most to manipulate), but through a process called extrusion an artist can create as much geometry as is needed for their object. Extrusion essentially duplicates anything that an artist has selected to extrude, but instead of simply creating a copy of the object, extrusion forces the duplicated object to remain attached to the original at all of the vertices that define the edge of the shape. An alternative to extrusion is to work from a very low polygon object like a cube and slowly subdivide that object until the desired amount of detail is achieved (Fig. 6).

Fig.6

Normally, a mix of these processes is actually used rather than a single style (Polygonal Modeling).

Because of the limitations with hardware and how quickly images must be rendered on-screen, game artists are presented with a unique challenge in their work. This challenge comes in the form of trying to balance the amount of polygons that are used with achieving the desired appearance of volume in an object. The reason that the amount of polygons makes a difference is that when the computer has to draw an image on the screen 60 times per second or more, it’s much easier if it has to calculate the positions of very few verts in each frame. The art of striking this balance is called low-polygon modeling, where the artist attempts to reduce the amount of polygons to where each and every one counts by expressing some form of volume in the mesh (Fig. 7).

Fig.7 http://poopinmymouth.com/3d/agust_construction.jpg

While this balance is fairly specific, it really represents a trade-off that is woven throughout all aspects of game art - the trade off of optimization versus visual fidelity. It’s a fine and difficult line that all game artists must tread in order to create great looking content that can also be displayed thousands of times per minute.

 

A Splash of Color

Once the modeler has created a mesh that he or she is satisfied with, it’s time to give it some color, texture and shading, since, up to this point, it’s been manipulated as a flat-looking grey with minimal shading. This stage is executed by a specialized artist called a texture artist. There are a couple of options for adding color, but one is far more popular than the other.

The first method of applying color and shading (not texture so much), is vertex painting the mesh. This method is far less expensive in terms of how much memory the computer must allocate to the task of calculating the color data. In fact, the computer need not use any more memory to compute the color than it does with the default grey mesh. This is because each vertex within the mesh already has a color variable assigned to it, so by vertex painting the artist is simply changing the value of that variable from a flat grey at default to a more helpful color (Technical brief vertex).

The major drawback to this type of coloring, because it’s vertex-based, is that the detail that is achievable is limited by the amount of vertices within the mesh. And, since the artist is low poly modeling, there are not a lot of vertices. Because very little detail is actually achievable, this method is mostly relegated to games with very bright and colorful cartoony graphics that don’t require much detail. The open source “Zody the Slug” game is one example (Fig. 8).

Fig.8 Zody the Slug - by p00f

Vertex painting’s other application is to supplement shading within a world. Say you have a dark cave that you don’t want lit because it’s supposed to look foreboding. However, the lighting model that is employed naturally illuminates the cave. To darken it, it can be vertex painted black to add the necessary darkness. This technique was used in CDProjekt’s “The Witcher” at the bottom of select stairways to add darkness. This technique, while previously common, has fallen out of favor as lighting models have advanced (The Witcher).

The more popular alternative to vertex shading is texture painting, which is the industry norm at this point. This method adds theoretically unlimited amounts of detail (limited only by the resolution of the texture image) to the mesh with a much easier way of manipulating that detail. But before a texture artist can create a texture map[4], he must first unwrap his or her mesh.

While this is an odd process to conceive of, it’s really fairly simple in practice. The best way to think of it is once again by using paper objects as an example. If you take a paper cube as your mesh and slice it along some of the edges, unwrap it, then paint on the unwrapped squares, and fold it back together, you’ll have colored cube. This process is called UV unwrapping (Fig. 9).

Fig.9

The computer does most of the legwork for the artist by using algorithms to unwrap the mesh in such a way as to preserve the original geometric shapes of the polygons. While this works really well for architectural or very box-like geometry, when you get into organic meshes, the computer will do it’s best to compute the UV coordinates with occasionally mixed results, and the artist will often have to rearrange the computed UV map (UV Mapping).

Once the UV map is finalized, it can be exported as an image file and drawn on in a program such as Photoshop, adding the detail that the artist is interested in achieving. This unwrapping, exporting, and painting process is known as skinning an object. Once again, tablets are the tool of choice here. This is very much like the digital portion of the conceptual phase of production in terms of the way the textures are painted, but there are a few noticeable differences.

First, and probably most importantly, the artist who is painting on a UV map, while painting in 2D, really has to imagine the way the mesh fits together three dimensionally, which is a process that is simple to understand, but very difficult to master. One of the reasons that this is so hard is because traditionally in a diffusion map (though much less common now with the advent of normal mapping) the texture artist will draw in the lighting information that affects the object right into the texture. This means that they have to imagine both how the object is shaded and how the detail that they’ve drawn into the mesh is shaded. Without proper attention to this shading, skinned objects will look flat and unattractive, but over-shading will cause the object to read poorly in different light (for example: if the artist draws shadows on the map coming from X direction, but when the object is used in-game the light comes from Y direction, the effect is quite poor), so it’s best to use shading to indicate depth and height on details only, rather than to cast shadows (Fig .10).

Fig.10 http://www.pioroberson.com/modelpages/info_spellborn2.htm

Secondly, filters can be used. While this method is fairly specific to environmental game art, it is used in all disciplines. Filters are simply a mathematical algorithm (or series of algorithms) that are applied to the image that is being painted. These algorithms affect the image in any number of ways by acting on the individual pixels or groups of pixels: moving them, changing color etc. These are important because they can save an artist an immense amount of time. For example, to create a decent rocky looking texture, using the solid noise filter, sharpening it up, and applying an emboss filter will give a decent result (Fig. 11).

Fig.11

While this is very simple, when used in conjunction with many other effects, it can be incredibly helpful and more importantly, very fast (Gordon).

An interesting fact about textures is that they may not actually negatively affect the performance of the game in real-time, no matter how large[5] they are as long as all the textures in the visible scene do not exceed the random access memory of the computer that the game is played on. This is because in most engines, all of the textures are loaded in to the machine’s RAM before the level is initialized. There are exceptions to this however. Recently, as the explorable area within games grows because of technology, some engines are using a technique that loads textures into the game on-the-fly during game play as they are viewed, and drops unused ones from memory. In this case, texture size can certainly make an impact. Both the Dunia Engine by UbiSoft Montreal and the Gamebryo engine by Emergent use this method (Elder Scrolls).

 

Dropping the Ball

So now that the artist has a fully 3D textured object, it’s time to make it move with object animation. Animation at its most fundamental is the expression of a transformation[6] over time. With computer animation (as with traditional animation), time is measured in frames. To make a transformation happen, the object must have a start point in time (frames) and another point in time that will indicate how the object has been transformed. These points in time are called keyframes and are user defined (meaning the artist places them manually). So for example, an artist, called an animator[7], could indicate that at frame 1, the object is located at (3, 5, -10) using a keyframe. Then the animator would move forward to frame 21 and keyframe the object to now be located at (3, 5, 10). This means that in the span of 20 frames, the object would have moved vertically (the Z axis) 20 units (3D Animation: Keyframing).

This is all well and good, but what happens between keyframes? How does the computer control the transformation the object undertakes between these frames? The answer is through interpolation[8]. There are three types of interpolation that are used to define transformations between keyframes.

The first is constant interpolation. The graph would be expressed as a constant graph, i.e. y=b (where b = any constant) (Fig. 12).

Fig.12

What this means basically is that there is no real interpolation, the object mentioned above would start at (3, 5, -10) at frame 1, and move instantly to (3, 5, 10) at frame 21. There’s no real motion; it’s instantaneous.

 

The second method of interpolation is called linear interpolation. Linear interpolation is different from constant interpolation in that any transformation, instead of being instant, is now continuous. The graph of linear interpolation between two points (keyframes) can be expressed as y=ax+b (Fig. 13).

Fig.13

So, for example, the object’s motion from (3, 5, -10) at frame 1 to (3, 5, 10) at frame 21 would remain constant over time.

 

The third method of interpolation is bezier interpolation. The graph of bezier interpolation would follow the form y=ax(squared)+bx+c (Fig. 14).

Fig.14

The bezier method is generally preferable (though not always as the other two types of interpolation have niche uses) because the transformation it represents is both very smooth and completely continuous. So, in the example above, the object would begin at (3, 5, -10) at frame 1 and move slowly at first, and build speed as it rose, then slow down as it came to rest at (3, 5, 10) at frame 21. The user also has a great deal of control over how the bezier graph behaves.

As mentioned, bezier interpolation is generally used for most animation because it most closely mimics physical phenomena like acceleration, inertia, and weight. Imagine bouncing a ball with constant interpolation, it would remain perfectly stationary in the air, then magically teleport to the ground and remain stationary, then again magically teleport to the air, etc. Imagine the same scenario with linear interpolation. The ball would move towards the ground, strike it, and immediately switch directions and continue moving at the same speed until it hit the peak of the bounce, then again instantly switch directions and head back towards the ground, all at the same speed. While this is slightly better, the real motion of a bouncing ball would be very fast as it rebounds off the ground, then it would slow as it reached the peak of the bounce, until it fell once again towards the ground to bounce again, accelerating until it hits. Bezier interpolation can easily mimic this. Linear and constant interpolations have their uses as well. Imagine a cog wheel that turns at a constant speed forever. This would be a linear animation. Or perhaps imagine if an object needed to teleport about. This would be an example of constant interpolation (Interpolation).

Thus far, transforming the location of an object through motion has mostly been discussed. However, through object animation, both the size and rotation of an object can also be transformed. What’s more, all of these three transformations can be constrained to occur on anywhere between one and three of the axes. For location this would simply mean moving the object along any of the axes, for scale, this would mean squashing or stretching along any of the axes, and for rotation, it would mean rotating around one or more of the available axes. The point that the object is transformed around is user defined.

An interesting bit about object animation is that in one way it is similar to vertex shading. No matter what transformation occurs, what actually happens is that all of the vertices within the object are moving in relation to the point they are being transformed about. Like vertex shading, all vertices already have a location variable so simply changing the value of that variable does not significantly impact the memory required to render the scene in real-time[9].

 

Twist and Shout

Object animation works for major transformations to an object, meaning its use is constrained to objects like car tires, a swinging pendulum, or a squashed ball; it’s hardly suited to character or organic animation where the object will actually have to deform in a specific manner.

Before animating a character, however, the artist will need to define a way that he or she will interact with the vertices inside of the mesh in order to animate them. While it may seem like using the object animation process on a selected group of vertices would work, it would become far too complicated and inefficient on all but the absolutely simplest meshes. To circumvent this production bottleneck, animators use what’s called a rig to group wanted vertices together in order to easily select and manipulate them.

The first part of the rig is the armature. This is a series of control rods known as bones connected to each other. As the name suggests, bones act essentially as real bones would within the mesh. So for example, if the animator’s mesh is of a human, the animator would create an armature such that all the bones within it would correspond approximately to where real human bones would be (Fig. 15).

Fig.15 http://www.caligari.com/news/news_april2007/CPCP_200704.html

Once the armature is established and the bones are in place, the animator assigns vertices to the bones that they correspond with. So, using the human example, for the leg, the artist would assign the vertices in the mesh’s foot to the foot bone, the vertices in the shin and calf to the lower leg bone, etc. Now, when the bone is transformed, all of the vertices that are assigned to it are transformed as well.

 

There is also an important parent/child relationship with bones. Bones that are a child of a parent bone are transformed identically as the parent is, and around the same point. However the child can then transform around its own point. The reason that this is important is that, using the human example, once you transform the upper leg bone, the animator would then want the lower leg bone to follow that transformation. Say the animator rotated the upper leg bone forwards and now the entire leg is sticking straight out forwards. The animator would then rotate the lower leg bone backwards, essentially bending the knee. Then, bending the foot bone back upwards, the character now looks like it’s taking a giant step (Fig. 16).

Fig.16

Once the rig is created, it can be animated much like object animation, except for a few differences. Firstly, within one object, you have many bones that must all be manipulated to form a pose before that pose is keyframed. This is an incredibly difficult job, and it is the reason why animators have their own profession. Once the pose is laid out and keyframed, the computer then interpolates all of the bones’ transformations between keyframes. The second major difference between character animation and object animation is that, unlike object animation, character animation is very taxing on the computer as far as memory allocation in real-time is concerned. Unlike object animation, where the changing of the locations of vertices occurs around one point, in character animation, the position of each vertex over time is now a function of the relative influence each bone has over it (called weight), and it’s very dependant on the parent/child relationship of all the bones within the entire armature. As a general rule, the more bones within an armature, the more of a memory impact that armature will make.

 

Graphics Pipeline

Fundamental to understanding how computer games operate is to understand the relationship between vector graphics and raster graphics. Vector graphics (or object oriented graphics) are “…the use of geometric primitives such as points, lines, curves, and polygons … [and] are all based upon mathematical equations to represent images…” (Vector Graphics). This describes the geometry that all 3D shapes in computer graphics are made of. This is what all the 3D assets within a game are made up of before they reach the raster phase of the graphics pipeline. Once they reach that point in the pipeline, they are converted to raster graphics and displayed on a monitor. Raster (or bitmap) graphics are a grid of pixels of varying colors, or what we know as traditional image files (.jpg, .png, etc.). These are possible to display on a monitor as it too is made of pixels, and it represents the final format that a vector graphic (most commonly a polygonal mesh) must achieve to be visible on a display (Fig. 17)

Fig.17 Inkscape

(Raster Graphics).

Before going any further, it’s important to understand how the computer interprets the information that all of the artists feed to it and how it synthesizes that information into the final product displayed on the screen. This process is known as the graphics pipeline.

1. Once the scene is defined as input, the pipeline begins with higher-order surface tessellation. This breaks down any object within the scene that is non-polygonal (a curve for instance) and forces it into triangles or lines (though triangles are preferred). This step is important, because triangles and lines are the only computable geometric shapes in real-time rendering.
2. Next, the input geometry is transformed according to animation or a vertex shader using matrix multiplication.
3. The geometry then passes on to be lit per vertex. The lighting takes into account the relative location of the object in relation to the lightsource, as well as the specularity of the individual vertices, and the brightness of the light.
4. Until now, the scene has not been rendered from a particular point of view. However, at this stage viewing transformation occurs, and the objects are transformed from global coordinates to coordinates based on the location and viewing angle of the camera. This results in the scene being viewed from the camera.
5. Now that a camera view is established, the scene’s geometry is transformed from 3D coordinate space to a 2D plane viewable from the camera during projection transformation.
6. Any objects outside of the viewable area are clipped, or discarded.
7. From here, all the geometric triangles are broken down into fragments of pixels in order to be ready for any pixel shading operations. This stage is called the triangle setup.
8. All the complicated pixel-level calculations occur at this point to determine what the final color of each pixel will be when it is displayed on the screen. Both traditional texturing and any wanted pixel shaders occur at this point.
9. The entire visible scene is then converted into raster format and sent to the display for the user to view (Technical brief vertex shaders, Graphics Pipeline).

The piece of software that allows the entire pipe-line process to occur is called the game engine. The engine is the piece of the puzzle that supervises the rendering process. Different engines are better at different things, depending on what they were built for. id’s Doom 3 engine for example looks fantastic indoors in small spaces, but it is incapable of rendering large areas or places with lots of people or dynamic objects. On the other hand, the Total War engine by Creative Assembly is capable of rendering thousands of men simultaneously on screen with an incredible view distance (Fig. 18).

 

Fig.18 Creative Assembly/ID Software

Game engines also typically come with an integrated development environment that aids artists and designers with creating content. Engines are also generally licensable for a fee. In this way, a single engine can often be used in many games (Game Engine).

 

Lighting and the Miracle of Shaders

While polygonal modeling, character animation, UV mapping etc. certainly manage to explain a considerable amount of the processes artists use to create their content, there are a couple more important things to consider. First, it’s important to understand how content is lit in real-time, and secondly, there are the little understood but vitally important shaders. Both of these topics are more difficult to understand as they are somewhat technical, but they both are fairly intertwined.

 

Flip the Switch

With today’s technology, there is far more flexibility than there has been in the recent past with regard to the way that things are lit. Because of this flexibility, there are many different ways to light objects in a game world. The way in which the method is determined depends largely on what the object being lit is, as well as how much performance the developer is willing to lose as a trade-off for more intense visuals. Once again this quintessential optimization versus visual fidelity question is raised, and it can be even more poignant with regards to lighting since. If every object in the game world receives a true shadow, even with today’s technology, it can be incredibly taxing on a machine and cause undesired visual stutters. To circumvent this, most mainstream developers will light different objects in a game world in many different ways depending on how important the quality of the object is.

The easiest and ‘cheapest’ method (in terms of memory impact), is to use vertex shading (Fig. 19).

 

Fig.19

Vertex shading is a subset of vertex shaders which will be addressed in the shader section. This lighting model computes lighting information based on the vertices within the mesh. Just like vertex painting and object animation, all vertices inherently contain a variable known as specularity, or the brightness of light reflected off the object from the light source. When light is cast on the mesh, the value of the vertices’ specularity is changed. However, unlike both vertex painting and object animation, because the specularity of the vertex is adjusted as a function of both the location relative to the light source, and the brightness of that light source, a small amount of memory is lost to those calculations, though it is fairly negligible. The visual problems with this technique are very similar to the main problem with vertex painting in that the level of lighting detail possible is limited by the amount of vertices within the mesh. Once again, because objects in the game world should have the fewest possible faces (and thus the fewest possible vertices), the level of achievable detail with this method is very small, leaving objects looking flat and uninteresting. Additionally, it is impossible to cast shadows with vertex shading because the effect that the light has on the object is constrained entirely to its vertices and cannot be transposed to another mesh. While this method came into fairly common use just prior to the year 2000 in games like Valve’s venerable “Half-Life” and slightly later with Monolith’s “Aliens Versus Predator 2”, because of its glaring limitations it is not regularly used in the mainstream industry today. However, this method still finds common use in independent games because of its inexpensive nature (Technical brief vertex).

The modern alternative that has pretty much replaced vertex shading is pixel shading (Fig. 20).

Fig.20

Like vertex shading, pixel shading is a subset of pixel shaders which will be addressed in the shader section. Pixel shading is much like vertex shading, except that lighting data is calculated on a per pixel basis, which, like textures, offers nearly unlimited detail. However, the main difference between the two shading models is that pixel shading is not calculated as part of the mesh, it is calculated in the final stages of the rendering process, after all the 3D shapes have been drawn. The pixel shading method will recalculate the color of each pixel on the screen in order to reflect the lightness of an object relative to the location of the light source and the brightness of that light. Obviously, this process is far more memory intensive as every pixel on the screen must be recalculated based on several functions, and modern screens can have in excess of 2 million pixels on the display depending on the resolution. Additionally, like vertex shading, pixel shading cannot cast shadows. Almost all developers are more than happy to use the pixel shading technique however as it is an easy trade off for the incredibly intricate lighting effects that are possible (Technical brief pixel).

The two models above are examples of dynamic shading techniques, that are determined in real-time based on the location of the lit object relative to the lightsource as well as the lightsources’ brightness. This means that as the object moves away from the light, it will get darker, or as it moves past the light the dark areas will move across its surface in order to stay on the surface that faces away from the light. While having a dynamic shading model is incredibly important, if the object being shaded doesn’t move and is static, then it’s essentially wasted calculation. A technique came into practice for use on static objects known as light mapping (Fig. 21).

Fig.21 Epic Megagames

Essentially, light mapping simply allows the artist to bake[10] all the lighting information onto the UV coordinates of the mesh as a texture file (Lightmap). This means that, once it is loaded into memory at the beginning of a level, it essentially has no impact on the memory used for the game. Light mapping is also beneficial because, as a texture, all the lighting information can be manually altered in a program like Photoshop. Light maps only became possible once game engines could support multiple textures being applied to the same mesh. This can introduce a lot of desirable effects like softly blurred or colored shadows. As mentioned, the only drawback to light mapping is that it cannot be used on dynamic objects. For this reason, it’s usually used for the environments in games (Iones).

How then are shadows cast from dynamic objects in games? There are two methods to approaching this, but they are both very memory intensive. The calculation strain comes essentially from the need for the computer to use an object’s silhouette to occlude the light that is cast on it so that objects behind the occluding object do not receive light. While both methods have similar results, their methods of achieving those results are very different.

The first method is for the machine to create shadow volumes[11](Fig. 22).

Fig.22 Monolith Games

This method absorbs more hardware resources, but offers a very precise cast shadow. Using shadow volumes is somewhat similar to off-line ray tracing techniques in that lines are projected from the lightsource through all the vertices of the illuminated object. All of the newly projected vertices are then filled in geometrically. This can be very memory intensive depending on the amount of vertices within an object. Also, as more and more light sources are introduced, the calculations become more intense. To have variation in the shadow (so that it’s not completely black), the scene is rendered twice[12] per viewable frame: once completely unlit, and once completely lit. These two scenes are then blended accurately together in the final stage of rendering (on a pixel-by-pixel basis) using the ray traced geometric shadow to erase areas in the unlit render that are not in shadow (Shadow Volume).

The other method of creating dynamic shadows is shadow mapping[13](Fig. 23).

Fig.23 Relic Entertainment

Shadow mapping renders a depth map from the light’s perspective to compute what regions are occluded. The scene is then redrawn from the camera viewpoint with the shadow drawn last as a pixel shader. Because the depth map is rendered as a texture, its accuracy is dependant on the resolution of the texture. Because of this, it’s possible to add interesting effects to the shadow, like blurring it to achieve a realistic soft shadow. Additionally, the technique can be modified to have the effect of a projector. This method is quite a bit faster than shadow volumes, but is also generally less accurate because it is drawn as a texture relative to the view rather than on an absolute pixel-by-pixel basis (Shadow Mapping).

 

The Reflecting Pool

What about all the interesting special effects achieved in games these days? How is water made? How do flags wave? Shaders are the answer to these questions. A shader in computer graphics terms is simply a software program that gives the artist (or, more likely, a specialized programmer) unprecedented control over both the final rendered output of a frame, as well as a fine level of control over individual vertex data depending on the type. Shaders can be broken down into two different types: vertex shaders and pixel or fragment shaders (Shaders).

Vertex shaders are programmed in such a way that they “…perform mathematical operations on the vertex data for objects” (Technical brief vertex). This means that all of the previously discussed variables contained within vertex data are at the fine control of a programmer. The programmer will write code that performs functions on the variables associated with the vertices within the chosen object. An example of this would be a waving flag (Fig. 24).

 

Fig.24 Epic Megagames

The programmer would specify that the vertices within the mesh would transform according to a sine wave, and voila, a waving flag. Other variables that a programmer has at his disposal to toy with include position, color, texture, fog, and specularity. One thing that cannot be done with a vertex shader is the creation or elimination of geometry. Until very recently (with the advent of geometry shaders) it was not possible to create additional geometry while in real-time. Vertex shaders have the added benefit of being relatively cost effective in terms of processing power, though this is quite dependant on the actual algorithms that the programmer has assigned (Vertex Shader).

 

Pixel shaders operate somewhat similarly to vertex shaders except that they perform operations on the individual pixels, and all their operations are performed at a different phase in the graphical pipeline. Pixel shaders operate on the lighting, shading and coloring values of individual pixels. Because there can often be more than 2 million pixels on a screen (depending on the resolution of the display), this can be a difficult processing task for the computer depending on the complexity of the algorithm. Pixel shading wasn’t available until 1998 when the nVidia Shading Rasterizer was introduced. Since then, it’s become a staple of video game art (Technical brief pixel).

Probably the most significant pixel shader to impact gaming is the sophisticated dot3 bump map[14], also known as a normal map, though it wasn’t used in a published title until 2003 with id Software’s Doom 3. Normal mapping simulates a perturbed surface on a polygon, making for incredibly detailed lighting models by applying an oddly colored texture to a surface. The colors within the texture itself determine the way the surface is perturbed (Normal Mapping). This technique is now incredibly popular and is used in most current-gen games because of the incredible realism that it can achieve. Since normal mapping became popular, increasing technology has driven it forward through several iterations. While it originally began as bump mapping, normal mapping has several current variations. Firstly, there’s parallax bump mapping (or virtual displacement mapping). Parallax mapping creates additional apparent depth by displacing the texture coordinates as a function of the viewing angle (Parallax Mapping). Normal mapping has progressed to a point where, with parallax occlusion mapping, not only do we have a perturbed and parallax surface, but that surface is also self-occluding so that the texture casts shadows across it’s own surface (Fig. 25)

 

Fig.25 http://ati.amd.com/developer/techreports/2006/I3D2006/Tatarchuk-POM-SI3D06.pdf

(Zink).

While there are unlimited shader possibilities (toon shading, desert mirages, mirrors, fish-eye and other lens effects, glow, fog, etc), the one that is by far the most difficult (by its very physical nature) is water. Water has a constantly altering perturbed surface, is reflective, refractive, contains depth-of-view limiting particles which also cause light shafts, has a Fresnel[15] effect, and is highly specular at low angles to the surface (Fig. 26).

 

Fig.26 Valve Software

This is an impressive list, and it still doesn’t cover the movement of the water. To create the perturbed surface, the programmer would employ an animated normal mapped surface. The reflection would be created with a cube map (a 360 degree view of the entire scene) projected onto the reflective surface. The suspended particles could be faked with the fog variable in the vertex data with the light shafts simply being made of a single polygon with a transparent texture that is set to always face the camera. The Fresnel effect would have to be taken care of with a special pixel shader simulating its physical phenomena, as would the specularity at low camera angles. To add small waves, a vertex shader could be implemented to transform the vertices on the surface of the water. This is one example of how a complicated material can be achieved (Lombard). Water has been wrestled with in real-time for a long time and has recently developed far enough that it is capable of being used in real time. Water began receiving significant attention in 2005 with the release of Microsoft Game Studios’ “Age of Empires III”. The effects in this game were unprecedented. Water continues to be an important consideration in games. 2K’s “Bioshock” contains some of the best water effects available at the moment, and “Crysis” contains incredible oceanic effects (Fig. 27).

 

Fig.27 Crytek/2K

 

Conclusion

With all of these tools, a coordinated team of artists can create a world that not only looks fantastic, but lives and breathes and lets the player interact with it fully as well. Once the audience for these games understands the scope of the game artists’ professions, it can facilitate the advancement of computer games as an art form instead of simply an entertainment medium. The sheer breadth of what a computer game artist has to be proficient in artistically should provide a lay audience with enough understanding of the creation process to be appreciative of the games that they play. When they boot up the next big title, they will know that a team of conceptual artists, mesh modelers, texture artists, animators, and shader programmers spent many months or even years creating the look and feel of the world that they may now interact with. It is up to the artists themselves to push the limits on people’s perceptions of what a computer game can be visually, and expand the creativity of the medium (Figs. 28.1-28.5).

Fig.28.1 CD Projekt Fig.28.2 Capcom
Fig.28.3 Ubisoft Montreal
Fig.28.4 Ubisoft Montreal
Fig.28.5 Crytek
Fig.28.5 Crytek

So if that makes a few zombies and scantily clad elven women become unemployed, then I’m willing to live with that; we will have risen to the tier of true artists.

 

Glossary

Armature: A system of bones that allows an animator to simulate the motion of a human or other creature.

Bake: Baking in computer graphics terms means that something that would normally be resolved dynamically within a graphics engine is pre-determined and made static.

Bezier interpolation: A system of interpolation represented by y=ax(squared)+bx+c.

Bitmap: See raster graphic.

Bone: A visual controller of groups of defined vertices. Used in character animation to deform a mesh in a desirable fashion.

Bump map: A technique used to create irregular, "bumpy" shading effects. I.e. adding cracks to stone, scratches to metal, etc.

Character animation: A subcategory of animation in which the animator uses an armature system to control the motion or deformation of a character of creature.

Child: An object in a parent/child relationship that’s transformation is dependant on the parent.

Clipping: A process used to optimize a scene by discarding geometry that falls outside of the camera view.

Constant interpolation: A system of interpolation represented by y=b (where b represents any constant).

Deformation: In computer graphics, deformation occurs when vertices within a mesh are transformed independently of the other vertices.

Depth map: A grayscale texture map that indicates the depth of view based on the gradient value of the map.

Dot3 bump map: See normal map.

Dynamic: In computer game graphics, a value that changes in real-time based on active input.

Edge: The distance between two vertices.

Extrusion: A method of creating new geometry still connected to the extruded geometry.

Face: The area inside of a set of three or four edges. In computer graphics, faces also contain texture coordinates.

Filter: An algorithm or set of algorithms that act on image data to alter it in some way.

Fragment shader: See pixel shader.

Frame: An image, that, when displayed in a sequence at high speed, can produce the illusion of motion.

Game engine: A piece of software that interprets input from an application and informs the computer's GPU about the nature of the images it has to draw.

Geometry shader: A type of shader that creates new geometry in real-time.

Graphics pipeline: Stages required to display a three dimensional image on a two dimensional screen.

Higher-order surface tessellation: A computational method for decomposing complex shapes into polygonal shapes.

Interpolation: The generation of values between a set of two known points.

Keyframe: Two frames that set the starting and ending point of a transformation.

Light-mapping: Baking lighting information into a texture map that can then be displayed in game.

Linear interpolation: A system of interpolation represented with y=ax+b.

Low-polygon modeling: A style of modeling that involves trying to use the smallest possible amount of geometry while still retaining the necessary illusion of volume.

Mesh: A three dimensional polygonal object.

Normal mapping: As bump mapping, except that it contains greater detail because it uses RGBA values rather than a grayscale image.

Object animation: The transformation of an object over time.

Object-oriented graphic: See vector graphic.

Organic animation: See character animation.

Parallax bump mapping: As normal mapping, except that it contains more apparent depth because it is rendered as a function of the view in relation to the tangent of the surface.

Parallax occlusion mapping: As parallax normal mapping, except that it is also self occluding, meaning that it can shade itself.

Parent: An object in a parent/child relationship that’s transformation is independent of the transformation of the child.

Pixel shader: Executed last in the graphics pipeline, a pixel shader alters the final output of pixels based on predefined algorithms.

Pixel shading: See pixel shader.

Primitive: A simplistic three dimensional shape such as a cube, cylinder, or sphere, that an artist will use to create more complex shapes.

Projection transformation: A mathematical process for projecting three dimensional space onto two dimensions.

Raster graphic: A grid of bits that represent the parts of an image. These bits correspond with pixels on a display.

Rig: A complete armature system, including the bones and weighted vertices.

Shader: A set of instructions sent to the GPU to dictate the way in which a scene is rendered.

Shadow mapping: A process for creating dynamic shadows, where each pixel is tested against a depth map to determine whether it's visible from the lightsource. If it is not, then it is in shadow.

Shadow volume: A process for creating dynamic shadows, where geometry is used to describe the space where light is occluded from a lightsource.

Skinning: The process of unwrapping a mesh, saving the UV coordinates, painting on them, then reapplying them to the mesh.

Specularity: A value that determines the amount and size of the light that an object reflects.

Static: An unmoving and non-interactive object or value in a computer game.

Subdivision: The process of creating additional geometry by dividing the current geometry.

Texture painting: The process of adding color to a texture by hand (i.e. not using filters or procedurals).

Triangle setup: The phase in the graphics pipeline in which geometric triangles are broken into fragments of pixels.

UV unwrapping: The process of cutting a three dimensional mesh in such a way that it can be displayed two dimensionally with as much of the original geometry intact as possible.

Vector graphic: An image that is displayed in terms of vectors rather than pixel information.

Vertex: An arbitrary point in three dimensional space. The basic building block of a mesh.

Vertex painting: Computing the change of color between two vertices based on an artists input.

Vertex shader: A programmable function that lets a programmer assign values to the variables contained in the vertex data by executing algorithms on it.

Viewing transformation: Forces the three dimensional scene to be rendered from the view point of the games camera.

Virtual displacement mapping: See parallax bump mapping.

Weight: The relative influence of a certain bone on its assigned vertices.