So today was Academic Day, in particular, for faculty members, who had been invited to hear about what students want, directly from students. (That was the theme)

A couple of weeks previously, Andrew Parsons (@MrAndyPuppy), the local Academic Developer Evangelist asked me to do a presentation on Windows Phone 7, with a split between the experience promised by Microsoft, and developing for the phone. After I learned that I really cannot do a reasonable phone demo in 10 minutes, I cut it down to a demo of pre-made code.

Either way, some big highlights arose today. I managed to get Andy to let me see and use a developer phone, and I must say, awesome, however I cannot go into specifics, nor can I take photos unfortunately.

The other highlight was the awesome keynote presented by Michael Kordahi (@delic8genius) with a special appearance by August De Los Reyos (@augustdlr) presenting his experiences developing User experiences on the Surface, and even the Windows key on the keyboard. Some awesome demos for a theoretical Foxtel experience (Cable TV in Australia) that uses all devices, including a Surface table (jeez they love that thing). They then showed off the big devices for this year, Kinect and Windows Phone 7, and seeing this in person was really awesome. I cannot wait for next week when I will get to try out a device. (Unfortunately they took the device back to Sydney right after the keynote.)

Photos will follow as I can sort them out, and maybe videos depending on what I’m allowed to post, and what connection I can get.

Exciting days remain ahead, I have to present Windows Phone 7 to students on Thursday, and of course I must experience the great TechEd sessions.

The key to deferred rendering and other techniques is the ability to use a depth map to store world position. As we know a depth map consists of floating point values (optimally) and so normally we would use those values with our Clip Space coordinates and the Inverse of the ViewProjection matrix to get the position in World Space.

However there is another way to do this, and the big benefit is that it does not require a matrix multiplication to do so.

Presented by Crytek during a presentation on Atmospheric Scattering, this method uses just a Multiply + Add to get the position in World Space from our depth value at that point. To do this we need to get the far view frustum corners, in View Space, and pass them to the shader.

Getting the Corners

This is easily done in XNA through the BoundingFrustum.GetCorners() method. Note that we only need the far corners, so we can take the last 4 in the 8 element array, and we also need these corners in View space, so be sure to transform them by the View matrix before you pass them through.

private void getFrustumCorners(out Vector3[] corners)
{
    corners = new Vector3[4];

    Vector3[] temp = CurrentCamera.Frustum.GetCorners();
    for (int i = 0; i < 4; i++)
    {
        corners[i] = Vector3.Transform(temp[i + 4], CurrentCamera.View);
    }
}

Since we are in the CPU/XNA code, be sure to also pass the depth of the far clip plane, and the camera’s World matrix. (Simply invert the View matrix if you do not already have a World matrix)

Writing out Depth

Now for some shader code! First in the depth shader, you need to change how you write out the depth value, instead we take the Linear View Space depth, which means that we need to get the Z value from Position * World * View rather than Position * World * View * Projection.

float4x4 wv = mul( World, View );
float4 posVS = mul( pos, wv );

output.Position = mul( posVS, Projection );
output.Depth = posVS.z;

Then in the pixel shader, take this depth value, negate it and divide it by the Far Clipping plane.

output.Depth = 1.0f - ( -depth / FarClip );

Now when you want to reconstruct this, we need to find the correct frustum corner inside the vertex shader. (Note: This code only applies if you are rendering a fullscreen quad)

Preparing and Choosing the Right Corner

The best way to do this is to pass along the array index as a 3rd value with your texture coordinates. If you do not want to modify the vertex type, you can also apply some maths to determine the right index.

For brevity I will assume that the index is provided with the texture coordinates.

Take that ray and multiply it by the camera World matrix, to move the ray into World space, then pass it to the pixel shader.

output.FrustRay = mul( FSQ_GetFrustumRay( tcri.z ), CamWorld );

Getting the Position

Now that you have the ray, simply sample the depth texture for the depth value, and multiply it by the ray. Then take that and add it to the camera position to get the point in world space. (Rather than just a direction)

float depth = 1.0f - tex2D( depthSampler, tc ).x;
float3 pos = CamPosition + depth * frustRay;

Now you have your position!

I will include the technique to do the same thing with arbitrary world geometry when I write an article on Point Lights or Spotlights. At the moment I am working on implementing shadows, and probably won’t touch those topics until I solve some pressing issues.

Final Notes

Here is a code sample for passing the index as a part of the texture coordinate value.

private struct VertexPositionTexCoordRayIndex
{
    private Vector3 position;
    public Vector3 Position
    {
        get { return position; }
        set { position = value; }
    }

    private Vector3 texCoordRayIndex;
    public Vector3 TexCoordRayIndex
    {
        get { return texCoordRayIndex; }
        set { texCoordRayIndex = value; }
    }

    public static int SizeInBytes { get { return sizeof(float) * 6; } }

    public static readonly VertexElement[] VertexElements =
        new VertexElement[] {
            new VertexElement(0, 0, VertexElementFormat.Vector3, VertexElementMethod.Default, VertexElementUsage.Position, 0),
            new VertexElement(0, sizeof(float)*3, VertexElementFormat.Vector3, VertexElementMethod.Default, VertexElementUsage.TextureCoordinate, 0)
        };

    public VertexPositionTexCoordRayIndex(Vector3 position, Vector3 texcoordRayindex)
    {
        this.position = position;
        this.texCoordRayIndex = texcoordRayindex;
    }
}

VertexPositionTexCoordRayIndex[] verts = new VertexPositionTexCoordRayIndex[4];
verts[0] = new VertexPositionTexCoordRayIndex(new Vector3(-1, 1, 1), new Vector3(0, 0, 0));
verts[1] = new VertexPositionTexCoordRayIndex(new Vector3(1, 1, 1), new Vector3(1, 0, 1));
verts[2] = new VertexPositionTexCoordRayIndex(new Vector3(-1, -1, 1), new Vector3(0, 1, 3));
verts[3] = new VertexPositionTexCoordRayIndex(new Vector3(1, -1, 1), new Vector3(1, 1, 2));

Remember that XNA provides the corners in a different order to just 1,2,3,4, so we can explicitly pass the correct index for the vertex we are working with here, and it will be chosen correctly in the shader.

Apologies for the lack of updates, I am in the process of finishing an internship required for my university course, and it is a challenge to find good time to tackle some XNA issues and write posts.

I am currently working on Directional light shadows for my engine, I have cascaded shadow maps implemented, and now I am just tackling some issues I have encountered, which is taking a while – I also need to find a good filtering method.

If I get bored of trying to fix those annoying issues, I will probably jump into re-implementing point lights, and spotlights, and write something about those.

Some housekeeping notes:

I encountered a pretty big issue with the “Reconstructing position from depth” technique, and I have taken that article down from the blog for now. At the moment my own code uses the old “Multiply my the Inverse of the View Projection matrix”, and probably will continue to until I have time to solve that problem.

I have also updated this blog to WordPress 3.0, so let me know if you encounter any issues.

Back to work.

I have found that the simplest way to integrate these ambient lights into the Light Pre Pass system, and allow for other elements to change the lights throughout the game is to create a new light type that fits into the normal lighting model.

This is probably the simplest type out there, you just create a full screen quad and render it using a shader that writes the colour information directly to the light buffer. All we have to pass to the shader is the colour and intensity. (you could always pre-compute this, but beware of the byte->int32 auto promotion C# does)

Here is the C# class:

public class AmbientLight
{
    public Color Color { get; set; }

    public bool Enabled { get; set; }

    private int intensity = 100;

    public int Intensity
    {
        get { return intensity; }
        set { intensity = value; }
    }

    private Effect shader;

    public AmbientLight(Color col, int intensity)
    {
        this.Color = col;
        this.Enabled = true;
        this.intensity = intensity;
    }

    public void LoadContent(ContentManager content)
    {
        shader = content.Load<Effect>(@"Effects\Lights\l_ambient");
    }

    public void DrawLight(Renderer caller)
    {
        if (!Enabled)
            return;

        shader.Begin();
        shader.Parameters.TrySet("LightColor", Color.ToVector3());
        shader.Parameters.TrySet("Intensity", intensity);

        shader.CurrentTechnique.Passes[0].Begin();
        caller.FSQ.Draw();
        shader.CurrentTechnique.Passes[0].End();

        shader.End();
    }
}

and the shader:

float3 LightColor;
float Intensity;

float4 vs_main(in float4 pos : POSITION) : POSITION
{
    return pos;
}

float4 ps_main() : COLOR
{
    return float4(LightColor * (Intensity / 100), 0);
}

technique AmbientLight
{
    pass p0
    {
        VertexShader = compile vs_2_0 vs_main();
        PixelShader = compile ps_2_0 ps_main();
    }
}

I am in the middle of a crunch at work, and so future posts will be slow until it is all over. (That is why this one took a while to come out) I have not had the chance to do much work on my own XNA code, let alone tutorials.

Thanks for your patience, I cannot wait to get back to XNA. (and I cannot wait for XNA 4.0 with HiDef)

I suggest you have a read through it if you are interested in the kinds of things I post on this blog, especially the Light Pre Pass technique.

One of the key parts of Shawn’s article are the changes that will be made to Render Targets, and the bucketing of the RenderTarget/Texture formats into the Reach and HiDef profiles.

It would be safe to say that future articles here will require the HiDef feature set. I make use of some of the Render Targets that do not exist in Reach, and while there may be a way to use a Reach Render Target, for the purposes of conveying the technique, I will not be adding extra code to support it. I may however note where alternatives can be used, and hint or mention how to use them.

I know I have previously used BGRA formats in my sample, and in future this will become RGBA, no big deal. I won’t change the 3.1 sample for now, but if I need to change other things to upgrade it to 4.0, then that will be “fixed”.

This shouldn’t be much of a problem, if you are going to be using the articles I plan to write, then you’ll have a computer that works with HiDef, or an XBOX.

As a side note, I no longer have a valid Creators Club subscription, so I have not tested the recent Light Pre-Pass article on the XBOX. If anyone has any issues let me know and I will try and look into it. I plan to get a premium subscription soon and test future articles on the XBOX.

This time around I will be including those changes into the implementation, and also releasing sample code for educational use. If you feel you have learned enough from the previous article, then feel free to skip this. For those new to the topic, please use this article instead of the older one.

This article will focus on the theory, and will attempt to be as API neutral as possible. I will be making some references to XNA SurfaceFormats as this article is aimed more towards XNA developers, but for reference here are the matching Direct3D formats:

See the sample for an XNA implementation.

General Overview

For a number of years now, the concept of Deferred Rendering has been a hot topic in games. The ability to have a large number of lights, with a low cost per light is an attractive option. However deferred rendering suffers from the inability to render transparent objects at the same time as opaque objects. A few “solutions” have appeared in recent years that approach deferred rendering in a different way to allow for some level of transparency, however none have solved the problem completely, and all require extra work in some form.

There are a few different types of Deferred Rendering, however the two key forms are Deferred Shading, and Deferred Lighting. These differ in exactly which stage of rendering is deferred.

Deferred Shading defers the entire shading and lighting process, handling geometry rendering once, and then using an Uber Shader to light and shade the objects. One of the key negatives here is that you are limited in your material selection – although there are a few alternative solutions that can help here. Some other negatives include the “Fat Framebuffer” that this technique requires. As Deferred Shading relies on Multiple Render Targets (MRTs) to render the geometry once and store all important details, we are hit with both a memory cost, as well as limitations in DirectX9 which prevent us from applying multi-sampling (AA) to MRTs.
Negatives aside, Deferred Shading allows us to get the benefit of many lights, and still only render the scene once, which can be good for performance in scenes where the triangle count is high, and you do not need a diverse set of materials.

Deferred Lighting on the other hand defers only the lighting step, and requires an extra pass over the geometry to apply the details. The benefit here is that you can apply a unique material to each of your objects, whilst retaining the ability to have many lights. This technique also makes use of MRTs, however it only uses 2 render targets during the Depth + Normals stage, so if you can afford another pass over geometry, you can split those into two passes and gain the ability to run code on older hardware that does not support MRTs.

Deferred Lighting Overview

Deferred Lighting consists of 3 key stages:

1. Render the Depth and World Space Normals of the Geometry
2. Render the lighting term using the Depth + Normals generated previously
3. Render the Geometry again, this time with unique materials, making use of the lighting term above.

Remember that if you retain your hardware depth stencil, you get a performance increase from the early-z tests that modern hardware provides.

One key optimisation that is used in Deferred Shading is to render the lights using bounding volumes, ie a Sphere for a Point Light. This prevents overdraw for areas that are not lit.

I will be following the steps mentioned above, with the aforementioned optimisation, for the rest of this article.

Stage 1: Depth + Normals

The key requirement for any lighting calculation is fragment position, and normal. This means that if we want to determine the contribution a light makes to a particular pixel, we need to know where in world space that pixel is, and the normal at that pixel.

As we are deferring the lighting calculations, we need to store this information in a format we can use later. Now to store a position, we could go the naive route and store one component per channel, i.e. R=X, G=Y and B=Z, however this only gives us 8 bits of precision for each component, and each component is a full 32bit floating point number.

Instead we store the depth of the pixel and later reconstruct the position using the depth, and information from the camera.

Unfortunately, as we are working with XNA, we are restricted to D3D9 technology, and even then cannot use the FOURCC driver hacks that modern GPUs provide. This means that we have to write out the depth buffer to a separate render target instead of making use of the depth buffer the graphics card always creates.

So begin by preparing two render targets, the same size as the backbuffer you are going to draw to in the end. Make the depth target use SurfaceFormat.Single, and the Normals target use SurfaceFormat.Bgra1010102.

Of course when implementing this in a game for Windows, you should always verify that the SurfaceFormats are supported, and in the case of the depth buffer you can fall back to HalfSingle if Single is not supported. The Normal buffer can fall back to SurfaceFormat.Color if the 1010102 format is not supported, and you can always take the depth buffer back to Color if you absolutely need to, however note that you will need to add in extra packing code to make use of the full buffer then.

After that, we need to render the visible scene using a special shader that writes the Depth and Normal information to our render targets.

There are two key things to note here:

1. If you decide to subtract the true depth from 1.0f, ensure you reverse that later on.
2. You need to remap the Normal from [-1, 1] to [0, 1] so that it stores correctly in the render target

To accomplish #2, you simply need to use the following equation:

output.Normal = (( input.Normal + 1.0f ) / 2.0f )

Later on we will return this value to the [-1, 1] range by using the reverse of the above equation.

Stage 2: Lighting

Now that we have the Depth and Normal for each pixel, we can calculate the lighting from every light that touches each pixel. If you plan to include some form of shadow mapping, this is where you would handle rendering the shadow maps, and make use of them when lighting the object.

You can approach lighting in two ways:

1. Render every type of light as a fullscreen quad.
2. Render each light as an appropriate light volume [Hargreaves04]

We will focus on #2, which allows us to reduce pixel draw and improve performance dramatically.

Although I will only be implementing a Directional Light in the sample, you can use the following primitives to render each type of light:

As you can see, the two light types that do not have a position or volume, are still rendered as fullscreen quads, whilst the other two use approximations of their light volumes to prevent overdraw

In the case of the ambient light, you may choose to implement this as an extra term in your material shader instead, however I choose to implement it as a light object so that I may alter/combine/disable at will.

Here you need to set a new render target, which will hold our Light Buffer. Once that is done, render each visible light volume using a special shader based on the light type.

This is where the magic happens. Inside these special shaders, we reconstruct the world space position of the pixel from the depth value, and using the position and the normal, calculate the lighting term as you would during forward rendering.

There are a couple of ways you can reconstruct the position from the depth value. The most common and obvious way would be to use the X/Y value of the pixel in Post-Projection Space, combined with the depth as Z, and multiply that by the inverse of the ViewProjection matrix. This will give you the position of the pixel in World space, which you can use.

There are other techniques, some which end up cheaper than the matrix multiply mentioned above. For a full overview of most (if not all) of the techniques available for you to use, simply read through these two articles:

http://mynameismjp.wordpress.com/2009/03/10/reconstructing-position-from-depth/

http://mynameismjp.wordpress.com/2009/05/05/reconstructing-position-from-depth-continued/

In the sample I chose to use the Frustum Ray method, which is a cheaper way to get the position, but requires you to do some processing on the CPU to get the rays ready. In the sample this is calculated every time the camera parameters are set, however realistically you only need to do this any time your bounding frustum changes or moves.

Our light buffer, which is a SurfaceFormat.Color, can take four 8bit integers. The original, and (to my knowledge) recommended layout (from Wolfgang Engel) is to store the N.L * Color terms in the RGB channels, and the specular term in the Alpha channel. This leaves you with the following layout:

At this point you can also take into account the shadow map for each light, and use that to choose whether this light should illuminate or do nothing to the pixel. Now when rendering the lights, we need to set a couple of Render States to ensure things go smoothly.

First of all, we need to ensure that the depth buffer is disabled. This has to happen because we may need to render lights behind/inside other lights, and if the depth buffer is enabled, the graphics card will reject the contribution from some of those lights via the Early-Z feature.

Since we are using geometry to render the lighting, we need to handle the case when the camera is inside the light volume. Here we change the CullMode render state so that it either culls interior faces when the camera is outside the object, or it culls exterior faces when the camera is inside the object. Normally this would be set to Counter-Clockwise, which remains our default when outside the object.

Simply detect when the camera is inside the object and set the CullMode to Clockwise to handle the other case.

Another issue you may notice with this is when the camera is entering a light volume, especially the point light sphere. At this point, part of the screen is inside the object, and the rest is outside, so we need to set CullMode to None here to handle both cases without creating visual glitches.

Finally we need to enable alpha blending and set the blend mode to Add, this allows multiple lights that illuminate the same pixel to combine correctly (see Phong Shading).

Do not forget to re-enable the Depth Buffer and revert the CullMode to Counter-Clockwise before continuing, otherwise you might encounter hard to debug issues when rendering the materials.

Stage 3: Materials

Once the light buffer has been filled with all of the visible lights, you can move on to rendering the final image using each object’s material. First you need to make sure that all of your Render States are back to normal after the changes made during the Lighting stage.

To render the materials, you need to re-render every object that was drawn during the Depth + Normals stage, however this time around each object can use its own shader to draw itself, and can take the Light buffer as a texture, which can then be used as the lighting term when shading.

There are numerous effects that can be used to shade an object. In the sample I will only provide a Blinn-Phong shader, however as long as you understand the basic concepts of lighting in modern graphics, you can adapt the lighting values into any other technique that can use them.

This is where I find the big benefits of this technique appear. By allowing the object to choose its own shader/material, you can regain the diverse range of materials that forward rendering allows, providing your artists with plenty of possibilities. Not only that, but you do not have to manage a monster Uber Shader, balancing it against potentially restrictive instruction/texture count limits. (Depending on your Shader Model)

The sample will show how to implement a Blinn-Phong lighting model, with a textured object.

On the other hand, the big negative of this technique appears here as well. You now have to re-render every object a second time. Depending on your scene, this could be a reasonable performance hit. So ensure you weigh up both options when choosing your renderer for your game, and consider what you will need when choosing.

Transparency/Anti-aliasing

Transparency and Anti-Aliasing have traditionally been a major problem when it comes to Deferred Rendering. The need to render a Depth buffer has prevented the ability to do any reasonable form of transparency aside from the basic Invisible/Opaque states. There are a few solutions to this issue, although none solve it perfectly.

1. Render your transparent objects using traditional forward rendering, with a limited number of lights, and composite the transparency scene into your deferred scene.
2. Use stippling to alternate between layers on a high resolution render target. This is used in the recent articles on Inferred Rendering.
3. Use Order Independent Transparency/Depth Peeling, which can be fairly expensive. (See DX SDK Feb 2010 Samples)

I would recommend using option #1 and constructing your scenes so that the only transparent objects are particles, which can be lit with a small number of lights for little to no visual impact.

Anti-Aliasing has also been a problem for Deferred Renderers, primarily because DirectX 9 does not allow for AA with multiple render targets. DirectX 10 allows this, and fixes the issue, however since my target here is XNA, this is not possible.

Traditionally the AA issue was solved by using an Edge Detection filter + Blur to soften the edges of objects, this is relatively cheap and can be implemented as a post processing effect.

However for those interested in taking advantage of the AA modes of modern graphics cards, with very crisp edges, LPP provides a way to have your cake and eat it. (With a small cost)

As there are only 2 Render Targets needed during the Depth + Normals pass, and one at a time after that, you can split that pass up and render the scene an extra time, this time with AA enabled for each step. Of course if you have high scene complexity this might be extremely expensive and not worth it, but the option is there, and it is up to you and the needs of your game.

Conclusion

You now have a backbuffer (or Render Target) filled with your shaded scene, ready for further post processing, or presentation to the user. By using this technique you gain the ability to have thousands of lights in your scene at once, which allows level designers and artists to really let loose with their creativity.

Simply having this sheer number of lights also opens up other possibilities for lighting, including fake indirect lighting, a technique that as not been easy to do dynamically. You can assign point lights to every particle in your scene, and really make your worlds much more vibrant.

Also remember that you have a Depth and Normal buffer available for free (you needed it for LPP anyway) for use in post processing, which allows you to add other techniques like SSAO to your game.

Deferred Rendering seems to be the "cool thing” in game graphics today, and with the Compute Shader/OpenCL becoming readily available, some of these techniques can be adapted to make use of the general purpose capabilities of modern graphics cards.

Whilst there are plenty of benefits, there are also negatives to using this technique, and you really need to think about what you want out of your renderer before making your final decision. There have been plenty of debates in the graphics world about whether Deferred Rendering is better than Forward Rendering, and I am not going to go into one of those today.

Further Reading

For more information about Deferred Lighting/Light Pre Pass, you can check out the following books/websites:

• http://diaryofagraphicsprogrammer.blogspot.com/
• Section8, Chapter 5; ShaderX7, Wolfgang Engel, Course Technology [ISBN: 1584505982]
• http://www.bungie.net/images/Inside/publications/siggraph/Engel/LightPrePass.ppt (SIGGRAPH 2009)

Sample

I have written a sample implementation of the Deferred Lighting/Light Pre Pass renderer using XNA 3.1. Feel free to learn from this implementation, and if you have any questions, just ask in the comments.

The code provided in the sample is for Academic Use Only, and cannot be copied into a Commercial program.

This sample shows off 6 directional lights around a single model taken from the DirectX SDK. Feel free to add more lights as you please. I have only implemented Directional lights for this sample, to keep things simple. I will provide information on implementing other lights soon, however in the meanwhile feel free to look at my previous (now obsolete) article for a point light implementation, which should be similar.

You can download the sample here.

Controls:

1 : Show Depth Buffer

2 : Show Normal Buffer

3 : Show Lights Buffer

4 : Show Material Buffer

W : Move Forward

S : Move Backward

A : Strafe Left

D : Strafe Right

Left Arrow : Rotate Left

Right Arrow : Rotate Right

I am almost done with the LPP article, just one more section and the sample to finish. If you are interested in more timely and fast updates, follow me on twitter: @quandtm

One thing I am missing from Writer is tag support, if anyone knows how to get it in Word, please let me know; or conversely if you know of a great plugin for automatically taking and inserting screenshots of the desktop for writer, let me know as well. (I know about the snipping tool, I currently use it, but it has a few steps in between that I don’t really want)

One of the main delays was getting the sample code out of my engine, and cleaning it up so it can be used as a learning tool. Unfortunately most of the code was hacked on as I fixed issues in the LPP renderer, and added features.

I have a fair bit of the article written, however recently I was implementing shadows and realised that I never thought about how they would integrate into the system, so I decided to rewrite my own renderer (which should not take long) and at the same time keep the code clean so it can serve as a sample as well.

This means I will probably also have Directional Light shadows in the sample. This then allows me to write about point lights, and spotlights later on, and include shadows for both.

So again, sorry for the delay, I am working hard to get it out soon.

As an aside, I noticed that Game Programming Gems is getting an 8th Edition, something I was not expecting, so with that, GPU Pro, and Game Engine Gems, I might be able to find some more cool things to write about.

Thank You for your patience.

For any readers who are in Sydney, this would be a great event to attend. E&D covers everything from the XBOX to the Surface, even Keyboards + Mice from MS hardware.

Hopefully there will be a Natal unit on display, and maybe a demo, although no confirmation yet.

RSVP by emailing kumarman[at]microsoft.com