One of the key methods to achieving great results in CGI is ambient occlusion (AO), it adds extra depth to a render and takes a nice image that one step further. There are a few different ways to go about adding AO to your scenes but which is the right way to do it? The quick answer is that there is no 'right' solution for every project, as with most things it is a case of finding out which one will work best in the circumstances. So with that in mind I thought I would write a bit about the methods I think are the best and when, in my opinion, you should use them.
Just before we go over these different methods though we should probably touch on what AO actually is. Essentially Ambient Occlusion is a crude global illumination solution which works out how much shading geometry should receive based on how 'occluded' (or blocked) it is by other geometry. I have often heard AO described as a 'nook and cranny' shader and it helps to think of it as such; basically wherever surfaces meet you will get some shading, if a surface doesn't have anything around it then it will be pure white.
Basic method:
3DS Max can generate an AO effect using any renderer but the one I prefer (and have by far the most experience with) is Mental Ray. The basic method of adding AO to a scene with Mental Ray is simple:
1: Create a mental ray material
2: Plug the 'Ambient/Reflective Occlusion' shader into the surface slot
3: In the settings for the 'Ambient/Reflective Occlusion' shader find the 'Max distance' option. This is critically important as it determines how far surfaces will 'look' for occlusion - the smaller the value the smaller your shadows will be, the larger the value the larger they will be. This setting will be different depending on your scene and what you think looks right but as a general rule-of-thumb I go with 5-10mm for close-up prod-viz shots, 90-100mm for arch-viz scenes and, depending on the camera distance, higher for cityscapes etc.
4: Next find the samples setting. This controls the amount of noise in your shadows, so if your AO looks a bit splotchy you put a higher value in. 16 samples are fine for test renders but for production renders you will need to up it to 64
5: Finally I always change the falloff setting to 0.75 but that's just a personal preference, you can leave it set to 1 if you like.
Once your AO shader is setup you need to apply it to your scene. The simplest way to do this is to use the 'material override' option which is under the 'processing' tab of the render setup window. Just drag your mental ray material to the material slot in the material override section and drop an instance in there; this will now automatically apply your AO material to everything in the scene, easy! You can apply the material by selecting everything in the scene and applying the mat to all, but it is usually better to do it with the material override as you get to keep all of your current material settings and can just switch the AO mat off when you're done.
One other thing to remember is that if you are applying this to a scene you have already textured and lit (which is usually the case) then you will need to switch off/delete all the lights, turn off MR exposure control, remove any environment maps, set the environment to white and turn off final gather - the ambient occlusion calculation doesn't require ANY lighting, it is using the geometry only to work out how much shading should be in your scene so having any of the above on will just screw it up.
Bells and whistles:
So we have covered how to do a 'pure' AO render with the method outlined above but there are some problems with that, yes it does give you great 'connecting shadows' but since everything is black and white there is no colour bleed from the materials, also since the previous AO option has no lighting solution (and is purely ambient) there are no indirect light bounces...how do we solve this?
Well firstly all of our scene materials need to be MR A&D mats - these can be quite daunting at first but once you get into using them and setting up your scenes with physically correct settings you will see how versatile they are. In each material you will need to scroll down the options to the 'special effects' group and in here you will find an 'ambient occlusion' setting, once this is turned on it will apply ambient occlusion to the material.
The settings are similar to the other AO option in that you have a 'samples' option to control quality and the 'max distance' setting; however, you then have a check box to 'use colour from other materials (exact AO)', this is the option that gives us shadows with bounced light so, as a general rule, I have this checked. The only other thing to check is that the 'shadow colour' is set to 'global ambient light colour' - this makes sure that the shadows generated will use the global settings from the 'Environment and Effects' tab rather than a custom setting per material.
NOTE: If you are working on a large scene with lots of materials in it is quite important to get your AO settings right in the first instance, clicking through hundreds of mats to tweak settings can make you lose the will to live! There is a very useful script available on Joe Gunn's website called 'Material Tweaker' which lets you specify material sets (which work in the same way as selection sets) and then make global changes to them. It works with Standard, MR A&D and V-Ray mats and whilst you can only make general changes to them the MR A&D set does allow you to switch AO on or off and adjust the radius (the script uses different terminology to the actual materials, basically AO radius = max distance). You can't adjust things like sampling quality unfortunately but in that case I would leave it at 16, do some renders to see where there are problems with noise and then adjust specific mats accordingly.
Once all your materials are setup in your scene you will get a very subtle ambient occlusion effect which, because it is part of a true final gather calculation, has indirect bounced light and colour bleeding in the shadows. But as technically accurate as this calculation is I tend to find that the end results look a bit washed out and weedy, yes the indirect bounces and colour bleeding look beautiful BUT they also take some of the shadow impact and edge definition out which, to my eyes, makes it look wrong...so what do we do?
Mix and match:
What I tend to do on almost every project (but not all of course - this is another thing where artistic license comes into the equation and you work out from project to project when it is relevant or not) is use both AO methods so that you have beautiful shadows with indirect bounces AND nice connecting shadows with well-defined edges.
If you know about rendering in passes then this will seem very basic, but what I do is setup my scene first with MR A&D materials (with occlusion switched on) and photometric lighting then render out the first pass. I then open another version of that scene and save it as an 'AO' version, setup my Mental Ray material with the ambient/reflective occlusion shader and plug it into the 'material override' section, delete all the lights, set the background to white, turn off FG and turn off exposure control. It may seem strange to do this in a separate file but I find it easier to keep things organised this way, I don't like having to switch everything off and on all the time in the main file, keeping them separate means I don't forget to turn things back on again or accidentally delete some lights. There are other reasons why having your AO version as a separate file is a good idea but we'll go over those in the next section. Once this is all setup and you have your renders it's time to stick them together so we switch to the compositing software. I use Photoshop for stills and After Effects for animation (although if I'm rendering Exr files I will always use After Effects as it deals with them better than Photoshop) but you can use whatever compositing software you're comfortable with, the concept is the same no matter what you use:
1: In Photoshop open up your 2 files, the first pass and the occlusion pass
2: Copy the occlusion pass and paste it onto a new layer over the first pass
3: Change the layer setting for the occlusion pass to 'multiply' and adjust the opacity to suit your taste
4: Done! If you switch off the occlusion layer's visibility you can see how much of an affect it has on the final image, it really adds definition and gives your shadows a great sense of depth.
The fiddly bits:
There are a couple of instances where the methodology used above doesn't really work, these are also when having a separate file for the AO starts to makes a lot of sense so let's briefly go over them:
1: Custom AO radii
The basic method of adding a global ambient occlusion shader to your entire scene is great for getting quick results but, as discussed earlier, depending on their size/detail objects sometimes need to have different 'max distance' values - a scene where every single object has the same setting just doesn't look quite right. So the answer is to ditch the 'material override' option and apply custom AO settings to your objects. You don't need to set them up individually of course, you can go with a few AO mats that have different distance values - one for fine detail and small objects, one for medium and one for large. Setting your AO up this way is more time consuming but it does give better results.
2: Reflective AO
In the 'ambient/reflective occlusion' shader there is a check box titled 'reflective', this can be very useful in enhancing the realism of your AO solution when it comes to highly reflective objects. Obviously a surface should not have a completely uniform amount of reflection, specularity maps help with this but the reflection occlusion shader can be used as well to get better results. Instead of sampling directly out from the surface normal as with standard AO, when you check the 'reflective' box the shader works in a different way and takes samples from the reflection direction of the surface normal. This calculation then focuses on areas that should have very little or no reflection, such as in tight corners etc, and will give you a map to use in compositing which reduces the reflection in these areas.
3: AO with masks
There are some instances where you have objects with transparency; the leaves on a tree model for example where you have a basic plane and the leaf shape is cut out with an alpha channel. Now obviously if you setup your AO pass as described above the alpha channel will be lost and your AO pass will render the full planes rather than the leaf shape, to get around this we need to, again, abandon the 'material override' method and go for a more custom setup. All you need to do is apply your basic AO to everything in the scene then, for the leaves (or whatever else it may be) do the following:
1: Create a blend material
2: Plug the AO material into slot 1
3: Create a standard material with opacity, specular level and glossiness set to 0
4: Take the alpha channel used in the original material and plug it into the mask channel of the blend material
That's it, you should now get an AO render which has your transparency masks included.
4: AO with round corners
One of the neat features of the MR A&D materials is the 'round corners' option, this lets you create an edge fillet effect without actually generating any extra geometry. It is useful for simple things like walls or the edges of a book...basically it can work quite well with any straight edge (although for most things I still prefer to actually model fillets into them). If you do have any of these in your scene then you will obviously need them to be in your occlusion pass as well, if you don't you will have some strange looking edges that are both rounded and straight at the same time! Again we will need to ditch the 'material override' option and apply AO to the scene selectively. Once you have applied the standard AO shader to everything that doesn't have the edge effect on do the following:
1: Create an MR A&D material with the colour set as pure white
2: Turn the reflectivity down to 0
3: In the special effects tab turn 'round corners' on and set fillet radius as desired
4: Plug the 'ambient/reflective occlusion' map into the 'additional colour map' channel
That should now give you an AO render with round corners where required.
Conclusion:
I hope this post proves to be useful to people, I started writing it thinking that I would just do a basic summary of the AO methods I personally use but, as I wrote, it got more and more detailed. Even so the methods outlined above are by no means exhaustive and the 'fiddly' options I have listed, which will undoubtedly give you better results, are ones I don't use that often - if anything I tend to find myself just doing the quick AO + MR A&D AO option as it gives good results with a minimum of fuss. The other options are good to know though for when you need to go that little bit further.
