Multicolor soft outline in Unreal Engine

This tutorial is a bit special, mostly because it merely combines two brilliant tutorials by Tom Looman, namely Multicolor outline material and Soft outlines. You should definitely go and visit these, as I'll be building from the provided resources. However, as I haven't seen any tutorial on how to combine these two, I decided to try it out and write it down myself. Hope you will find it useful.

Idea 

I will not explain the idea behind how to create binary outline (see the above mentioned blog for that), nevertheless, the algorithm works with Custom Depth buffer, where we store all objects that we want to outline. Additionally, we also assign these object to stencil buffer, to decide which color the outline should have. The soft outline effect is achieved by blurring the custom depth and interpolating between outline color and background based on the intensity of the blurred depth.

Resources

In order to get this material working, please download the Multicolor outline material from Tom Looman's page. We will also need Spiral Blur code from Soft outlines post, although this blog provides a custom code that was tested in UE 4.20, where work with scene textures and post-processing materials changed from the way it was described in the above mentioned blogs.

Setup

We will be editing PP_OutlineColored.uasset that you can download from Multicolor outline material. Simply extract Content folder from provided .zip to Content folder in your project. The material itself should look something like this.

To test out it is working, use PPI_OutlineColored material instance as Post processing material. You will also need to assign objects to be highlighted into Custom depth buffer. You can do it by clicking on a specific object and searching for Render CustomDepth Pass in Rendering section. To choose the color of the outline, assign appropriate CustomDepth Stencil Value as seen in the picture.

By default, values start with 252 as follows:

 252 -> Color1 (Green)
 253 -> Color2 (Blue)
 254 -> Color3 (Red)
 255 -> Color4 (White)

If everything was set up correctly you should get a binary outline around you models that may look something like this:

You can find a bit more thorough explanation of the Custom Depth and its use in the post about Partial post-process effect application.

Soft outline - Custom depth blur

The idea of the soft outline is to blur values in Custom depth buffer and then assign alpha values for outline color based on how blurred the depth is. To do this, we first have to create a Material function that will blur out the Custom depth. I tried out several types of blurring methods, but in the end decided on Spiral blur that works the best for the needed effect. You may notice that UE already has node for Spiral blur, however this only works on scene texture, or texture sample, not custom depth.

Create a new material function (in my case name MF_OutlineBlur) and set it up as follows:

In our case, beside Input nodes, we have two Custom nodes. Node UV returns uv coordinates of currently processed pixels. It contains following code:

return GetDefaultSceneTextureUV(Parameters, 14);

Blur Custom node blurs Custom depth buffer and returns blurred values for given pixel.

 float2 nUV = UV;
 int i=0;
 float StepSize = Detail / (int) DetailIterations;
 float CurDetail=0;
 float2 CurOffset=0;
 float3 CurColor=0;
 float SubOffset = 0;
 float TwoPi = 6.283185;
 float accumdist=0;
 int TexIndex = 13;

 if (DetailIterations < 1)
 {
   return SceneTextureLookup(UV, TexIndex, false);
 }
 else
 {
   while (i < (int) DetailIterations)
   {
     CurDetail += StepSize;
     for (int j = 0; j < (int) RadialSteps; j++)
     {
       SubOffset +=1;
       CurOffset.x = cos(TwoPi*(SubOffset / RadialSteps));
       CurOffset.y = sin(TwoPi*(SubOffset / RadialSteps));
       nUV.x = UV.x + CurOffset.x * CurDetail;
       nUV.y = UV.y + CurOffset.y * CurDetail;
       float distpow = pow(CurDetail, KernelPower);
       CurColor += ceil(SceneTextureLookup(nUV, TexIndex, false))*distpow;
       accumdist += distpow;
     }
     SubOffset +=RadialOffset;
     i++;
   }
   CurColor = CurColor;
   CurColor /=accumdist;
   return CurColor;
 }

To get blurred Custom depth add following nodes to PP_OutlineColored material.

Note that Divide node and Brightness Correction parameter are used to sample Custom Depth.

To make sure the effect is working, plug the output from the second Clamp node to Emission color in the output node. This will show you how your blurred depth looks.

Soft outline - adding color

Next step is to connect blurred Custom depth with our current outline. Find the highlighted section in PP_OutlineColor material.

And add Lerp node and connect it with blurred Custom depth as follows. Additionally add Divide node and Float parameter node.

Note that Divide node and Outline Divider parameter are optional, but can help you if you plan to animate and/or increase/decrease brightness of the outline.

The last step is to Lerp between background and outline. You can do this by connecting Divide node to Lerp node as follows. 

With this set up, you should have everything working out as required!

Results

Here are some of the results with following settings.

Issues

- As this is post-processing effect, the outline is decide based on a 2D texture rather than 3D scene. For this reason when two objects that are in Custom Depth buffer are occluded by one another they might create unwanted effect, as shown in the picture

- Currently the final effect is created by combining two sources, blurred depth and binary outline. If your Outline thickness is not large enough, you might get green colored outline along with the color you chosen. To fix this just change Outline thickness parameter. Additionally, you can add white "bumper" color that is less distractive. Change Stencil value starts to 251 in Material instance and Outline color computation to following. 

- Note that Outline thickness and Distance parameters might not work if you zoom the camera in and out, try out different values, but as a general rule of thumb, smaller the object you want to highlight, smaller the Outline thickness and Distance parameters should be

References

Multicolor outline - by Tom Looman, containing starting material for this tutorial

Soft outlines - by Tom Looman, containing tutorial on single colored soft outline