Oil Painting Post-process Effect in Unreal Engine

Oil painting post-effect is a rather common filter routinely found in the graphics software such as GIMP, or Photoshop. It allows you to give your image a more "blocky" and artistic feeling. It is also surprisingly easy to create via post-processing material in Unreal Engine.

Idea  

The idea behind the algorithm is relatively simple and rather similar to the Guassian blurring filter. For each pixel we need to search its neighborhood and find intensities of the neighboring pixels. The intensities are then divided into bins and the most common intensity is used to create new pixel color. 

I suggest reading this (with pseudocode) or this (with C++ code) page for deeper understanding of the issue.

Code

Directly within Unreal Engine, you will only need a simple setup to make the shader work. We will use SceneTexture:PostprocessInput0  to get the image that is to be rendered to the user, a constant to tweak with the radius of the effect, texture coordinates of the processed pixel and the custom node that will hold the majority of the computation.

Setup the nodes as follows:

Use following code within Custom Expression code field:

 int TexIndex = 14;

 int intensityCount[10];
 float avgR[10];
 float avgG[10];
 float avgB[10];
 

 for (int iLevel = 0; iLevel < 10; iLevel++){
     intensityCount[iLevel] = 0;
     avgR[iLevel] = 0.0;
     avgG[iLevel] = 0.0;
     avgB[iLevel] = 0.0;
  }


 //COUNT INTENSITIES
 uv *= 0.5;
 for (int i = 0; i < radius; ++i)
 {
     int offsetI = -1 *(radius / 2) + i;
     float v = uv.y + offsetI * invSize.y;
     int temp = i * radius;
     for (int j = 0; j < radius; ++j)
    {
         int offsetJ = -(radius / 2) + j;
         float u = uv.x + offsetJ * invSize.x;
         float2 uvShifted = uv + float2(u, v);
         float3 tex = SceneTextureLookup(uvShifted, TexIndex, false);

        float currentIntensity = ((tex.r + tex.g + tex.b) / 3 * 10);
        intensityCount[currentIntensity]++;
        avgR[currentIntensity] += tex.r;
        avgG[currentIntensity] += tex.g;
        avgB[currentIntensity] += tex.b;
    }
  }


 float maxIntensity = 0;
 int maxIndex = 0;

 //FIND CORRECT MOST COMMON INTENSITY LEVEL
 for(int cLevel = 0; cLevel < 10; cLevel++){
     if(intensityCount[cLevel] > maxIntensity){
         maxIntensity = intensityCount[cLevel];
         maxIndex = cLevel;
     }
  }

 float newR = avgR[maxIndex] / maxIntensity;
 float newG = avgG[maxIndex] / maxIntensity;
 float newB = avgB[maxIndex] / maxIntensity;

 float4 res = float4(newR, newG, newB, 1.0);

 return res;

The Custom node then returns the float4 representing the color of the processed pixel that can be directly connected to the Output node, or further processed.

Result

You can see the effect using Effect Radius = 10 picture below.

While following image shows the scene without applying the filter.

Following video shows the filter running on Soul City map.

Issues

- Since the shader code needs to search the neighborhood, it's obvious that with increased "Effect Radius" parameter this computation complexity increases. Parameters between 5 and 10 should make the effect pronounced enough for the most cases and still keep performance relatively stable.
- Currently the shader doesn't use adaptive radius based on the scene depth, so objects in background seem more blurred than images in foreground (this may or may not be desirable)
- If you are using UE 4.19+ you may get a strange issue where part of the screen is black. Refer to this article for solution

References

Oil Paint Effect by Santhosh_G - C++ CPU code for the Oil Painting effect along with the explanation of the filtering process.

The Supercomputing Blog on Oil Painting Algorithm - pseudocode CPU code of which my code was based off, along with short explanation of the filtering process.

Gaussian Blur Example - containing GPU code for UE4 used as the base for neighborhood search

Soul: City - free assets from Epic Games used for testing of the filter