Reflection Removal Techniques : Review

What is Reflection in an Image?

When a person takes photographs through a transparent material such as glass or windows, undesired reflections often ruin the images. To obtain clear images, users may make a dark situation or change the camera position, but it is not effective for removing reflections because of the limitation on space. The reflection degrades the image quality and also affects the results of applications such as segmentation or classification.

Table of Content

1. What is need of Reflection Removal?
2. Several Techniques for Reflection Removal
3. Results
4. Limitations
5. Conclusion

What is need of Reflection Removal?

When we take a picture through glass windows, the photographs are often degraded by undesired reflections. To separate the reflection layer and background layer is an important problem for enhancing image quality, removal is the reflection from the image is necessary. Thus, to solve this ill-posed problem, multiple input images or a video is used in many methods, but in actual cases, it is difficult to prepare adequate multiple images because additional devices are required, even if it is not required, users have to take several photographs from a different view and this is difficult when the space is limited or the object is not stationary, so the study of single-image reflection removal is important.

Some methods were proposed to remove reflection based on solving the Optimization Problem. Since understanding the image structure is important in removing reflections, Convolutional Neural Network (CNN) was applied. In particular, Generative Adversarial Networks (GAN)-based methods have produced good results and can generate more realistic images since adversarial loss encourages networks to generate images that follow natural image distribution.

Now, let’s move to techniques used for reflection removal.

Several Techniques for Reflection Removal

Averaging

It is a simple technique. It performs averaging over a set of images pixel by pixel to remove the reflection.

Independent Component Analysis

We can think of the image behind the glossy surface and the image formed by the reflection over the surface to be a linear combination of two independent images. If we want to separate independent components in the image, and we know that convolved independent components make a somewhat Gaussian product, we can separate the components by maximizing “non-gaussianness.”

To separate the reflection from the desired image we first photograph a scene through a linear polarized at two distinct orientations. To solve the under-constrained set of equations we make the modest assumption that the image of the painting and reflection are independent. Intuitively this means that for each spatial position the pixel intensity in one image provides no predictive information about the pixel intensity in the second image. This is a reasonable assumption since there is no reason to expect a correlation between the image of objects behind the glass and the image of objects reflected by the glass. Given the linear model of image formation and this assumption of independence, we perform independent components analysis (ICA) to separate the reflection from the desired image.

In short, based on the concept of reflection of light and independence of underlying distribution is reflected image, two images captured at different polarizing orientation are taken and solved for: Y = MX, where Y = [y1, y2], two images, M is mixing matrix [a,b; c,d] (each denoting the amount of reflection) and X= [x1,x2] are painting and reflecting components in the two images.

Relative Smoothness

It models the input image as a linear combination of transmission and smooth reflection layer, I = T + R, where R is a smooth function. Based on sparsity prior to natural scenes, the objective function is to maximize the probability of joint distribution of T and R. The desired layer is more in focus while the reflection is blurred. This can be expressed as

I = T + (R ∗ h )

where the reflection layer is convolved with the depth of field kernel h modeled as a Gaussian blur.

By introducing priors on the two layers’ gradients and R is smoother than T, then large gradients are more likely to belong to T. To solve the layer separation problem, we adapt a probabilistic model to seek the most likely explanation of the input image using the probabilities of the two layers . In essence, we are maximizing the joint probability P(T, R). By using directional first order derivative filters and a second order Laplacian filter, and from experiments it is found that the first order derivative filter helps to recover the significant edges in T, while the Laplacian filter encodes smooth variations in R. The objective function is non-convex for T and we can calculate R as I-T.

Reflect Suppression Via Convex Optimization

Convex model is used to suppress the reflection from a single input image. It implies a partial differential equation with gradient thresholding, which is solved efficiently using Discrete Cosine Transform.

We formally express our assumption using the following equation

Y = w*T + (1 − w)*(κ ∗ R)

where Y is the input camera image, T is the transmission layer and R is the reflection layer. w is a parameter that measures the weight between the two layers. κ is a Gaussian blurring kernel.

Our model minimizes the data fidelity term which is the difference on the edges between the output and input images . The edge information of an image is obtained by applying the Laplacian operator . Also, an prior of the image gradient is added to the objective function. It encourages smoothing of the image while maintaining the continuity of large structures. The model removes more gradients as the regularization parameter λ increases. Essentially, it sets a threshold on the gradients of the input image and removes the gradients whose magnitudes are larger than the given threshold. The gradient-thresholding step appears as a closed-form solution in each iteration of the algorithm.

Generative Adversarial Networks

Single-Image Reflection Removal method based on generative adversarial networks uses ‘‘gradient constraint loss’’ as training loss to preserve texture and structure information effectively while separating background and reflection layer.

Network Model :

The generator architecture is shown in Figure.It is a combination of convolutional layer, batch normalization layer, leaky ReLU layer, max pooling layer, and bilinear interpolation layer. We use B as a main output and the other outputs are used for computing pixel loss. The filter size of the convolutional layers are set to 3 × 3. Our discriminator is composed of the enumeration of convolutional layer, batch normalization layer, and leaky ReLU layer. The stride of convolutional layers is set to 2 in every two convolutional layers.

Loss Function for Generator :

Pixel Loss : Pixel loss is applied to compare the pixel-wise difference between generated image and ground truth image. Since minimizing the mean squared error (MSE) is effective for avoiding vanishing gradient problem in training GAN, we use MSE loss function to calculate the pixel loss.

Feature Loss : In the reflection removing task, it is important to preserve the structure of the image. Since the pixel loss cannot optimize the semantic feature of the image, We calculate the L1 difference between the feature vector of generated image and ground truth image. Since background layer and reflection layer have different image structure, the feature loss is applied to both background and reflection layer.

Adversarial Loss : It is known that simple CNN-based networks with MSE loss tend to generate blurry and unnatural images. It is because the images generated by those methods are the average of several natural solutions . To avoid this problem, adversarial loss is proposed.

Gradient Loss : To minimize the correlation, we applied a novel loss function called gradient constraint loss.

Loss Function for Discriminator:

Since our method is based on GAN, discriminator has to be trained while generator is trained. The discriminator is trained by minimizing the loss L(θ) described below:

L(θ) = X Ni (||D(Bi; θ)||² + ||V − D(Bi; θ)||²),

where V is a random valued matrix which follows Gaussian distribution with an average of 1.

After creating a GAN model, training is done.

Results

Averaging

Independent Component Analysis

Relative Smoothness

Reflect Suppression Via Convex Optimization

Generative Adversarial Networks

Limitations

1. Averaging requires a set of images to remove the reflection.
2. The techniques employed here, work under their underlying assumptions, ICA requires two same layers being at different polarisation angles, which is a rare phenomenon.
3. Relative smoothness removes reflections that are smooth in nature.
4. Reflective suppress tries to suppress reflection, as parameter h increases the image tends to more blur.

Conclusion

In this article, we review various techniques for reflection removal from multiple/single considering from optimization techniques to deep learning algorithms. Several techniques like Ghosting Cues, Perceptual Reflection Removal, and ERRNet has been proven effective for removing reflection.

References

1. Separating reflections and lighting using independent components analysis.
2. Single image layer separation using relative smoothness.
3. Fast Single Image Reflection Suppression via Convex Optimization.
4. Single Image Reflection Removal Based on GAN With Gradient Constraint.

The code is available at github.com/Devashi-Choudhary/Reflection-Removal. For any questions or doubts, feel free to contact me directly at github.com/Devashi-Choudhary.