Introduction

Convolutional Neural Networks (CNN) are primarily used in image-related tasks like image recognition and classification. Image Segmentation involves classifying each pixel into an object instance, that is going to each pixel and asking which object class does it belong to.

The CNN architecture outputs SoftMax probability matrix and we classify the pixel corresponding to the highest SoftMax value. We have a likelihood loss function between the SoftMax values generated by the CNN architecture and the ground truth labels. The likelihood loss function plays a key role in the performance and needs to be chosen carefully because

• Loss function can help you to adapt to your end goal. You can augment your own devised loss function with general loss functions like cross-entropy or dice score to get superior results compared to general loss functions used alone.
• The loss function can help in faster convergence

So in this article, we will discuss different challenges in segmentation and how different research papers, given their end goal tried to modify existing loss function or start from scratch devising their loss function, to get superior results. Also, we will try to classify the proposed loss function into three different categories, Boundary-based, Region-based and Topology preserving.

Challenges in segmentation

• Hard to collect data

  • The ground truth data is labelled by annotators, who mark the segmentation boundary of different object instances.
  • Accurate boundaries are important in segmentation, but it is expensive and time consuming to manually annotate accurate boundaries. We will discuss some of the methods on how to get accurate boundaries later in the article

• Unbalanced dataset segmentation

  

  • Unbalanced dataset can be of two types, frequency based or size based
    • Frequency based unbalanced, when an object instance is more pronounced or dominating compared to other instances, we have a frequency imbalance.
      • We pass a mini-batch to our network, because it is computationally efficient than passing individual pixels, and then we take the average of all the errors in the minibatch and the error backpropagates to update our gradients.
        • In case of frequency imbalance, we have our weight updates biased to high frequency class. The network learns that it is safer to predict the high frequency class because it will give higher accuracy.
      • Work-arounds won't help us like,
        • Assigning weights to classes inversely proportional to frequency of the class, may lead to noise amplification. Medical images which are merely CT scans are noisy and weights would exacerbate the problem.
        • Down-sampling high frequency class may lead to loss of information
    • Size based unbalanced dataset, when an object instance is disproportionately smaller than other instances, we have size imbalance. Even if they have same frequency, the results can be skewed to bigger size instances.

• Per-pixel accuracy

  • Passing a batch of pixel to network is computationally efficient and it helps the network to learn about relative positions of different objects which conserves localization. But as we take average of errors from individual pixels, so some pixel level error is tolerable thus compromising per-pixel accuracy
  • But if we have pixel level changes, then pixel level error is intolerable and we need topology preserving models.

As we have seen in our introduction section, the importance of loss function. Loss function provides a platform (in deep learning terminology, an objective function) which helps the network to minimize its inaccurate predictions. Choosing a pertinent loss function, can help you in getting accurate results even with small number of parameters, thus decrease computational burden and improve performance. So, now we will discuss different segmentation loss function used in state-of-the-art models.

Segmentation Losses Tree diagram

We will start with boundary based loss function

BOUNDARY BASED LOSS FUNCTIONS

HAUSDORFF DISTANCE BASED LOSS FUNCTION

• Hausdorff distance (HD) is the maximum among the minimum distances between two curves. It is the highest segmentation error. It is used for benchmarking segmentation results and it is a well-known evaluation metric.
• HD is sensitive to outliers and noise. Thus it is unstable while training, so we will discuss three HD "inspired" loss function where we mildly penalize the HD based loss function. The main idea is to estimate the HD in an indirect way and then try to minimize the error.