Subset selection for visualization of relevant image fractions for deep learning based semantic image segmentation

Semantic image segmentation is a challenging problem from image processing where deep convolutional neural networks (CNN) have been applied with great success in the recent years. It deals with pixel-wise classification of an input image, dividing it into regions of multiple object classes. However, CNNs are opaque models. Given a trained CNN, it is hard to tell which information encoded in the input image is important for the network to perform segmentation. Such information could be useful to judge whether a trained network learned to segment in a plausible way or how its performance can be improved.For a trained CNN, we formulate an optimization problem to extract relevant image fractions for semantic segmentation. We try to identify a subset of pixels that contain the relevant information for the segmentation of one selected object class. In experiments on the Cityscapes dataset, we show that this is an easy way to gain valuable insight into a CNN trained for semantic segmentation. Looking at the relevant image fractions, we can identify possible limits of a trained network and draw conclusions about possible improvements.