Steps For The Normalized Cut Method

Published Date: 02 Nov 2017

Chapter 1

Introduction

Image segmentation is an important image processing, and it seems everywhere if we want to analyze what inside the image. For example, if we seek to find if there is a chair or person inside an indoor image, we may need image segmentation to separate objects and analyze each object individually to check what it is. Image segmentation usually serves as the pre-processing before image pattern recognition, image feature extraction and image compression. Researches of it started around 1970, while there is still no robust solution, so we want to find the reason and see what we can do to improve it [10].

Image segmentation is a process of dividing an image into different regions such that each region is nearly homogeneous, whereas the union of any two regions is not. It serves as a key in image analysis and pattern recognition and is a fundamental step toward low-level vision, which is significant for object recognition and tracking, image retrieval, face detection, and other computer-vision-related applications [1]. Color images carry much more information than gray-level ones [24]. In many pattern recognition and computer vision applications, the color information can be used to enhance the image analysis process and improve segmentation results compared to gray-scale-based approaches. As a result, great efforts have been made in recent years to investigate segmentation of color images due to demanding needs. Existing image segmentation algorithms can be generally classified into three major categories, i.e., feature-space-based clustering, spatial segmentation, and graph-based approaches. Feature-space-based Clustering approaches [12], [13] capture the global characteristics of the image through the selection and calculation of the image features, which are usually based on the color or texture. By using a specific distance measure that ignores the spatial information, the feature samples are handled as vectors, and the objective is to group them into compact, but well-separated clusters. The Normalized Cuts algorithm requires as input the number of regions into which the image should be partitioned. The number of regions used was the same as the number of regions produced by our algorithm. We set the parameters of the Local Variation algorithm once, and used the same parameters to process all test images.

In computer vision, segmentation refers to the process of partitioning a digital image into multiple segments (sets of pixels, also known as super pixels).The objective should instead be to use the low-level coherence of brightness, color, texture, or motion attributes to sequentially come up with hierarchical partitions. Mid- and high-level knowledge can be used to either confirm these groups or select some for further attention. This attention could result in further repartitioning or grouping [10],[26].

The goal of segmentation is to simplify and/or change the representation of an image into something that is more meaningful and easier to analyze. Image segmentation is typically used to locate objects and boundaries (lines, curves, etc.) in images. More precisely, image segmentation is the process of assigning a label to every pixel in an image such that pixels with the same label share certain visual characteristics. The result of image segmentation is a set of segments that collectively cover the entire image, or a set of contours extracted from the image (see edge detection). Each of the pixels in a region are similar with respect to some characteristic or computed property, such as color, intensity, or texture. Adjacent regions are significantly different with respect to the same characteristic(s).Some of the practical applications of image segmentation are:

Medical imaging

Locate tumors and other pathologies

Measure tissue volumes

Computer-guided surgery

Diagnosis

Treatment planning

Study of anatomical structure

Locate objects in satellite images (roads, forests, etc.)

Face recognition

Fingerprint recognition

Traffic control systems

Brake light detection

Image segmentation is frequently regarded as an application of pattern recognition and analysis methods. Algorithms of this group operate in a vectorial space considering image elements as independent vectors. As in pattern recognition, segmentation methods may be further separated into two categories; classification [12], [13], [21], [25] and unsupervised clustering [3], [5], [23], [28], [31]. According to the first category, a training data set is required to set the classifier and the unlabeled data. Several typical classifiers [6] used in image segmentation approaches are bayesian, maximum likelihood, expectation-maximization, and hidden markov models. Classification methodologies can be divided into parametric or nonparametric, depending on whether initial assumptions have been imposed on the specific application or not. These methods are application dependent and they regularly require separate training data sets for different types of images.

Mean Shift Method

Mean Shift, a simple iterative procedure that shifts each data point to the average of data points in its neighborhood, is generalized and analyzed in this dissertation. This generalization makes some mean shift clustering algorithms its special cases. It is shown that mean shift is a mode-seeking process on a surface constructed with a shadow kernel. For Gaussian kernels, mean shift is a gradient mapping. Convergence is studied for mean shift iterations. Cluster analysis is treated as a deterministic problem of finding a fixed point of mean shift that characterizes the data. Mean Shift [6], [11] is also considered as an evolutionary strategy that performs multi start global optimization. The mean shift algorithm is a powerful technique for image segmentation [28]. The algorithm recursively moves to the kernel smoothed centroid for every data point. The quadratic computational complexity of the algorithm is a significant barrier to the scalability of this algorithm to practical applications. The fast Gauss transform (FGT) has successfully accelerated the kernel density estimation to linear running time for low-dimensional problems. Unfortunately, the cost of a direct extension of the FGT to higher-dimensional problems grows exponentially with dimension, making it impractical for dimensions above 3. We develop an improved fast Gauss transform to efficiently estimate sums of Gaussians in higher dimensions, where a new multivariate expansion scheme and an adaptive space subdivision technique dramatically improve the performance. The improved FGT has been applied to the mean shift algorithm achieving linear computational complexity. Mean Shift (MS) Algorithm:

Choose a search window size.

Choose the initial location of the search window.

Compute the mean location (centroid of the data) in the search window.

Center the search window at the mean location computed in Step 3.

Repeat Steps 3 and 4 until convergence.

Mean Shift Segmentation Algorithm:

Convert the image into tokens (via color, gradients, texture measures etc).

Choose initial search window locations uniformly in the data.

Compute the mean shift window location for each initial position.

Merge windows that end up on the same "peak" or mode.

The data these merged windows traversed are clustered together.

Normalized Cuts (NC) method

The set of points in an arbitrary feature space can be represented as a weighted undirected complete graph, where the nodes of the graph are the points in the feature space. The weight of an edge is a function of the similarity between the nodes and. In this context, we can formulate the image segmentation problem as a graph partitioning problem that asks for a partition of the vertex set , where, according to some measure, the vertices in any set have high similarity, and the vertices in two different sets have low similarity. In the normalized cuts approach, for any cut in , measures the similarity between different parts, and measures the total similarity of vertices in the same part. Since, a cut that minimizes also maximizes .Computing a cut that minimizes is an NP-hard problem. However, we can find in polynomial time a cut of small normalized weight using spectral techniques [16]. Efficient and effective image segmentation is an important task in computer vision and object recognition. Since fully automatic image segmentation is usually very hard for natural images, interactive schemes with a few simple user inputs produce good solutions.

Steps for the Normalized Cut method:

Given an image sequence, construct a weighted graph that’s each node is each pixel of the image. Let be the number of nodes (pixels), i.e..

Step 1

Construct an symmetric similarity matrix is given in equation (1.1)

(1.1)

where, is the spatial location of node , i.e., the coordinates in the original image , and is a feature vector defined as:

for segmenting point sets,

, the intensity value, for segmenting brightness (gray scale) images,

, where are the HSV values, for color segmentation,

Where, the are DOOG filters at various scales and orientations, for texture segmentation.

Letbe the total connection from node to all other nodes.

Construct diagonal matrix with on its diagonal using equation (1.1).

Step 2 Solve generalized eigensystem and get a eigenvector with the second smallest eigenvalue. To solve eigensystem we have, matlab function eigs.

Step 3 Use the eigenvector to bipartition the graph. In the ideal case, the eigenvector should only take on two discrete values, and the signs tell us exactly how to partition the graph.However, is relaxed to take real values, therefore, we need to choose a splitting point. There are several ways choose a splitting point such as

We choose a splitting by take ncut value= 0;

We choose a splitting point by take median;

We choose a splitting point to Search a results in which is minimized

The splitting point which minimizes value can be found by repeating calculation (1.2)

where, where, where,

(1.3)

where, is an dimensional indicator vector, if node is in and , otherwise. The optimal splitting point is generally around the mean value of the obtained eigenvector. Fortunately, matlab has a function, fminsearch, for this purpose.

Step 4 Repeat bipartition recursively. Stop if value is larger than a prespecified threshold value (Large Ncut value means that there is no clear partition point any more). Furthermore, stop if the total number of nodes in the partition is smaller than a prespecified threshold value (this is another criteria added newly to the dissertation algorithm.)

By using four different steps we can find similarity matrix using equation (1.1) and using matlab function eigs, to solve eigensystem to solve eigenvalue and eigenvector. The normalized cut method is used to represent the bipartite graph.

Image Segmentation Algorithms

Segmentation algorithms differ greatly in their ultimate objective. Segmentation is generally carried out as an intermediate process to some high level visual tasks such as object recognition and localization, reconstruction. In these circumstances; segmentation is seen as a tool to break a scene into regions that are fewer in number and more distinctive than pixels. Here, segmentation is just a means to an end. Segmentation as an intermediate process is relatively less constrained and has to output just a suitable decomposition containing regions that are not too big or too small. A decomposition containing large regions is called under-segmentation. It is usually undesirable because the regions contain pixels from surfaces at different depths which are usually unrelated. It is the other kind of decomposition, called over-segmentation [21], that is used as an intermediate processing step. Even here, an overly refined segmentation is useless as each region has just too few pixels to be discriminative, but a boundary preserving over-segmentation [34] is often used to obtain the regions for the next level of processing.

1.4.1 Feature space based clustering

The data to be clustered are represented by an undirected adjacency graph with arc capacities assigned to reflect the similarity between the linked vertices. Clustering is achieved by removing arcs of 9 to form mutually exclusive sub graphs such that the largest inter-sub graph maximum flow is minimized. It captures the global characteristics of the image through the selection and calculation of the image features, which are usually based on the color or texture. By using a specific distance measure that ignores the spatial information [6], [16] the feature samples are handled as vectors, and the objective is to group them into compact, but well-separated clusters. Image segmentation is frequently regarded as an application of pattern recognition and analysis methods. Algorithms of this group operate in a vectorial space considering image elements as independent vectors. According to the first category, a training data set is required to set the classifier and the unlabeled data. Several typical classifiers [6] used in image segmentation approaches are bayesian, maximum likelihood, expectation-maximization, and hidden markov models. Classification methodologies can be divided into parametric or nonparametric, depending on whether initial assumptions have been imposed on the specific application or not. These methods are application dependent and they regularly require separate training data sets for different types of images. Segmentation of an image renders a partition of the image into distinct regions. Features are extracted from each of the individual regions to characterize the objects implied by those regions. Region size is the area of the region normalized by the size of the image. The idea is to encode the amount of space occupied by the region in the image using a single number.

Spatial segmentation

Although the data clustering approaches are efficient in finding salient image features, they have some serious drawbacks as well. The spatial structure and the detailed edge information of an image are not preserved, and pixels from disconnected regions of the image may be grouped together if their feature spaces overlap. Given the importance of edge information, as well as the need to preserve the spatial relationship between the pixels on the image plane, there is a recent tend encrypt handle images in the spatial domain [11], [28]. The spatial segmentation method is also referred to as region-based when it is based on region entities. The watershed algorithm [19] is an extensively used technique for this purpose. It may seem easy to define watersheds on digital pictures, since this notion is a quite natural one, however, when looking closer at it, it turns out that there exist many particular cases, so that this definition task must be achieved very carefully. However, it may undesirably produce a very large number of small but quasi-homogenous regions. Therefore, some merging algorithm should be applied to these regions [20], [28].The watershed transformation considers the gradient magnitude of an image as a topographic surface. Pixels having the highest gradient magnitude intensities (GMIs) correspond to watershed lines, which represent the region boundaries. Water placed on any pixel enclosed by a common watershed line flows downhill to a common local intensity minimum (LIM). Pixels draining to a common minimum form a catch basin, which represents a segment. In multivariate statistics and the clustering of data, spectral clustering techniques make use of the spectrum (eigenvalue) of the similarity matrix of the data to perform dimensionality reduction before clustering in fewer dimensions.

Graph based approach

Graph-based approaches can be regarded as image perceptual grouping and organization methods based on the fusion of the feature and spatial information. In such approaches, visual group is based on several key factors such as similarity, proximity, and continuation [3], [5], [21], [25]. The common theme underlying these approaches is the formation of a weighted graph, where each vertex corresponds to image pixel or a region, and the weight of each edge connecting two pixels or two regions represents the likelihood that they belong to the same segment. The weights are usually related to the color and texture features, as well as the spatial characteristic of the corresponding pixels or regions. A graph is partitioned into multiple components that minimize some cost function of the vertices in the components and/or the boundaries between those components. So far, several graph cut-based methods have been developed for image segmentations [8], [14], [22], [23], [27], [30], [31]. For example, Shi and Malik [23] proposed a general image segmentation approach based on normalized cut (Ncut) by solving an eigensystem, and Wang and Siskind [8] developed an image-partitioning approach by using a complicated graph reduction. Besides graph-based approaches, there are also some other types of image segmentation approaches that mix the feature and spatial information [4], [29].

In this dissertation, a Normalized cut method in a large scale. It has been empirically shown that the Ncut method can robustly generate balanced clusters and is superior to other spectral graph partitioning methods, such as average cut and average association [23]. The Ncut method has been applied in video summarization, scene detection [17], and cluster-based image retrieval [18]. However, image segmentation approaches based on Ncut, in general, require high computation complexity and, therefore, are not suitable for real-time processing [23]. An efficient solution to this problem is to apply the graph representation strategy on the regions that are derived by some region segmentation method. For example, Makrogiannis et al. [20] developed an image segmentation method that incorporates region based segmentation and graph-partitioning approaches. This method first produces a set of over segmented regions from an image by using the watershed algorithm, and a graph structure is then applied to represent the relationship between these regions.

The overall segmentation performance of the region-based graph-partitioning approaches is sensitive to the region segmentation results and the graph grouping strategy. The inherent over segmentation effect of the watershed algorithm used in [20] and [28] produces a large number of small but quasi-homogenous regions, which may lead to a loss in the salient features of the overall image and, therefore, yield performance degradation in the consequent region grouping.

To overcome these problems, we proposed this dissertation, that provides effective and robust image segmentation with low computational complexity by incorporating the mean shift (MS) and the Ncut methods. In the dissertation method, we first perform image region segmentation by using the MS algorithm [4], and we then treat these regions as nodes in the image plane and apply a graph structure to represent them. The final step is to apply the Ncut method to partition these regions.

The MS algorithm is a robust feature-space analysis approach [4], [16] which can be applied to discontinuity preserving smoothing and image segmentation problems. It can significantly reduce the number of basic image entities, and due to the good discontinuity preserving filtering characteristic, the salient features of the overall image are retained. The latter property is particularly important in the partitioning of natural images, in which only several distinct regions are used in representing different scenes such as sky, lake, sand beach, person, and animal, whereas other information within a region is often less important and can be neglected. However, it is difficult to partition a natural image into significative regions to represent distinct scenes, depending only on the MS segmentation algorithm. The main reason is that the MS algorithm is an unsupervised clustering-based segmentation method, where the number and the shape of the data cluster are unknown a priori. Moreover, the termination of the segmentation process is based on some region-merging strategy applied to the filtered image result, and the number of regions in the segmented image is mainly determined by the minimum number of pixels in a region, which is denoted as (i.e., regions containing less than pixels will be eliminated and merged into its neighboring region). In this dissertation approach, therefore, an appropriate value of is chosen to yield a good region representation in the sense that the number of segmented regions is larger than the number of typical scenes, but is much smaller than the number of pixels in the image, and the boundary information of the typical scenes is retained by the boundaries of the regions [1].

The Ncut method [23], on the other hand, can be considered as a classification method. In most image segmentation applications, the Ncut method is applied directly to the image pixels, which are typically of very large size and thus require huge computational complexity. For example, to use the Ncut method in [26], a gray image has to be decimated into a size of pixels or smaller. It is difficult to get real-time segmentation using only Ncut method. In this dissertation, the Ncut method is applied to the segmented regions instead of the raw image pixels. As such, it eliminates the major problem of the Ncut method that requires prohibitively high complexity. By applying the Ncut method to the preprocessed regions rather than the raw image pixels, the dissertation method achieves a significant reduction of the computational cost. On the other hand, due to some approximation in the implementation of the Ncut method, the segmentation processing of a graph exploiting the lower dimensional region-based weight matrix also provides more precise and robust partitioning performance compared to that based on the pixel-based weight matrix.

Adjacency Matrix: W

Figure 1.1 Graph representations with Adjacency Matrix

In figure 1.1 shows a graph, where is the set of nodes, and is the set of edges connecting the nodes and is a weight assign on each edge. Represent tokens (which are associated with each pixel) using a weighted graph. Cut up this graph to get sub graphs with strong interior links and weaker exterior links [17], [24].

We defined the bipartite graph into two disjoint sets. This dissertation is to solve the graph-partitioning problem and produce the final segmentation result. Graph partitioning methods can effectively be used for image segmentation [25]. In these methods, the image is modeled as a weighted, undirected graph. Usually a pixel or a group of pixels are associated with nodes and edge weights define the (dis)similarity between the neighborhood pixels. The graph (image) is then partitioned according to a criterion designed to model "good" clusters. Each partition of the nodes (pixels) output from these algorithms are considered an object segment in the image. Some popular algorithms of this category are normalized cuts, random walker, minimum cut, isoperimetric partitioning and minimum spanning tree-based segmentation [17], [22], [24].

The strength of graph-based approaches derive from the solid mathematical foundation it relies on, since most of the well-established graph theory [6] can be directly used to handle the image segmentation problem. For instance, spectral graph theory [7] has been the basic tool for the so called spectral cuts method [1], [8], [9], [10], which exploits the eigen structures of the image affinity graph so as to accomplish the image segmentation. In fact, spectral graph theory enables great flexibility in the segmentation process, as different choices can be made towards defining the similarity graph connectivity as well as the assignment of weights to the edges of the graph. Such flexibility has leveraged a multitude technique, turning out spectral cuts an attractive image segmentation approach. To define the measure of dissimilarity between neighboring regions, we first define an appropriate feature space. Features like color, texture, statistical characteristics, and shape are useful for segmentation purposes and can be extracted from an image region. We adopt the color feature in [1] because it is usually the most dominant and distinguishing visual feature and adequate for a number of segmentation tasks. The average color components are computed over a region’s pixels and are described by a three-element color vector. Proper selection of the color spaces is important to the development of a good region-merging algorithm. To obtain meaningful segmentation results, the perceived color difference should be associated with the Euclidean distance in the color space. Under a graph representation, region grouping can be naturally formulated as a graph-partitioning problem. In the proposed method, the Ncut algorithm is used to solve such a problem. The major difference between the proposed method and the conventional Ncut algorithm is that the construction of the weight matrix is based not on the pixels of the original image but rather on the segmentation result of the MS algorithm [1], [4].

Problem Statement

In this dissertation we use natural scene color image database, because color images carries much more information than gray scale images. The main objective of present research work is to segment color image for different regions. Segmentation is used when we need to automate a particular activity. To design of an efficient image segmentation technique using fusions of Mean Shift segmentation and Normalized cut approach, to measure dissimilarity between different groups and similarity within group which will provide:

Effective and robust feature space analysis

Good discontinuity preserving characteristics, and

Reduction in the time of execution.

The segmentation performed following steps:

(1) First step, it preprocesses an image by using mean shift algorithm which gives segmented regions and original characteristics of an image remain same.

(2) Secondly, then these segmented regions represented by using bipartite graph structure. This graph is used to represent two disjoint sets and.

(3) Finally, we apply normalized cut method on this graph to partitioning the image.

Organization of Report

This dissertation report is organized as follows:

Chapter 1 explains the basics concept of segmentation, natural color images database and defines problem statement.

Chapter 2 deals with literature survey of related research work of the color image segmentation.

Chapter 3 explains the Mean shift method and normalized cut method with all steps apply on natural scene color images to get segmented results.

Chapter 4 deals with experimental results and discussion using mean shift and normalized cut methods applied on natural scene color images.

Chapter 5 contains concluding remarks and future work.