Most Interesting And Vivid Research Areas

Published Date: 02 Nov 2017

Abstract

Image retrieval has been one of the most interesting and vivid research areas in the field of computer vision over the last few decades. Image Retrieval systems are used in order to automatically index, search, retrieve and surf image databases. Gathering of large collections of digital images has created the need for efficient and intelligent schemes for classifying and retrieval of images of our choice within a short span of time and with high accuracy.

Many methods that have been introduced in the last ten years for image retrieval are based on the similarity, size of database, image classification and similarity extraction in groups of images or performance of the retrieval process. In the field of Computer Vision, Content-Based Image Retrieval (CBIR) systems are used in order to search, retrieve and browse images from large databases.

The main aim of this work is to extract similar images based on the intrinsic properties of a query image. In our thesis, we are using the Clustering algorithm for retrieving the images from huge volumes of data with better performance. Application of this algorithm requires image processing methods like colour-histogram feature extraction, classification of images, retrieval and indexing steps in order to develop an efficient image retrieval system. Each Image is converted into greyscale images and the threshold values are calculated for 10000 images. Edges are then extracted and connecting points of each image is calculated. All connected points are stored into the database then based on query image feature values similar images are retrieved. Processing is done through the image clustering method. Feature extraction and classification is done using the K-means classification algorithm. For retrieval of images, Euclidian distance method values are calculated between query image and database images.

For retrieval of images, mean values are calculated between the query image and database images and all clustered mean values are considered as a sorted order. When the statistical distances are calculated between the images, in our observation we found excellent performance and could extract similarities, thus leading to an efficient search.

In the field of content-based image retrieval, interactive systems have attracted a lot of research interest in recent years. Comparatively early systems, on the contrary, focused on fully automatic strategies. Another approach for the proposed system focuses on the retrieval of images within a large image collection based on color projections and different geometric / mathematical approaches. In this work, RGB colour combinations considered for retrieval of images and reduced comparison steps. This technique also resulted in fast and efficient results compared to previous methods.

Contents

Chapter 1

Introduction

It is widely said that "A picture is worth a thousand words". The meaning of an image is highly Individual and subjective. There is a rapid growth in the size of digital image collection. In many areas of commerce, government, academia, and hospitals, large collections of digital images are being created. Many of these collections are the product of digitizing existing collections of analogue photographs, diagrams, drawings, paintings, and prints and huge amount of information is out there. However, we can’t access to or make use of the information unless it is organized so as to allow an efficient browsing, searching and retrieval. Retrieving image from such large collections is a challenging problem, and thus methods for organizing a database of images and for efficient retrieval have become important. Content-based document image retrieval (CBIR), a technique which uses visual contents to search images from large scale image databases according to users’ interests, has been an active and fast advancing research area. During the past decade, remarkable progress has been made in both theoretical research and system development. In Content-based image retrieval, uses the visual contents of an image such as color, shape, texture and decides the similarity between two images by reckoning the closeness of these different regions. The problems of image retrieval are becoming widely recognized, and the search for solutions increasingly active area for research and development.

Most of the approaches make use of either of the clustering algorithms for faster retrieval of images, while most of the time it doesn’t come out with most appealing results .Similarly while retrieving the images using shape based methods there is a huge effort required to retrieve the images from the entire database.

In this thesis, we applied clustering algorithms on image and collected features of the images to fiend the similar groups and similar images from the data base we approached mathematical methods and analyzed cluster mean values. All approaches are given good results comparatively existing systems .this system performance, method implementations and conclusions are discussed in coming sections.

The term content based image retrieval (CBIR) is the application of computer vision techniques to the image related problems. Experiments into automatic retrieval of images from a large collection database are based on the colours, shapes and texture and image features. At current stage, there is a gap between low-level features of the retrieval system and the high-level semantic concepts of the user, called semantic gap. Compared with text-Information Retrieval, this problem result in very poor performance of CBIR system .All techniques and algorithms that are used originate from fields such as statistics, pattern recognition and computer vision.

Content-Based Image Retrieval (CBIR) is according to the user-supplied in the bottom characteristics, directly find out images containing specific content from the image library The basic process: First of all, do appropriate pre-processing of images like size and image transformation and noise reduction is taking place, and then extract image characteristics needed from the image according to the contents of images to keep in the database. When we retrieve to identify the image, extract the corresponding features from a known image and then retrieve the image database to identify the images which are similar with it, also we can give some of the characteristics based on a query requirement, then retrieve the required images based on the given suitable values. In the whole retrieval process, feature extraction is essential; it is closely related to all aspects of the feature, such as color, shape, texture and space.

Image Retrieval

In today’s world, data forms a very important part of our day-to-day lives. A huge amount of data is created on a regular basis and stored on digital media every day. This, coupled with the ease of procuring storage space, has led to a data boom, where we are deluged with a very high volume of data, stored in various locations that may be scattered all over the world. However, all the data that is available around us may not be useful for our particular requirements. We have to devise intelligent and effective methods to sift through the available data and extract useful data from the pile.

One important category of data that is very useful in today’s context is image data. With millions of images available in every library, we need to use effective algorithms that will retrieve the images that we really need. Such algorithms and systems can be broadly classified under the category of image retrieval systems.

Typically, image retrieval systems target the browsing, searching and retrieval of images similar in property to a given reference image from a large database of images. Traditional methods of image retrieval add some kind of metadata like captioning, keywords, or descriptions to the images and then match the annotation words with the search criteria. However, these kinds of methods are very manual in nature, subjective, time consuming and inefficient. It is difficult to apply these methods on large databases and if done, they do not always give satisfactory results, as the annotation of the search image might not match with those on the searched images.

This operational difficulty has led to large scale research in the area of image retrieval and more objective and automated methods of image retrieval have been discovered. Out of these methods, one of the most effective is one that uses statistical information regarding the contents of the image for retrieval similar images. This method is called content-based image retrieval (CBIR) and this thesis concentrates on providing new and more effective algorithms for CBIR.

A broad classification of all the image search and retrieval systems available today (with special focus on content-based image retrieval) is given in figure 1.1.

Figure 1.1: Classification of Image Retrieval Systems

Data sources

It is important to understand the scope and nature of the available image data (in the library) in order to determine the complexity of image search system design. No image retrieval system can be designed without some knowledge of the input data available. The design also depends on factors such as the diversity of user-base and expected user traffic. Overall, the search data can be classified into the following broad heads.

Personal data: This kind of data is mostly available on the user’s computer and is often small in size. They are also expected to be fairly homogeneous in nature, thus facilitating easy searches.

Enterprise data: This type of data is often large and heterogeneous. It refers to the entire collection of data for an enterprise and would be available on the company’s own servers or intranet.

Domain-specific data: This type of collection is large, and is specific to a controlled set of users with a common objective. This type of data is business-specific and will be available on public storages.

Archive data: These are large collections of structured or partially structured homogeneous data that has been collected over a long duration for a specific purpose.

Web data: This type of data is available on the internet and is the most common. They are available to everyone and are generally large in volume, may just partially structured. It is estimated that about 10 billion images are added to the web on an average every year. The target of this thesis is this data set.

Applications of Image Retrieval systems

Image retrieval systems have become an integral part of software systems today. There are many applications that require these systems for retrieving the correct image. More often than not, the requirement is for image retrieval in a completely automated manner with no requirement for human intervention. Some of the applications that require this technology are:

Employee recruitments

Personal identification

Astronomy

Population of picture galleries

Preparation of catalogs

Therapeutic diagnosis

Crime investigation

Military instruments

Architectural design works

Content-based image retrieval

CBIR concentrates on avoiding the subjectivity of keyword-based searches through use of computer-vision-based searches. In this technique, the algorithm extracts internal image property data (like texture, colour or shape of the image) and tries to match these properties with those of the library images to find a result set with similar properties.

"Content-based" means that the search will analyze the actual contents of the image rather than the metadata such as keywords, tags or other descriptions associated with the image. Content refers to colours, shapes, textures, or any other information that can be derived from the image itself. CBIR has a great advantage because most web based image search engines based on other techniques rely purely on metadata and this produces a lot of garbage in the results. Also having humans manually enter keywords for images in a large database can be inefficient, expensive and may not capture every keyword that describes the image. Thus a system that can filter images based on their content would provide better indexing and return more accurate results.

The methods of CBIR were first applied around the year 1980, when the text based retrieval process was replaced with content based image retrieval. The first projects with this idea were carried out by IBM at its Almaden Research center. In these projects, visual features of an image were considered rather than keywords in text based retrieval.

CBIR techniques can generally attack high and low level features. Low level visual features include shape, colour, texture and spatial elements in an image or a combination of these features. Out of these features, colour is one of the most widely used visual features used in content-based image retrieval. It is easy to use and understand. In its elementary form, every image is first segmented into the three basic colours (RGB) for its retrieval. Alternative colour spaces and their respective codes have also been proposed for identifying every pattern and differentiating it from the user’s query. The most frequently used technique is the colour histogram technique which calculates the relative strengths of the red, green and blue channels and their properties. Due to the uneven distribution of colours, different images rarely have matching histograms. An appropriate algorithm is required for standardization, extraction and analysis of the colour content and for increasing the efficiency and accuracy with which the images are retrieved.

Another very successful content-based image retrieval (CBIR) approach is the shape-based image retrieval technique. Based on shape of the image contents, the retrieval accuracy rate has been much better than corresponding colour based techniques.

Only low-level features will never lead to a perfect match during an image retrieval exercise. So high level features must also be included for increasing the accuracy rate. For this purpose, semantic features have been utilized in conjugation with the low level features to increase the accuracy of retrieval.

Query Techniques

Different implementations of CBIR use different types of user queries. Query by example is a query technique that involves providing the CBIR system with an example image that will act as the basis of the search process. The underlying search algorithms may vary depending on the application, but result images should all share common elements with the provided example.

Options for providing example images to the system include:

User supplies a pre-existing image to the query algorithm.

The user draws a rough approximation of the image they are looking for, and uses it for the search. Examples would include the user drawing a circle, thus searching for images that have a round shape in them.

This query technique removes the difficulties that can arise when trying to describe images with words.

Semantic Retrieval

The ideal CBIR system from a user perspective would involve a semantic retrieval, where the user makes a request like "find pictures of whales" or "find pictures of Mahatma Gandhi". This type of open-ended task is very difficult for computers to perform - pictures of Blue Whales and the Great White look very different, and Mahatma Gandhi may not always be facing the camera or in the same pose. Current CBIR systems therefore generally make use of lower-level features like texture, color, and shape, although some systems take advantage of very common higher-level features like faces. Not every CBIR system is generic. Some systems are designed for a specific domain, e.g. shape matching can be used for finding parts inside a CAD-CAM database.

Other query methods

Other query methods include

Browsing for example images,

Navigating customized / hierarchical categories,

Querying by image region (rather than the entire image),

Querying by multiple example images,

Querying by visual sketch,

Querying by direct specification of image features,

Multimodal queries (e.g. combining touch, voice, etc.)

CBIR systems can also make use of relevance feedback, where the user progressively refines the search results by marking images in the results as "relevant", "not relevant", or "neutral" to the search query, then repeating the search with the new information.

Content comparison using image distance measures

The most common method for comparing two images in content based image retrieval (typically an example image and an image from the database) is using an image distance measure. An image distance measure compares the similarity of two images in various dimensions such as color, texture, shape, and others. For example a distance of 0 signifies an exact match with the query, with respect to the dimensions that were considered. As one may intuitively gather, a value greater than 0 indicates various degrees of similarities between the images. Search results then can be sorted based on their distance to the queried image.

Color

Computing distance measures based on color similarity is achieved by computing a color histogram for each image that identifies the proportion of pixels within an image holding specific values (that humans express as colors). Current research is attempting to segment color proportion by region and by spatial relationship among several color regions. Examining images based on the colors they contain is one of the most widely used techniques because it does not depend on image size or orientation. Color searches will usually involve comparing color histograms, though this is not the only technique in practice.

Texture

Texture measures look for visual patterns in images and how they are spatially defined. Textures are represented by texels which are then placed into a number of sets, depending on how many textures are detected in the image. These sets not only define the texture, but also where in the image the texture is located.

Texture is a difficult concept to represent. The identification of specific textures in an image is achieved primarily by modeling texture as a two-dimensional gray level variation. The relative brightness of pairs of pixels is computed such that degree of contrast, regularity, coarseness and directionality may be estimated (Tamura, Mori & Yamawaki, 1978). However, the problem is in identifying patterns of co-pixel variation and associating them with particular classes of textures such as silky, or rough.

Shape

Shape does not refer to the shape of an image but to the shape of a particular region that is being sought out. Shapes will often be determined first applying segmentation or edge detection to an image. Other methods like [Tushabe and Wilkinson 2008] use shape filters to identify given shapes of an image. In some case accurate shape detection will require human intervention because methods like segmentation are very difficult to completely automate.

Edge Detection

Edge detection is a necessary preprocessing step in most of computer vision and image understanding systems. The accuracy and reliability of edge detection is critical to the overall performance of these systems. Earlier researchers paid a lot of attention to edge detection, but up to now, edge detection is still highly challenging. In this section, we will briefly illustrate two common edge detection methods, and point out their drawbacks. In addition, we introduce a simple and efficient method for edge detection.

Literature Review

An extensive amount of work has been carried out in the field of CBIR (Content Based Image Retrieval) all over the world in the last 15 years or so. Researchers have tried using various internal properties of image contents to make the process of image retrieval more efficient and effective. The early stages of the work in this field has been very effectively summarized by Smeulders, A.W.M et. al. [2] in their tutorial paper, in which they discuss some 200 publications in this field.

This very important paper starts with discussing the working conditions of CBIR, that includes patterns of use, types of pictures, the role of semantics, and the sensory gap. Then, they present various computational steps for image retrieval systems. Step one of the review is image processing for retrieval sorted by colour, texture, and shapes. Next, they present the features, sorted by accumulative and global features, salient points, object and shape features, signs, and structural combinations. Each feature type is discussed in terms of the feedback the user of the systems is capable of giving through interaction. The paper also presents aspects of system engineering, including databases, system architecture, and evaluation. Finally, the authors present their views on the driving force of the field, the heritage from computer vision, the influence on computer vision, the role of similarity and of interaction, the need for databases, the problem of evaluation, and the role of the semantic gap.

The traditional way of image retrieval using colour histograms was studied in detail and refined by Hafner et. al [5] in 1995, where they used a weighted distance between colour histograms of two images, represented in a quadratic form, to look for matches. This approach was further refined in 1997 by Jung Huang et. al., who introduced the idea of colour ‘correlograms’ for image indexing and comparison. In this approach, they have distilled the spatial correlation of the colour space of the images, to bring out a robust and effective model for CBIR.

Similar work was also done by Pass and Zabih [11] who used a technique for comparing images called histogram refinement, which imposes additional constraints on histogram based matching. This was necessitated because in some cases, very dissimilar images might have similar-looking histograms. Histogram refinement splits the pixels in a given bucket into several classes, based upon some local property and uses these bucket (segments) to compare the images instead of entire images at one go.

At a much later stage, Khanh Vu et. al [19] refined the correlogram approach by segmenting the images (especially the query image) based on various regions of interest. Instead of using the entire image for the query, the authors identify the region of interest (ROI), which is a subset of the entire image (like the Eiffel Tower separated from its background). This smaller ROI is then used for the histogram matching rather than the entire image. This method provides significantly better performance and efficiency and is particularly suitable for large image databases.

Carson et. al. [27] also did similar work where they present a new image representation which provides a transformation from the raw pixel data to a small set of localized coherent regions in colour and texture space.

The idea of using a combination method of content and text based image retrieval was explored by Ogle and Stonebraker in 1995 [7], with some amount of success, but largely not applicable in today’s scenario.

Post 2000, the focus of image retrieval has shifted from libraries of images present on the desktops (normally limited in number and complexity) to high-capacity libraries that are available online with higher image size and huge numbers of images. As a result, efficiency of the algorithm has become a key factor in ensuring its success.

In the year 2000, significant work was done by Gevers and Smeulders [8] and Wei-Ying and Manjunath [9], who tried different combinations of features to improve the hit rate of image retrieval systems. In the prior paper, a combination of colour and shape invariant features was used. Colour models were proposed independent of the object geometry, object pose, and illumination. From these colour models, colour invariant edges were derived. This, in turn, led to the computation of shape invariant features. Computational methods were tuned to cater to about 500 images in a database and significantly good results were obtained. In the latter paper, a scheme utilizing a predictive coding model to identify the direction of change in colour and texture at each image location at a given scale was used to construct an edge flow vector. By propagating the edge flow vectors, the boundaries were detected at image locations which encounter two opposite directions of flow in the stable state. Aslandogan and Yu [13] and Seyoon Jeong et. al [34] also used a combination of image and text properties for image and video retrieval.

Yining and Manjunath [22] also worked in this space and implemented a system for content-based search and retrieval of video based on low-level visual features. The system described by them consisted of three parts - automatic video partition, feature extraction, video search and retrieval. The three primary features, colour (histograms), texture (Gabor features) and motion (histograms) were used for indexing.

Significant innovative work was done by Zhuozheng Wang et. al. [1] in their paper in 2009, where they used a SIFT (Scale Variant Feature Transform) based algorithm to carry out CBIT on web libraries. In this paper, they have used SIFT descriptors, rather than the common features like colour, texture, shape and spatial properties to identify matching images. To improve performance, they have used the process of storing extracted keypoints in XML files for easy reference, thus improving the overall stability of the system.

Colour constancy is the ability to measure colours of an image irrespective of the colour of the light source that is shining on the object. A well-known colour constancy method is based on the gray-world assumption which states that the average reflectance of surfaces in the world is achromatic. In a paper authored by Weijer, Gebers and Gijsenji [14], in 2007, they propose a new hypothesis for colour constancy called the gray-edge hypothesis. This hypothesis assumes that the average edge difference in any real scene is achromatic. Based on this hypothesis, an algorithm is proposed for colour constancy which is based on the derivative structure of images. They also propose a framework which unifies a variety of known (gray-world, max-RGB, Minkowski norm) and the newly proposed gray-edge and higher order gray-edge algorithms.

A new colour edge detector method based on vector differences was proposed by Evans and Liu [18]. This basic technique outputs the maximum distance between the vectors within a mask. When applied to scalar-valued images, the method reduces to the classic morphological gradient. The method was shown to be computationally efficient and applicable on other vector-valued images. A quantitative evaluation of this technique using Pratt's figure of merit was used to prove its efficiency as compared to other similar methods available in the market.

An important problem in colour-based image retrieval and video segmentation is the lack of information regarding how the colour map of the image is spatially distributed. To solve this problem and enhance the performance of image and video analyses, Ho, Ho and Yeong [26] proposed a spatial colour descriptor that involves a colour adjacency histogram and colour vector angle histogram. The colour adjacency histogram represents the spatial distribution of colour pairs at colour edges in an image, thereby incorporating spatial information into the proposed colour descriptor. Meanwhile, the colour vector angle histogram represents the global colour distribution of smooth pixels in an image. The colour descriptor proposed by the authors in this paper includes spatial adjacency information between colours. Thereby, it can significantly reduce the effect of any major change in appearance and shape in image and video analyses. Moreover, since the colour adjacency histogram is simply represented by binary streams, the storage space required for the image histogram values can be effectively reduced.

T Vers [29] published their work on content-based image retrieval of textured objects in natural scenes under varying illumination and viewing conditions. In order to achieve this, their image retrieval method was based on matching feature distributions derived from colour invariant gradients. To cope with object cluttering, region-based texture segmentation was applied on the target images prior to the actual image retrieval process. They have also verified their retrieval scheme empirically on colour images taken from textured objects under different lighting conditions. This work was extended by Smith and Chang [30] who proposed a method for automatic colour extraction and indexing to support colour queries of image and video databases. This approach identifies the regions within images that contain colours from predetermined colour sets. By searching over a large number of colour sets, a colour index for the database is created in a fashion similar to that for file inversion. This allows very fast indexing of the image collection by the color contents of the images. Furthermore, information about the identified regions, such as the colour set, size, and location, enables a rich variety of queries that specify both colour content and spatial relationships of regions. They present the single colour extraction and indexing method and contrast it to other colour approaches. This approach was applied and tested on a database of 3000 colour images

As time passed, more statistical methods were being used to identify matching images for retrieval. Lin Hsin-Chih et. al. [31]used Hidden Markov Models for colour image retrieval. Mignotte [32] used a segmentation method by fusion of histogram-based K-Means clusters in different colour spaces to propose a new, efficient method for image segmentation and retrieval.

A cluster based approach was taken for image retrieval by Kyoung-Mi and Steet [40] where, in addition to adequate retrieval techniques, it is also important to enable some form of adaptation to users' specific needs. In this approach, a new refinement method for retrieval based on the learning of the users' specific preferences was introduced. The proposed system indexes objects based on shape and groups them into a set of clusters, with each cluster represented by a prototype. Clustering constructs taxonomy of objects by forming groups of closely-related objects. The proposed approach to learn the users' preferences is to refine corresponding clusters from objects provided by the users in the foreground, and to simultaneously adapt the database index in the background. Queries can be performed based solely on shape, or on a combination of shape with other features such as colour. Experimental results showed that the system successfully adapts queries into databases with only a small amount of feedback from the users. The quality of the returned results was also shown to be superior to that of a simple colour-based query, and continues to improve with continued use.

In summary, it is evident that a number of approaches have been tried by different researchers to improve on the efficiency and hit-rate of the CBIR systems. Overall, the following features are still most widely used for image retrieval purposes:

Colour

Shape

Texture

The method of extracting this data and comparing with that of the query image may be absolute or statistical in nature, with the latter taking centre-stage with rising complexity and data volume. In addition, image segmentation is also playing a large part in ensuring that we deal with manageable chunks of data rather than irrelevant, huge data volumes.

The CBIR process can be represented using the following diagram

Figure 1.2: Content Based Image Retrieval Process

Motivation

Recent times have seen a rapid increase in the size of digital image collections. Everyday imaging equipment generates several giga-bytes of images. Surveys (1) have estimated that world-wide 2,600 new images are created per second (equivalent to 80 billion per year) with an estimated 10 billion of which are available on the internet. Finding the correct image has become an expensive problem. It is necessary to have these data organized so as to allow efficient browsing, searching, and retrieval. Image retrieval has been a very active research area since the 1970s, with the thrust from two major research communities; database management and computer vision (2).Image retrieval can be divided into text-based image retrieval (TBIR) and content-based image retrieval (CBIR). The text-based image retrieval technique first annotates the images by text, and then uses text-based database management systems to perform image retrieval. However, there exist two major difficulties, especially when the size of image collections is large. One is the vast amount of labour required in manual image annotation. The other difficulty is the subjectivity of human perception, that is, for the same image content different people may perceive it differently. The perception subjectivity and annotation impreciseness may cause unrecoverable mismatches in later retrieval processes. This leads to more efforts being required on content-based image retrieval. Using this technique, images are indexed by their own visual content, such as colour, texture or shape. CBIR has become more and more important with the advance of computer technology, and provides the answer to the two drawbacks of the TBIR system mentioned above. It is important to stress that CBIR is not a replacement of, but rather a complementary component to new system. Only the integration of the two can result in satisfactory retrieval performance at present. In this thesis, the application of content-based image retrieval system is applied to various image collections. The powerful techniques like colour projection, image clustering edge detection methods are implemented .image processing and analysis will be used as the main tool for this research thesis.

Limitations of Current Image Retrieval Methods

The current results in CBIR very limited in spite of over 20 years of research efforts. There are various reasons for this incompleteness, especially for the discrepancy between high quality results shown in papers and poorer results in practice. The main reason is that the lessons about feature selection and the "curse of dimensionality" in pattern recognition have been ignored in CBIR. Because there is little connection between pixel statistics and the human interpretation of an image (the "semantic gap") the use of large number of generic features makes highly likely that results will not be scalable, i.e. they will not hold on collections of images other than the ones used during the development of the method. In other words, the transformation from images to features (or other descriptors) is many-to-one and when the data set is relatively small, there are no collisions. But as the size of the set increases unrelated images are likely to be mapped into the same features. There are severe limitations imposed by color and edge strength histograms respectively for image characterization and this is the feature that we have addressed in the current thesis.

The CBIR processes need improvements in both quality (efficiency) of algorithms and advances in computer hardware. In this thesis, our efforts have been focused on retrieval of images in specific applications where it is feasible to derive semantically meaningful features.

Problem Identification

The problem that is addressed in this thesis can be summarized in the following bullet points

Finding similar images from the image database in an efficient manner.

Minimizing computing time using various Techniques.

Maximizing the hit rate while retrieving images

These problems have been addressed in this thesis and the solutions obtained thereby have been illustrated through suitable examples to prove their superiority