What Is Google Now

Published Date: 02 Nov 2017

Amol Verma - 2010013

Vinayak Chopra - 2010096

Predictive assistance systems have been a pipe dream for decades, such as the smart, computer-augmented homes depicted in futuristic cartoons like The Jetsons; Jarvis, Tony Stark’s smart computer assistant which anticipates his foolhardy actions, from Iron Man; and KITT, the intelligent (and sarcastic) car in Knight Rider. Google Now is the first concrete step towards bridging this sort of fantasy and reality.

What is Google Now?

Google Now is an intelligent digital personal assistant service made for the Android operating system. It was launched by Google along with Android 4.1 in July 2012, but gained its predictive capabilities with the release of Android 4.2, in November 2012. It was developed in response to a call by Google’s CEO, Larry Page, to make products which do work for humans, rather than the other way around. It uses a natural language interface to do give predictive recommendations and perform actions in response to user requests, such as answering questions.

Its functionality includes:

Conventional search, with natural language query processing

Predictive and ambient recommendations

Google Now has two modes: active and passive. Conventional search is a part of the active mode whereas the predictive recommendation system is a part of the passive mode.

Active Mode

Google Now’s active mode involves the user invoking Google Now, giving it a query, and Google Now coming up with an appropriate response. This is powered by all the technology behind Google Search, and the active component of Google Now essentially acts as a mobile interface to Google Search.

Passive Mode

Google Now’s passive mode is a novel feature. It is designed to give the user ambient information before the user asks for it, based on predictions Google Now makes using all information it has on the user, such as location, calendar, schedule, emails, music library, browsing history, and past search queries. Google Now’s passive component accesses Google’s impressive repertoire of products for this information.

Cards

Google Now uses cards for all interaction with the user. Cards display specially formatted responses consisting of textual as well as graphical elements.

Examples of Active Responses

Google Now responds to us with the definition of battletoads.

Google Now responds to us with the writer of The Call of Cthulhu.

Google Now recognizes us saying "goog", and realizes we mean the stock index of Google.

Google Now is able to perform basic phone tasks such as dictating a message to email to a contact.

Examples of Passive Suggestions

Google Now can warn users of bad weather without them asking about it, based on their location.

Google Now can read tracking numbers from emails, and automatically show a card about its shipping status.

Google Now can automatically alert users of nearby concerts by their favorite artists based on their music library.

Google Now can automatically display the route map to a destination based on daily travel patterns of users.

Inner Workings of Natural Language Interface

The search process of Google Now can be divided into three stages:

Speech recognition

Natural language processing

Entity recognition

Basically, Google Now takes voice input from the user, converts it to the equivalent text using speech recognition algorithms, analyses the text for words which indicate the type of query, recognizes words which represent real-world entities (based on the Google Knowledge Graph), and formulates an appropriate response.

The example on the left is a graphical representation of this process for a specific query.

Speech Recognition and Neural Networks

Google Now performs speech recognition using neural networks. Voice queries are offloaded to Google’s servers, where they are processed by neural networks, which convert each spoken word into its phonetic representation. This phonetic representation is then converted to text, which is sent for natural language processing and entity recognition.

Neural networks attempt to simulate the human brain, and its process of learning. They are used for learning concepts without human intervention, and require very large amounts of training data. Google managed to build a robust speech recognition system by training it on the vast amount of speech data available to it, through services like Google Voice and YouTube.

Neural networks are composed of many layers of software "neurons", which process small amounts of information and perform small amounts of pattern recognition. The first few layers in a neural network perform simple tasks, such as looking for a color in an image. The latter layers recognize more complex patterns, such as male and female faces in an image. This makes it easy to distribute work, such as recognition of different environments, different accents, different microphones, and different background noise.

Google recently dropped the speech recognition system it had been using for 5 years in favor of neural networks, and they have reported a 25% reduction in error rate upon doing so. This is evident upon using Google Now, with it being able to easily recognize queries like "give me directions to hauz khas" delivered in an Indian accent. In our testing, Google Now rarely failed to recognize speech, even when it contained entities with Hindi names.

Natural Language Processing

Google uses natural language algorithms such as POS (parts of speech) tagging, topic extraction, and keyword extraction to determine the context of a query, such as whether it’s about a person, a place, or a request for directions.

Entity Recognition and the Knowledge Graph

Google uses the Knowledge Graph for entity recognition. The Knowledge Graph is a massive effort by Google to document entities in a manner similar to that of the human brain. The Knowledge Graph contains entities Google has recognized by crawling the web, information about these entities specific to the type of entity, and correlations of various entities.

For example, searching for Tom Cruise leads to Google Now giving information about the actor, and not vacation cruises. They also provide relevant information such as the movies he’s acted in, and how tall he is. They also suggest follow up searches based on actors and directors he’s worked with. Similarly, the Leonardo Da Vinci entity in the Google Knowledge Graph is related to the Mona Lisa entity, the Italy entity, and the Michelangelo and Raphael entities.

This example illustrates Google Now’s entity recognition capability.

Searching for "Directions to interior today exhibition in New Delhi" causes Google Now to:

Find that exhibition.

Understand we care about where it is being shown.

Deduce our current location.

Present us with a map card to the museum itself along with a button to immediately get directions.

Privacy Concerns Related to Google Now

Though Google Now is extremely innovative and useful, its predictive capabilities are an invasion of user privacy. Google goes through every single scrap of stored personal information and monitors all our online activities to come up with predictions. It goes through our email, Google+ data, browsing history and much more.

Google Now massive capability is a direct result of Google’s intrusion into our personal lives. It is disconcerting to know about the amount of data Google has on us, and Google Now serves to highlight that. We assume our information is safe with Google, but we relinquished control over all of it when we signed up for any Google service.

Conclusion

Google Now is an intelligent personal digital assistant, an ambient knowledge supply, an extension of Google Search and much more. It has excellent real-world performance. It has the potential to make our lives easier and simpler, but sparks privacy concerns.