Frowns, Sighs, and Advanced Queries -- How does search behavior change as search becomes more difficult?

Posted by Anne Aula, Rehan Khan, and Zhiwei Guan, User Experience Team

How does search behavior change as search becomes more difficult?

At Google, we strive to make finding information easy, efficient, and even fun. However, we know that once in a while, finding a specific piece of information turns out to be tricky. Based on dozens of user studies over the years, we know that it’s relatively easy for an observer to notice that the user is having problems finding the information, by watching changes in language, body language, and facial expressions:



Computers, however, don’t have the luxury of observing a user the way another person would. But would it be possible for a computer to somehow tell that the user is struggling to find information?

We decided to find out. We first ran a study in the usability lab where we gave users search tasks, some of which we knew to be difficult. The first couple of searches always looked pretty much the same independent of task difficulty: users formulated a query, quickly scanned the results and either clicked on a result or refined the query. However, after a couple of unsuccessful searches, we started noticing interesting changes in behavior. In addition to many of them sighing or starting to bite their nails, users sometimes started to type their searches as natural language questions, they sometimes spent a very long time simply staring at the results page, and they sometimes completely changed their approach to the task.

We were fascinated by these findings as they seemed to be signals that the computer could potentially detect while the user is searching. We formulated the initial findings from the usability lab study as hypotheses which we then tested in a larger web-based user study.

The overall findings were promising: we found five signals that seemed to indicate that users were struggling in the search task. Those signals were: use of question queries, use of advanced operators, spending more time on the search results page, formulating the longest query in the middle of the session, and spending a larger proportion of the time on the search results page. None of these signals alone are strong enough predictors of users having problems in search tasks. However, when used together, we believe we can use them to build a model that will one day make it possible for computers to detect frustration in real time.

You can read the full text of the paper here.

Read More

Focusing on Our Users: The Google Health Redesign

Posted by Hendrik Mueller, User Experience Researcher

When I relocated to New York City a few years ago, some of the most important health information for me to have on hand was my immunization history. At the time, though, my health records were scattered, and it felt like a daunting task to organize them -- a not-uncommon problem that many people face. For me, the solution came when Google Health became available in May of 2008, and I started using it to organize my health information and keep it more manageable. I also saw the potential to do much more within Google Health, such as tracking my overall fitness goals. When I joined the Google Health team as the lead user experience researcher, I was curious about the potential for Google Health to impact people’s lives beyond things like immunization tracking and how we could make the product a lot easier to use. So I set out to explore how to expand and improve Google Health.

Here at Google, we focus on the user throughout the entire product development process. So before Google Health was first launched, we interviewed many people about how they managed their medical records and other health information to better understand their needs. We then iteratively created and tested multiple concepts and designs. After our initial launch, we followed up with actual Google Health users through surveys, interviews, and usability studies to understand how well we were meeting their needs.



From this user research, we learned what was working in the product and what needed to be improved. Here are some of the things our users found especially useful:

• Organizing and tracking health-related information in a single place that is accessible from anywhere at any time
• Sharing medical records easily with loved ones and health care providers, either by allowing online access or by printing out health summaries
• Referencing rich information about health topics, aggregated from trusted sources and Google search results

Our users also described to us the benefits they saw from using Google Health:

“Google Health gives me many tools to research my prescriptions and symptoms, and to track all of the many tests I keep having. Google Health made several necessary and cumbersome tasks easy and worry free.”

“For years now, I've tried to remember my son’s allergies and medications, but the list has grown so long, that I kept forgetting one or two when a doctor asked me about them. That can't happen again because I now have a single place to keep up with them. And I love the fact that I can print off information for situations when I really need it.”

“I really like that I can share my profile with others. I want my mom to know my medical information, just in case anything ever happens to me.”

While we learned that our users were clearly getting positive results from using Google Health, our research also taught us that more was needed. We learned that we needed to make fundamental changes to fully meet the needs of all of our current and prospective users, such as those that are chronically ill, those who care for family members, and especially those users looking to track and improve their wellness and fitness.

On this last point, our user surveys already pointed out that there was more we could do to help our users track and manage their wellness, not just their sickness, so we conducted further research about how people collect, monitor, track, and analyze their wellness data. We interviewed several people in their homes and invited others into our usability labs. As a result, we identified several areas where we could improve Google Health to make it a more useful wellness tool, including:

• Dedicated wellness tracking including pre-built and custom trackers
• Efficient manual data entry as well as automatic data collection through devices
• A customizable summary dashboard of wellness and other health topics
• Goal setting and progress tracking using interactive charts
• Personalized pages for each topic with rich charts, journaling, and related information

These insights led us to a whole new set of design proposals. We gathered feedback on the resulting sketches, wire-frames, and screenshots from active and new Google Health users. The results throughout this process were eye-opening. While we were on the right track for some parts of the design, other parts had to be corrected or even redesigned. We went through several iterations until we had a design that tested well and we felt met the user needs our research had uncovered. Finally, we conducted several usability studies with a functioning prototype throughout the product development process to continuously improve usability and function.



At the end, the collaboration between the user experience, engineering, and product management teams resulted in an entirely new user experience for Google Health combined with a set of new functionality that is now available for you to try out at www.google.com/health. See for yourself how the old and new versions compare. Here is a screenshot of a health profile in the new version:



And this is how the same account and profile looked in the old user interface:



As a Google Health user, I am excited to take advantage of the new design and have already started using it for my own exercise and weight tracking. And on behalf of the user experience team and the entire Google Health team, we’re excited about being able to bring you a new design and more powerful tool that we think will meet more of your health and wellness needs.

We look forward to continuing to explore how we can make Google Health even more useful and easier to use for people like you. As you use Google Health, you may see a link to a feedback survey at the top of the application. If you do, please take the time to fill it out - we will be listening to your input!

Read More

Discontinuous Seam Carving for Video Retargeting

Posted by Matthias Grundmann and Vivek Kwatra, Google Research

Videos come in different sizes, resolutions and aspect ratios, but the device used for playback, may it be your TV, mobile phone, or laptop, only has a fixed resolution and form factor. As a result, you cannot watch your favorite old show that came in 4:3 on your new 16:9 HDTV without having black bars on the side, referred to as letterboxing. Likewise, widescreen movies and user-videos uploaded on YouTube are shot using various cameras with wide-ranging formats, so they do not fit completely on the screen. As an alternative to letterboxing, several devices try to upscale the content uniformly, which either changes the aspect ratio, making everything look stretched out, or simply crop the frame, thereby discarding any content that cannot fit the screen after scaling.

At Google Research, together with collaborators from Georgia Tech, we have developed an algorithm that resizes (or retargets) videos to fit the form factor of a given device without cropping, stretching or letterboxing. Our approach uses all of the screen’s precious pixels, while striving to deliver as much video-content of the original as possible. The result is a video that adapts to your needs, so you don’t have to adapt to the video.


Six frames from the result of our retargeting algorithm applied to a sub-clip of “Apologize”, © 2006 One Republic. Original frame is shown on the left, our resized result on the right. The original content is fit to a new aspect ratio.

The key insight is that we can separate the video into salient and non-salient content, which are then treated differently. Think of salient content as actors, faces, or structured objects, where the viewer anticipates specific, important details to perceive it as being correct and unaltered. We cannot change this content beyond uniform scaling without it being noticeable. On the other hand, non-salient content, such as sky, water or a blurry out-of-focus background can be squished or stretched without changing the overall appearance or the viewer noticing a dramatic change.

Our technique, which we call discontinuous seam carving -- named so because it modifies the video by adding or removing disconnected seams (or chains) of pixels -- allows greater freedom in the resizing process than previous approaches. By optimizing for the retargeted video to be consistent with the original, we carefully preserve the shape and motion of the salient content while being less restrictive with non-salient content. The key innovations of our research include: (a) a solution that maintains temporal continuity of the video in addition to preserving its spatial structure, (b) space-time smoothing for automatic as well as interactive (user-guided) salient content selection, and (c) sequential frame-by-frame processing conducive for arbitrary length and streaming video. The outcome is a scalable system capable of retargeting videos featuring complex motions of actors and cameras, highly dynamic content and camera shake. For more details, please refer to our paper or visit the project web-site.

Read More

Google Search by Voice: A Case Study

Posted by Johan Schalkwyk, Google Research

Wind the clock back two years with your smart phone in hand. Try to recall doing a search for a restaurant or the latest scores of your favorite sports team. If you’re like me you probably won’t even bother, or you’ll suffer with tiny keys or fat fingers on a touch screen. With Google Search by Voice all that has changed. Now you just tap the microphone, speak, and within seconds you see the result. No more fat fingers.

Google Search by Voice is a result of many years of investment in speech at Google. We started by building our own recognizer (aka GReco ) from the ground up. Our first foray in search by voice was doing local searches with GOOG-411. Then, in November 2008, we launched Google Search by Voice. Now you can search the entire Web using your voice.

What makes search by voice really interesting is that it requires much more than a just good speech recognizer. You also need a good user interface and a good phone like an Android in the hands of millions of people. Besides the excellent computational platform and data availability, the project succeeded due to Google’s culture built around teams that wholeheartedly tackle such challenges with the conviction that they will set a new bar.

In our book chapter, “Google Search by Voice: A Case Study”, we describe the basic technology, the supporting technologies, and the user interface design behind Google Search by Voice. We describe how we built it and what lessons we have learned. As the product required many helping hands to build, this chapter required many helping hands to write. We believe it provides a valuable contribution to the academic community.

The book, Advances in Speech Recognition, is available for purchase from Springer.

Read More

Towards Energy-Proportional Datacenters

Posted by Dennis Abts, Michael R. Marty, Philip M. Wells, Peter Klausler, and Hong Liu

This is part of the series highlighting some notable publications by Googlers.

At Google, we operate large datacenters containing clusters of servers, networking switches, and more. While this gear costs a lot of money, an increasingly important cost -- both in terms of dollars and environmental impact -- is the electricity that drives the computing clusters and the cooling infrastructure. Since our clusters often do not run at full utilization, Google recently put forth a call to industry and researchers to develop energy proportional computer systems. With such systems, the power consumed by our clusters would be directly proportional to utilization. Servers consume the most electricity, and therefore researchers have responded to Google’s call by focusing their attention towards servers. As the servers become increasingly energy proportional, however, the “always on” network fabric that connects servers together will consume an increasing fraction of datacenter power unless it too becomes energy proportional.

In a paper recently published at the International Symposium on Computer Architecture (ISCA), we push further towards the goal of energy-proportional computing by focusing on the energy usage of high-bandwidth, highly-scalable cluster networking fabrics. This research considers a broad set of architectural and technological solutions to optimize energy usage without sacrificing performance. First, we show how the Flattened Butterfly network topology uses less power since it uses less switching chips and fewer links than a comparable-performance network built using the more conventional Fat Tree topology. Second, our approach takes advantage of the observation that when network demand is low, we can reduce the speed at which links transmit data. We show via simulation, that by tuning the speeds of the links very rapidly, we can reduce power consumption with little impact on performance. Finally, our research is a further call to action for the academic and industry research communities to make energy efficiency, and energy proportionality in particular, a first-class citizen in networking research. Put together, our proposed techniques can reduce energy cost for typical Google workloads seen in our production datacenters by millions of dollars!

Read More

Google North American Faculty Summit - Day 2

Posted by Andrew Tomkins, Director of Engineering, Google Research

Friday at the Google Faculty Summit, we discussed ideas around online social capabilities. Chris Messina opened the discussion with a talk about open initiatives for the social web. Damon Horowitz, founder of Aardvark, gave a talk about the Aardvark experience. But in this post, I’d like to talk about a panel I moderated on the future of the social web. The panel consisted of four experts in the area. Joseph Smarr came to Google after eight years as CTO of social networking site Plaxo. Lada Adamic is on the faculty at University of Michigan, where she studies the nature of social and information networks. Eytan Adar is also on the faculty at the University of Michigan, where he studies the evolution of production and consumption of data over time. Luis von Ahn is on the faculty of Carnegie Melon University and also an employee at Google; he studies mechanisms to connect significant human efforts to interesting problems.

One theme that received a lot of attention from panelists and audience members alike was the benefits and pitfalls of social personalization. In the context of an activity stream, there seems to be general agreement that passing lightweight updates among friends is a valuable tool for “social grooming,” or keeping light contact with friends as a way of maintaining the state of the friendship. For information discovery, however, the topic received more debate: real-world social networks have always been used to both push and pull information, but in conjunction with high-quality search, it's reasonable to ask which types of information needs can be best addressed by your friends. Social network connections typically display homophily (similarity) in the dimensions of geography and interests, so your friends are more likely to have something interesting to say about your local area and your longstanding hobbies or interests, along with other subjects. If so, the answer you receive has two added bonuses. First, your background knowledge about your friend will aid you in assessing the quality of the answer. And second, an answer from a friend satisfies not just an information need but also a human need to interact and share experiences. This socially augmented information can arrive through a push channel in which your friend already posted (for example) a review for a restaurant, or through a pull channel in which you send to your friends a request for information. The same mechanisms for social information sharing may also operate powerfully in the context of a group coming together around a shared interest or goal, rather than just in the context of an individual. Consider for example a group of students working together to understand some new material. The same two mechanisms apply: knowledge about the other students helps you evaluate their contributions, and the interactions in the group have value beyond the pure information transmitted.

There was considerable discussion about social networks' capacity to funnel information to a user through the lens of a particular viewpoint or ideology. Imagine an individual who arrives on the web as a supporter or detractor of a particular political figure or mindset, and then surrounds him or herself with like-minded people online, enjoying positive and supportive discussions but failing to encounter a diverse set of views and counter-opinions. Literature in the social sciences, beginning with the famous Asch conformity experiments from the 1950s, details the mechanisms that cause people to conform to group expectations and even abandon normal personality traits based on the norms of the new situation. And work by Nobel Prize-winning economist Thomas Schelling shows that very small and "reasonable" biases we might have towards avoiding becoming an extreme minority might lead a system to evolve into a highly balkanized state. Similar models have been proposed and evaluated in the Internet domain, and some preliminary measurements have been performed. While faculty members in the audience surmised that personalization could lead to more extremism, the group agreed there is no conclusive evidence.

Another topic we touched on is mechanism design: the problem of designing systems so that agents in the system, each acting selfishly, will together produce some desired outcome. Consider a social networking game. If the desired outcome is revenue for the game manufacturer, then the actions that increase status in the game (using real-world currency to purchase items in the game; inviting friends to join and participate in the game; clicking on advertisements in the game) are designed well to support this goal. Rewarding the action of bringing new friends into the game is one obvious approach to increasing the total user population. More subtly, any game system must provide sufficient fun to be worth the expense to users. The dramatic success of casual online games of this form (6 percent of U.S. pageviews come from these games, according to a study by Ravi Kumar and myself in the WWW 2010 conference) is a testimony to the presence of successful mechanisms of this form.

Finally, here is a small sampling of other issues that arose in the panel as controversial points or interesting areas for future research:

1. Social networks draw massive amounts of user time. We are beginning to get some limited visibility into exactly how this attention is allocated, which raises the research question of how much utility users are actually deriving from this investment of time, either in information, entertainment, social grooming or other intangibles.
2. In certain online communities, we see behavioral norms that are skewed towards public visibility of essentially all activity. Do these norms reflect the desires of the populations that choose to join the community, or do they emerge specifically because of the technical tools offered by the website that hosts the community?
3. Social networks are increasingly offering richer tools to users in an attempt to capture nuances of interactions that exist in the real world. In the fullness of time, how close will we get, and when will this happen?
4. Social networks formalize the status of a friendship, with significant breakpoints at initiation, acceptance and removal of a binary tie. The visibility of these events leads to both "overfriending" and offense when friendships are refused or removed. Are there improved mechanisms to produce and manage the relationships in online social networks, and if so, what are these mechanisms?
5. Social network graphs are notoriously difficult to partition into large regions with few edges between them (the sole exception being parts of a network that interact using different languages). A series of computational challenges arise when attempting to shard these networks for distributed analysis or serving from multiple computers.

One thing is clear from the discussion on Friday: social networks are increasingly becoming a valuable area for academic study. Faculty from widely disparate areas of computer science have thought deeply about the issues and implications of these tools; active research is ongoing in essentially all top institutions; and social network dynamics are appearing in the undergraduate curriculum. On top of that, they are an interdisciplinary phenomenon, involving not only many aspects of CS (UX, mechanism design, intense system requirements, security and privacy) but also psychology, economics and ethics, to name a few. There is much to study in order to understand these networks and maximize their societal value.

Read More

Google North American Faculty Summit - cloud computing

Posted by Brian Bershad, Director of Engineering, Site Director, Google Seattle

Of the three themes of our 2010 Faculty Summit, cloud computing was the one that pervaded all others, from security in the cloud to the presumption of cloud infrastructure behind the social web. But in our more focused discussion on cloud computing last Thursday, we started with the premise of “prodigiousness,” a concept introduced by Afred Spector, VP of Research and Special Initiatives.

While we all know that systems are huge and will get even huger, the implications of this size on programmability, manageability, power, etc. is hard to comprehend. Alfred noted that the Internet is predicted to be carrying a zetta-byte (10²¹ bytes) per year in just a few years. And growth in the number of processing elements per chip may give rise to warehouse computers of having 10¹⁰ or more processing elements. To use systems at this scale, we need new solutions for storage and computation. It was these solutions we focused on throughout our discussions.

In the plenary talk, Andrew Fikes spoke on storage system opportunities. Among many topics, he talked about shifting engineering foci to storage management and optimization not just on an individual cluster of co-located systems, but across geographically distributed clusters. The goal is so-called planetary-scale systems. This brings up all manner of diverse challenges ranging from the need to continually balance storage vs. transmission costs, the need to account for variable network latency characteristics, and the desire to optimize storage (e.g., by physically storing only one copy of a file that many feel they have rights to, or own).

We had a few roundtables in the afternoon for deeper discussions. In the table I led, we discussed two systems for “programming the data center” developed by systems researchers at Google Seattle/Kirkland. The first, Dremel, is a scalable, interactive ad-hoc query system for analysis of read-only nested databases. Dremel was recently presented in a paper at VLDB (Dremel: Interactive Analysis of Web-Scale Datasets, Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey Romer, Shiva Shivakumar, Matt Tolton, Theo Vassilakis. In Proceedings of the 36th Int'l Conf on Very Large Data Bases, 2010). The system serves as the foundational technology behind BigQuery, a product launched in limited preview mode at Google I/O in May.

We also discussed FlumeJava, a Java library that makes it easy to develop, test and run efficient data-parallel pipelines at data center scale. FlumeJava was developed by programming languages researchers at Google Seattle, and is currently in widespread use within Google. It was presented at the recent PLDI conference (FlumeJava: easy, efficient data-parallel pipelines, Craig Chambers, Ashish Raniwala, Frances Perry, Stephen Adams, Robert R. Henry, Robert Bradshaw, Nathan Weizenbaum. In Proceedings of the 2010 ACM SIGPLAN conference on Programming language design and implementation). The work reflects Google’s commitment to programming language and compiler technologies at scale.

The field of data center programming has progressed substantially in the last 10 years. Dremel and FlumeJava systems represent abstractions of a higher level than the MapReduce construct we previously introduced, and we think they are easier to use (within their domain of applicability) and more automatically optimizable. With time, the field will discover new “instructions” and even better abstractions leading us to a point where computations which run on nearly unlimited processors can be expressed as easily as sequential programs. We are working hard to make progress here, and I look forward to reporting on our progress in the future.

Read More