Evaluation of the Secret Shopper Model for E-commerce

Karl Thomas Rees

April 30, 1999

Introduction

In their proposal to adapt the "secret" or "mystery" shopper model (as it is sometimes called) to e-commerce, SecretWebShopper, Inc. presents a convincing solution to business concerns over customer satisfaction. While the number of online shoppers is increasing dramatically, recent market surveys show that customers are frustrated with their online shopping experience. As the amount of companies offering products over the web continues to grow, customer satisfaction will become a key factor in profits. Secret shopper services can provide online merchants with the quality end-user feedback essential for online success.

My research of numerous Internet databases and e-commerce journals shows that there are few companies currently offering services similar to those proposed by SecretWebShopper, Inc. Compared against the strategies of those companies that do offer similar services, SecretWebShopper's secret shopper model has a number of advantages that will guarantee its success. This report will emphasize this conclusion as it proceeds in the following format:

Consumer Problems with E-commerce

The 1998 Christmas shopping season proved how insufficiently Internet shops are meeting the expectations of online shoppers. By June 1998, online shopping had increased 37 percent over the previous September (Hewlett-Packard,1998, November). A Shop.org survey of November and December 1998 showed the increase becoming even more dramatic, with revenues growing 230 percent over the previous year (Shop.org, 1999, '98 Online . . .). However, as early as November 1998, research firms such as Zona Research recognized that despite the increase in customers, many more potential customers were being lost due to lack of customer satisfaction (Hewlett-Packard, 1998, November). Ken Pawlak of eFusion, Inc. cited this decrease in customer satisfaction, as well as a decrease in completed sales per customer, in declaring the current web sales model inadequate for converting potential customers into shoppers (Pawlak, 1999).

This customer dissatisfaction is better understood in the statistical findings of the Shop.org survey. Only five percent of unique visitors to a site will become a customer, with only 1.6 percent of all visitors making a purchase (Shop.org, 1999, Data Highlights . . .). If these numbers can be improved only slightly, revenues will greatly increase. For example, consider two companies with similar e-commerce revenues. If the first company can increase visitor to customer conversion rates by just one percent, it will have a 20 percent revenue advantage over its competitor. Since customer satisfaction is the most important factor in the customer's decision to make a purchase, improving customer satisfaction is in the best interests of any company serious about e-commerce success.

In order to improve these figures, companies must first understand why their customers are dissatisfied. Market research suggests many areas where e-commerce needs improvement. The previously cited Zona Research survey shows that many customers have difficulty finding a specific product. Over 62 percent of online shoppers gave up their search for a particular product on the web at least once in a two-month period because of this difficulty (Hewlett-Packard, 1998, November).

Click-Here Commerce, a February 1999 study by Shelley Taylor and Associates (http://www.infofarm.com), also recognized this problem, suggesting that online shops should invest in better navigation systems Other problems identified in the survey included incompatibility with older browsers, lack of shopping carts on every page, and a deficit in available product information. The greatest problem, however, proved to be a lack of customer support services. To these problems, Click-Here commerce offered the solution of learning from and returning to the traditional land-based retail models (Sell it on the Web, 1999, February 26).

Many other surveys identify the problem of customer support. A recent NFO Interactive survey showed that 25 percent of customers would be likely to buy more if they could interact with a sales representative as they made their purchase. Of those who have never made an online purchase, 13.7 percent said they would buy products from online stores if they knew that "human" support was available (Internet.com, 1999, April 16). Ken Pawlak also addresses this problem, suggesting a solution in a combination of web personalization and call centers (Pawlak, 1999).

The source of these problems lies in the inability of merchants to get inside the head of their customer. Merchants are already familiar with their web sites, and therefore have difficulty understanding how a new customer might respond to it. Shop.org's survey reaches this same conclusion: "Online retailers must understand how Web site customers think and behave when they shop. How do these customer decide to make a purchase? Understanding customer behavior and likes and dislikes provides the context necessary to build a compelling shopping experience that turns browsers into buyers" (Shop.org, 1999, shop.org/BCG Study: Strategic Insight). This problem has opened the way for a new market of e-commerce evaluation from the customer perspective.

Current Solutions

In order to understand SecretWebShopper's prospects for success, we must first examine current market solutions, especially potential competitors. These include a variety of services, from annual industry-wide surveys to individually oriented evaluations. This section will first of all, examine industry-wide surveys. Then it will focus on two services oriented towards individual online merchants: BizRate and WebCriteria.

Surveys

Evaluation of the effectiveness of e-commerce began years ago as a collection of independently sponsored surveys. From these, online merchants could gain insight on what strategies generally worked and then apply those strategies to their web sites. The most general of these surveys included traffic counts such as Media Matrix (http://www.mediamatrix.com) and Web 21's Hot 100 list of Web shopping sites (http://www.hot100.com). These surveys, however, provided far too little information to be of value.

Newer surveys have focused on more specific aspects of e-commerce. EMarketer (http://www.emarketer.com) is one example of many companies finding success in providing demographical reports on online shopping. EMarketer periodically attempts to classify online shopping trends, including such information as who is buying what for how much and from where. Their purpose is much as that declared by Shop.org: "To develop a set of performance benchmarks that could be used by retailers to track and improve performance for business planning, and to compare their performance with that of competitors" (Shop.org, 1999, shop.org/BCG Study of Online . . ."). Such reports provide information influential for online marketing strategies, thus preventing potential sources of customer dissatisfaction. However they are hardly adequate as an indicator of consumer reaction to online stores.

Mark Hurst, founder of Creative Good (http://www.creativegood.com) and InfoWorld's Netrepreneur of the Year, took such evaluations to a higher level. In his report "In Search of E-Commerce," Hurst looked at the web sites of seven different companies from the customer's perspective. Dylan Tweney of InfoWorld summarizes the report: "The report pulled no punches. In it, Hurst and Seidman pointed out how, in many cases, these sites actively discouraged their customers from making purchases. Confusing Web-page designs, excessive graphics, and complicated order forms all served to turn customers away in droves" Hurst says that these companies later corrected many of the problems that he cited in the report. Tweney suggests that Hurst's approach may lead to a "usability revolution," greatly improving customer-merchant relationships (Tweney, 1999, February 15).

Ironically, Creative Good's own site fails the usability test, offering little information besides the fact that they consider themselves Web's leading customer experience firm. They offer web evaluation services, but excepting the previously named report, for which they charge $600, they do not offer examples or list prices (Creative Good, Inc., 1999). Were it not for this lack of information, I would consider Creative Good to be a greater competitor and go into greater depth about its services.

BizRate.com

Several other companies have attempted to implement an approach similar to Hurst's. While evaluation of the customer experience is a relatively new industry with few competitors, Binary Compass has been offering such services through BizRate.com since December 1996 (Guglielmo, 1997, August 4). BizRate's evaluation works at two different levels. The first is on a consumer level, suggesting potential customers which company would offer the best services. The second is on a merchant level, offering comparison reports of other companies in addition to the consumer-level reports.

BizRate also implements their evaluation at two levels. Originally, BizRate relied on a team of professionals to secretly shop at selected online shops. They then proceed to evaluate the selected site in ten areas, including product selection, product availability, and timeliness of delivery. BizRate then made this information available to customers. Companies with the best evaluation receive the highest recommendation. All companies that agreed to be evaluated in this fashion received a silver medal, meaning they were automatically ranked higher than those companies that did not (Marable, 1997, June 2 ).

The purpose of this was, of course, to inform the consumer. However, BizRate soon saw the opportunity to market themselves to online companies as a source of customer evaluation. Once a month, they offer a report of their findings to the evaluated businesses. Starting in summer 1997, BizRate added evaluations directly from customers. As a customer concludes a purchase, participating web sites can now include a pop-up window containing a BizRate survey form. This survey asks questions concerning the customer's shopping experience and decision to buy the purchased product. To customers who choose to complete the survey, BizRate will e-mail the follow-up second half of the survey, asking questions regarding delivery and customer support. BizRate then collects this data and uses it for its rating system. For companies that agree to undergo this level of evaluation, BizRate awards a gold medal, meaning that they will automatically be ranked higher than silver medal companies (BizRate.com, 1999, BCE: Survey Demo). It is unclear whether BizRate still implements the information collected from the professional shoppers into its ratings for gold medal companies.

The service, which BizRate keeps free as to avoid allegations of bribery, currently has 500 participating merchants (BizRate.com, 1999, Frequently . . .). Between 8 and 12 percent of customers respond to the three-minute survey, far surpassing the 1.5 percent return rate for land-based post-product mail surveys (Andrews, 1999, February 1). That BizRate has become a respected source of information about consumer attitudes towards online merchants is evident in its widely cited report on the Christmas 1998 shopping season. Appearing in such places as the New York Times, the report gave a five-star rating to only one web site -- The SitStay GoOut Store (http://www.sitstay.com), a site specializing dog products. The report has been cited as a key indicator of the current level of customer dissatisfaction (Tedeschi, 1999, April 30).

Furthermore, BizRate has an already established presence on the Internet. In September of 1998, BizRate reached a deal with American Online, Inc. to provide its services to AOL subscribers (ZD Net, 1998 September 21). In January 1999, BizRate made a similar deal with Microsoft to provide its services over the Microsoft Network (ZD Net, 1999, January 20). Clearly, BizRate has an immediate advantage over potential competition.
BizRate's surveys focus on ten categories. These are as follow: price, product selection, product information, ease of ordering, web site navigation and looks, on-time delivery, product representation, customer support, posted privacy policies, and product shipping and handling (BizRate.com, 1999, Help . . .). While they offer short examples of sample criteria for these categories, the criterion is left mostly to the customer's judgement. The rest of the survey asks for demographic information such as age group and occupation. A sample survey can be found at http://www.binarycompass.com/scripts/survey.asp?t=home&b=surveyweb.

In the reports it issues to merchants, BizRate arranges this data with various graphs and figures. Besides giving a summary of performance over the last year, the report offers no customer satisfaction information not already available to the customer. A sample merchant report can be found at http://www.binarycompass.com/docs/sample_perform.pdf. For an unspecified price, BizRate also offers summarized reports for the entire industry. In its Flash Survey Reports, BizRate also allows for more customized surveys. BizRate will perform these surveys for an unspecified price (BizRate.com, 1999, Research Products). It is from these last two types of reports that BizRate earns it money (BizRate.com, 1999, Frequently . . .).

WebCriteria

In their sample merchant reports, BizRate claims to be "the only industry level aggregator of customer satisfaction" (p. 3). Though this is close to the truth, newly formed companies such as Creative Good are seeking a share of the largely untouched market. WebCriteria is the most recent company to enter the market. Since late 1998, WebCriteria has sought to "enable interactive agencies and web site owners to objectively compare any web site to its direct competition, its industry and the ever-changing performance standards of the web" (WebCriteria, 1999, Background). Unlike BizRate, their approach is oriented only towards online merchants.

WebCriteria offers end-user evaluation of online shops through its SiteProfile service. SiteProfile's mission is to solve the problem of customer-merchant relations, which problem it describes to its customers in four key points:

Their approach is clearly more oriented towards solving the problem outlined at the beginning of this report than consumer-oriented BizRate. Because of this, they are both a better model and a more formidable competitor for SecretWebShopper.

SiteProfile's report concentrates on web site performance in the areas that have a "significant impact on the percentage of visitors that return to a web site." These areas include load time, accessibility, freshness, and composition. Delivered within 72 hours of the initial request, the report is 15 pages in length and compares the evaluated online merchant with three competing web sites and one industry benchmark. Available for free are twelve industry benchmarks, summarizing evaluation results for over 100 leading e-commerce web sites (WebCriteria, 1999, Background).

Their sample report, located at http://www.webcriteria.com/product/docs/SiteProfile_Report.pdf, better explains how SiteProfile evaluates these four areas of web site performance. Load times are calculated from both office computer and home computers. Accessibility is calculated in terms of the percentage of pages on the site that can be downloaded in five minutes. Freshness is judged according to the number of date-stamped HTML pages and graphics. Composition is basically a summary of what type of files (audio, video, text, etc. . .) make up the site. The report concludes with recommendations on how to improve web site performance, an overview of the SiteProfile's methodology, and an appendix including tables of the previously summarized figures.

The problem with WebCriteria's approach is its methodology. Although the company claims to have 35 years of experience in the measurement business and 20 years of experience with behavior, it chooses to implement that experience through a web robot rather than a live customer. Called the Max Engine, SiteProfile's robot is capable of analyzing 3 million links every 24 hours. WebCriteria describes Max "as a browsing behavior agent that models aspects of perceptual, cognitive, and motor behavior - that is, how we see, think and move - as they relate to web site browsing" (WebCriteria, 1999, Technology Overview). Naturally, the use of a robot to emulate customer behavior raises some questions regarding the accuracy of SiteProfile.

WebCriteria charges $495 for its SiteProfile service. It offers packages of four and ten reports over a twelve-month period at the discounted rates of $1195 and $1950 respectively (WebCriteria, 1999, Pricing).

The Secret Shopper Model

A more appropriate solution is the secret shopper model. The model has proven extremely effective in land-based retail. If the object of online retail should be to emulate the success of land-based retail, as the Click-Here Commerce survey suggested, then it would make sense to apply this model to e-commerce. This section will first of all clarify what the secret shopper model is and how it could apply to the Internet. Then it will show that the model is currently not being implemented on the Internet. Finally, it will compare the model with the previously mentioned solutions, coming to the ultimate conclusion that by implementing the model, SecretWebShopper would have significant advantages over potential competitors.

Clarification of Secret Shopper Concept

Several land-based secret shopper services have web sites for advertising purposes. Some services also use their web sites as a collection point for the data collected by their secret shoppers. Most of the information in this section comes from these web sites. Interestingly enough, none of these services has considered offering secret shopper services to online merchants. One of them, 21^st Century Solutions (http://www.21stcenturysolutions.com), even offers e-commerce consulting, but has yet to merge the two areas of service.

Secret ShopNET describes the purpose of the secret shopper as follows: "to anonymously observe and document the quality of service at a store or business on a given day. Clients can then evaluate a sample of service delivery, product knowledge and facility maintenance at corporate stores or franchises" (Secret ShopNET, 1996, Welcome to . . .). The audience of a secret shopper report is only the merchant, thus assuring that the report is both confidential and geared towards helping the merchant.

Unlike regular customers, secret shoppers are paid to notice every aspect of the merchant's operation, and therefore are capable of giving feedback to the average customer. The previous work experience of a secret shopper varies. Secret ShopNET requires no previous experience from its employees, opening soliciting new employees on its web site (Secret ShopNET, 1996, Mystery Shopping . . .). On the other hand, Howard Services' Service Sleuths advertises that most of their secret shoppers have management background in the type of store they are shopping at (Howard Services, 1999). Secret shoppers usually work out of their own homes, receiving assignments and submitting reports over the Internet (Secret ShopNET, 1996, Mystery Shopping . . .). Naturally, secret web shoppers would complete all of their work from the home.

Reports generated by secret shoppers are far more valuable as an indicator of customer satisfaction than are surveys from live customers. The reason for this is that most dissatisfied customers simply do not give feedback. This is one of Service Sleuth's key selling points. On their web page, they offer the following facts:

Certainly these facts also hold true for e-commerce, hinting at the advantages of SecretWebShopper's secret shopper model.

Secret shopping services generate a variety of reports. Service Sleuths, for instance, provides Competitive Shopping, Customer Retention, Employee Integrity, and Loss Prevention reports (Howard Services, 1999). Another Secret Shopping group, Ken-Rich, offers similar services, but also includes an in-depth employee analysis, where the secret shopper will focus specifically on a single employee for at least one-half hour (Ken-Rich, 1999, April 17). Secret shoppers, in pretending to be a regular customer, have the special advantage of being able to evaluate how efficiently merchants support their customers through friendliness and product knowledge.

The secret shopper model can be applied to the web with but few modifications. Obviously, there would be little need for reports on loss prevention or employee integrity. Data security, on the other hand, would be an important consideration. Secret web shoppers would still evaluate availability of information about products and the online store's product selection. Especially important would be a consideration of the web page interface and its ease of navigation. Secret web shoppers would also need to evaluate page layout and accessibility from the perspective of a first time customer in order to determine how easily customers can understand the site. Load time, availability of customer support, and punctual product delivery would be other major concerns.

One minor difficulty in adapting the secret shopper model to the web is that it is often used to determine how well a certain franchise or store is complying with company procedures. In a web environment, there is only one store for the entire company. Therefore, such services are not necessary. However, as I have shown so far, the secret shopper model has enough benefits for this remain only a minor difficulty.

Research Results: Secret Shopper Model Not in Use for E-Commerce

My research, constituting altogether about 20 hours of examination of online search engines and e-commerce journals, found that there are currently no secret shopper services in implementation for online shopping. I also found that there are few companies offering customer perspective research services comparable to the Secret Shopper model. I have listed all such services in the previous sections of the report. If any other services do exist, it is my opinion that they do not market themselves well enough to be seriously considered as potential competition.

I performed my search on the following databases: Yahoo, AltaVista, and Netscape. I searched through the archives of the following internet sites and e-commerce journals: the New York Times on the Web (http://www.nytimes.com), InfoWorld (http://www.infoworld.com), E-Commerce Times (http://www.ecommercetimes.com), Sell it on the Web (http://www.sellitontheweb.com), Wilson Internet Services' Web Commerce Today (http://www.wilsonweb.com), ZD Net's Inter@ctive Week (http://www.zdnet.com/intweek), Internet.com (http://www.internet.com), Internet Week (http://www.internetworld.com), All Net Research (http://www.allnetresearch.com), and The E-Commerce Knowledge Center (http://www.knowledgecenters.org/eccenter.asp).

The searches involved various versions and combinations of the following keywords: secret shopper, mystery shopper, e-commerce, market report, market research, customer research, customer satisfaction, consumer report, and benchmarking. Once similar services were found, such as BizRate, they were also included in the search as to get a variety of opinions on the service. I then scanned through the search results, looking in depth at all sites that seemed connected to the project. Finally, if the site suggested another source of information, I proceeded to examine the new site. I have reported all of the significant information that I found.

Comparison of Secret Shopper Model with Current Market Solutions

As a method of informing merchants of customer attitudes towards their web sites, SecretWebShopper's approach would possess important advantages when compared to alternative solutions. This last section will explore these advantages and possible disadvantages in context of BizRate and WebCriteria, respectively. Because of their inability to inform of customer attitudes towards specific merchants, I have deliberately chosen to ignore industry-wide surveys such as those sponsored by Shop.org. I have also ignored Creative Good because of a lack of information available on their web site. Should they market their services better, however, I suspect that of all the previously examined services, theirs would be the most comparable to SecretWebShopper, and therefore they should be considered as a potentially major competitor.

The primary difference between the approaches of SecretWebShopper and BizRate are the intended audiences. BizRate's primary audience is the consumer. It is because of BizRate's popularity among consumers that companies are willing to undergo its surveys. Companies understand that using BizRate gives them free publicity. In turn, this consumer traffic continues to increase, allowing BizRate to collect some of the most detailed demographic information currently available. BizRate then turns a profit on selling this information. Though much of the process is directed at informing businesses, BizRate's top priority must be maintaining customer confidence. Otherwise, they would lose their valuable input.

This is the source of both many advantages and disadvantages. First, I shall examine BizRate's advantages over potential competitors. It already has a strong hold on the market, as evident in its contracts with AOL and Microsoft. It will be difficult to crack into this market share, especially if merchants see SecretWebShopper's services as being similar to BizRate's. BizRate's services are free, and the opportunity for free advertisement is also enticing. BizRate offers continual feedback, meaning that merchants constantly receive up-to-date information on consumer attitudes. And finally, BizRate has strength in numbers. Whereas the Secret Shopper model leaves the evaluation to a handful of consumers, BizRate's evaluations are the compiled results of thousands of customers.

In order to overcome BizRate, SecretWebShopper must make it clear that its services are not in place of the consumer information BizRate offers, but in addition to. The Secret Shopper model provides for input that BizRate cannot provide. BizRate's services are only general indicators of consumer attitudes. From their surveys, merchants can understand that their web site is not having as much success as their competitor's, but they cannot find out why this is so. The secret shopper model allows for a much more detailed and specific analysis of a web site, including recommendations for improvement.

In some aspects, SecretShopper's services are also clearly better than BizRate's. Rather than gearing the reports to inform customers, secret shoppers can concentrate specifically on the business, offering a much more helpful analysis. There is also the question of whether BizRate's surveys are truly an accurate and random sample of the consumer population. Since, as we have previously mentioned, dissatisfied customers do not generally complain about service or purchase a product (which is a necessity for BizRate's survey to work), the secret shopper model can offer insight into a population of dissatisfied customers not represented in BizRate's surveys. And finally, BizRate's surveys can eventually become burdensome and annoying to customers and inconvenient to merchants. Because e-commerce is so new, customers may be willing to respond to surveys now. In the future, however, they may tire of such surveys.

WebCriteria's audience is much more comparable to that of SecretWebShopper. Their one advantage is purely financial. Because their survey relies mostly on computer-gathered information, they require a much smaller staff than would SecretWebShopper. SecretWebShopper, on the other hand, would have much more to offer by way of customer insight than WebCriteria. WebCriteria's reports do nothing more than attach numbers to factors such as load time, accessibility, and content. Any secret shopper would easily be able to pick up on problems in these areas. Meanwhile, WebCriteria's robot cannot respond to factors more significant in the customer's decision process (such as aesthetic qualities and availability of information). Even if it does become possible for a computer to make evaluations in these categories, it is doubtful that merchants would trust a computer's opinion over that of a live customer.

Conclusion

As I have shown throughout the report, there is a growing demand for SecretWebShopper's services. In order to find success in e-commerce, online merchants need a method of increasing customer satisfaction. Only through understanding the customer's experience with their web site, can a company expect to make improvements necessary to increase customer satisfaction. The current market is far from saturated with solutions to this problem. With the number of online merchants increasing daily, even if SecretWebShopper's services were inferior to other such services, they would still find an anxious market ready to support it.

Fortunately, SecretWebShopper's approach is superior to other market solutions. By collecting information from professional shoppers who anonymously evaluate online web sites, they are capable of offering the best advice on how to improve a customer's experience. In doing so, merchants that contract SecretWebShopper's services will be able to improve customer satisfaction and company revenues. SecretWebShopper's proposal is a guaranteed success for both its founders and the companies that will use its services.

References